Open Data: How We Got Here, and Where We’re Going

Rufus Pollock at the Lift 12 conference. More info in http://okfn.org/opendata/ , where the video is embedded.

Mock track of subtitles to check a possible issue with the player. Please don't use, edit or translate it!

(lift)

(lift 12 - Feb 24 2012 - Geneva)

(Rufus Pollock - Stories)

[Rufus Pollock] Just to say for those of you who don't know:

the Open Knowledge Foundation is a not-profit -- not for profit

founded in 2004

and which builds tools and communities

to create, use and share open information

and that's information that anyone can use, reuse and redistribute.

And as such, we've been working on open data for quite a long time

since we started in 2004.

And today, I want to start the story by going back in time 5000 years,

to ancient Mesopotamia.

There, between the Tigris and the Euphrates rivers,

flourished the Sumerian civilization.

And they were confronted by a problem.

They were confronted by the limitations of human memory

in the recording of taxes, food and other goods.

And those ancient civil servants and businessmen hit on a novel solution:

What they decided to do was they would start counting things with small clay chits,

which they would bake inside of a clay -- a little clay box

and then mark, on the outside of that box, what they were counting.

You know, was it grain, was it tax payments, whatever.

And so, born out of a necessity for a state and a society,

came one of the great information technology revolutions of all time: writing.

The Sumerians invented writing via cuneiform.

And if we fast-forward from that a few thousand years, we come to the UK census.

Again, it's always interesting that states, governments are often at the forefront

of at least driving information technology and information systems innovations.

The UK census: again, the state, this is during the Napoleon Wars,

desired to count the population more accurately:

and we have the first UK census in 1801.

And in the US, they also had censuses, in fact starting in 1790.

And one of the problems encountered in the 1880 census

was they tabulated the census by hand.

And by the 1880 census, it was taking seven years to tabulate the census.

So after it got taken in 1880, it wasn't until 1887 they actually had any data they could use.

And they calculated that for the next census in 1890,

they wouldn't be finished by 1900.

They still wouldn't have the results of the census by the time they started the next one.

They had a crisis of information technology.

And what they went and did is they commissioned Herman Hollerith

to build the first automatic tabulator.

And for those of you who know your company history, of course,

Herman Hollerith's company went on to be one of the founders,

if you like, one of the companies that came and created IBM.

And IBM, by the sixties, were building this -- they replaced those hand --

those kind of wooden, mechanical tabulators with this stuff:

digital tabulators, the modern computer of this age

and again, much of this -- I don't know if you guys know --

IBM would have gone bankrupt

if it hadn't been for Franklin Roosevelt passing the Social Security Act in the States,

which necessitated a huge amount of new tabulation.

So, again, a lot of innovation in this space came out of government need

and also, of course, the nuclear program,

the other great needer of computational power.

And today, today, we find ourselves again in the midst of a revolution.

It's a revolution driven by two needs:

ones that have been the same throughout history as I've just shown,

information complexity, which is the necessity,

and information technology, which is the opportunity.

And what we're doing in this case is a policy innovation, if you like.

We are innovating by opening up information.

So just take the obvious example, government, as I said often the innovator.

In the last 3 years ago, you go back 3 years,

there's almost no open government data initiatives in the world.

Today there are dozens.

The UK, the US, Finland, Kenya, The Netherlands,

and there's new ones almost every week.

There's been a launch of an official kind of movement as a part of the UN

called the Open Government Partnership

in which countries sign up, and among other things, they open up their data.

And of course, it's been, in the UK and other countries, Tim Berners-Lee has been involved.

I've helped advise the government around this in the UK.

But it's not just government.

It's also companies.

Companies are opening up data.

Very interestingly, last year, Nike started an open data initiative there

to open up supply chain and sustainability data,

for themselves and also for their suppliers,

which I think is a very interesting change.

And it's also communities.

Often, in fact, back there in the beginning,

this incredible map that you saw in an earlier slide,

is a OpenStreetMap activity, around the world.

People adding to this crowd-built map of the world.

And in the last 6 years, OpenStreetMap,

from a bottom-up community,

have built a complete, comprehensive, map of the world, of fully open data.

So I've just gone on about Open Data,

and one thing I'm aware of, of this audience,

is you might not all know what it is.

So I'm going to take a brief moment, a brief moment, to say what it is.

What does it mean when I say 'open'?

And was it, you know, what's different from anything else?

What's different from simply public data?

So there's actually a definition,

a definition we the Open Knowledge Foundation helped write, it's very simple.

In a nutshell, a piece of information, a piece of data, is open

if anyone is free to use, reuse, and redistribute it

subject only at most to a requirement to attribute and share alike.

And anyone means anyone!

It doesn't mean -- there can't be any commercial restrictions.

You can't say: hey, here's this data,

but only people using it for non-commercial purposes.

Or only people working in education.

Or only people living in the developing world, or the developed world.

There can't be any restrictions like that.

And there's a reason for this, by the way,

and it isn't just because one's obsessed about, if you like, trademarking an attractive term.

It's because it's about interoperability.

One of my experiences at this conference,

I remember from previous trips to Geneva,

is I've been unable to plug in my laptop!

Even though I have a French adaptor, in fact,

these wonderful Swiss plugs here, are you know, these wonderful, small octagonal shape.

And even with my adaptor I can't plug in. Right?

And it's called interoperability.

When we travel around to different countries, our power adaptors don't necessarily fit in.

We have to buy something.

And the point about this definition,

and the point about caring about Open Data, is, it's about interoperability.

The dream of Open Data is interoperability.

Of seamlessly being able to share and interweave information.

And if every time I get information from two different people

I have to consult a lawyer, I have to work out whether I'm allowed to do it,

whether I'm allowed to put these things together,

we lose that dream, that dream is shattered.

And the key point is, this definition, and those conditions, ensure interoperability.

If you comply with them,

we know that any piece of info, of Open Data,

will work with any other piece of Open Data.

And also, it's worth saying for a quick moment, what kind of data.

And to emphasize a point,

just to foreclose those kinds of questions, otherwise I always get asked.

When we talk about opening up data,

in general, we're not talking about personal data.

We're not talking about opening up your private health records

or opening up your personal tax information.

We're talking about information that is non-personal in nature.

And for the government for example: transport, geodata, statistics, electoral, legal.

Stuff that the UK has, in fact, for example been opening up over the last few years.

This financial information on government spending,

this information on health outcomes, on prescriptions,

This information on educational outcomes, this information on the law.

This information -- statistical information.

That's the kind of thing that we're talking about.

Now, I want to say... because in this story, we have this story, this story of over-time.

But why governments are doing it now?And why Open Data?

So, okay, for thousands of years, governments innovate,

but why do they innovate at this particular moment and in this way?

So I want to start here with a quick story, a story of medicine gone wrong.

It is from a great book by a guy called Stephen Klaidman.

It's in fact one of the things that made me think quite deeply about this:

why I was interested in Open Data.

In that picture there, you can see what was the Redding Medical Centre in Northern California.

There, in 2002, in the Summer of 2002, John Corapi, in typical American style,

an ex-accountant from Vegas turned Catholic priest,

[scattered laughter]

...arrived at the Redding Medical Centre having been referred by his doctor for having chest pains.

He had a cardiogram by the local cardiologist

and was told that he needed an immediate heart bypass,

that he was at serious risk, and that he should come back later that day,

or at the latest, tomorrow, to have open heart surgery.

Rather shocked, and dazed by this news, he returned home to pack his bags in order to return to hospital.

He called up his best friend, who was still an accountant in Vegas,

whose partner was a hospital nurse.

And who advised him that he should get a second opinion.

That, according to his partner, it was not, you know,

it was very unusual that you would need to have immediate open heart surgery,

and that he should get a second opinion.

Rather doubtful about this, because he was extremely worried, he did get on a plane.

He went to Vegas, he got seen by another specialist...

who, to his complete surprise told him there was nothing wrong with his heart.

He saw another specialist, just to make sure. They told him also, there was nothing wrong with his heart.

Relieved, and rather, you know, happy, he returned home and just wanted to really forget about it.

But his friend said: "No, what's going on here? Somethings wrong".

And they went in to see the CEO of Tenet Healthcare, the people running this hospital

(which, by the way, was a private hospital).

And said: "Look, something's wrong, what's going on, what are you going to do about this?"

And basically they were told: not very much.

You know, mistakes get made, it's bad luck,

don't worry about it, we'll look into it, but thank you very much.

They weren't convinced by this, and eventually they decided to contact the FBI.

The reason they contacted the FBI, by the way,

is it was a private healthcare provider in the United States,

they provide Medicare provision of healthcare to the Federal Government.

So, if the Federal Government is getting defrauded, the FBI can get involved.

The FBI started investigating.

Eventually it turned out that hundreds, probably thousands of people,

over a ten, or longer year period, had been operated on unnecessarily.

Most of them had had serious procedures performed on them,

open heart surgery, some had died as a result.

Obviously it's quite a serious operation.

Some people had basically been condemned to a lifetime of pain.

One of the most traumatic examples was a 36-year-old, he had been cut open,

which is obviously what happens in open heart surgery,

and his chest had never knitted back together correctly.

Basically, he would be in pain for the rest of his life.

So, hundreds, thousands of people had been harmed.

One of the interesting things was that in this community

there was already some suspicion, there were anecdotes,

I mean, one of the ones I really liked from this book was the story that went:

don't get a flat tyre outside of Redding Medical Centre because you'll end up with a heart bypass.

[scattered laughter]

You know, but the thing was, there was no data.

People were you know, a bit suspicious, but it was among doctors,

and you, you know, in the community, and who wants to doubt it.

And guess what?

Also, Redding Medical Centre had one of the best mortality rates

for cardiac procedures in the United States,

because if you operate on healthy people, you have a good mortality rate!

[scattered laughter]

So, the other thing, though,

and this is the point that comes to Open Data for me,

the other red flag if you had been looking at the data, was these two things:

one is incredibly low mortality rate,

and (B) that it had almost the highest number of procedures

for the population that it covered in the United States. ////

Which should be a red flag, right? Because, one, it's just a massive outlier on that basis, and also, the more people you should be operating on, the more you're doing marginal cases, the higher should be your mortality rate unless something very odd is going on.

The thought was: what if people had been looking at this data? What if we'd - if this data had been open and public? And not maybe just for particular researchers to look at or the government.

And it kind of reminded me of a phrase that's very famous in Open Source software, which is: "to many eyes, all bugs are shallow". What's great about Open Source software is lots of people can look at it, lots of people can fix it.

And for me, what this was saying was: to many eyes, all anomalies are noticeable. It's somewhat of an exaggeration, but what happens if rather than ten or twenty people who worked in monitoring Medicare provision in the US government, we'd had thousands or millions of people?

If the local journalists or citizens, who had suspicions, had been able to go and look at that data and say: "Whoa! What's going on here? This isn't just anecdotes, there's some data".

And so, and it's not just then, about kind of spotting healthcare errors, or issues, or risks, it's also about things like apps and services.

That you can build with Open Data.

This is a great app built by mySociety in the UK called Mapumental. And the question is, I don't know if people know, London house prices are very expensive, I don't know whether they rival Geneva's, but they're, it's a pretty difficult thing.

And one of the questions was, if I have to work somewhere, and I want to know where I can live, and afford, and I can commute to work in a certain time, and it's not too ugly, this is what this app does.

You can choose the price, you can say where you're going to work, you can choose the commute time, and you can choose the scenicness. And it will show you, on this map, where you can live.

Another example, more about transparency, is a project we did called "Where Does My Money Go?". It's an interactive version, you can kind of draw it out, so what it starts with, is one, is it tells you what your tax is, something that most people often don't know. And it will tell you how much you're paying each day to a particular area of society.

And the dream for me, a dream that we're on the way to realising, is in this visualisation, you can drill down into areas. And my dream is to keep drilling down.

So depending on what day we have, I want to go down, right down through those bubbles, step by step, until I see the money spent on street lights on my street. On filling in potholes. On collecting my rubbish.

And for two reasons: One, obviously there's a question, particularly in some countries, of inefficiency or corruption, but also, just because most of us don't feel very happy about paying tax.

It's not one of those things people welcome! But it's something that we should. Government does an awful lot for us, and having a better sense of where it's going could make us feel an awful lot better about paying that tax.

In the way that when we go to a restaurant, we don't, when we get the bill we don't necessarily feel bad. We feel "Wow, that was a great meal. That was worth it."

But why Open? I've given you examples, and you know, we see a lot of apps and services. Why is Open relevant here?

This goes back to what I said about the information technology, the revolution. So it's the challenge and the opportunity. It's the challenge that we see today, is exploding informational complexity.

I mean, another great story, in the 1820s, all bank clearing in the largest financial centre in the world was done in a single room, where one person from each bank gathered and they'd go round the room pulling out gold, and swapping it around, between different banks.

And that's how they did bank clearing.

Today we have billions of transactions a minute. And the way we as humans deal with complexity is by dividing and conquering it. We split it up into manageable chunks that we deal with.

The other answer, and this answer's particularly relevant about Open Data, is information technology. Today, a smartphone has as much computing power as the system that ran the Apollo moon landings.

And in an even better example of storage, one terabyte of storage today is a hundred dollars. In 1994, this would have cost 400,000 dollars. I can have every financial transaction the UK government, or the US government made last year, or even for the last decade, on my laptop.

That was not possible for an average citizen a decade ago.

So it's mass participation information access, processing, and production. It's decentralisation. And the claim here is that openness is key.

It's because it's about scaling. What we are doing is weaving data together. Like I said, we deal with complexity by splitting it up, we componentise, we split data up into blocks that we recombine. But if we are going to recombine information, we need to put Humpty Dumpty back together again, it won't work most of the time if it is closed.

We need Open Data to scale and to componentise.

And it's a point just to make here in this respect, that you might think: "Well you know, you're talking about Open Data, you know, this could be true of anything! Why don't we have like, Open Cars, and Open Shoes, and you know, why don't we just share everything, man! It would be so beautiful!".

Right? And the sad thing is, is that that hasn't generally worked as a way of organising most production in our society.

Instead, we have private property, and so we don't do that much openness relatively. But there's something different about digital information. We all know it, but it's worth emphasising, which is, it's very cheaply copied.

I mean, give me a copy of your data isn't a problem if you're the government. Give me a copy of your car, or your house, or whatever, is.

And it's also about innovation here. I mean, in a way it's almost the purest aspect of markets. Markets are about moving things to the person who could use them most best.

And that's true of data. The best thing to do with your data will likely be thought of by someone else.

And vice versa! You will think of the best thing to do with someone else's data. And Open Data allows us, in the most frictionless, easiest way, to move data to where it can be most optimally used.

Particularly, if you're government.

So in short, it's about better understanding. It's about better government. It's about better research. It's about better economy.

And something also for companies and governments, I think it's about better engagement. It's about a closer relationship, sometimes, between your citizens and you as the government.

Between you, even possibly, as a company, and your users.

So I wanted to kind of finish here by saying where we're going.

The story was, of this talk, was, you know, where are we? Why have we got here? And where are we going?

So one answer is just more use. So right now, I just said at the beginning, Open Data is relatively young. This vast outpouring, for example, of government data, that anyone can freely use, reuse, and redistribute, is really new. Even if it's done three years ago.

For example, in the UK, much of the most useful data that could be released has only been released in the last six months or a year.

You want prescription data? Are you a pharmaceutical company? And you want to know what kind of prescription habits are going on in the UK? I would emphasise: at an anonymised or somewhat aggregate level.

Do you want to know about what crimes are going on? Are you building a real estate website and you want data on environment, or you want data on unemployment, or other information about where properties are situated?

You can now get that. So I think there's going to be a lot more use from business. There'll be a lot more use from everyone.

But I think particularly business is going to wake up to the opportunities here.

I think it's also going to lead to more data. One is, government is going to be more data. I think also businesses are going to realise, and communities, that they want to share back some of that data, some of the data they have.

It's not going to be their kind of crown jewels, and it's not going to, it's often going to start out with data that's not core to their business. Right? It's kind of like Nike, they realised that by opening and sharing data, they can scale in a way they can't on their own.

And does it mean that richer data, going back -- how could I leave out Hegel and Marx in a talk like this -- "Quantity changes quality" as Hegel told us. And more data, going back to that woven ball, more data actually means better data.

It means richer data, it's a qualitative difference in what we can do. Geodata on it's own isn't that useful. Transport data on it's own isn't useful. Geodata plus transport data is useful!

And we're going to be seeing data refining. Data is the new oil, right? So, we're going to refine it. And that's going to be a big business. Higher quality data.

And I want to leave you with a couple of thoughts. So, one is: Some people say, "well, okay, but, you know, selling data is big business". And it is, but going forward in some of these things like software, data is going to be a platform.

It's not a commodity. Businesses built purely on selling data, I just don't think I'm going to make it.

You need to be building on your data, not attempting to purely sell it.

And the other answer is to be modest. So I said: where are we going? I don't know if people know, and this takes us back to an earlier age, an age of electricity and steam, of Faraday.

So he's demonstrating electricity at the Royal Society, and Gladstone, the future Prime Minister of England, sees him do this stuff, you know, the frog legs move, and Gladstone's like: "well, I mean, this is party trick, Faraday, it's great, but, what's really, you know, what's electricity going to amount to?"

And Faraday says to him: "Well, what's the use of a baby?"

You know, a baby when it's young is not very useful.

[scattered laughter]

But it grows up into something! And that is where we are going today. We are the beginning of the Open Data journey. And partly is, we don't know what it's going to grow up into.

Thank you very much!

[Applause]

QUESTIONER: Um, citizens and um, I guess patients in hospitals, assume that the institutions have all this data and it's very well organised, and it's a question of will. Have you encountered cases in which they simply don't have it, or they have it, and it's just such a mess that they're too embarrassed to give it out?

RUFUS: Absolutely. I mean, one story that kind of intrigues me, is we've been building this "Where Does My Money Go?" open spending project. And one of the things the government mandated was giving out, rather than just high-level financial information, giving out information at a detailed level. You know, so they now publish, for example, spending data from each government department monthly, every transaction over 25,000 pounds.

Every purchase they make, every mobile phone provider they contract with, we get that data. And one of the intriguing things, of their mandating this, was it turned out, before, they had no way, before they did this, of actually seeing, on any regular basis, what their department spent money on.

Because in fact, the only thing they reported up onto, in central government to Treasury, was kind of like, how much did you spend against Project X that you were allocated budget for?

You know, departments, were actually really intrigued, they say "Oh, well that other department's going with Vodafone, and we're with Orange, and look how much they're paying per month!"

So I think in essence it is really driving changes in government, and yeah, there are people, I think you'd been worried about giving out data quality. I was just talking to the Department of Education last week and they said -- you know, one of the things -- they had financial information from schools, and which they were slowly being mandated to publish.

And schools are suddenly all ringing up, saying: "Well we never really bothered to really update that information to be accurate! Uh, we really want to do it right now".

So I think that definitely does happen, yep.

QUESTIONER: Are you seeing now new roles in government, to help facilitate this?

RUFUS: Yeah. I mean, to take another example, I, sorry. Both in government, so the UK government has a transparency kind of 'czar' if you like. Also I learnt, is Nike hired an Open Data evangelist. One of the things they, while they were implementing this programme, they actually hired explicitly, an Open Data evangelist.

So yeah, I think we are, we're definitely seeing this in government. Both in the tech level, but also at the policy level.

And I think it's not just government, it will also be companies doing this, and so on, who will be saying: "We need an Open Data expert. We need to be aware of what's going on here and be able to plan it as part of our strategy."

QUESTIONER: A final question. You mentioned that, kind of outsourcing, almost, some of this data refining, outside government or the big institutions, has helped them. Can you tell us any stories of kind of gratitude being expressed by the government?

RUFUS: Well, I mean, to kind of, yeah. I mean there was an interesting example actually where we had some complaint because the open spending data I told you about, where we're aggregating the government spending and financial data -- you know, the site had a few performance issues, occasionally, as we loaded more data in.

I remember kind of getting this call kind of going : "Well, you know, we're a little bit upset". You know, data.gov.uk, and it turned out the reason was, the Treasury kept looking at this data, and they were annoyed when the site was going down.

So that was really intriguing to me, that we were kind of one of the best, at least, up-to-date aggregators out there.

Um, I think you are already seeing people doing stuff with the data and kind of doing stuff, sometimes for free. And you don't have to have the shiny front-end.

I mean, one of the things, we went about, we went on about, I know Tim Berners-Lee went on about -- raw data now, you know, you can build fewer shiny front-ends, and just release raw data. And you know, someone else will help you build the app, the front-end, the interface.

And help you innovate about it. What is the best way to provide healthcare data to citizens, or education data to citizens, so they make better and more informed choices?

I don't know, and the government probably doesn't know.

But somewhere out there, someone is going to innovate and really provide the best way for us to deliver that kind of information to citizens.

QUESTIONER: Thank you very much.

RUFUS: Thank you.

[Applause]

lift _ Video Production ACTUA

Copyright (c) 2012 Lift conference