cdn.media.ccc.de/.../wikidatacon2019-13-eng-OpenStreetMap_and_Wikidata_hd.mp4

Hello. So, good afternoon.

Welcome to the OpenStreetMap
and Wikidata workshop.

My name is Eugene.

And together with Edward,
we'll be talking about OpenStreetMap

and how it can work together
with Wikidata.

So, just a show of hands.

Who here has an OpenStreetMap account?

Okay, some. So, probably
this is not new to you.

But for those who are not familiar
with OpenStreetMap,

I'll give an introduction
to OpenStreetMap and its data model.

So, basically, what is OpenStreetMap?

It is basically a crowdsourced project
to map the whole world.

And the usual way we introduce
OpenStreetMap to people

is like OpenStreetMap
is like Wikipedia for maps.

But actually, a more accurate way
to introduce OpenStreetMap

is that it is like Wikidata
for geographical data.

But that presupposes that the audience
already knows or is familiar

with what Wikidata is.

And why do we say
that OpenStreetMap is like Wikidata?

And that's because both
have quite a lot of things in common,

both being crowdsourced
and open data projects.

So, you know Wikidata--

it has items, statements,
properties, et cetera.

In the same way, OpenStreetMap
has things like nodes, ways, relations,

that have members and roles,
and these have tags

that are composed of keys and values.

So, as more detail, nodes, ways,
and relations model the geometry

and topology of objects.

And then, we have tags,
which are actually key value strings

that describe the actual things
that those objects represent.

So, to give an example,
here's the Wikidata item for Berlin.

So, we can have property
like population, 3 million something,

with a qualifier, point in time,
and references.

And they have counterparts
in OpenStreetMap.

So, for example, the Berlin relation
in OpenStreetMap

has the tag population is equal
to 3.4 million something,

and it has another tag, source:population
equals this URL and that date.

So, unlike in Wikidata,

wherein you can have qualifiers
and references for your statements,

in OpenStreetMap, the tag is quite flat.

There's no secondary levels of tags.

Everything is flat.

And that's why we have to put
what you call secondary tags.

So, for example here, source:population

to indicate that the population tag
has this source.

Another thing is that OpenStreetMap's tags
are not strictly controlled.

Unlike with Wikidata, wherein you have
to have approval process

before properties are created,

here, OpenStreetMap mappers

can invent and add
any tags that they like.

However, there is a tagging
proposal process

in order to propose common tags

that will be used by mappers
all over the world.

Okay, data modeling discussions
on the Wikidata:Project chat page

are actually quite similar
to the discussions

in OpenStreetMap's tagging mailing list.

For example, here's an example
of discussion on the project tag,

how do we model a building
that has changed its use?

In OpenStreetMap,
we have similar discussions.

How do we tag these sorts of buildings?

So, I've given an introduction
of what OpenStreetMap is.

I'd love to discuss it more,
but we don't have enough time.

So, we'll go into how do we link

between OpenStreetMap
and Wikidata together.

I don't have to explain
why linking is a good thing.

We're all Wikidatans, and we know
that linking data is a good thing.

So, how do we actually link
Wikidata with OpenStreetMap?

So, from Wikidata to OpenStreetMap,

Wikidata items on places can link
to OpenStreetMap relations

using the OSM relation ID,
or the P402 property.

So, the question is: why only relations?

That's because OSM IDs are not stable.

For example, you can change nodes
to represent a different object.

Ways can be split

to add new information about those ways.

However, relatively, relations
in OpenStreetMap are relatively stable.

At least for major relations,
such as administrative boundaries,

or highway routes,
or public transportation routes.

That way, you can link at least the,
for example, here,

Berlin Wikidata data edit item,

can link to the relation
representing the boundary

in OpenStreetMap via its ID.

In terms of the ontology,
Wikidata items and properties

for geographical features
can link to "equivalent," in quotes,

OpenStreetMap classes
using the OSM tag or key property.

For example, the lighthouse item
in Wikidata has the value

for OpenStreetMap tag or key
Tag:man_made=lighthouse.

That means that lighthouses are equivalent
to objects that are tagged

in OpenStreetMap
with man-made lighthouse.

Going in the other direction,
OpenStreetMap objects can link

to corresponding Wikipedia articles
and Wikidata items

using the Wikipedia
and Wikidata tags, respectively.

So, here's an example.

The OpenStreetMap relation for Berlin.

We have the Wikidata tag, Q64,

and the Wikipedia article
linking to the German article for Berlin.

There are also several Wikidata
secondary tags,

such as for example, brand:wikidata,
architect:wikidata, artist:wikidata,

or name:etymology:wikidata.

We use this in order to exactly specify

what we are referring to.

For example, on the top part here,
we have the example.

There's an artwork in OpenStreetMap

that was created
by the artist named Herakut,

but who is that?

So, in order to specify exactly,
we use artist:wikidata,

and that Q ID number.

So that you can be exactly sure
which Herakut artist it really is.

This is also useful, for example,
if you're tagging, for example,

objects in OpenStreetMap
that are in a different language.

For example, in Japan, you might have
a fast-food restaurant

called Makudonarudo,
which is actually McDonald's.

So, you can tag that using
the brand:wikidata tag

pointing to the McDonald's
item in Wikidata.

So, in terms of ontology,
we define and describe the tags

in OpenStreetMap
on the OpenStreetMap Wiki,

and we can add links to corresponding
Wikipedia articles and Wikidata items

so that we can sort of explain
the correspondences and relations

between these tags and items in Wikidata.

Okay, so how do OpenStreetMap
and Wikimedia use each other's data?

So, first, we have the interactive maps.

So, OpenStreetMap data powers

the Wikimedia Foundation's
Kartotherian map tile service,

which is used by the Kartographer
MediaWiki extension.

So, basically, any time you see
an interactive map

or almost any interactive map
on any Wikimedia project,

that is usually powered
by the Kartotherian map tile service.

For example, here's the interactive map
for Berlin in the English Wikivoyage.

So, the base map there is all coming
from OpenStreetMap.

So, another thing that
the Kartographer extension can do

is it can pull and overlay geometry
from OpenStreetMap.

So, here's the infobox on Commons
for the Berlin category.

And the map there, you can see
an outline for Berlin, there.

That outline comes from OpenStreetMap.

In 2008, the foundation released
localized map tiles for Kartotherian,

and this leveraged the multilingual
name tags in OpenStreetMap,

so that you can view those maps
that you see on Wikimedia projects

in the user's language.

Then, how do we use Wikidata
in OpenStreetMap?

For example, when tagging brands,

for example, in shops
and restaurants or banks,

OpenStreetMap's Name Suggestion Index
uses Wikidata to provide brand identity

and improved tagging.

So, for example, if you tag
an object in OpenStreetMap

with brand Wikidata pointing
to the McDonald's item in Wikidata,

the name field is now automatically locked

so that users cannot just change that
to, for example, Burger King.

And then, you can edit or also pull icons,

the McDonald's icon there

that is taken from the Facebook
item property

in Wikidata.

So, yeah.

So, that, at least, when users are tagging
these shops in OpenStreetMap,

they can be sure
that they're doing it correctly.

Okay, so Sophox is a SPARQL endpoint
for OpenStreetMap data.

So, this service can use RDF federation
to also query linked Wikidata items.

So, actually, in OpenStreetMap,
we usually use other query services,

such as Overpass.

But if you want to also query
using Wikidata items,

we have the Sophox endpoint
that you can use.

And for geocoding,
if you're not familiar with geocoding,

basically, that's the technology
wherein given an address,

you are returned geocoordinates.

So, we have what we call Nominatim,
which is the usual service

in OpenStreetMap for doing geocoding.

And previously, it already uses
Wikipedia tags in OpenStreetMap.

But this year, we added
Google Summer of Code project code

to integrate using Wikidata tags
in Nominatim,

so that search results
can become more relevant for users

who are doing the searches.

And for localization, Mapbox and MapTiler,
which are third-party companies

that extensively use OpenStreetMap,

they use Wikidata to power
their localized map products.

So, basically, if there
are missing name tags in OpenStreetMap,

and if that object is linked to Wikidata,
they can pull the labels from Wikidata,

and use that to show multilingual labels,
if that is missing in OpenStreetMap.

The reason for that is because
we have a philosophy in OpenStreetMap

that we do not try to add too many tags,
especially if that can be automated.

For example,
for automatic transliterations,

if that can be automated, we don't need
to add that to OpenStreetMap.

But in Wikidata, that's no problem.

So, you can do that by doing this linking
between OpenStreetMap and Wikidata.

You don't have to do
that transliteration on your own.

You can just pull it from Wikidata.

And also, the OpenStreetMap Wiki
has the Wikibase extension installed.

So, the idea here is that we want
the tag information or the description

or the description of the tags,
that we use in OpenStreetMap

to be machine-readable.

Hopefully, this will be used
by software and editors

that use OpenStreetMap data to better see

how objects are described
in OpenStreetMap.

Currently, this is not used as much,

but hopefully, as the tagging information
becomes more complete and better,

this can be used
by OpenStreetMap software,

thanks to the Wikibase installation.

Okay, some copyright and IP issues.

Wikidata can't import coordinates
from OpenStreetMap.

The reason for that is because
OpenStreetMap is licensed

under the Open Database License.

And also, we have conflicting doctrines.

Here in the European Union
and the United Kingdom,

we have database rights.

Whereas, in the US, we have the idea
that facts are not copyrightable.

So, we cannot just--
even though you cannot say--

you cannot copyright the fact
that this restaurant or this bank

or this place is at this location,

doing that as an import
or as a batch job

is not allowed

because OpenStreetMap
is protected by database rights

being hosted in the United Kingdom.

Conversely, OpenStreetMap
will not import geodata from Wikidata,

despite the CC0 license,
because of data provenance issues.

If you're not familiar
with how geocoordinates are added

into Wikipedia articles,
usually users just go to Google Maps,

search, and then copy the coordinates
that show up in the results,

and place that
into the Wikipedia articles.

In OpenStreetMap, we, as much as possible,

avoid copying data
from third-party sources

that are proprietary, such as,
for example, Google Maps.

And because of that, we will never,
in OpenStreetMap, never import data

from Wikipedia and also Wikidata,

because most coordinates in Wikidata
have been imported from Wikipedia.

So, it's an established principle
on OpenStreetMap

that we don't import from Wikipedia.

Okay, I'll just then
turn it over to Edward.

(Edward) I'm going to talk
about the process for adding links

from OpenStreetMap to Wikidata.

So, I've written a tool
for automating this process.

Like, it's user-assisted editing.

So, it's not a fully automated tool.

It's available. Anyone can use it.

There's the address.

So, when I run the tool on Berlin,
it finds 2,800 matches.

So, these are Wikidata items

where it thinks it has found
the same OpenStreetMap objects.

So, the matcher is using this criteria.

It looks for things
that are the same entity type.

They've got the same coordinates,

and then either they've got the same name,
street address, or identifier.

So, with the name, it's doing lots
of kind of normalization,

like lower casing, removing spaces,
all kinds of bits and pieces

to try and match up
slightly different ways

that you could write a name.

And I'm looking at names
from different sources,

like the labels, and the aliases,

but also the site links,
the article titles,

and I pull anything in bold
from the Wikipedia article,

so lots of sources for names.

These are the identifiers
that I'm matching on.

So, we've got lots
of identifiers in Wikidata.

OpenStreetMap has identifiers, as well.

So, I've got a mapping
between the name of the tag

for the identifier in OpenStreetMap,
and the property in Wikidata,

and I look for things
that have the same identifier.

So, my first version of this,
I tried to completely automate it,

and the OpenStreetMap community
was not impressed.

So, better to have
a semi-automated process,

so people put in a place name,
and then they see a list of matches,

and they go through
and they check the matches,

and when they're happy, they hit save.

And the OpenStreetMap community
is much happier with that.

It does make mistakes, the software,
it tries very carefully,

but there are errors in there.

So you have to have
someone checking them.

I've got a question of like,
when I designed this,

I felt like there should be
a one-to-one mapping

between things
in OpenStreetMap and Wikidata,

and it doesn't really work.

Like for my example, tunnels
often get represented as two objects

in OpenStreetMap--
one for each tunnel bore,

or each road, lane within the tunnel--

whereas in Wikidata they tend
to be represented as a single item,

so I need to change my software
to take account of this.

And I have difficulties with tram stops.

So one item in Wikidata for a tram stop,

but in OpenStreetMap, it's represented

as a relation with nodes

where the tram stops
on either side of the road.

But I'm using a piece of software
called osm2pgsql

to do the OpenStreetMap side of things.

And it doesn't really support
these relations.

So, I'm struggling with tram stops.

And so, people are using this tool.

There's almost 10,000 changesets
uploaded to OpenStreetMap.

Edits on OpenStreetMap
are grouped into changesets;

they're not individual edits
like on Wikidata.

And I've got over 200 users.

And using this tool, there's been
a quarter of a million links added

to OpenStreetMap.

But overall, those people are adding
Wikidata links by hand,

or with other tools, and there's now
1.4 million OpenStreetMap objects

with a Wikidata tag.

Yeah, so that is our presentation.

Any questions?

And just while we're taking questions,

I'll see if I can do
a live demo of the tool.

Any questions?

(woman) I'm very interested in sort of
surpassing this license problem.

And can you tell me about strategies,
that can be already used, such as--

I understand that there
are some contributions that aren't CC0--

or like the users, or whatever they are--

that can facilitate the exchange
of information between the systems.

(Edward) It's true that when you sign up
to OpenStreetMap,

you can tick a box saying,
"My edits are CC0."

But the difficulty is that you
are often editing something

that somebody else has edited.

And so, it's difficult
to isolate the CC0 edits.

(woman) Maybe like-- further question,
like what can we do about that?

Like, can we discuss-- I mean,
I think it has been tried,

but I don't think it's necessarily
doomed to fail.

(Eugene) Well, the best thing we can do
is try to link items together

using Edward's tool and other tools.

But for the moment, we just try
to map these things separately.

Maybe we can coordinate--

for example, if the third-party database
that we want to import is compatible

with both Wikidata and OpenStreetMap,
you can do a coordinated import to both.

But otherwise, we really have
to respect the license,

because in order
for OpenStreetMap to work,

it really respects intellectual
property and copyright.

(man) Thank you. Is it possible
to change the language of the background

when you go to the map?

Because it appears the language
of the local place

that you are looking for.

(Eugene) So, in OpenStreetMap,
basically, we tag the default name

according to the local language
of that place.

So, for example, if you go to Japan
in OpenStreetMap,

you will see Japanese names.

You cannot do that using
the OpenStreetMap website,

but there are third-party services
or tile services

that provide multilingual maps.

As I mentioned, there's Mapbox,
there's MapTiler.

They provide multilingual maps
so that you can use that

instead of the default layer
in OpenStreetMap.

(man) [inaudible]

or from the OpenStreetMap [inaudible]?

Yeah, for example--

(man) [inaudible]

not actually to this tool,
but also [inaudible].

Well, currently, OpenStreetMap,
as a project does not--

no project to provide this service,

because the idea
is that we provide the data,

and other people can build on that
to provide the services

that users will be able to use.

(man 2) Yeah, this is a great project
for all to [inaudible] on Wikidata.

Say, in Wikidata,
we have a lot of locations,

which is already coded and it is CC0.

So, there are a lot of images,
a lot of other things are in Wikidata.

So, if we integrate
this Wikidata Q items to OSM,

can we pull this,
all the other information

from Wikidata directly to OpenStreetMap,

any kind of tool, or something like that?

Or can we add an image
which is in Commons?

Can we add the link of the image
in Commons to OpenStreetMap,

like this Wikidata ID?

I feel like you don't need to.

Just leave the data in Wikidata,
and access it through the link.

Like just add the link
from OpenStreetMap to Wikidata,

and then, if you want the images--
don't duplicate the data,

don't have the same thing in both places.

And like Eugene was saying,
it's a bit tricky copying the data.

It's true that it's CC0,

but if we just start
importing lots of data,

then OpenStreetMap's going to ask
what's the provenance of the data,

where has all this come from.

I mean, I don't know if Eugene--
if you got anything to add to that.

(Eugene) Well, OpenStreetMap
does have an image tag.

So, you can add that image tag
pointing to a Commons file,

if you really want to.

But if you link it to the Wikidata item,
you don't need that,

because the Wikidata item

would probably have
a Commons category statement,

and that provides you links
to several images, as well.

You don't need to add that
in OpenStreetMap.

Can I just show this quick demo.

This is the software that I built.

So, I've looked up
Orange County in Indiana.

You can see, I've already run the software
in 2017, and I added 43 elements.

It guesses the language is English,

by looking at the number of languages
that the Wikidata labels are in.

And so the software
has found five matches,

and it's got a list of matches
with tick boxes.

There's a map.

It shows you the first paragraph
from the Wikipedia article

in the currently selected language.

If I say, show tags, these are the tags
from OpenStreetMap,

so it's matched--the name is the same.

It says it's found eight name matches,
and it's a hotel which matches.

I can say, show on map.

And the pin is the location
of the Wikidata coordinates,

and it's matched
this hotel building polygon.

So, I can go through,
and you can see some others.

There's a school.

It's failed with the airport.

The airport is represented
both as a node and as a way,

and it can't figure out which one to use,

so it isn't going to do the airport.

Here's a historic bank building
that it's managed to match.

There's an old name tag in OpenStreetMap,

that it's matched the old name,
with the name that is in Wikidata.

And then it's also matched up
public library.

So, if I click on add wikidata tags
to OpenStreetMap,

it gives me a change comment field
where I could edit it--

change comment if I wanted.

And it shows me the same matches again.

And I hit save, and it's edited the map,

and it's added
the Wikidata tags to the map.

(applause)

([Muhammad]) It's actually not a question.

But first, thank you for this project.

My name is [Muhammad Hidjal]
from Palestine.

I am a civil engineer,
and I do some special statistics.

A few months ago, a magazine in my country
asked me to do some statistics

on Nobel Prize winners.

So, for that, I used
Wikidata Query Service,

and ArcGIS program for geographic
information system analyzation.

I extracted the place of birth
for all Nobel Prize winners,

and projected them on the map
using ArcGIS program,

and then they asked me,
"How many of them--

how many of the winners were born
in the north part of the world,

how many of them were born
in the south part of the world?"

The problem here is
that ArcGIS program is not free

and I don't want to use it anymore.

Can I do these statistics
using OpenStreetMap

after projecting the special
information of these?

Okay, so the problem is
that what you're doing--

what you're trying to do is
what we call a geospatial analysis.

However, OpenStreetMap is a data project.
We provide data.

And what you can do is, for example,
take the data from OpenStreetMap,

take the data from
your Nobel Prize place of births,

and use a tool, like ArcGIS,
which is not free,

but there's an open source tool,
called QGIS,

which you can use to do
that spatial analysis that you want.

So, you can combine, for example,

the boundaries for northern countries
versus southern countries,

put that in QGIS, then put your data
with the Nobel Prize place of births,

and then do an intersection
tool or function.

So, yeah.

So, I think we're out of time,
and it's lunch now.

Thanks, everyone.

(applause)