cdn.media.ccc.de/.../wikidatacon2019-1091-eng-Wikidata_Statistics_What_Where_and_How_hd.mp4

Yes, Wikidata Statistics:
What, Where, and How?

This has been an attempt of an overview
for analytical systems

focusing on what was developed
with the Wikimedia Deutschland

in the previous almost three years

since I started doing data science
for Wikidata and thе dictionary.

So, during this presentation,
I will try to switch from the presentation

to the dashboards
and show you the end data products.

However, if that causes any trouble,

so this is actually the URL
of the analytics portal.

So everything that
I will be presenting here,

whatever you can see on the slides,
you can also check out later

from the presentation,
go and play with the real thing.

Otherwise, you will see only
the screenshots here from the slides.

So the goal-- well, the talk
will be a failed attempt to communicate

an almost endlessly
technically complicated field

in terms that can actually motivate
people to start making use

of this analytical product

in which development
we are really putting a lot of effort.

So, as I said, I will try
to provide an overview

of the Wikidata Statistics
and Analytics systems.

So I will try to exemplify the usage
of some of them, not all.

And also I will try to go just
a little bit under the hood

to try to illustrate how it is done,
what is done here,

because I thought it might be
interesting to the audience.

Okay, so say...

In analytics and data science,
you always start with formulating

as clearly as possible
your goals and motivations.

Otherwise, you enter into endless cycles
of developing analytical tools

and data science products
that actually do something,

but nobody really understands
what they're being built for.

In 2017, in Wikimedia Deutschland,
a request, a demand was formulated--

we said that we needed
an analytical system

that will give an insight into the ways

that Wikidata items are reused
across the Wikimedia projects,

meaning across the Wikipedia universe--
all the encyclopedias,

and then Wikivoyage,
Wikibooks, WikiCite, etc.--

all the websites, approximately 800
that we are actually managing.

So just to explain the differences
between the data.

On the left, for example, you see a small
or very small substitute Wikidata.

These are the languages,
some of the Slavic, I think, languages,

and in Wikidata they are connected,

but they are properties and belong
to different classes, etc.

But we were looking
for a different kind of mapping.

So what you see here,
on the right side, is a set of items

all belonging to the class
of architectural structures, I would say.

And this here is the result
of their empirical embeddings.

So the items related here--

they are linked by their similarity
of usage across Wikipedias, for example.

So what does it mean-- the similarity?

To be similar in terms of how an item
is used across the Wikipedias.

So imagine you take an area of numbers,

and each element of an area
is one project-- it's English Wikipedia,

it is French Wikivoyage,
it is Italian Wikipedia, etc.

And then, you count how many times

a particular item has been used
in that project.

So you use an area of numbers
to describe the item that way.

It's a little bit more complicated
in practice.

And then, you can describe all items
in Wikidata that were ever used

across the websites at all
by such areas of numbers,

called embeddings, technically, right?

From those data,
using different distance metrics,

applying machine learning methods,
doing dimensionality reduction,

and similar things,

you can actually figure out
what is the similarity pattern.

And here items are connected,

but how similar are their patterns
of usage across different Wikipedias.

Once again, every visualization,
every result that I show--

there is a link on the presentation,
so you can go and check for yourself.

You can play
with this thing interactively.

Similarly, we will be able to derive
a graph like this one.

This one does not connect
the Wikidata items, it connects projects.

But looking at how similar they are

in terms of how they use
different Wikidata items.

To be precise as possible,

the data that we use to do this--
they do not live in Wikidata,

they are not a part of the Wikidata,

data does not at all [locate] here.

We have the Wikidata,
we have formulated our motivational goals,

and immediately we started talking
about the data model and the structures.

What structures and data models
you need to answer the questions

that you have initially proposed.

So there is Wikibase
and the client site tracking mechanism,

which is installed in all those wikis,

that actually tracks the Wikidata usage
on a project, on Wikipedia, for example.

So every time an item is used
in [meaningful works]

or in a different way--
there is a role in a huge sequel table

that enters and checks
the usage of that number.

Now, immediately, we had to face
a data-engineering problem, of course,

because we are talking
about hundreds of huge sequel tables,

and we had to do
machine learning and statistics

across all the data together,
not separately,

in order to be able to produce structures,
like this one or like this one.

So in cooperation with the Analytics
Engineering Team of the Foundation,

we started transferring
those data from Wikibase

to the Wikimedia Foundation Data Lake
which is actually a big data storage.

The data do not live there
in a relational database.

They live in something similar--

its Hadoop, and Hive tables
are there, etc.,

but it's a huge,
huge engineering procedure.

So not all data in analytics,
especially in big games like this

that we have to play
with Wikidata and Wikipedia,

are immediately available to you.

One source of complication

is before you actually start solving
the problem in a scientific way,

to put it that way, is to engineer
the data stats to prepare the structures

that you actually need for doing
machine-learning statistics

and similar things.

This is a full design of the system
called the Wikidata Concepts Monitor

that tracks their reuse statistics.

I will not go
into details here, of course.

The obvious complication
is that-- as I wrote it up--

many systems need to work together.

You have to synchronize
many different sources of data,

many different infrastructures

just in order to make it happen,
even before starting thinking

in terms of methodologies, science,
statistics, and similar.

As I said, we started
with our goals and motivation,

then, typically, the data model
and the structures that you need--

they correspond to those goals
and motivations that should always be--

your first step in developing
an analytics project.

Then you figure out
it's really too complicated,

it cannot be done when one person--

It cannot be done on one computer,
to put it that way.

So we needed to work
with the analytics infrastructure

and then add an additional layer
of complication--

that's communication
with external teams and cooperators

because, obviously, such a system
cannot be managed easily by one person.

Actually, I think
it would be pretty impossible.

So, as I mentioned,
there is this Data Lake,

our big data storage in Hadoop,

and the team of awesome data engineers
in the Foundation

called the Analytics Engineering Team.

To data science, data engineers are people
who actually watch your back

while you're trying to do your things.

If you cannot rely on
a good engineering team,

there's not much you will be able
to do by yourself.

This infrastructure is actually
maintained by the Foundation,

so you enter through
several statistical servers--

these blue boxes down there.

You can communicate
with the relational database systems.

We used the MariaDB.

You can communicate with the Data Lake.

And, of course, for your computations,
you go to the so-called Analytics Cluster

where you do things
like Apache Spark that actually--

it's the only really efficient way
to process the data

that we need to process.

When I started doing this back in 2017,
I remember when I saw

only the schema of the infrastructure
for the first time.

If I could not rely on my colleague
Adam Shorland--

who is still with us
in Wikimedia Deutschland--

I would never make it, I wouldn't even
know how to navigate the structure.

As you start building a project
to do analytics for Wikidata,

you see how increasingly it gets
more and more complicated

because you have to deal
with synchronizing different systems,

different teams, infrastructures,
different data stats.

However, it pays off,

that synchronization and all the pain.

It can get really nasty sometimes,
and the most recent example

is the production
of the Data Quality Report for Wikidata.

That's an initial assessment
of the quality of work we had in Wikidata.

In order to produce it,

we had to rely on the Quality Predictions
from the ORES system,

the machine learning system,
developed by Aaron Halfaker,

and the scoring platform

to combine that with the Wikidata
Concepts Monitor reuse statistics.

We the revision history, the full revision
history of all Wikipedias

is available in one single
huge big data table

called the MediaWiki History.

That lives in the Data Lake.

And also we had to process
the JSON Dump in HDFS.

So we're talking about form
as in structures,

like two machine learning systems
with their complexities,

and two huge data sets.

Everything needs to work in sync in order
to be able to produce the Quality Report

that we're presenting
this year at WikidataCon.

But if we didn't do, if we [listed]
or something like that,

we couldn't say, we couldn't show
beautiful things like this.

So on the horizontal axis, you have
the ORES Quality Prediction score.

We use five categories.

And you can inform yourself-- just google
"Wikidata data quality categories."

You will find the description.

The A-class to the left--
the best items that we have,

and at the same time--
that's the green box--

they are the most
reused items in Wikipedia.

So it's not like,
as Lydia explained yesterday,

it's not like all our items
are of the highest quality.

To the contrary, we have many items
that are not of that high quality,

but at least we know
what we're doing with them.

And you can see the regularity.

As the quality of an item
decreases from left to right,

the items tend to be less and less reused.

So also this synchronization
helped us learn things like this.

To the right, for example,
this five-time series here.

Each time series corresponds
to one of the quality categories--

A, B, C, or D.

And the time is on the horizontal axis
running from left to right.

And you can see here how many items
from each quality-class

received their latest revision when.

So the top quality class A
that is this [inaudible] line

which is found, say,
at the most right position here,

and the shortest line.

So those are the best items that we have.

And what you can see
is actually that there is no item

that did not receive at least
one revision after December 2018,

meaning one thing-- if you want quality
in Wikidata, you have to work on it.

So the best items that we have
are actually the items

that we're really paying attention to.

If you look at the classes
of lower quality, the other time-series,

you will see that we have items
that were revised in 2012

for the last time.

So it tells a story of responsibilities--

how much work we put
into the items [that actually work].

What brings quality.

While we do these things,
we also try to make use

of the byproducts
of these procedures as possible.

So, for example, in order
to develop the project

called Wikidata Languages Landscape--

I think I mentioned it yesterday
during the Birthday Presentation--

I had to perform a quite thorough study

of the sub-ontology
in Wikidata of languages.

And you know what?
There are problems in that ontology.

I tried not to miss
to give you an opportunity.

So this is the dashboard actually
about the languages

called the Wikidata Languages Landscape.

Once again, you have all the URLs
in the presentation.

So for example, you want to take a look
at a particular language.

Say, English, okay.

So the dashboard will generate
its local ontological context

and mark all the relations
of the form instance

of subclass often part of.

Why did I choose to do this?

To help you fix the language ontology.

Why? Because you will find many languages,
for example, my native language

which used to be Serbo-Croatian,

and for silly reasons now we have Serbian
and Croatian-- it's a political thing.

I don't want to go into it,
but you realize

that Serbian is now, for example,
at the same time

a subclass of Serbo-Croatian
and a part of Serbo-Croatian.

Which still holds for the Croatian--

Croatian is also a part
and a subclass of Serbo-Croatian.

So Serbo-Croatian used to be a language.

Now we don't have
normative support for it.

But still, it's not a language class,
it's a language.

Can it be a part of it
or can it be a subclass of it?

So it's a confusion of [methodological]
and set-theoretic relations,

and I think it should be fixed somehow.

In other words, don't say

that you don't have the tool
to fix the ontology.

Just find some time and go play with it.

Speaking of languages, I mentioned,
just to show you this project.

Many people liked this thing
what I published online on Twitter.

That's one of the things, you know.

Data science is usually
sold via visualizations.

People like to visualize things,

and, of course,
we do pay attention to that.

Aesthetics is a part of communication.

It's not the most important thing
for a scientific finding

to show you something beautiful,

but if you can show something beautiful,
you shouldn't miss the opportunity.

So here we did
with the languages in Wikidata

the same thing that we do
with items and projects

in the Wikidata Concepts Monitor.

We actually group languages by similarity,
and the similarity was defined

as how much do they overlap
across the items.

So if I can talk about
the same things in English

and in some West-African
language, for example,

then those two things, those two languages

are similar in terms
of their reference sets.

What they can refer to.

Each language here

points to its closest neighbor,
nearest neighbor--

to the most which is most similar to it.

And, of course, you can see
these groupings actually occur naturally.

So it's not a fully-connected graph.

Clustering this thing
was nothing like [there is].

Also, what you can learn
from the Languages Landscape project

is when you combine our data
with external resources.

So this is also very informative for us,

for the whole, I would say,
Wikimedia community.

We have the UNESCO language status

which Wikidata actually gets from UNESCO,

its websites and databases,

and the Ethnologue language status
on the vertical axes.

We have the Concepts Monitor
reuse statistic.

So we look at all the items that have
a label in a particular language,

and then we look at
how popular those items are,

how many times people used them.

Of course, those safe national languages,
languages that are not endangered,

they have a slight advantage.

But the situation is not really that bad.

Say, for example, take a look
at the Ethnologue category

of "Second language only"--
that's the rightmost one.

You will see three languages
there being reused

in a way comparable to the most favorable,

not endangered category
of national languages.

It's not like a gender bias.

Wikipedia seems to be really reflecting
the gender bias that exists in the world.

Then we have nice initiatives like women
who are trying to fix this thing.

With languages, well, of course,
some languages are a little bit favored,

but it's not that bad,

and that finding really brought
a lot of joy to ourselves.

Now, speaking of external resources,
every time that I look at this graph,

I say to myself, "We know
who is the queen of the databases."

You know the external identifiers
property in Wikidata.

So here we take all external identifiers
that were present in August,

JSON Dump of Wikidata, which we processed.

Then, once again,
did some statistics on it

and grouped all the external identifiers
by how much they overlap across the items.

Aha, here we are.

That visualization, except for maybe
being aesthetically pleasing,

is not that useful,

but you have an interactive version
developed in the dashboard.

If you go and inspect
the interactive version,

you can learn, for example,
one obvious fact

that they really follow
some natural semantics.

They are grouped in intuitive ways.

We should be perfectly expecting them
to give some feedback on the quality

of the organizational data in Wikidata,

telling that situation
is really not that bad.

What I am saying is
that all the external identifiers

from the databases
on sports, for example,

you will find to be in one cluster.

And then, for example, you will even
be able to figure out what sport.

Databases on tennis are here,
databases on football are here, etc.

Yes, these external resources

are things that we really try
to pay a lot of attention to.

All right, as I said, the final thing
is communication and aesthetics.

We do pay attention to it.

So, for example, this thing--
many people liked it.

It's a little bit rescaled for aesthetics,

the same network of external identifiers
that you were able to see.

But you don't get
to these results for free, of course.

For example, this one was obtained
by running a clustering algorithm

on Jaccard distances--
technical terms, I'm not going into it.

And first, we had to start from a matrix
actually derived from 408 languages

that are reused across the Wikimedia.

Wikidata knows about
many languages, not only 400.

But only 400 of them are actually
labels of the items that get reused

across 60 million items contingency
matrix-- that's a lot of computations.

To add an additional layer of complication

and, of course, the most beautiful part
of your work as a data scientist,

but it doesn't get to occupy

more than, say, 10% or 15% of your time,

because everything else
goes to data engineering

and synchronization of different systems.

With the machine learning
and statistic things,

we use plenty of different algorithms.

I don't think this is now time to go
and talk about details of these things.

I have plenty of opportunities
to discuss them,

but it's typically
a highly technical topic,

better suited for a scientific conference.

Here are old layers of complexity.

In the end, we have to add
deployment and dashboards,

because they won't build
themselves to this thing.

And all these things, all these phases

of development of analytics
of data science project

need to fit together in order
to be able to derive empirical results

on the system of Wikidata's complexity.

The true picture is that you cannot
really just run through these cycles.

All the phases of the process
are interdependent

because you really
have to plan very early on

what visualizations are you going to use,
what technology you will use

to render those visualizations in the end.

What machine learning algorithms
you will be using,

because all of them have their own taste
about what data structures they like.

And then you hit the constraints
of infrastructure-- similar things.

I am not complaining,
I'm really enjoying this.

This is the most beautiful playground
I've ever seen in my life.

Thanks to you and people
who built Wikidata.

Thank you very much!

That would be it.

(moderator) Thank you, Goran.

(applause)

(moderator) You have time
for a couple of questions.

(man) Well, you did a lot of research,
I can see that.

(Goran) Sorry?

(man) You did a lot of research,
I can see that.

I'm wondering if there anything
that you discovered during the research

that surprised you.

Thank you for that question.

Actually, I wanted to focus
on that in this talk

until I realized that we simply
won't have enough time

to explain everything.

Most of the time
when you're analyzing big datasets

structured in a way like Wikidata.

Even when you're going to the wild,
meaning study the reuse of data

across Wikipedia,

where actually people can do
whatever they like with those items,

you have a lot of data,
a lot of information.

Of course, you see structure.

Most of the time, 90% of the time,
you see things that are expected.

Things like what projects
we make the most use of Wikidata.

And you can almost--
you didn't have to do too much statistics,

you can rely on the expectations
of all the world and see what's happening.

Many things were surprising,

and those things that were surprising
are really the most informative things.

When one communicates the findings
from analytics and such systems,

it's important, people typically expect
either "wow" visualizations

and have tons of data so we can always
deliver "wow" visualizations,

or they expect to learn things like,

"Our project is doing better
than this project"

or "Yes, we are rocking!" etc.,

while the goal of the whole game
should actually be to learn

what is wrong, what is not working,
what could be done better.

Many things were surprising.

For example, the distribution
of item usage across languages--

that was surprising to me.

This thing.

So I did not really expect
that the situation with languages

will be this good, I would say.

My expectation would be that languages
that have less economic support,

normative support,
even political support--

that's a fact when you talk
about languages--

will not be so widely reused
across the Wikimedia universe.

In fact, it turns out
that the differences-- we can see them,

but it's far away from gender bias
which is really bad, I think,

we need to work there.

That was surprising, for example.

It was a positive surprise,
to put it that way.

Then from time to time,
we discover projects

that actually do a great job by reusing
the Wikidata content and Wikimedia.

We're totally surprised to learn that
such a project can do it.

Then you start thinking, you figure out
there is a community of people

actually doing it.

And it's a strange feeling because I get
to see all these things through machines,

through databases,
through visualizations and tables,

and it's always that strange feeling
when I realize this result was produced

by a group of people, they don't even know
the time looking at their result now.

(moderator) Another question?

Thank you.

Is that it? Thank you very much!

(moderator) Thank you.

(applause)