36C3 Wikipaka WG: Interactively Discovering Implicational Knowledge in Wikidata

36C3 preroll music

Herald Angel: We have Tom and Max here.
They have a talk here with a very

complicated title that I don't quite
understand yet. It's called "Interactively

Discovering Implicational Knowledge in
Wikidata. And they told me the point of

the talk is that I would like to
understand what it means and I hope I

will. So good luck.
Tom: Thank you very much.

Herald: And have some applause, please.

applause

T: Thank you very much. Do you hear me?
Does it work? Hello? Oh, very good. Thank

you very much and welcome to our talk
about interactively discovering

implicational knowledge in Wikidata. It
is more or less a fun project we started

for finding rules that are implicit in
Wikidata – entailed just by the data it

has, that people inserted into the
Wikidata database so far. And we will

start with the explicit knowledge. So the
explicit data in Wikidata, with Max.

Max: So. Right. What what is Wikidata?
Maybe you have heard about Wikidata, then

that's all fine. Maybe you haven't, then
surely you've heard of Wikipedia. And

Wikipedia is run by the Wikimedia
Foundation and the Wikimedia Foundation

has several other projects. And one of
those is Wikidata. And Wikidata is

basically a large graph that encodes
machine readable knowledge in the form of

statements. And a statement basically
consists of some entity that is connected

– or some some entities that are connected
by some property. And these properties

can then even have annotations on them.
So, for example, we have Donna Strickland

here and we encode that she has received a
Nobel prize in physics last year by this

property "awarded" and this has then a
qualifier "time: 2018" and also "for:

Chirped Pulse Amplification". And all in
all, we have some 890 million statements

on Wikidata that connect 71 million items
using 7000 properties. But there's also a

bit more. So we also know that Donna
Strickland has "field of work: optics" and

also "field of work: lasers" so we can use
the same property to connect some entity

with different other entities. And we
don't even have to have knowledge that

connects the entities. We can have a date
of birth, which is 1959. Nineteen ninety.

No. Nineteen fifty nine. Yes. And this is
then just a plain date, not an entity. And

now coming from the explicit knowledge
then, well, we have some more we have

Donna Strickland has received a Nobel
prize in physics and also Marie Curie has

received the Nobel prize in physics. And
we also know that Marie Curie has a Nobel

prize ID that starts with "phys" and then
"1903" and some random numbers that

basically are this ID. Then Marie Curie
also has received a Nobel prize in

chemistry in 1911. So she has another
Nobel ID that starts with "chem" and has

"1911" there. And then there's also
Frances Arnold, who received the Nobel

prize in chemistry last year. So she has a
Nobel ID that starts with "chem" and has

"2018" there. And now one one could assume
that, well, everybody who was awarded the

Nobel prize should also have a Nobel ID.
So everybody who was awarded the Nobel

prize should also have a Nobel prize ID,
and we could write that as some

implication here. So "awarded(nobelPrize)"
implies "nobelID". And well, if you

look sharply at this picture, then there's
this arrow here conspicuously missing that

Donald Strickland doesn't have a Nobel
prize ID. And indeed, there's 25 people

currently on Wikidata that are missing
Nobel prize IDs, and Donna Strickland is

one of them. So we call these people that
don't satisfy this implication – we call

those counterexamples and well, if you
look at Wikidata on the scale of really

these 890 million statements, then you
won't find any counterexamples because

it's just too big. So we need some way to
automatically do that. And the idea is

that, well, if we had this knowledge that
while some implications are not satisfied,

then this encodes maybe missing
information or wrong information, and we

want to represent that in a way that is
easy to understand and also succinct. So

it doesn't take long to write it down, it
should have a short representation. So

that rules out anything, including complex
syntax or logical quantifies. So no SPARQL

queries as a description of that implicit
knowledge. No description logics, if

you've heard of that. And we also want
something that we can actually compute on

actual hardware in a reasonable timeframe.
So our approach is we use Formal Concept

Analysis, which is a technique that has
been developed over the past several years

to extract what is called propositional
implications. So just logical formulas of

propositional logic that are an
implication in the form of this

"awarded(nobelPrize)" implies "nobleID".
So what exactly is Formal Concept

Analysis? Off to Tom.
T: Thank you. So what is Formal Concept

Analysis? It was developed in 1980s by a
guy called Rudolf Wille and Bernard Ganter

and they were restructuring lattice
theory. Lattice theory is an ambiguous

name in math, it has two meanings: One
meaning is you have a grid and have a

lattice there. The other thing is to speak
about orders – order relations. So I like

steaks, I like pudding and I like steaks
more than pudding. And I like rice more

than steaks. That's an order, right? And
lattices are particular orders which can

be used to represent propositional logic.
So easy rules like "when it rains, the

street gets wet", right? So and the data
representation those guys used back then,

they called it a formal context, which is
basically just a set of objects – they

call them objects, it's just a name –, a
set of attributes and some incidence,

which basically means which object does
have which attributes. So, for example, my

laptop has the colour black. So this
object has some property, right? So that's

a small example on the right for such a
formal context. So the objects there are

some animals: a platypus – that's the fun
animal from Australia, the mammal which is

also laying eggs and which is also
venomous –, a black widow – the spider –,

the duck and the cat. So we see, the
platypus has all the properties; it has

being venomous, laying eggs and being a
mammal; we have the duck, which is not a

mammal, but it lays eggs, and so on and so
on. And it's very easy to grasp some

implicational knowledge here. An easy rule
you can find is whenever you endeavour a

mammal that is venomous, it has to lay
eggs. So this is a rule that falls out of

this binary data table. Our main problem
then or at this point is we do not have

such a data table for Wikidata, right? We
have the implicit graph, which is way more

expressive than binary data, and we cannot
even store Wikidata as a binary table.

Even if you tried to, we have no chance to
compute such rules from that. And for

this, the people from Formal Context
Analysis proposed an algorithm to extract

implicit knowledge from an expert. So our
expert here could be Wikidata. It's an

expert, you can ask Wikidata questions,
right? Using this SPARQL interface, you

can ask. You can ask "Is there an example
for that? Is there a counterexample for

something else?" So the algorithm is quite
easy. The algorithm is the algorithm and

some expert – in our case, Wikidata –, and
the algorithm keeps notes for

counterexamples and keeps notes for valid
implications. So in the beginning, we do

not have any valid implications, so this
list on the right is empty, and in the

beginning we do not have any
counterexamples. So the list on the left,

the formal context to build up is also
empty. And all the algorithm does now is,

it asks "is this implication, X follows Y,
Y follows X or X implies Y, is it true?"

So "is it true," for example, "that an
animal that is a mammal and is venomous

lays eggs?" So now the expert, which in
our case is Wikidata, can answer it. We

can query that. We showed in our paper we
can query that. So we query it, and if the

Wikidata expert does not find any
counterexamples, it will say, ok, that's

maybe a true, true thing; it's yes. Or if
it's not a true implication in Wikidata,

it can say, no, no, no, it's not true, and
here's a counterexample. So this is

something you contradict by example. You
say this rule cannot be true. For example,

when the street is wet, that does not mean
it has rained, right? It could be the

cleaning service car or something else. So
our idea now was to use Wikidata as an

expert, but also include a human into this
loop. So we do not just want to ask

Wikidata, we also want to ask a human
expert as well. So we first ask in our

tool the Wikidata expert for some rule.
After that, we also inquire the human

expert. And he can also say "yeah, that's
true, I know that," or "No, no. Wikidata

is not aware of this counterexample, I
know one." Or, in the other case "oh,

Wikidata says this is true. I am aware of
a counterexample." Yeah, and so on and so

on. And you can represent this more or
less – this is just some mathematical

picture, it's not very important. But you
can see on the left there's an exploration

going on, just Wikidata with the
algorithm, on the right an exploration, a

human expert versus Wikidata which can
answer all the queries. And we combined

those two into one small tool, still under
development. So, back to Max.

M: Okay. So far for that to work, we
basically need to have a way of viewing

Wikidata, or at least parts of Wikidata,
as a formal context. And this formal

context, well, this was a binary table, so
what do we do? We just take all the items

in Wikidata as objects and all the
properties as attributes of our context

and then have an incidence relation that
says "well, this entity has this

property," so it is incident there, and
then we end up with a context that has 71

million rows and seven thousand columns.
So, well, that might actually be a slight

problem there, because we want to have
something that we can run on actual

hardware and not on a supercomputer. So
let's maybe not do that and focus on

a smaller set of properties that are
actually related to one another through

some kind of common domain, yeah? So it
doesn't make any sense to have a property

that relates to spacecraft and then a
property that relates to books – that's

probably not a good idea to try to find
implicit knowledge between those two. But

two different properties about spacecraft,
that sounds good, right? And then the

interesting question is just how do we
define the incidence for our set of

properties? And that actually depends very
much on which properties we choose,

because it does – for some properties, it
makes sense to account for the direction

of the statement: So there is a property
called parent? Actually, no, it's child,

and then there's father and mother, and
you don't want to turn those around, as do

you want to have "A is a child of B," that
should be something different than "B

is a child of A." Then there's the
qualifiers that might be important for

some properties. So receiving an award for
something might be something different

than receiving an award for something
else. But while receiving an award in 2018

and receiving one in 2017, that's probably
more or less the same thing, so we don't

necessarily need to differentiate that.
And there's also a thing called subclasses

and they form a hierarchy on Wikidata. And
you might also want to take that into

account because while winning something
that is a Nobel prize, that means also

winning an award itself, and winning the
Nobel Peace prize means winning a peace

prize. So there's also implications going
on there that you want to respect. So,

to see how we actually do that, let's look
at an example. So we have here, well, this

is Donald Strickland. And – I forgot his
first name – Ashkin, this is one of the

people that won the Nobel prize in physics
with her last year. And also Gérard

Mourou. That is the third one. They all
got the Nobel prize in physics last year.

So we have all these statements here, and
these two have a qualifier that says

"with: Gérard Mourou" here. And I don't
think the qualifier is on this statement

here, actually, but it doesn't actually
matter. So what we've done here is,

put all the entities in the small graph as
rows in the table. So we have Strickland

and Mourou and Ashkin, and also Arnold and
Curie that are not in the picture. But you

can maybe remember that. And then here we
have awarded, and we scaled that by the

instance of the different Nobel prizes
that people have won. So that's the

physics Nobel in the first column, the
chemistry Nobel Prize in the second column

and just general Nobel prizes in the third
column. There's awarded and that is scaled

by the "with" qualifier, so awarded with
Gérard Mourou. And then there's field of

work, and we have lasers here and
radioactivity, so we scale by the actual

field of work that people have. And well
then, if we look at what kind of incidence

we get for Donna Strickland, she has a
Nobel prize in physics and that is also a

Nobel prize, and she has that together
with Mourou. And she has "field of work:

lasers," but not radioactivity. Then,
Mourou himself: he has a Nobel prize in

physics, and that is a Nobel prize, but
none of the others. Ashkin gets the Nobel

prize in physics, and that is still a
Nobel prize, and he gets that with Gérard

Mourou. And also he works on lasers, but
not in radioactivity. So Frances Arnold

has a Nobel prize in chemistry, and that
is a Nobel prize. And Marie Curie, she has

a Nobel prize in physics and one in
chemistry, and they are both a Nobel

prize. And she also works on
radioactivity. But lasers didn't exist

back then, so she doesn't get "field of
work: lasers." And then basically this

table here is a representation of our
formal context. So and then we've actually

gone ahead and started building a tool
where you can interactively do all these

things, and it will take care of building
the context for you. You just put in the

properties, and Tom will show
you how that works.

T: So here you see some first screenshots
of this tool. So please do not comment on

the graphic design. We have no idea about
that, we have to ask someone about that.

We're just into logics, more or less. On
the left, you see the initial state of the

game. On the left you have five boxes:
they're called countries and borders,

credit cards, use of energy, memory and
computation – I think –, and space

launches, which are just presets we
defined. You can explore, for example, in

the case of the credit card, you can
explore the properties from Wikidata which

are called "card network," "operator," and
"fee," so you can just choose one of them,

or on the right, "custom properties," you
can just input the properties you're

interested in Wikidata, whatever one of
the seven thousand you like, or some

number of them. On the right, I chose then
the credit card thingy and I now want to

show you what happens if you now explore
these properties, right? The first step in

the game is that the game will ask – I
mean, the game, the exploration process –

will ask, is it true that every entity in
Wikidata will have these three properties?

So are they common among all entities in
your data, which is most probably not

true, right? I mean, not everything in
Wikidata has a fee, at least I hope. So,

what I will do now, I would click the
"reject this implication" button, since

the implication "Nothing implies
everything" is not true. In the second

step now, the algorithm tries to find the
minimal number of questions to obtain the

domain knowledge, so to obtain all valid
rules in this domain. So next question is

"is it true that everything in Wikidata
that has a 'card network' property also

has a 'fee' and an 'operator' property?"
And down here you can see Wikidata says

"ok, there are 26 items which are
counterexamples," so there's 26 items in

Wikidata which have the "card network"
property but do not have the other two

ones. So, 26 is not a big number, this
could mean "ok, that's an error, so 26

statements are missing." Or maybe that
that's, really, that's the true case.

That's also ok. But you can now choose
what you think is right. You can say, "oh,

I would say it should be true" or you can
say "no, I think that's ok, one of these

counterexamples seems valid. Let's reject
it." I in this case, rejected it. The next

question it asks: "is it true that
everything that has an operator has also a

fee and a card network?" Yeah, this is
possibly not true. There's also more than

1000 counterexamples, one being, I think a
telecommunication operator in Hungary or

something. And so we can reject this as
well. Next question, everything that has

an operator and a card network – so card
network means Visa, MasterCard, whatever,

all this stuff – is it true that they have
to have a fee?" Wikidata says "no," it has

23 items that contradict it. But one of
the items, for example, is the American

Express Gold Card. I suppose the American
Express Gold Card has some fee. So this

indicates, "oh, there is some missing data
in Wikidata," there is something that

Wikidata does not know but should know to
reason correctly in Wikidata with your

SPARQL queries. So we can now say, "yeah,
that's, uh, that's not a reject, that's an

accept," because we think it should be
true. But Wikidata thinks otherwise. And

you go on, we go on. This is then the last
question: "Is it true that everything that

has a fee and a card work should have an
operator," and you see, "oh, no counter

examples." This means Wikidata says "this
is true," because it says there is no

counterexample. If you're asking Wikidata
it says this is a valid implication in the

data set so far, which could also be
indicating that something is missing, I'm

not aware if this is possible or not, but
ok, for me it sounds reasonable. Everyone

has a fee and a card network should also
have an operator, which meens a bank or

something like that. So I accept this
implication. And then, yeah, you have won

the exploration game, which essentially
means you've won some knowledge. Thank

you. And the knowledge is that you know
which implications in Wikidata are true or

should be true from your point of view.
And yeah, this is more or less the state

of the game so far as we programmed it in
October. And the next state will be to

show you some – "How much does your
opinion of the world differ from the

opinion that is now reflected in the
data?" So is what you think about the data

true, close to true to what is true in
Wikidata. Or maybe Wikidata has wrong

information. You can find it with that.
But Max will tell me more about that.

M: Ok. So let me just quickly come
back to what we have actually done. So we

offer a procedure that allows you to
explore properties in Wikidata and the

implicational knowledge that holds between
these properties. And the key idea's here

that when you look at these implications
that you get, while there might be some

that you don't actually want because they
shouldn't be true, and there might also be

ones that you don't get, but you expect to
get because they should hold. And these

unwanted and/or missing implications, they
point to missing statements and items in

Wikidata. So they show you where the
opportunities to improve the knowledge in

Wikidata are, and, well, sometimes you
also get to learn something about the

world, and in most cases, it's that the
world is more complicated than you thought

it was – and that's just how life is. But
in general, implications can guide you in

your way of improving Wikidata and the
state of knowledge therein. So what's

next? Well, so what we currently don't
offer in the exploration game and what we

definitely will focus next on is having
configurable counterexamples and also

filterable counterexamples – right now you
just get a list of a random number of

counterexamples. And you might want to
search through this list for something you

recognise and you might also want to
explicitly say, well, this one should be a

counterexample, and that's definitely
coming next. Then, well, domain specific

scaling of properties, there's still much
work to be done. Currently, we only have

some very basic support for that. So you
can have properties, but you can't do the

fancy things where you say, "well,
everything that is an award should be

considered as one instance of this
property." That's also coming and then

what Tom mentioned alread: compare your
knowledge that you have explored through

this process against the knowledge that is
currently on Wikidata as a form of seeing

"where do you stand? What is missing in
Wikidata? How can you improve Wikidata?"

And well, if you have any more suggestions
for features, then just tell us. There's a

Github link on the implication game page.
And here's the link to the tool again. So,

yeah, just let us know. Open an issue and
have fun. And if you have any questions,

then I guess now would be the time to ask.
T: Thank you.

Herald: Thank you very much, Tom and Max.

applause

Herald: So we will switch microphones now
because then I can hand this microphone to

you if any of you have a question for our
two speakers. Are there any questions or

suggestions? Yes.
Question: Hi. Thanks for the nice talk. I

wanted to ask what's the first question,
what's the most interesting implication

that you've found?
M: Yeah. That would have made for a

good back up slide. The most interesting
implication so far –

T: The most basic thing you would expect
everything that is launched in space by

humans – no, everything that landed from
space, that has a landing date, also has a

start date. So nothing landed on earth,
which was not started here.

M: Yes.
Q: Right now, the game only helps you find

out implications. Are you also planning to
have that I can also add data like for

example, let's say I have twenty five
Nobel laureates who don't have a Nobel

laureate ID. Is there plans where you
could give me a simple interface for me to

Google and add that ID because it would
make the process of adding new entities to

Wikidata itself more simple.
M: Yes. And that's partly hidden

behind this "configurable and filterable
counterexamples" thing. We will probably

not have an explicit interface for adding
stuff, but most likely interface with some

other tool built around Wikidata, so
probably something that will give you

QuickStatements or something like that.
But yes, adding data is definitely on the

roadmap.
Herald: Any more questions? Yes.

Q: Wouldn't it be nice to do this in other
languages, too?

T: Actually it's language independent, so
we use Wikidata and then as far as we

know, Wikidata has no language itself. You
know, it has just items and properties, so

Qs and Ps, and whatever language you use,
it should be translated in the language of

the properties, if there is a label for
that property or for that item that you

have. So if Wikidata is aware of your
language, we are.

Herald: Oh, yes. More!
M: Of course, the tool still needs to be

translated, but –
T: The tool itself, it should be.

Q: Hi, thanks for the talk. I have a
question. Right now you only can find

missing data with this, right? Or surplus
data. Would you think you'd be able to

find wrong information with a similar
approach.

T: Actually, we do. I mean, if we Wikidata
has a counterexample to something we would

expect to be true, this could point to
wrong data, right? If the counterexample

is a wrong counterexample. If there is a
missing property or missing property to an

item.
Q: Ok, I get to ask a second question. So

the horizontal axis in the incidence
matrix. You said it has 7000, it spans

7000 columns, right?
M: Yes, because there's 7000 properties in

Wikidata.
Q: But it's actually way more columns,

right? Because you multiply the properties
times the arguments, right?

M: Yes. So if you do any scaling then of
course that might give you multiple

entries.
Q: So that's what you mean with scaling,

basically?
M: Yes. But already seven thousand is way

too big to actually compute that.
Q: How many would it be if you multiply

all the arguments?
M: I have no idea, probably a few million.

Q: Have you thought about a recursive
method, as counterexamples may be wrong by

other counterexamples, like in an
argumentative graph or something like

this?
T: Actually, I don't get it. How can a

counterexample be wrong through another
counterxample?

Q: Maybe some example says that cats can
have golden hair and then another example

might say that this is not a cat.
T: Ah, so the property to be a cat or

something cat-ish is missing then. Okay.
No, we have not considered so far deeper

reasoning. This horn-propositional logic,
you know, it has no contradictions,

because all you can do is you can
contradict by counterexamples, but there

can never be a rule that is not true, so
far. Just in your or my opinion, maybe,

but not in the logic. So what we have to
think about is that we have bigger

reasoning, right? So.
Q: Sorry, quick question. Because you're

not considering all the 7000 odd
properties for each of the entities,

right? What's your current process of
filtering? What are the relevant

properties? I'm sorry, I didn't get that.
M: Well, we basically handpick those. So

you have this input field? Yeah, we can go
ahead and select our properties. We also

have some predefined sets. Okay. And
there's also some classes for groups of

properties that are related that you could
use if you want bigger sets,

T: for example, space or family or what
was the other?

M: Awards is one.
T: It depends on the size of the class.

For example, for space, it's not that
much, I think it's 10 or 15 properties. It

will take you some hours, but you can do
because they are 15 or something like

that. I think for family, it's way too
much, it's like 40 of 50 properties. So a

lot of questions.
Herald: I don't see any more hands. Maybe

someone who has not asked the question yet
has another one we could take that,

otherwise we would be perfectly on time.
And maybe you can tell us where you will

be for deeper discussions where people can
find you.

T: Probably at the couches.
Herald: The couches, behind our stage.

M: Or just running around somewhere. So
there's also our DECT numbers on the

slides; it's 6284 for Tom and 6279 for me.
So just call and ask where we're hanging

around.
H: Well then, thank you again. Have a

round of applause.
applause

T: Thank you.
M: Well, thanks for having us.

Applause

postroll music

subtitles created by c3subtitles.de
in the year 2020. Join, and help us!