36C3 Wikipaka WG: Modernizing Wikipedia

36C3 preroll music

Daniel: Good morning! I'm glad you all
made it here this early on the last day. I

know it can can't be easy wasn't easy for
me I have to warn you that the way I

prepared for this song is a bit
experimental. I didn't make a slide set I

just made a mind map and I'll just click
through it while I talk to you. So,

this talk is about modernizing Wikipedia
as you probably have noticed visiting,

Wikipedia can feel a bit like visiting a
website from 10-15 years ago but before I

talk about any problems or things to
improve, I first want to revisit that the

software and the the infrastructure we
build around it has been running Wikipedia

and its sister sites for the last... well
nearly 19 years now and it's extremely

successful. We serve 17 billion page
views a month, yes?

Person in the audience: Could you make it
louder or speak up and also make the image

bigger?

inaudible dialogue

Daniel: Is this better? Like if I speak up
I will loose my voice in 10 minutes it's

already in it, no it's fine. We have
technology for this. I can... the light

doesn't help, yeah the contrast could be
better. Is it better like this? Okay cool.

All right so yeah we are serving 17
billion page views a month, which is quite

a lot. Wikipedia exists in about 100
languages. If you attended the talk about

the Wikimedia infrastructure yesterday, we
talked about 300 languages. We actually

support 300 languages for localization but
we have Wikipedia in about 100, if I'm not

completely off. I find this picture quite
fascinating. This is a visualization of

all the places in the world that are
described on Wikipedia and sister projects

and I find this quite impressive although
it's also a nice display of cultural bias

of course. We, that is Wikimedia
Foundation, run about 900 to a 1000 wikis

depending on how you count, but there are
many many more media wiki installations

out there, some of them big and many many
of them small. We have actually no idea

how many small instances there are. So
it's a very powerful very flexible and

versatile piece of software but, you know, but
sometimes it can feel like... you can do a

lot of things with it, right, but
sometimes it feels like it's a bit

overburdened and maybe you should look at
improving the foundations. So one of the

things that make MediaWiki great but also
sometimes hard to use is that kind of

everything is text, everything is markup,
everything is done with with wikitext,

which has grown in complexity over the
years so if you look at the autonomy of a

wiki page it can be a bit daunting. You
have different syntax for markup at

different kinds of transclusion or
templates and media and some things

actually, you know, get displayed in
place, some things show up in a completely

different place on the page it can be
rather confusing and daunting for

newcomers. And also things like having a
conversation just talking to people like,

you know, having a conversation thread
looks like this. You open the page you

look through the markup and you indent to
make a conversation thread and then you

get confused about the indenting and
someone messes with the formatting and

it's all excellent. There have been many
attempts over the years to improve the

situation, we have things like echo which
notifies you, for instance when someone

mentions your name or someone... It is
also used to to welcome people and do this

kind of achievement unlocked
notifications: hey, you did your first

edit, this is great welcome! To make
people a bit more engaged with the system

but it's really mostly improvements around
the fringes. We have had a system called

Flow for awhile to improve the way
conversations work. So you have more like

a thread structure that the software
actually knows about but then there are

many, well quite a few people who have
been around for a while that are very used

to the manual system and also there's a
lot of tools to support this manual system

which of course are incompatible with
making things more modern. So we use this

for instance on MediaWiki.org which is a
site which is basically a self

documentation site of MediaWiki but on
most Wikipedia this is not enabled or at

least not used for default everywhere. The
biggest attempt to move away from the text

only approach is Wikidata, which we
started in 2012. The idea of Wikidata of

course, if you didn't attend many great
talks we had about it here over of the

course of the Congress, is a way to
basically model the world using structured

data, using a semantic approach instead of
natural language which has its own

complexities but at least it's a way to
represent the knowledge of the world in a

way that machines can understand. So this
would be an alternative to wiki text but

still the vast majority of things
especially on Wikipedia are just markup.

And this markup is pretty powerful and
there's lots of ways to extend it and to

do things with it. So a lot of things on
MediaWiki are just DIY, just do it

yourself. Templates are a great example of
this. Infoboxes of course, the nice blue

boxes here on the right side of pages, are
done using templates but these templates

are just for formatting, there is not data
processing there's no the data base or

structured data backing them. It's just
basically, you know, it's still just

markup. It's still... you have a predefined
layout but you're still feeding a text not

data. You have parameters but the values
of the parameters are still again maybe

templates or links or you have markup in
them, like you know HTML line breaks and

stuff. So it's kind of semi structured.
And this of course is also used to do

things like workflow. The template... Oh
no, this was actually an infobox, wrong

picture, wrong capture. This is also used
to do workflows, so if a page on Wikipedia

gets nominated for deletion you put manual
put a template on the page that defines

why this is supposed to be deleted and
then you have to go to a different page

and put a different template there, giving
more explanation and this again is used

for discussion. It's a lot of structure
created by the community and maintained by

the community, using conventions and tools
built on top of what is essentially just a

pile of markup. And because doing all this
manually is kind of painful, only on there

we created a system to allow people to add
JavaScript to the site, which is then

maintained on wiki pages by the community
and it can tweak and automate. But again,

it doesn't really have much to work with,
right? It basically messes with whatever

it can, it directly interacts with the DOM
of the page, whenever the layout of the

software changes, things break. So this is
not great for for compatibility but it's

used a lot and it is very important for
the community to have this power. Sorry, I

wish there was a better way to show these
pictures. Okay, that's just to give you an

idea of what kind of thing is implemented
that way and maintained by the community

on their site. One of the problems we have
with that is: these are bound to a wiki

and I just told you that we run over 900
of these not over 9,000 and it would be

great if you could just share them between
wikis but we can't. And again, there have

been... we have been talking about it a
lot and it seems like it shouldn't be so

hard, but you kind of need to write these
tools differently, if you want to share

them across sites, because different sites
use different conventions, they use

different templates. Then it just doesn't
work and you actually have to write decent

software that uses internationalization if
you want to use it across wikis. While

these are usually just you know one-off
hacks with everything hard-coded we would

have to put in place an
internationalization system and it's

actually a lot of effort and there's a lot
of things that are actually unclear about

it. So, before I dive more deeply into the
different things that will make it hard to

improve on the current situation and the
things that we are doing to improve it do

we have any questions or do you have any
other - do you have any things you may

find particularly, well, annoying or
particularly outdated, when interacting

with Wikipedia? Any thoughts on that?
Beyond what I just said?

Microphone: The strict separation, just in
Wikipedia, between mobile layout and

desktop layout.
Daniel: Yeah. So, actually having a

reactive layout system that would just
work for mobile and desktop in the same

way and allowing the designers and UX
experts, who work on the system to just do

this once and not two or maybe even three
times - because of course we also have

native applications for different
platforms - would be great and it's

something that we're looking into at the
moment. But it's not, you know , it's not

that easy we could build a completely new
system, that does this but then again you

would be telling people: "You can no
longer use the old system", but now they

have build all these tools that rely on
how the old system works and you have to

port all of this over so there's a lot of
inertia. Any other thoughts? Everyone is

still asleep that's excellent. So I can
continue. So, another thing that makes it

difficult to change how MediaWiki works or
to improve it is that we are trying to do

well to be at least two things at once: on
the one hand we are running a top 5

website and serving over 100,000 requests
per second using the system and you on the

other hand, at least until now, we have
always made sure that you can just

download MediaWiki and install it on a
shared hosting platform you don't even

need root on the system, right? You don't
even need administrative privileges you

can just set it up and run it in your web
space and it will work. And, having the

same piece of software do both, run in a
minimal environment and run at scale, is

rather difficult and it also means that
there's a lot of things that we can't

easily do, right? All this modern micro
service architecture separate front-end

and back-end systems, all of that means
that it's a lot more complicated to set up

and needs more knowledge or more
infrastructure to set up and so far that

meant we can't do it, because so far there
was this requirement that you should

really be able to just run it on your
shared hosting. And we are currently

considering to what extent we can continue
this, I mean, container based hosting is

picking up. Maybe this is an alternative
it's still unclear but it seems like this

is something that we need to reconsider.
Yeah, but if we make this harder to do

then a lot of current users of MediaWiki
would maybe not, well, maybe no longer

exist or at least would not exist as they
do now, right. You probably have seen

this nice MediaWiki instance the Congress
wiki. Which - with a completely customized

skin and a lot of extensions installed to
allow people to define their sessions

there and making sure these sessions
automatically get listed and get put into

a calendar - this is all done using
extensions, like Semantic MediaWiki, that

allow you to basically define queries in
the wiki text markup. Yeah, another thing

that, of course, slows down development is
that Wikimedia does engineering on a,

well, comparatively a shoestring budget,
right? The budget of the Wikimedia

Foundation, the annual budget is something
like a hundred million dollars, that

sounds like a lot of money, but if you
compare it to other companies running a

top five or top ten website it's like two
percent of their budget or something like

that, right? It's really, I mean, 100
million is not peanuts but compared to

what other companies invest to achieve
this kind of goal it kind of is, so , what

this budget translates into is something
like 300, depending on how you count,

between three hundred and four hundred
staff. So, this is the people who run all

of this, including all the community
outreach all the social aspects all the

administrative aspects. Less than half of
these are the engineers who do all this.

And we have like, something like 2,500
servers, bare-metal, so, which is not a

lot for this kind of thing. Which also
means that we have to design the software

to be not just scalable but also quite
efficient. The modern approach to scaling

is usually scale horizontally make it so
you can just spin up another virtual

machine in some cloud service, but, yeah,
we run our own service, we run our own

servers, so we can design to scale
horizontally, but it means ordering

hardware and setting it up and it's going
to take half a year or so. And we don't

actually have that many people who do
this, so, scalability and performance are

also important factors when designing the
software. Okay. Before I dive into what we

are actually doing - any questions? This
one in the back. Wait for the mic, please.

In the very...
Q: Hi!

Daniel: Hello.
Q: So, you said you don't have that many

people, but how many do you actually have?
Daniel: For... it's something like 150 engineers

worldwide. It always depends on what you
count, right? So you count the people, who

- do you count engineers, who work on the
native apps, do you account engineers, who

work on the Wikimedia cloud services -
actually we do have cloud services, we

offer them to the community to run their
own things, but we don't run our stuff on

other people's cloud. Yeah, so depending
on how you count or something and whether

you count the people working here in
Germany for Wikimedia Germany, which is a

separate organization technically - it's
something like 150 engineers.

Q: Thanks!
Q: I'm interested: What are the reasons

that you don't run on other people's
services like on the cloud. I mean, then

it will be easy to scale horizontally,
right?

Daniel: There's, well, one reason is being
independent, right? If we, yeah, I imagine

we ran all our stuff on Amazon's
infrastructure and then maybe Amazon

doesn't like the way that the Wikipedia
article about Amazon is written - what do

we do, right? Maybe they shut us down,
maybe they make things very expensive,

maybe they make things very painful for
us, maybe there is some at least like it

self-censorship mechanism happening and we
want to avoid that. There are there are

thoughts about this there are thoughts
like maybe we can do this at least for

development infrastructure and CI, not for
production or maybe we can make it so that

we run stuff in the cloud services by more
than one vendor, so we basically we spread

out so we are not reliant on a single
company. We are thinking about these

things but so far the way to actually stay
independent has been to run our own

servers.
Q: You've been talking about scalability

and changing the architecture, that kind
of seems to imply to me that there's a

problem with scaling at the moment or that
it's foreseeable that things are not gonna

work out if you just keep doing what
you're doing at the moment. Can you maybe

elaborate on that.
Daniel: So, there's, I think there's two sides

to this. On the one hand the reason I
mentioned it is just that a lot of things

that are really easy to do basically for
me, right? Works on my machine are really

hard to do if you want to do them at
scale. That's one aspect. The other aspect

is MediaWiki is pretty much a PHP monolith
and that means getting it always means

copying the monolith and breaking it down
so you have smaller units that you can

scale and just say, yeah, I don't know, I
need more instances for authentication

handling or something like that. That
would be more efficient, right, because

you have higher granularity, you can just
scale the things that you actually need

but that of course needs rearchitecting.
It's not like things are going to explode

if we don't do that very soon, it's not,
so there's not like an urgent problem

there. The reason for us to rearchitect is
more, to gain more flexibility in

development, because if you have a
monolith that is pretty entangled, code

changes are risky and take a long time.
Q: How many people work on product design

or like user experience research to, like,
sit down with users and try to understand

what their needs are and from there
proceed.

A: Across... I don't have an exact number,
something like five.

Audience: Do you think that's sufficient?
Herald: The question was, whether it's

sufficient. So just...
Daniel: Probably not? But it's more than,

that's more people than we have for
database administration, and that's also

not sufficient.
Herald: Are the further questions? I don't

think.
Daniel: Okay. So, one of the things, that

holds us back a bit, is that there's
literally thousands of extensions for

MediaWiki and the extension mechanism is
heavily reliant on hooks, so basically on

callbacks. And, we have - I don't have a
picture, I have a link here - we have a

great number of these. So, you see, each
paragraph is basically documenting one

callback that you can use to modify the
behavior of the software and, I mean,

there's, I have never counted, but
something like a thousand? And all of them

are of course interfaces to extra - to
software that is maintained externally, so

they have to be kept stable and if you
have a large chunk of software that you

want to restructure but you have a
thousand fixed points that you can't

change, things become rather difficult.
It's basi.. yeah, these hook points kind

of, like, they act like nails in the
architecture and then you kind of have to

wiggle around them - it's fun. We are
working to change that. We want to

architect it so the interface that is
exposed to these hooks become much more

narrow and the things that these hooks or
these callback functions can do is much

more restricted. There's currently an RSC
open for this, has been open for a while

actually. The problem is that in order to
assess whether the proposal is actually

viable you have to survey all the current
users of these hooks and make sure that we

can, the use case is still covered in the
new system and, yeah, we have like a

thousand hook points and we have like a
thousand extensions that's quite a bit of

work. Another thing that I'm currently
working on is establishing a stable

interface policy. This may sound pretty
obvious - it has a lot of pretty obvious

things like, yeah, if you have a class and
there's a public method then that's a

stable interface it will not just change
without notice, we have deprecation policy

and all that. But if you have worked with
extensible systems that rely on the

mechanisms of object-oriented programming,
you may have come across the question

whether a protected method is part of this
stable interface of the software or not,

or maybe the constructor? I don't know, if
you have worked in environments that use

dependency injection the idea is basically
that the construction signature should be

able to change at any time but then you
have extensions that you're subclassing and

things break. So, this is why we are
trying to establish a much more

restrictive stable interface policy, that
would would make explicit things like

constructor signatures actually not being
stable and that gives us a lot more wiggle

room to restructure the software.
MediaWiki itself has grown as a software

for the last 18 years or so and, at least
in the beginning, was mostly created by

volunteers. And in a monolithic
architecture there's a great tendency to

just, you know, find and grab the thing
that you want to use and just use it.

Which leads to, well, structures like this
one: everything depends on everything. And

if you change one bit of code everything
else may or may not break. And with, yeah.

And if you don't have great test coverage
at the same time this just makes it so

that any change becomes very risky and you
have to do a lot of manual testing a lot

of manual digging around, touching a lot
of files and we are for the last year,

year and a half we have started a
concerted effort to tie the worst - to cut

the worst ties, to decouple these things
that are, basically that have most impact

there's a few objects in the software that
rep... - for instance one that represents

the user and one that represents a title
that are used everywhere and the way

they're implemented currently also means
that they depend on everything and that of

course is not a good situation. On a,
well, a similar idea on a higher level is

decomposition of the software so the
decoupling was about the software

architecture this is about the system
architecture breaking up the

monolith itself into multiple services that
serve different purposes. The specifics of

this diagram are not really relevant to
this talk. This is more to, you know, give

you an impression of the complexity and
the sort of work we are doing there. The

idea is that perhaps we could split out
certain functionality into its own service

into a separate application, like maybe
move all the search functionality into

something separate and self-contained, but
then the question is how do you, again,

compose this into the final user interface
- at some point these things have to get

composed together again - and again this
is a very trivial trivial issue if you

only want to want this to work on your
machine or you only need to serve a

hundred users or something. But doing this
at scale doing it at the rate of something

like 10,000 page views a second, I said a
hundred thousand requests earlier but that

includes resources, icons, CSS and all
that. So, yeah, then you have to think

pretty hard about what you can cache and,
thank you, how you can recombine things

without having to recompute everything and
this is something that we are currently

looking into - coming up with a
architecture that allows us to compose and

recombine the output of different
background services. Okay. Before I

started this talk I said I would probably
roughly use half of my time going through

the presentation and I guess I just hit
that spot on. So, this is all I have

prepared but I'm happy to talk to you more
about the things I said or maybe any other

aspects of this that you may be interested
in. If any comments or questions. Oh!

Three already.
Q: First of all thanks a lot for the

presentation, such a really interesting
case of a legacy system and thanks for the

honesty. It was really interesting as a,
you know, software engineer to see how

that works. I have a question about
decoupling, so, I mean, I kind of, you

have like, probably your system is
enormous and how do you find, so to say,

the most evil, you know, parts which
sort of have to be decoupled. Do you use other

software, with, you know, this, like, what
a metrics and stuff or do you just know,

kind of intuitively..
Daniel: Yeah, it's actually, this is quite

interesting and maybe I can, maybe we can
talk about it a bit more in depth later.

Very quickly: it's a combination on the
one hand you just have the anecdotal

experience of what is actually annoying
when you work with the software and you

try to fix it and on the other hand I try
to find good tooling for this and the

existing tooling tends to die when you
just run it against our code base. So, one

of the things that you are looking for are
cyclic dependencies but the number of

possible cycles in a graph grows
exponentially with a number of nodes. And

if you have a pretty tightly knit graph
that number quickly goes into the

millions. And, yeah, the tool just goes to
100% CPU and never returns. So, I spend

quite a bit of time trying to find
heuristics to get around that - was a lot

of fun. I can, yeah, we can talk about
that later, if you like. Okay, thanks.

Q: So what exactly is this Wikidata you
mentioned before. Is it like an extension

or is it a completely different project?
Daniel: Wiki - so there's an extension called

Wikibase, that implements this, well I
would say, ontological modeling interface

for MediaWiki and that is used to run a
website called Wikidata which has

something like 30 million items modeled
that describe the world and serve as a

machine-readable data back-end to other
wiki project, other Wikimedia projects.

Yeah, I used to work on that project for
Wikimedia Germany. I moved on to do

different things now for a couple of
years. Lukas here in front is probably the

person most knowledgeable about the latest
and greatest in the Wikidata development.

Q: You've shortly talked about test
coverage. I will be into history..

Daniel: Sorry?
Q: You talked about test coverage.

Daniel: Yes.
Q: I would be interested in if you amped

your efforts to help you modernize it and
how your current situation is with test

coverage.
Daniel: Test coverage for MediaWiki core is below

50%. In some parts it's below 10% which is
very worrying. One thing that we started

to look into, like half a year ago, is
instead of writing unit tests for all the

code that we actually want to throw away,
before we touch it, we tried to improve

the test coverage using integration tests
on the API level. So we are currently in

the process of writing a suite of tests,
not just for the API modules, but for all

the functionality, all the application
logic behind the the API. And that will

hopefully cover most of the relevant code
paths and will give us confidence when we

refactor the code.
Q: Thanks.

Herald: Other questions?
Q: So you said that you have this legacy

system and eventually you have to move
away from it but are there any, like, I

don't know, plans for the near future to,
I don't know. At some point you have to

cut the current infrastructure to your
extensions and so on and it's a hard cut, I

see. But are there any plans to build it
up from scratch or what are the plans?

Daniel: Yeah, we are not going to rewrite from
scratch - that's a pretty sure fire way to

just kill the system. We will have to make
some tough decisions about backwards

compatibility and probably reconsider some
of the requirements and constraints we

have, well, with respect to the platforms
we run on and also the platforms we serve.

One of the things that we have been very
careful to do in the past for instance is

to make sure that you can do pretty much
everything with MediaWiki with no

JavaScript on the client side. And that
requirement is likely to drop. You will

still be able to read of course, without
any JavaScript or anything, but the extent

of functionality you will have without
JavaScript on the client side is likely to

be greatly reduced - that kind of thing.
Also we will probably end up breaking

compatibility to at least some of the
user-created tools. Hopefully we can offer

good alternatives, good APIs, good
libraries that people can actually port

to, that are less brittle. I hope that
will motivate people and maybe repay them

a bit for the pain of having their tool
broken. If we can give them something that

is more stable, more reliable, and
hopefully even nicer to use. Yeah, so,

it's small increments, bits, and pieces
all over the system there's no, you know,

no great master plan, no big change to
point to really.

Herald: Okay, okay, further questions?
Daniel: I plan to just sit outside here at

the table later if you just want to come
and chat so we can also do that there.

Herald: Okay, so, last call are there any
other questions? It does not appear so,

so, I'd like ask for a huge applause for
Daniel for this talk.

Applause

36C3 postroll music

Subtitles created by c3subtitles.de
in the year 2020. Join, and help us!