36C3 Wikipaka WG: Infrastructure of Wikipedia

Music

Herald:Hi! Welcome, welcome in Wikipaka-
WG, in this extremely crowded Esszimmer.

I'm Jakob, I'm your Herald for tonight
until 10:00 and I'm here to welcome you

and to welcome these wonderful three guys
on the stage. They're going to talk about

the infrastructure of Wikipedia.
And yeah, they are Lucas, Amir, and Daniel

and I hope you'll have fun!
Applause

Amir Sarabadani: Hello, my name is
Amir, um, I'm a software engineer at

Wikimedia Deutschland, which is the German
chapter of Wikimedia Foundation. Wikimedia

Foundation runs Wikipedia. Here is Lucas.
Lucas is also a software engineer, at

Wikimedia Deutschland, and Daniel here is
a software architect at Wikimedia

Foundation. We are all based in Germany,
Daniel in Leipzig, we are in Berlin. And

today we want to talk about how we run
Wikipedia, with using donors' money and

not lots of advertisement and collecting
data. So in this talk, first we are going

to go on an inside-out approach. So we are
going to first talk about the application

layer and then the outside layers, and
then we go to an outside-in approach and

then talk about how you're going to hit
Wikipedia from the outside.

So first of all, let's some,
let me get you some information. First of

all, all of Wikimedia, Wikipedia
infrastructure is run by Wikimedia

Foundation, an American nonprofit
charitable organization. We don't run any

ads and we are only 370 people. If you
count Wikimedia Deutschland or all other

chapters, it's around 500 people in total.
It's nothing compared to the companies

outside. But all of the content is
managed by volunteers. Even our staff

doesn't do edits, add content to
Wikipedia. And we support 300 languages,

which is a very large number. And
Wikipedia, it's eighteen years old, so it

can vote now. And also, Wikipedia has some
really, really weird articles. Um, I want

to ask you, what is your, if you have
encountered any really weird article

in Wikipedia? My favorite is a list of
people who died on the toilet. But if you

know anything, raise your hands. Uh, do
you know any weird articles in Wikipedia?

Do you know some?
Daniel Kinzler: Oh, the classic one….

Amir: You need to unmute yourself. Oh,
okay.

Daniel: This is technology. I don't know
anything about technology. OK, no. The, my

favorite example is "people killed by
their own invention". That's yeah. That's

a lot of fun. Look it up. It's amazing.
Lucas Werkmeister: There's also a list,

there is also a list of prison escapes
using helicopters. I almost said

helicopter escapes using prisons, which
doesn't make any sense. But that was also

a very interesting list.
Daniel: I think we also have a category of

lists of lists of lists.
Amir: That's a page.

Lucas: And every few months someone thinks
it's funny to redirect it to Russel's

paradox or so.
Daniel: Yeah.

Amir: But also beside that, people cannot
read Wikipedia in Turkey or China. But

three days ago, actually, the block in
Turkey was ruled unconstitutional, but

it's not lifted yet. Hopefully they will
lift it soon. Um, so Wikipedia, Wikimedia

projects is just not Wikipedia. It's lots
and lots of projects. Some of them are not

as successful as the Wikipedia. Um, uh,
like Wikinews. But uh, for example,

Wikipedia is the most successful one, and
there's another one, that's Wikidata. It's

being developed by Wikimedia Deutschland.
I mean the Wikidata team, with Lucas, um,

and it's being used – it's infobox – it
has the data that Wikipedia or Google

Knowledge Graph or Siri or Alexa uses.
It's basically, it's sort of a backbone of

all of the data, uh, through the whole
Internet. Um, so our infrastructure. Let

me… So first of all, our infrastructure is
all Open Source. By principle, we never

use any commercial software. Uh, we could
use a lots of things. They are even

sometimes were given us for free, but we
were, refused to use them. Second

thing is we have two primary data center
for like failovers, when, for example, a

whole datacenter goes offline, so we can
failover to another data center. We have

three caching points of presence or
CDNs. Our CDNs are all over the world. Uh,

also, we have our own CDN. We don't have,
we don't use CloudFlare, because

CloudFlare, we care about the privacy of
the users and is very important that, for

example, people edit from countries that
might be, uh, dangerous for them to edit

Wikipedia. So we really care to keep the
data as protected as possible.

Applause

Amir: Uh, we have 17 billion page views
per month, and, which goes up and down

based on the season and everything, we
have around 100 to 200 thousand requests

per second. It's different from the
pageview because requests can be requests

to the objects, can be API, can be lots of
things. And we have 300,000 new editors

per month and we run all of this with 1300
bare metal servers. So right now, Daniel

is going to talk about the application
layer and the inside of that

infrastructure.
Daniel: Thanks, Amir. Oh, the clicky

thing. Thank you. So the application layer
is basically the software that actually

does what a wiki does, right? It lets you
edit pages, create or update pages and

then search the page views. interference
noise The challenge for Wikipedia, of

course, is serving all the many page views
that Amir just described. The core of the

application is a classic LAMP application.
interference noise I have to stop

moving. Yes? Is that it? It's a classic
LAMP stack application. So it's written in

PHP, it runs on an Apache server. It uses
MySQL as a database in the backend. We

used to use a HHVM instead of the… Yeah,
we…

Herald: Hier. Sorry. Nimm mal das hier.
Daniel: Hello. We used to use HHVM as the

PHP engine, but we just switched back to
the mainstream PHP, using PHP 7.2 now,

because Facebook decided that HHVM is
going to be incompatible with the standard

and they were just basically developing it
for, for themselves. Right. So we have

separate clusters of servers for serving
requests, for serving different requests,

page views on the one hand, and also
handling edits. Then we have a cluster for

handling API calls and then we have a
bunch of servers set up to handle

asynchronous jobs, things that happen in
the background, the job runners, and…

I guess video scaling is a very obvious
example of that. It just takes too long to

do it on the fly. But we use it for many
other things as well. MediaWiki, MediaWiki

is kind of an amazing thing because you
can just install it on your own shared-

hosting, 10-bucks-a-month's webspace and
it will run. But you can also use it to,

you know, serve half the world. And so
it's a very powerful and versatile system,

which also… I mean, this, this wide span
of different applications also creates

problems. That's something that I will
talk about tomorrow. But for now, let's

look at the fun things. So if you want to
serve a lot of page views, you have to do

a lot of caching. And so we have a whole…
yeah, a whole set of different caching

systems. The most important one is
probably the parser cache. So as you

probably know, wiki pages are created in,
in a markup language, Wikitext, and they

need to be parsed and turned into HTML.
And the result of that parsing is, of

course, cached. And that cache is semi-
persistent, it… nothing really ever drops

out of it. It's a huge thing. And it's, it
lives in a dedicated MySQL database

system. Yeah. We use memcached a lot for
all kinds of miscellaneous things,

anything that we need to keep around and
share between server instances. And we

have been using redis for a while, for
anything that we want to have available,

not just between different servers, but
also between different data centers,

because redis is a bit better about
synchronizing things between, between

different systems, we still use it for
session storage, especially, though we are

about to move away from that and we'll be
using Cassandra for session storage. We

have a bunch of additional services
running for specialized purposes, like

scaling images, rendering formulas, math
formulas, ORES is pretty interesting. ORES

is a system for automatically detecting
vandalism or rating edits. So this is a

machine learning based system for
detecting problems and highlighting edits

that may not be, may not be great and need
more attention. We have some additional

services that process our content for
consumption on mobile devices, chopping

pages up into bits and pieces that then
can be consumed individually and many,

many more. In the background, we also have
to manage events, right, we use Kafka for

message queuing, and we use that to notify
different parts of the system about

changes. On the one hand, we use that to
feed the job runners that I just

mentioned. But we also use it, for
instance, to purge the entries in the

CDN when pages become updated and things
like that. OK, the next session is going

to be about the databases. Are there, very
quickly, we will have quite a bit of time

for discussion afterwards. But are there
any questions right now about what we said

so far? Everything extremely crystal
clear. OK, no clarity is left? I see. Oh,

one question, in the back.
Q: Can you maybe turn the volume up a

little bit? Thank you.
Daniel: Yeah, I think this is your

section, right? Oh, its Amir again. Sorry.
Amir: So I want to talk about my favorite

topic, the dungeons of, dungeons of every
production system, databases. The database

of Wikipedia is really interesting and
complicated on its own. We use MariaDB, we

switched from MySQL in 2013 for lots of
complicated reasons. As, as I said,

because we are really open source, you can
go and not just check our database tree,

that says, like, how it looks and what's
the replicas and masters. Actually, you

can even query the Wikipedia's database
live when you have that, you can just go

to that address and login with your
Wikipedia account and just can do whatever

you want. Like, it was a funny thing that
a couple of months ago, someone sent me a

message, sent me a message like, oh, I
found a security issue. You can just query

Wikipedia's database. I was like, no, no,
it's actually, we, we let this happen.

It's like, it's sanitized. We removed the
password hashes and everything. But still,

you can use this. And, but if you wanted
to say, like, how the clusters work, the

database clusters, because it gets too
big, they first started sharding, but now

we have sections that are basically
different clusters. Uh, really large wikis

have their own section. For example,
English Wikipedia is s1. German Wikipedia

with two or three other small wikis are in
s5. Wikidata is on s8, and so on. And

each section have a master and several
replicas. But one of the replicas is

actually a master in another data center
because of the failover that I told you.

So it is, basically two layers of
replication exist. This is, what I'm

telling you, is about metadata. But for
Wikitext, we also need to have a complete

different set of databases. But it can be,
we use consistent hashing to just scale it

horizontally so we can just put more
databases on it, for that. Uh, but I don't

know if you know it, but Wikipedia stores
every edit. So you have the text of,

Wikitext of every edit in the whole
history in the database. Uhm, also we have

parser cache that Daniel explained, and
parser cache is also consistent hashing.

So we just can horizontally scale it. But
for metadata, it is slightly more

complicated. Um, metadata shows and is
being used to render the page. So in order

to do this, this is, for example, a very
short version of the database tree that I

showed you. You can even go and look for
other ones but this is a s1. s1 eqiad this

is the main data center the master is this
number and it replicates to some of this

and then this 7, the second one that this
was with 2000 because it's the second data

center and it's a master of the other one.
And it has its own replications

between cross three replications because
the master, that master data center is in

Ashburn, Virginia. The second data center
is in Dallas, Texas. So they need to have a

cross DC replication and that happens
with a TLS to make sure that no one starts

to listen to, in between these two, and we
have snapshots and even dumps of the whole

history of Wikipedia. You can go to
dumps.wikimedia.org and download the whole

reserve every wiki you want, except the
ones that we had to remove for privacy

reasons and with a lots and lots of
backups. I recently realized we have lots

of backups. And in total it is 570 TB of data
and total 150 database servers and a

queries that happens to them is around
350,000 queries per second and, in total,

it requires 70 terabytes of RAM. So and
also we have another storage section that

called Elasticsearch which you can guess
it- it's being used for search, on the top

right, if you're using desktop. It's
different in mobile, I think. And also it

depends on if you're rtl language as well,
but also it runs by a team called search

platform because none of us are from
search platform we cannot explain it this

much we don't know much how it works it
slightly. Also we have a media storage for

all of the free pictures that's being
uploaded to Wikimedia like, for example,

if you have a category in Commons. Commons
is our wiki that holds all of the free

media and if we have a category in Commons
called cats looking at left and you have

category cats looking at right so we have
lots and lots of images. It's 390 terabytes

of media, 1 billion object and uses Swift.
Swift is the object is storage component

of OpenStack and it has it has several
layers of caching, frontend, backend.

Yeah, that's mostly it. And we want to
talk about traffic now and so this picture

is when Sweden in 1967 moved from a left-
driving from left to there driving to

right. This is basically what happens in
Wikipedia infrastructure as well. So we

have five caching layers and the most
recent one is eqsin which is in Singapore,

the three one are just CDN ulsfo, codfw,
esams and eqsin. Sorry, ulsfo, esams and

eqsin are just CDNs. We have also two
points of presence, one in Chicago and the

other one is also in Amsterdam, but we
don't get to that. So, we have, as I said,

we have our own content delivery network
with our traffic or allocation is done by

GeoDNS which actually is written and
maintained by one of the traffic people,

and we can pool and depool DCs. It has a
time to live of 10 minute- 10 minutes, so

if a data center goes down. We have - it
takes 10 minutes to actually propagate for

being depooled and repooled again. And we
use LVS as transport layer and this layer

3 and 4 of the Linux load balancer for
Linux and supports consistent hashing and

also we ever got we grow so big that we
needed to have something that manages the

load balancer so we wrote something our
own system is called pybal. And also we -

lots of companies actually peer with us. We
for example directly connect to

Amsterdam amps X. So this is how the
caching works, which is, anyway, it's

there is lots of reasons for this. Let's
just get the started. We use TLS, we

support TLS 1.2 where we have K then
the first layer we have nginx-. Do you

know it - does anyone know what nginx-
means? And so that's related but not - not

correct. So we have nginx which is the free
version and we have nginx plus which is

the commercial version and nginx. But we
don't use nginx to do load balancing or

anything so we stripped out everything
from it, and we just use it for TLS

termination so we call it nginx-, is an
internal joke. So and then we have Varnish

frontend. Varnish also is a caching layer
and this is the frontend is on the memory

which is very very fast and you have the
backend which is on the storage and the

hard disk but this is slow. The fun thing
is like just CDN caching layer takes 90%

of our requests. Its response and 90% of
because just gets to the Varnish and just

return and then with doesn't work it goes
through the application layer. The Varnish

holds-- it has a TTL of 24 hours so if you
change an article, it also get invalidated

by the application. So if someone added the
CDN actually purges the result. And the

thing is, the frontend is shorted that can
spike by request so you come here load

balancer just randomly sends your request
to a frontend but then the backend is

actually, if the frontend can't find it,
it sends it to the backend and the backend

is actually sort of - how is it called? -
it's a used hash by request, so, for

example, article of Barack Obama is only
being served from one node in the data

center in the CDN. If none of this works it
actually hits the other data center. So,

yeah, I actually explained all of this. So
we have two - two caching clusters and one

is called text and the other one is called
upload, it's not confusing at all, and if

you want to find out, you can just do mtr
en.wikipedia.org and you - you're - the end

node is text-lb.wikimedia.org which is the
our text storage but if you go to

upload.wikimedia.org, you get to hit the
upload cluster. Yeah this is so far, what

is it, and it has lots of problems because
a) varnish is open core, so the version

that you use is open source we don't use
the commercial one, but the open core one

doesn't support TLS. What? What happened?
Okay. No, no, no! You should I just-

you're not supposed to see this. Okay,
sorry for the- huh? Okay, okay sorry. So

Varnish has lots of problems, Varnish is
open core, it doesn't support TLS

termination which makes us to have this
nginx- their system just to do TLS

termination, makes our system complicated.
It doesn't work very well with so if that

causes us to have a cron job to restart
every Varnish node twice a week. We have a

cron job that this restarts every Vanish
node which is embarrassing, but also, on

the other hand then the end of Varnish
like backend wants to talk to the

application layer, it also doesn't support
terminate - TLS termination, so we use

IPSec which is even more embarrassing, but
we are changing it. So we call it, if you

are using a particular fixed server which
is very very nice and it's also open

source, a fully open source like in with
Apache Foundation, Apache does the TLS,

does the TLS by termination and still
for now we have a Varnish frontend that

still exists but a backend is also going
to change to the ATS, so we call this ATS

sandwich. Two ATS happening between and
there the middle there's a Varnish. The

good thing is that the TLS termination
when it moves to ATS, you can actually use

TLS 1.3 which is more modern and more
secure and even very faster so it

basically drops 100 milliseconds from
every request that goes to Wikipedia.

That translates to centuries of our
users' time every month, but ATS is going

on and hopefully it will go live soon and
once these are done, so this is the new

version. And, as I said, the TLS and when
we can do this we can actually use the

more secure instead of IPSec to talk about
between data centers. Yes. And now it's

time that Lucas talks about what happens
when you type in en.wikipedia.org.

Lucas: Yes, this makes sense, thank you.

So, first of all, what you see on the
slide here as the image doesn't really

have anything to do with what happens when
you type in wikipedia.org because it's an

offline Wikipedia reader but it's just a
nice image. So this is basically a summary

of everything they already said, so if,
which is the most common case, you are

lucky and get a URL which is cached, then,
so, first your computer asked for the IP

address of en.wikipedia.org it reaches
this whole DNS daemon and because we're at

Congress here it tells you the closest
data center is the one in Amsterdam, so

esams and it's going to hit the edge, what
we call load bouncers/router there, then

going through TLS termination through
nginx- and then it's going to hit the

Varnish caching server, either frontend or
backends and then you get a response and

that's already it and nothing else is ever
bothered again. It doesn't even reach any

other data center which is very nice and
so that's, you said around 90% of the

requests we get, and if you're unlucky and
the URL you requested is not in the

Varnish in the Amsterdam data center then
it gets forwarded to the eqiad data

center, which is the primary one and there
it still has a chance to hit the cache and

perhaps this time it's there and then the
response is going to get cached in the

frontend, no, in the Amsterdam Varnish and
you're also going to get a response and we

still don't have to run any application
stuff. If we do have to hit any

application stuff and then Varnish is
going to forward that, if it's

upload.wikimedia.org, it goes to the media
storage Swift, if it's any other domain it

goes to MediaWiki and then MediaWiki does
a ton of work to connect to the database,

in this case the first shard for English
Wikipedia, get the wiki text from there,

get the wiki text of all the related pages
and templates. No, wait I forgot

something. First it checks if the HTML for
this page is available in parser cache, so

that's another caching layer, and this
application cache - this parser cache

might either be memcached or the database
cache behind it and if it's not there,

then it has to go get the wikitext, get
all the related things and render that

into HTML which takes a long time and goes
through some pretty ancient code and if

you are doing an edit or an upload, it's
even worse, because then always has to go

to MediaWiki and then it not only has to
store this new edit, either in the media

back-end or in the database, it also has
update a bunch of stuff, like, especially

if you-- first of all, it has to purge the
cache, it has to tell all the Varnish

servers that there's a new version of this
URL available so that it doesn't take a

full day until the time-to-live expires.
It also has to update a bunch of things,

for example, if you edited a template, it
might have been used in a million pages

and the next time anyone requests one of
those million pages, those should also

actually be rendered again using the new
version of the template so it has to

invalidate the cache for all of those and
all that is deferred through the job queue

and it might have to calculate thumbnails
if you uploaded the file or create a -

retranscode media files because maybe you
uploaded in - what do we support? - you

upload in WebM and the browser only
supports some other media codec or

something, we transcode that and also
encode it down to the different

resolutions, so then it goes through that
whole dance and, yeah, that was already

those slides. Is Amir going to talk again
about how we manage -

Amir: I mean okay yeah I quickly come back
just for a short break to talk about

managing to manage because managing 100-
1300 bare metal hardware plus a Kubernetes

cluster is not easy, so what we do is that
we use Puppet for configuration

management in our bare metal systems, it's
fun, five to 50,000 lines of Puppet code. I

mean, lines of code is not a great
indicator but you can roughly get an

estimate of how its things work and we
have 100,000 lines of Ruby and we have our

CI and CD cluster, we have so we don't
store anything in GitHub or GitLab, we

have our own system which is based on
Gerrit and for that we have a system of

Jenkins and the Jenkins does all of this
kind of things and also because we have a

Kubernetes cluster for services, some of
our services, if you make a merger change

in the Gerrit it also builds the Docker
files and containers and push it up to the

production and also in order to run remote
SSH commands, we have cumin that's like in

the house automation and we built this
farm for our systems and for example you

go there and say ok we pull this node or
run this command in all of the data

Varnish nodes that I told you like you
want to restart them. And with this I get

back to Lucas.
Lucas: So, I am going to talk a bit more

about Wikimedia Cloud Services which is a
bit different in that it's not really our

production stuff but it's where you
people, the volunteers of the Wikimedia

movement can run their own code, so you
can request a project which is kind of a

group of users and then you get assigned a
pool of you have this much CPU and this

much RAM and you can create virtual
machines with those resources and then do

stuff there and run basically whatever you
want, to create and boot and shut down the

VMs and stuff we use OpenStack and there's
a Horizon frontend for that which you use

through the browser and it's largely out
all the time but otherwise it works pretty

well. Internally, ideally you manage the
VMs using Puppet but a lot of people just

SSH in and then do whatever they need to
set up the VM manually and it happens,

well, and there's a few big projects like
Toolforge where you can run your own web-

based tools or the beta cluster which is
basically a copy of some of the biggest

wikis like there's a beta English
Wikipedia, beta Wikidata, beta Wikimedia

Commons using mostly the same
configuration as production but using the

current master version of the software
instead of whatever we deploy once a week so

if there's a bug, we see it earlier
hopefully, even if we didn't catch it

locally, because the beta cluster is more
similar to the production environment and

also the continuous - continuous
integration service run in Wikimedia Cloud

Services as well. Yeah and also you have
to have Kubernetes somewhere on these

slides right, so you can use that to
distribute work between the tools in

Toolforge or you can use the grid engine
which does a similar thing but it's like

three decades old and through five forks
now I think the current fork we use is son

of grid engine and I don't know what it
was called before, but that's Cloud

Services.
Amir: So in a nutshell, this is our - our

systems. We have 1300 bare metal services
with lots and lots of caching, like lots

of layers of caching, because mostly we
serves read and we can just keep them as a

cached version and all of this is open
source, you can contribute to it, if you

want to and there's a lot of configuration
is also open and I - this is the way I got

hired like I open it started contributing
to the system I feel like yeah we can-

come and work for us, so this is a -
Daniel: That's actually how all of us got

hired.
Amir: So yeah, and this is the whole thing

that happens in Wikimedia and if you want
to - no, if you want to help us, we are

hiring. You can just go to jobs at
wikimedia.org, if you want to work for

Wikimedia Foundation. If you want to work
with Wikimedia Deutschland, you can go to

wikimedia.de and at the bottom there's a
link for jobs because the links got too

long. If you can contribute, if you want
to contribute to us, there is so many ways

to contribute, as I said, there's so many
bugs, we have our own graphical system,

you can just look at the monitor and a
Phabricator is our bug tracker, you can

just go there and find the bug and fix
things. Actually, we have one repository

that is private but it only holds the
certificate for as TLS and things that are

really really private then we cannot
remove them. But also there are

documentations, the documentation for
infrastructure is at

wikitech.wikimedia.org and documentation
for configuration is at noc.wikimedia.org

plus the documentation of our codebase.
The documentation for MediaWiki itself is

at mediawiki.org and also we have a our
own system of URL shortener you can go to

w.wiki and short and shorten any URL in
Wikimedia structure so we reserved the

dollar sign for the donate site and yeah,
you have any questions, please.

Applause

Daniel: It's if you know we have quite a bit of
time for questions so if anything wasn't

clear or they're curious about anything
please, please ask.

AM: So one question what is not in the
presentation. Do you have any efforts with

hacking attacks?
Amir: So the first rule of security issues

is that we don't talk about security issues
but let's say this baby has all sorts of

attacks happening, we have usually we have
DDo. Once there was happening a couple of

months ago that was very successful. I
don't know if you read the news about

that, but we also, we have a infrastructure
to handle this, we have a security team

that handles these cases and yes.
AM: Hello how do you manage access to your

infrastructure from your employees?
Amir: So it's SS-- so we have a LDAP

group and LDAP for the web-based
systems but for SSH and for this ssh we

have strict protocols and then you get a
private key and some people usually

protect their private key using UV keys
and then you have you can SSH to the

system basically.
Lucas: Yeah, well, there's some

firewalling setup but there's only one
server for data center that you can

actually reach through SSH and then you
have to tunnel through that to get to any

other server.
Amir: And also, like, we have we have a

internal firewall and it's basically if
you go to the inside of the production you

cannot talk to the outside. You even, you
for example do git clone github.org, it

doesn't, github.com doesn't work. It
only can access tools that are for inside

Wikimedia Foundation infrastructure.
AM: Okay, hi, you said you do TLS

termination through nginx, do you still
allow non-HTTPS so it should be non-secure access.

Amir: No we dropped it a really long
time ago but also

Lucas: 2013 or so
Amir: Yeah, 2015

Lucas: 2015
Amir: 2013 started serving the most of the

traffic but 15, we dropped all of the
HTTP- non-HTTPS protocols and recently even

dropped and we are not serving any SSL
requests anymore and TLS 1.1 is also being

phased out, so we are sending you a warning
to the users like you're using TLS 1.1,

please migrate to these new things that
came out around 10 years ago, so yeah

Lucas: Yeah I think the deadline for that
is like February 2020 or something then

we'll only have TLS 1.2
Amir: And soon we are going to support TLS

1.3
Lucas: Yeah

Are there any questions?
Q: so does read-only traffic

from logged in users hit all the way
through to the parser cache or is there

another layer of caching for that?
Amir: Yes we, you bypass all of

that, you can.
Daniel: We need one more microphone. Yes,

it actually does and this is a pretty big
problem and something we want to look into

clears throat but it requires quite a
bit of rearchitecting. If you are

interested in this kind of thing, maybe
come to my talk tomorrow at noon.

Amir: Yeah one reason we can, we are
planning to do is active active so we have

two primaries and the read request gets
request - from like the users can hit

their secondary data center instead of the
main one.

Lucas: I think there was a question way in
the back there, for some time already

AM: Hi, I got a question. I read on the
Wikitech that you are using karate as a

validation platform for some parts, can
you tell us something about this or what

parts of Wikipedia or Wikimedia are hosted
on this platform?

Amir: I am I'm not oh sorry so I don't
know this kind of very very sure but take

it with a grain of salt but as far as I
know karate is used to build a very small

VMs in productions that we need for very
very small micro sites that we serve to

the users. So we built just one or two VMs,
we don't use it very as often as I think

so.
AM: Do you also think about open hardware?

Amir: I don't, you can
Daniel: Not - not for servers. I think for

the offline Reader project, but this is not
actually run by the Foundation, it's

supported but it's not something that the
Foundation does. They were sort of

thinking about open hardware but really
open hardware in practice usually means,

you - you don't, you know, if you really
want to go down to the chip design, it's

pretty tough, so yeah, it's- it's it- it's
usually not practical, sadly.

Amir: And one thing I can say but this is
that we have a some machine - machines that

are really powerful that we give to the
researchers to run analysis on the between

this itself and we needed to have GPUs for
those but the problem was - was there

wasn't any open source driver for them so
we migrated and use AMD I think, but AMD

didn't fit in the rack it was a quite a
endeavor to get it to work for our

researchers to help you CPU.
AM: I'm still impressed that you answer

90% out of the cache. Do all people access
the same pages or is the cache that huge?

So what percentage of - of the whole
database is in the cache then?

Daniel: I don't have the exact numbers to
be honest, but a large percentage of the

whole database is in the cache. I mean it
expires after 24 hours so really obscure

stuff isn't there but I mean it's- it's a-
it's a- it's a power-law distribution

right? You have a few pages that are
accessed a lot and you have many many many

pages that are not actually accessed
at all for a week or so except maybe for a

crawler, so I don't know a number. My
guess would be it's less than 50% that is

actually cached but, you know, that still
covers 90%-- it's probably the top 10% of

pages would still cover 90% of the
pageviews, but I don't-- this would be

actually-- I should look this up, it would
be interesting numbers to have, yes.

Lucas: Do you know if this is 90% of the
pageviews or 90% of the get requests

because, like, requests for the JavaScript
would also be cached more often, I assume

Daniel: I would expect that for non-
pageviews, it's even higher

Lucas: Yeah
Daniel: Yeah, because you know all the

icons and- and, you know, JavaScript
bundles and CSS and stuff doesn't ever

change
Lucas: I'm gonna say for every 180 min 90%

but there's a question back there
AM: Hey. Do your data centers run on green

energy?
Amir: Very valid question. So, the

Amsterdam city n1 is a full green but the
other ones are partially green, partially

coal and like gas. As far as I know, there
are some plans to make them move away from

it but the other hand we realized that if
we don't produce as much as a carbon

emission because we don't have much servers
and we don't use much data, there was a

summation and that we realized our carbon
emission is basically as the same as 200

and in the datacenter plus all of their
travel that all of this have to and all of

the events is 250 households, it's very
very small it's I think it's one

thousandth of the comparable
traffic with Facebook even if you just cut

down with the same traffic because
Facebook collects the data, it runs very

sophisticated machine learning algorithms
that's that's a real complicate, but for

Wikimedia, we don't do this so we don't
need much energy. Does - does the answer

your question?
Herald: Do we have any other

questions left? Yeah sorry
AM: hi how many developers do you need to

maintain the whole infrastructure and how
many developers or let's say head

developer hours you needed to build the
whole infrastructure like the question is

because what I find very interesting about
the talk it's a non-profit, so as an

example for other nonprofits is how much
money are we talking about in order to

build something like this as a digital
common.

Daniel: If this is just about actually
running all this so just operations is

less than 20 people I think which makes if
you if you basically divide the requests

per second by people you get to something
like 8,000 requests per second per

operations engineer which I think is a
pretty impressive number. This is probably

a lot higher I would I would really like
to know if there's any organization that

tops that. I don't actually know the whole
the the actual operations budget I know is

it two two-digit millions annually. Total
hours for building this over the last 18

years, I have no idea. For the for the
first five or so years, the people doing

it were actually volunteers. We still had
volunteer database administrators and

stuff until maybe ten years ago, eight
years ago, so yeah it's really nobody

did any accounting of this I can only
guess.

AM: Hello a tools question. I a few years
back I saw some interesting examples of

saltstack use for Wikimedia but right now
I see only Puppet that come in mentioned

so kind of what happened with that
Amir: I think we dished saltstack you -

I don't I cannot because none of us are in
the Cloud Services team and I don't think

I can answer you but if you look at the
wikitech.wikimedia.org, it's

probably if last time I checked says like
it's deprecated and obsolete we don't use

it anymore.

AM: Do you use the bat-ropes like the top

runners to fill spare capacity on the web
serving servers or do you have dedicated

servers for the roles.
Lucas: I think they're dedicated.

Amir: The job runners if you're asking job runners
are dedicated yes they are they are I

think 5 per primary data center so
Daniel: Yeah they don't, I mean do we do we

actually have any spare capacity on
anything? We don't have that much hardware

everything is pretty much at a hundred
percent.

Lucas: I think we still have some server
that is just called misc1111 or something

which run five different things at once,
you can look for those on wikitech.

Amir: But but we go oh sorry it's not five
it's 20 per data center 20 per primary

data center that's our job runner and they
run 700 jobs per second.

Lucas: And I think that does not include
the video scaler so those are separate

again
Amir: No, they merged them in like a month

ago
Lucas: Okay, cool

AM: Maybe a little bit off topic that can
tell us a little bit about decision making

process for- for technical decision,
architecture decisions, how does it work

in an organization like this: decision
making process for architectural

decisions for example.

Daniel: Yeah so Wikimedia has a

committee for making high-level technical
decisions, it's called a Wikimedia

Technical Committee, techcom and we run an
RFC process so any decision that is a

cross-cutting strategic are especially
hard to undo should go through this

process and it's pretty informal,
basically you file a ticket and start

this process. It gets announced
in the mailing list, hopefully you get

input and feedback and at some point it is
it's approved for implementation. We're

currently looking into improving this
process, it's not- sometimes it works

pretty well, sometimes things don't get
that much feedback but it still it makes

sure that people are aware of these high-
level decisions

Amir: Daniel is the chair of that
committee

Daniel: Yeah, if you want to complain
about the process, please do.

AM: yes regarding CI and CD across along the
pipeline, of course with that much traffic

you want to keep everything consistent
right. So is there any testing

strategies that you have said internally,
like of course unit tests integration

tests but do you do something like
continuous end to end testing on beta

instances?
Amir: So if we have beta cluster but also

we do deploy, we call it train and so
we deploy once a week, all of the changes

gets merged to one, like a branch and the
branch gets cut in every Tuesday and it

first goes to the test wikis and
then it goes to all of the wikis that are

not Wikipedia except Catalan and Hebrew
Wikipedia. So basically Hebrew and Catalan

Wikipedia volunteer to be the guinea pigs
of the next wikis and if everything works

fine usually it goes there and is like oh
the fatal mater and we have a logging and

then it's like okay we need to fix this
and we fix it immediately and then it goes

live to all wikis. This is one way of
looking at it well so okay yeah

Daniel: So, our test coverage is not as
great as it should be and so we kind of,

you know, abuse our users for this. We
are, of course, working to improve this

and one thing that we started recently is
a program for creating end-to-end tests

for all the API modules we have, in the
hope that we can thereby cover pretty much

all of the application logic bypassing the
user interface. I mean, full end-to-end

should, of course, include the user
interface but user interface tests are

pretty brittle and often tests you know
where things are on the screen and it just

seems to us that it makes a lot of sense
to have more- to have tests that actually

test the application logic for what the
system actually should be doing, rather

than what it should look like and, yeah,
we are currently working on making- so

yeah, basically this has been a proof of
concept and we're currently working to

actually integrate it in- in CI. That
perhaps should land once everyone is back

from the vacations and then we have to
write about a thousand or so tests, I

guess.
Lucas: I think there's also a plan to move

to a system where we actually deploy
basically after every commit and can

immediately roll back if something goes
wrong but that's more midterm stuff and

I'm not sure what the current status of
that proposal is

Amir: And it will be in Kubernetes, so it
will be completely different

Daniel: That would be amazing
Lucas: But right now, we are on this

weekly basis, if something goes wrong, we
roll back to the last week's version of

the code
Herald: Are there are any questions-

questions left? Sorry. Yeah. Okay, um, I
don't think so. So, yeah, thank you for

this wonderful talk. Thank you for all
your questions. Um, yeah, I hope you liked

it. Um, see you around, yeah.

Applause

Music

Subtitles created by c3subtitles.de
in the year 2021. Join, and help us!