Music
Herald:Hi! Welcome, welcome in Wikipaka-
WG, in this extremely crowded Esszimmer.
I'm Jakob, I'm your Herald for tonight
until 10:00 and I'm here to welcome you
and to welcome these wonderful three guys
on the stage. They're going to talk about
the infrastructure of Wikipedia.
And yeah, they are Lucas, Amir, and Daniel
and I hope you'll have fun!
Applause
Amir Sarabadani: Hello, my name is
Amir, um, I'm a software engineer at
Wikimedia Deutschland, which is the German
chapter of Wikimedia Foundation. Wikimedia
Foundation runs Wikipedia. Here is Lucas.
Lucas is also a software engineer, at
Wikimedia Deutschland, and Daniel here is
a software architect at Wikimedia
Foundation. We are all based in Germany,
Daniel in Leipzig, we are in Berlin. And
today we want to talk about how we run
Wikipedia, with using donors' money and
not lots of advertisement and collecting
data. So in this talk, first we are going
to go on an inside-out approach. So we are
going to first talk about the application
layer and then the outside layers, and
then we go to an outside-in approach and
then talk about how you're going to hit
Wikipedia from the outside.
So first of all, let's some,
let me get you some information. First of
all, all of Wikimedia, Wikipedia
infrastructure is run by Wikimedia
Foundation, an American nonprofit
charitable organization. We don't run any
ads and we are only 370 people. If you
count Wikimedia Deutschland or all other
chapters, it's around 500 people in total.
It's nothing compared to the companies
outside. But all of the content is
managed by volunteers. Even our staff
doesn't do edits, add content to
Wikipedia. And we support 300 languages,
which is a very large number. And
Wikipedia, it's eighteen years old, so it
can vote now. And also, Wikipedia has some
really, really weird articles. Um, I want
to ask you, what is your, if you have
encountered any really weird article
in Wikipedia? My favorite is a list of
people who died on the toilet. But if you
know anything, raise your hands. Uh, do
you know any weird articles in Wikipedia?
Do you know some?
Daniel Kinzler: Oh, the classic one….
Amir: You need to unmute yourself. Oh,
okay.
Daniel: This is technology. I don't know
anything about technology. OK, no. The, my
favorite example is "people killed by
their own invention". That's yeah. That's
a lot of fun. Look it up. It's amazing.
Lucas Werkmeister: There's also a list,
there is also a list of prison escapes
using helicopters. I almost said
helicopter escapes using prisons, which
doesn't make any sense. But that was also
a very interesting list.
Daniel: I think we also have a category of
lists of lists of lists.
Amir: That's a page.
Lucas: And every few months someone thinks
it's funny to redirect it to Russel's
paradox or so.
Daniel: Yeah.
Amir: But also beside that, people cannot
read Wikipedia in Turkey or China. But
three days ago, actually, the block in
Turkey was ruled unconstitutional, but
it's not lifted yet. Hopefully they will
lift it soon. Um, so Wikipedia, Wikimedia
projects is just not Wikipedia. It's lots
and lots of projects. Some of them are not
as successful as the Wikipedia. Um, uh,
like Wikinews. But uh, for example,
Wikipedia is the most successful one, and
there's another one, that's Wikidata. It's
being developed by Wikimedia Deutschland.
I mean the Wikidata team, with Lucas, um,
and it's being used – it's infobox – it
has the data that Wikipedia or Google
Knowledge Graph or Siri or Alexa uses.
It's basically, it's sort of a backbone of
all of the data, uh, through the whole
Internet. Um, so our infrastructure. Let
me… So first of all, our infrastructure is
all Open Source. By principle, we never
use any commercial software. Uh, we could
use a lots of things. They are even
sometimes were given us for free, but we
were, refused to use them. Second
thing is we have two primary data center
for like failovers, when, for example, a
whole datacenter goes offline, so we can
failover to another data center. We have
three caching points of presence or
CDNs. Our CDNs are all over the world. Uh,
also, we have our own CDN. We don't have,
we don't use CloudFlare, because
CloudFlare, we care about the privacy of
the users and is very important that, for
example, people edit from countries that
might be, uh, dangerous for them to edit
Wikipedia. So we really care to keep the
data as protected as possible.
Applause
Amir: Uh, we have 17 billion page views
per month, and, which goes up and down
based on the season and everything, we
have around 100 to 200 thousand requests
per second. It's different from the
pageview because requests can be requests
to the objects, can be API, can be lots of
things. And we have 300,000 new editors
per month and we run all of this with 1300
bare metal servers. So right now, Daniel
is going to talk about the application
layer and the inside of that
infrastructure.
Daniel: Thanks, Amir. Oh, the clicky
thing. Thank you. So the application layer
is basically the software that actually
does what a wiki does, right? It lets you
edit pages, create or update pages and
then search the page views. interference
noise The challenge for Wikipedia, of
course, is serving all the many page views
that Amir just described. The core of the
application is a classic LAMP application.
interference noise I have to stop
moving. Yes? Is that it? It's a classic
LAMP stack application. So it's written in
PHP, it runs on an Apache server. It uses
MySQL as a database in the backend. We
used to use a HHVM instead of the… Yeah,
we…
Herald: Hier. Sorry. Nimm mal das hier.
Daniel: Hello. We used to use HHVM as the
PHP engine, but we just switched back to
the mainstream PHP, using PHP 7.2 now,
because Facebook decided that HHVM is
going to be incompatible with the standard
and they were just basically developing it
for, for themselves. Right. So we have
separate clusters of servers for serving
requests, for serving different requests,
page views on the one hand, and also
handling edits. Then we have a cluster for
handling API calls and then we have a
bunch of servers set up to handle
asynchronous jobs, things that happen in
the background, the job runners, and…
I guess video scaling is a very obvious
example of that. It just takes too long to
do it on the fly. But we use it for many
other things as well. MediaWiki, MediaWiki
is kind of an amazing thing because you
can just install it on your own shared-
hosting, 10-bucks-a-month's webspace and
it will run. But you can also use it to,
you know, serve half the world. And so
it's a very powerful and versatile system,
which also… I mean, this, this wide span
of different applications also creates
problems. That's something that I will
talk about tomorrow. But for now, let's
look at the fun things. So if you want to
serve a lot of page views, you have to do
a lot of caching. And so we have a whole…
yeah, a whole set of different caching
systems. The most important one is
probably the parser cache. So as you
probably know, wiki pages are created in,
in a markup language, Wikitext, and they
need to be parsed and turned into HTML.
And the result of that parsing is, of
course, cached. And that cache is semi-
persistent, it… nothing really ever drops
out of it. It's a huge thing. And it's, it
lives in a dedicated MySQL database
system. Yeah. We use memcached a lot for
all kinds of miscellaneous things,
anything that we need to keep around and
share between server instances. And we
have been using redis for a while, for
anything that we want to have available,
not just between different servers, but
also between different data centers,
because redis is a bit better about
synchronizing things between, between
different systems, we still use it for
session storage, especially, though we are
about to move away from that and we'll be
using Cassandra for session storage. We
have a bunch of additional services
running for specialized purposes, like
scaling images, rendering formulas, math
formulas, ORES is pretty interesting. ORES
is a system for automatically detecting
vandalism or rating edits. So this is a
machine learning based system for
detecting problems and highlighting edits
that may not be, may not be great and need
more attention. We have some additional
services that process our content for
consumption on mobile devices, chopping
pages up into bits and pieces that then
can be consumed individually and many,
many more. In the background, we also have
to manage events, right, we use Kafka for
message queuing, and we use that to notify
different parts of the system about
changes. On the one hand, we use that to
feed the job runners that I just
mentioned. But we also use it, for
instance, to purge the entries in the
CDN when pages become updated and things
like that. OK, the next session is going
to be about the databases. Are there, very
quickly, we will have quite a bit of time
for discussion afterwards. But are there
any questions right now about what we said
so far? Everything extremely crystal
clear. OK, no clarity is left? I see. Oh,
one question, in the back.
Q: Can you maybe turn the volume up a
little bit? Thank you.
Daniel: Yeah, I think this is your
section, right? Oh, its Amir again. Sorry.
Amir: So I want to talk about my favorite
topic, the dungeons of, dungeons of every
production system, databases. The database
of Wikipedia is really interesting and
complicated on its own. We use MariaDB, we
switched from MySQL in 2013 for lots of
complicated reasons. As, as I said,
because we are really open source, you can
go and not just check our database tree,
that says, like, how it looks and what's
the replicas and masters. Actually, you
can even query the Wikipedia's database
live when you have that, you can just go
to that address and login with your
Wikipedia account and just can do whatever
you want. Like, it was a funny thing that
a couple of months ago, someone sent me a
message, sent me a message like, oh, I
found a security issue. You can just query
Wikipedia's database. I was like, no, no,
it's actually, we, we let this happen.
It's like, it's sanitized. We removed the
password hashes and everything. But still,
you can use this. And, but if you wanted
to say, like, how the clusters work, the
database clusters, because it gets too
big, they first started sharding, but now
we have sections that are basically
different clusters. Uh, really large wikis
have their own section. For example,
English Wikipedia is s1. German Wikipedia
with two or three other small wikis are in
s5. Wikidata is on s8, and so on. And
each section have a master and several
replicas. But one of the replicas is
actually a master in another data center
because of the failover that I told you.
So it is, basically two layers of
replication exist. This is, what I'm
telling you, is about metadata. But for
Wikitext, we also need to have a complete
different set of databases. But it can be,
we use consistent hashing to just scale it
horizontally so we can just put more
databases on it, for that. Uh, but I don't
know if you know it, but Wikipedia stores
every edit. So you have the text of,
Wikitext of every edit in the whole
history in the database. Uhm, also we have
parser cache that Daniel explained, and
parser cache is also consistent hashing.
So we just can horizontally scale it. But
for metadata, it is slightly more
complicated. Um, metadata shows and is
being used to render the page. So in order
to do this, this is, for example, a very
short version of the database tree that I
showed you. You can even go and look for
other ones but this is a s1. s1 eqiad this
is the main data center the master is this
number and it replicates to some of this
and then this 7, the second one that this
was with 2000 because it's the second data
center and it's a master of the other one.
And it has its own replications
between cross three replications because
the master, that master data center is in
Ashburn, Virginia. The second data center
is in Dallas, Texas. So they need to have a
cross DC replication and that happens
with a TLS to make sure that no one starts
to listen to, in between these two, and we
have snapshots and even dumps of the whole
history of Wikipedia. You can go to
dumps.wikimedia.org and download the whole
reserve every wiki you want, except the
ones that we had to remove for privacy
reasons and with a lots and lots of
backups. I recently realized we have lots
of backups. And in total it is 570 TB of data
and total 150 database servers and a
queries that happens to them is around
350,000 queries per second and, in total,
it requires 70 terabytes of RAM. So and
also we have another storage section that
called Elasticsearch which you can guess
it- it's being used for search, on the top
right, if you're using desktop. It's
different in mobile, I think. And also it
depends on if you're rtl language as well,
but also it runs by a team called search
platform because none of us are from
search platform we cannot explain it this
much we don't know much how it works it
slightly. Also we have a media storage for
all of the free pictures that's being
uploaded to Wikimedia like, for example,
if you have a category in Commons. Commons
is our wiki that holds all of the free
media and if we have a category in Commons
called cats looking at left and you have
category cats looking at right so we have
lots and lots of images. It's 390 terabytes
of media, 1 billion object and uses Swift.
Swift is the object is storage component
of OpenStack and it has it has several
layers of caching, frontend, backend.
Yeah, that's mostly it. And we want to
talk about traffic now and so this picture
is when Sweden in 1967 moved from a left-
driving from left to there driving to
right. This is basically what happens in
Wikipedia infrastructure as well. So we
have five caching layers and the most
recent one is eqsin which is in Singapore,
the three one are just CDN ulsfo, codfw,
esams and eqsin. Sorry, ulsfo, esams and
eqsin are just CDNs. We have also two
points of presence, one in Chicago and the
other one is also in Amsterdam, but we
don't get to that. So, we have, as I said,
we have our own content delivery network
with our traffic or allocation is done by
GeoDNS which actually is written and
maintained by one of the traffic people,
and we can pool and depool DCs. It has a
time to live of 10 minute- 10 minutes, so
if a data center goes down. We have - it
takes 10 minutes to actually propagate for
being depooled and repooled again. And we
use LVS as transport layer and this layer
3 and 4 of the Linux load balancer for
Linux and supports consistent hashing and
also we ever got we grow so big that we
needed to have something that manages the
load balancer so we wrote something our
own system is called pybal. And also we -
lots of companies actually peer with us. We
for example directly connect to
Amsterdam amps X. So this is how the
caching works, which is, anyway, it's
there is lots of reasons for this. Let's
just get the started. We use TLS, we
support TLS 1.2 where we have K then
the first layer we have nginx-. Do you
know it - does anyone know what nginx-
means? And so that's related but not - not
correct. So we have nginx which is the free
version and we have nginx plus which is
the commercial version and nginx. But we
don't use nginx to do load balancing or
anything so we stripped out everything
from it, and we just use it for TLS
termination so we call it nginx-, is an
internal joke. So and then we have Varnish
frontend. Varnish also is a caching layer
and this is the frontend is on the memory
which is very very fast and you have the
backend which is on the storage and the
hard disk but this is slow. The fun thing
is like just CDN caching layer takes 90%
of our requests. Its response and 90% of
because just gets to the Varnish and just
return and then with doesn't work it goes
through the application layer. The Varnish
holds-- it has a TTL of 24 hours so if you
change an article, it also get invalidated
by the application. So if someone added the
CDN actually purges the result. And the
thing is, the frontend is shorted that can
spike by request so you come here load
balancer just randomly sends your request
to a frontend but then the backend is
actually, if the frontend can't find it,
it sends it to the backend and the backend
is actually sort of - how is it called? -
it's a used hash by request, so, for
example, article of Barack Obama is only
being served from one node in the data
center in the CDN. If none of this works it
actually hits the other data center. So,
yeah, I actually explained all of this. So
we have two - two caching clusters and one
is called text and the other one is called
upload, it's not confusing at all, and if
you want to find out, you can just do mtr
en.wikipedia.org and you - you're - the end
node is text-lb.wikimedia.org which is the
our text storage but if you go to
upload.wikimedia.org, you get to hit the
upload cluster. Yeah this is so far, what
is it, and it has lots of problems because
a) varnish is open core, so the version
that you use is open source we don't use
the commercial one, but the open core one
doesn't support TLS. What? What happened?
Okay. No, no, no! You should I just-
you're not supposed to see this. Okay,
sorry for the- huh? Okay, okay sorry. So
Varnish has lots of problems, Varnish is
open core, it doesn't support TLS
termination which makes us to have this
nginx- their system just to do TLS
termination, makes our system complicated.
It doesn't work very well with so if that
causes us to have a cron job to restart
every Varnish node twice a week. We have a
cron job that this restarts every Vanish
node which is embarrassing, but also, on
the other hand then the end of Varnish
like backend wants to talk to the
application layer, it also doesn't support
terminate - TLS termination, so we use
IPSec which is even more embarrassing, but
we are changing it. So we call it, if you
are using a particular fixed server which
is very very nice and it's also open
source, a fully open source like in with
Apache Foundation, Apache does the TLS,
does the TLS by termination and still
for now we have a Varnish frontend that
still exists but a backend is also going
to change to the ATS, so we call this ATS
sandwich. Two ATS happening between and
there the middle there's a Varnish. The
good thing is that the TLS termination
when it moves to ATS, you can actually use
TLS 1.3 which is more modern and more
secure and even very faster so it
basically drops 100 milliseconds from
every request that goes to Wikipedia.
That translates to centuries of our
users' time every month, but ATS is going
on and hopefully it will go live soon and
once these are done, so this is the new
version. And, as I said, the TLS and when
we can do this we can actually use the
more secure instead of IPSec to talk about
between data centers. Yes. And now it's
time that Lucas talks about what happens
when you type in en.wikipedia.org.
Lucas: Yes, this makes sense, thank you.
So, first of all, what you see on the
slide here as the image doesn't really
have anything to do with what happens when
you type in wikipedia.org because it's an
offline Wikipedia reader but it's just a
nice image. So this is basically a summary
of everything they already said, so if,
which is the most common case, you are
lucky and get a URL which is cached, then,
so, first your computer asked for the IP
address of en.wikipedia.org it reaches
this whole DNS daemon and because we're at
Congress here it tells you the closest
data center is the one in Amsterdam, so
esams and it's going to hit the edge, what
we call load bouncers/router there, then
going through TLS termination through
nginx- and then it's going to hit the
Varnish caching server, either frontend or
backends and then you get a response and
that's already it and nothing else is ever
bothered again. It doesn't even reach any
other data center which is very nice and
so that's, you said around 90% of the
requests we get, and if you're unlucky and
the URL you requested is not in the
Varnish in the Amsterdam data center then
it gets forwarded to the eqiad data
center, which is the primary one and there
it still has a chance to hit the cache and
perhaps this time it's there and then the
response is going to get cached in the
frontend, no, in the Amsterdam Varnish and
you're also going to get a response and we
still don't have to run any application
stuff. If we do have to hit any
application stuff and then Varnish is
going to forward that, if it's
upload.wikimedia.org, it goes to the media
storage Swift, if it's any other domain it
goes to MediaWiki and then MediaWiki does
a ton of work to connect to the database,
in this case the first shard for English
Wikipedia, get the wiki text from there,
get the wiki text of all the related pages
and templates. No, wait I forgot
something. First it checks if the HTML for
this page is available in parser cache, so
that's another caching layer, and this
application cache - this parser cache
might either be memcached or the database
cache behind it and if it's not there,
then it has to go get the wikitext, get
all the related things and render that
into HTML which takes a long time and goes
through some pretty ancient code and if
you are doing an edit or an upload, it's
even worse, because then always has to go
to MediaWiki and then it not only has to
store this new edit, either in the media
back-end or in the database, it also has
update a bunch of stuff, like, especially
if you-- first of all, it has to purge the
cache, it has to tell all the Varnish
servers that there's a new version of this
URL available so that it doesn't take a
full day until the time-to-live expires.
It also has to update a bunch of things,
for example, if you edited a template, it
might have been used in a million pages
and the next time anyone requests one of
those million pages, those should also
actually be rendered again using the new
version of the template so it has to
invalidate the cache for all of those and
all that is deferred through the job queue
and it might have to calculate thumbnails
if you uploaded the file or create a -
retranscode media files because maybe you
uploaded in - what do we support? - you
upload in WebM and the browser only
supports some other media codec or
something, we transcode that and also
encode it down to the different
resolutions, so then it goes through that
whole dance and, yeah, that was already
those slides. Is Amir going to talk again
about how we manage -
Amir: I mean okay yeah I quickly come back
just for a short break to talk about
managing to manage because managing 100-
1300 bare metal hardware plus a Kubernetes
cluster is not easy, so what we do is that
we use Puppet for configuration
management in our bare metal systems, it's
fun, five to 50,000 lines of Puppet code. I
mean, lines of code is not a great
indicator but you can roughly get an
estimate of how its things work and we
have 100,000 lines of Ruby and we have our
CI and CD cluster, we have so we don't
store anything in GitHub or GitLab, we
have our own system which is based on
Gerrit and for that we have a system of
Jenkins and the Jenkins does all of this
kind of things and also because we have a
Kubernetes cluster for services, some of
our services, if you make a merger change
in the Gerrit it also builds the Docker
files and containers and push it up to the
production and also in order to run remote
SSH commands, we have cumin that's like in
the house automation and we built this
farm for our systems and for example you
go there and say ok we pull this node or
run this command in all of the data
Varnish nodes that I told you like you
want to restart them. And with this I get
back to Lucas.
Lucas: So, I am going to talk a bit more
about Wikimedia Cloud Services which is a
bit different in that it's not really our
production stuff but it's where you
people, the volunteers of the Wikimedia
movement can run their own code, so you
can request a project which is kind of a
group of users and then you get assigned a
pool of you have this much CPU and this
much RAM and you can create virtual
machines with those resources and then do
stuff there and run basically whatever you
want, to create and boot and shut down the
VMs and stuff we use OpenStack and there's
a Horizon frontend for that which you use
through the browser and it's largely out
all the time but otherwise it works pretty
well. Internally, ideally you manage the
VMs using Puppet but a lot of people just
SSH in and then do whatever they need to
set up the VM manually and it happens,
well, and there's a few big projects like
Toolforge where you can run your own web-
based tools or the beta cluster which is
basically a copy of some of the biggest
wikis like there's a beta English
Wikipedia, beta Wikidata, beta Wikimedia
Commons using mostly the same
configuration as production but using the
current master version of the software
instead of whatever we deploy once a week so
if there's a bug, we see it earlier
hopefully, even if we didn't catch it
locally, because the beta cluster is more
similar to the production environment and
also the continuous - continuous
integration service run in Wikimedia Cloud
Services as well. Yeah and also you have
to have Kubernetes somewhere on these
slides right, so you can use that to
distribute work between the tools in
Toolforge or you can use the grid engine
which does a similar thing but it's like
three decades old and through five forks
now I think the current fork we use is son
of grid engine and I don't know what it
was called before, but that's Cloud
Services.
Amir: So in a nutshell, this is our - our
systems. We have 1300 bare metal services
with lots and lots of caching, like lots
of layers of caching, because mostly we
serves read and we can just keep them as a
cached version and all of this is open
source, you can contribute to it, if you
want to and there's a lot of configuration
is also open and I - this is the way I got
hired like I open it started contributing
to the system I feel like yeah we can-
come and work for us, so this is a -
Daniel: That's actually how all of us got
hired.
Amir: So yeah, and this is the whole thing
that happens in Wikimedia and if you want
to - no, if you want to help us, we are
hiring. You can just go to jobs at
wikimedia.org, if you want to work for
Wikimedia Foundation. If you want to work
with Wikimedia Deutschland, you can go to
wikimedia.de and at the bottom there's a
link for jobs because the links got too
long. If you can contribute, if you want
to contribute to us, there is so many ways
to contribute, as I said, there's so many
bugs, we have our own graphical system,
you can just look at the monitor and a
Phabricator is our bug tracker, you can
just go there and find the bug and fix
things. Actually, we have one repository
that is private but it only holds the
certificate for as TLS and things that are
really really private then we cannot
remove them. But also there are
documentations, the documentation for
infrastructure is at
wikitech.wikimedia.org and documentation
for configuration is at noc.wikimedia.org
plus the documentation of our codebase.
The documentation for MediaWiki itself is
at mediawiki.org and also we have a our
own system of URL shortener you can go to
w.wiki and short and shorten any URL in
Wikimedia structure so we reserved the
dollar sign for the donate site and yeah,
you have any questions, please.
Applause
Daniel: It's if you know we have quite a bit of
time for questions so if anything wasn't
clear or they're curious about anything
please, please ask.
AM: So one question what is not in the
presentation. Do you have any efforts with
hacking attacks?
Amir: So the first rule of security issues
is that we don't talk about security issues
but let's say this baby has all sorts of
attacks happening, we have usually we have
DDo. Once there was happening a couple of
months ago that was very successful. I
don't know if you read the news about
that, but we also, we have a infrastructure
to handle this, we have a security team
that handles these cases and yes.
AM: Hello how do you manage access to your
infrastructure from your employees?
Amir: So it's SS-- so we have a LDAP
group and LDAP for the web-based
systems but for SSH and for this ssh we
have strict protocols and then you get a
private key and some people usually
protect their private key using UV keys
and then you have you can SSH to the
system basically.
Lucas: Yeah, well, there's some
firewalling setup but there's only one
server for data center that you can
actually reach through SSH and then you
have to tunnel through that to get to any
other server.
Amir: And also, like, we have we have a
internal firewall and it's basically if
you go to the inside of the production you
cannot talk to the outside. You even, you
for example do git clone github.org, it
doesn't, github.com doesn't work. It
only can access tools that are for inside
Wikimedia Foundation infrastructure.
AM: Okay, hi, you said you do TLS
termination through nginx, do you still
allow non-HTTPS so it should be non-secure access.
Amir: No we dropped it a really long
time ago but also
Lucas: 2013 or so
Amir: Yeah, 2015
Lucas: 2015
Amir: 2013 started serving the most of the
traffic but 15, we dropped all of the
HTTP- non-HTTPS protocols and recently even
dropped and we are not serving any SSL
requests anymore and TLS 1.1 is also being
phased out, so we are sending you a warning
to the users like you're using TLS 1.1,
please migrate to these new things that
came out around 10 years ago, so yeah
Lucas: Yeah I think the deadline for that
is like February 2020 or something then
we'll only have TLS 1.2
Amir: And soon we are going to support TLS
1.3
Lucas: Yeah
Are there any questions?
Q: so does read-only traffic
from logged in users hit all the way
through to the parser cache or is there
another layer of caching for that?
Amir: Yes we, you bypass all of
that, you can.
Daniel: We need one more microphone. Yes,
it actually does and this is a pretty big
problem and something we want to look into
clears throat but it requires quite a
bit of rearchitecting. If you are
interested in this kind of thing, maybe
come to my talk tomorrow at noon.
Amir: Yeah one reason we can, we are
planning to do is active active so we have
two primaries and the read request gets
request - from like the users can hit
their secondary data center instead of the
main one.
Lucas: I think there was a question way in
the back there, for some time already
AM: Hi, I got a question. I read on the
Wikitech that you are using karate as a
validation platform for some parts, can
you tell us something about this or what
parts of Wikipedia or Wikimedia are hosted
on this platform?
Amir: I am I'm not oh sorry so I don't
know this kind of very very sure but take
it with a grain of salt but as far as I
know karate is used to build a very small
VMs in productions that we need for very
very small micro sites that we serve to
the users. So we built just one or two VMs,
we don't use it very as often as I think
so.
AM: Do you also think about open hardware?
Amir: I don't, you can
Daniel: Not - not for servers. I think for
the offline Reader project, but this is not
actually run by the Foundation, it's
supported but it's not something that the
Foundation does. They were sort of
thinking about open hardware but really
open hardware in practice usually means,
you - you don't, you know, if you really
want to go down to the chip design, it's
pretty tough, so yeah, it's- it's it- it's
usually not practical, sadly.
Amir: And one thing I can say but this is
that we have a some machine - machines that
are really powerful that we give to the
researchers to run analysis on the between
this itself and we needed to have GPUs for
those but the problem was - was there
wasn't any open source driver for them so
we migrated and use AMD I think, but AMD
didn't fit in the rack it was a quite a
endeavor to get it to work for our
researchers to help you CPU.
AM: I'm still impressed that you answer
90% out of the cache. Do all people access
the same pages or is the cache that huge?
So what percentage of - of the whole
database is in the cache then?
Daniel: I don't have the exact numbers to
be honest, but a large percentage of the
whole database is in the cache. I mean it
expires after 24 hours so really obscure
stuff isn't there but I mean it's- it's a-
it's a- it's a power-law distribution
right? You have a few pages that are
accessed a lot and you have many many many
pages that are not actually accessed
at all for a week or so except maybe for a
crawler, so I don't know a number. My
guess would be it's less than 50% that is
actually cached but, you know, that still
covers 90%-- it's probably the top 10% of
pages would still cover 90% of the
pageviews, but I don't-- this would be
actually-- I should look this up, it would
be interesting numbers to have, yes.
Lucas: Do you know if this is 90% of the
pageviews or 90% of the get requests
because, like, requests for the JavaScript
would also be cached more often, I assume
Daniel: I would expect that for non-
pageviews, it's even higher
Lucas: Yeah
Daniel: Yeah, because you know all the
icons and- and, you know, JavaScript
bundles and CSS and stuff doesn't ever
change
Lucas: I'm gonna say for every 180 min 90%
but there's a question back there
AM: Hey. Do your data centers run on green
energy?
Amir: Very valid question. So, the
Amsterdam city n1 is a full green but the
other ones are partially green, partially
coal and like gas. As far as I know, there
are some plans to make them move away from
it but the other hand we realized that if
we don't produce as much as a carbon
emission because we don't have much servers
and we don't use much data, there was a
summation and that we realized our carbon
emission is basically as the same as 200
and in the datacenter plus all of their
travel that all of this have to and all of
the events is 250 households, it's very
very small it's I think it's one
thousandth of the comparable
traffic with Facebook even if you just cut
down with the same traffic because
Facebook collects the data, it runs very
sophisticated machine learning algorithms
that's that's a real complicate, but for
Wikimedia, we don't do this so we don't
need much energy. Does - does the answer
your question?
Herald: Do we have any other
questions left? Yeah sorry
AM: hi how many developers do you need to
maintain the whole infrastructure and how
many developers or let's say head
developer hours you needed to build the
whole infrastructure like the question is
because what I find very interesting about
the talk it's a non-profit, so as an
example for other nonprofits is how much
money are we talking about in order to
build something like this as a digital
common.
Daniel: If this is just about actually
running all this so just operations is
less than 20 people I think which makes if
you if you basically divide the requests
per second by people you get to something
like 8,000 requests per second per
operations engineer which I think is a
pretty impressive number. This is probably
a lot higher I would I would really like
to know if there's any organization that
tops that. I don't actually know the whole
the the actual operations budget I know is
it two two-digit millions annually. Total
hours for building this over the last 18
years, I have no idea. For the for the
first five or so years, the people doing
it were actually volunteers. We still had
volunteer database administrators and
stuff until maybe ten years ago, eight
years ago, so yeah it's really nobody
did any accounting of this I can only
guess.
AM: Hello a tools question. I a few years
back I saw some interesting examples of
saltstack use for Wikimedia but right now
I see only Puppet that come in mentioned
so kind of what happened with that
Amir: I think we dished saltstack you -
I don't I cannot because none of us are in
the Cloud Services team and I don't think
I can answer you but if you look at the
wikitech.wikimedia.org, it's
probably if last time I checked says like
it's deprecated and obsolete we don't use
it anymore.
AM: Do you use the bat-ropes like the top
runners to fill spare capacity on the web
serving servers or do you have dedicated
servers for the roles.
Lucas: I think they're dedicated.
Amir: The job runners if you're asking job runners
are dedicated yes they are they are I
think 5 per primary data center so
Daniel: Yeah they don't, I mean do we do we
actually have any spare capacity on
anything? We don't have that much hardware
everything is pretty much at a hundred
percent.
Lucas: I think we still have some server
that is just called misc1111 or something
which run five different things at once,
you can look for those on wikitech.
Amir: But but we go oh sorry it's not five
it's 20 per data center 20 per primary
data center that's our job runner and they
run 700 jobs per second.
Lucas: And I think that does not include
the video scaler so those are separate
again
Amir: No, they merged them in like a month
ago
Lucas: Okay, cool
AM: Maybe a little bit off topic that can
tell us a little bit about decision making
process for- for technical decision,
architecture decisions, how does it work
in an organization like this: decision
making process for architectural
decisions for example.
Daniel: Yeah so Wikimedia has a
committee for making high-level technical
decisions, it's called a Wikimedia
Technical Committee, techcom and we run an
RFC process so any decision that is a
cross-cutting strategic are especially
hard to undo should go through this
process and it's pretty informal,
basically you file a ticket and start
this process. It gets announced
in the mailing list, hopefully you get
input and feedback and at some point it is
it's approved for implementation. We're
currently looking into improving this
process, it's not- sometimes it works
pretty well, sometimes things don't get
that much feedback but it still it makes
sure that people are aware of these high-
level decisions
Amir: Daniel is the chair of that
committee
Daniel: Yeah, if you want to complain
about the process, please do.
AM: yes regarding CI and CD across along the
pipeline, of course with that much traffic
you want to keep everything consistent
right. So is there any testing
strategies that you have said internally,
like of course unit tests integration
tests but do you do something like
continuous end to end testing on beta
instances?
Amir: So if we have beta cluster but also
we do deploy, we call it train and so
we deploy once a week, all of the changes
gets merged to one, like a branch and the
branch gets cut in every Tuesday and it
first goes to the test wikis and
then it goes to all of the wikis that are
not Wikipedia except Catalan and Hebrew
Wikipedia. So basically Hebrew and Catalan
Wikipedia volunteer to be the guinea pigs
of the next wikis and if everything works
fine usually it goes there and is like oh
the fatal mater and we have a logging and
then it's like okay we need to fix this
and we fix it immediately and then it goes
live to all wikis. This is one way of
looking at it well so okay yeah
Daniel: So, our test coverage is not as
great as it should be and so we kind of,
you know, abuse our users for this. We
are, of course, working to improve this
and one thing that we started recently is
a program for creating end-to-end tests
for all the API modules we have, in the
hope that we can thereby cover pretty much
all of the application logic bypassing the
user interface. I mean, full end-to-end
should, of course, include the user
interface but user interface tests are
pretty brittle and often tests you know
where things are on the screen and it just
seems to us that it makes a lot of sense
to have more- to have tests that actually
test the application logic for what the
system actually should be doing, rather
than what it should look like and, yeah,
we are currently working on making- so
yeah, basically this has been a proof of
concept and we're currently working to
actually integrate it in- in CI. That
perhaps should land once everyone is back
from the vacations and then we have to
write about a thousand or so tests, I
guess.
Lucas: I think there's also a plan to move
to a system where we actually deploy
basically after every commit and can
immediately roll back if something goes
wrong but that's more midterm stuff and
I'm not sure what the current status of
that proposal is
Amir: And it will be in Kubernetes, so it
will be completely different
Daniel: That would be amazing
Lucas: But right now, we are on this
weekly basis, if something goes wrong, we
roll back to the last week's version of
the code
Herald: Are there are any questions-
questions left? Sorry. Yeah. Okay, um, I
don't think so. So, yeah, thank you for
this wonderful talk. Thank you for all
your questions. Um, yeah, I hope you liked
it. Um, see you around, yeah.
Applause
Music
Subtitles created by c3subtitles.de
in the year 2021. Join, and help us!