[Script Info]
Title: 
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,0:00:00.00,0:00:18.77,Default,,0000,0000,0000,,{\i1}Music{\i0}
Dialogue: 0,0:00:18.77,0:00:25.33,Default,,0000,0000,0000,,Herald:Hi! Welcome, welcome in Wikipaka-\NWG, in this extremely crowded Esszimmer.
Dialogue: 0,0:00:25.33,0:00:32.08,Default,,0000,0000,0000,,I'm Jakob, I'm your Herald for tonight\Nuntil 10:00 and I'm here to welcome you
Dialogue: 0,0:00:32.08,0:00:36.69,Default,,0000,0000,0000,,and to welcome these wonderful three guys\Non the stage. They're going to talk about
Dialogue: 0,0:00:36.69,0:00:44.71,Default,,0000,0000,0000,,the infrastructure of Wikipedia.\NAnd yeah, they are Lucas, Amir, and Daniel
Dialogue: 0,0:00:44.71,0:00:52.97,Default,,0000,0000,0000,,and I hope you'll have fun!\N{\i1}Applause{\i0}
Dialogue: 0,0:00:52.97,0:00:57.06,Default,,0000,0000,0000,,Amir Sarabadani: Hello, my name is\NAmir, um, I'm a software engineer at
Dialogue: 0,0:00:57.06,0:01:01.13,Default,,0000,0000,0000,,Wikimedia Deutschland, which is the German\Nchapter of Wikimedia Foundation. Wikimedia
Dialogue: 0,0:01:01.13,0:01:06.52,Default,,0000,0000,0000,,Foundation runs Wikipedia. Here is Lucas.\NLucas is also a software engineer, at
Dialogue: 0,0:01:06.52,0:01:10.30,Default,,0000,0000,0000,,Wikimedia Deutschland, and Daniel here is\Na software architect at Wikimedia
Dialogue: 0,0:01:10.30,0:01:15.11,Default,,0000,0000,0000,,Foundation. We are all based in Germany,\NDaniel in Leipzig, we are in Berlin. And
Dialogue: 0,0:01:15.11,0:01:21.42,Default,,0000,0000,0000,,today we want to talk about how we run\NWikipedia, with using donors' money and
Dialogue: 0,0:01:21.42,0:01:29.91,Default,,0000,0000,0000,,not lots of advertisement and collecting\Ndata. So in this talk, first we are going
Dialogue: 0,0:01:29.91,0:01:34.86,Default,,0000,0000,0000,,to go on an inside-out approach. So we are\Ngoing to first talk about the application
Dialogue: 0,0:01:34.86,0:01:39.83,Default,,0000,0000,0000,,layer and then the outside layers, and\Nthen we go to an outside-in approach and
Dialogue: 0,0:01:39.83,0:01:48.64,Default,,0000,0000,0000,,then talk about how you're going to hit\NWikipedia from the outside.
Dialogue: 0,0:01:48.64,0:01:53.32,Default,,0000,0000,0000,,So first of all, let's some,\Nlet me get you some information. First of
Dialogue: 0,0:01:53.32,0:01:57.26,Default,,0000,0000,0000,,all, all of Wikimedia, Wikipedia\Ninfrastructure is run by Wikimedia
Dialogue: 0,0:01:57.26,0:02:01.81,Default,,0000,0000,0000,,Foundation, an American nonprofit\Ncharitable organization. We don't run any
Dialogue: 0,0:02:01.81,0:02:07.96,Default,,0000,0000,0000,,ads and we are only 370 people. If you\Ncount Wikimedia Deutschland or all other
Dialogue: 0,0:02:07.96,0:02:12.50,Default,,0000,0000,0000,,chapters, it's around 500 people in total.\NIt's nothing compared to the companies
Dialogue: 0,0:02:12.50,0:02:19.53,Default,,0000,0000,0000,,outside. But all of the content is\Nmanaged by volunteers. Even our staff
Dialogue: 0,0:02:19.53,0:02:24.17,Default,,0000,0000,0000,,doesn't do edits, add content to\NWikipedia. And we support 300 languages,
Dialogue: 0,0:02:24.17,0:02:29.50,Default,,0000,0000,0000,,which is a very large number. And \NWikipedia, it's eighteen years old, so it
Dialogue: 0,0:02:29.50,0:02:37.95,Default,,0000,0000,0000,,can vote now. And also, Wikipedia has some\Nreally, really weird articles. Um, I want
Dialogue: 0,0:02:37.95,0:02:42.51,Default,,0000,0000,0000,,to ask you, what is your, if you have\Nencountered any really weird article
Dialogue: 0,0:02:42.51,0:02:47.97,Default,,0000,0000,0000,,in Wikipedia? My favorite is a list of\Npeople who died on the toilet. But if you
Dialogue: 0,0:02:47.97,0:02:54.62,Default,,0000,0000,0000,,know anything, raise your hands. Uh, do\Nyou know any weird articles in Wikipedia?
Dialogue: 0,0:02:54.62,0:02:58.75,Default,,0000,0000,0000,,Do you know some?\NDaniel Kinzler: Oh, the classic one….
Dialogue: 0,0:02:58.75,0:03:03.60,Default,,0000,0000,0000,,Amir: You need to unmute yourself. Oh,\Nokay.
Dialogue: 0,0:03:03.60,0:03:09.55,Default,,0000,0000,0000,,Daniel: This is technology. I don't know\Nanything about technology. OK, no. The, my
Dialogue: 0,0:03:09.55,0:03:13.90,Default,,0000,0000,0000,,favorite example is "people killed by\Ntheir own invention". That's yeah. That's
Dialogue: 0,0:03:13.90,0:03:20.51,Default,,0000,0000,0000,,a lot of fun. Look it up. It's amazing.\NLucas Werkmeister: There's also a list,
Dialogue: 0,0:03:20.51,0:03:24.81,Default,,0000,0000,0000,,there is also a list of prison escapes\Nusing helicopters. I almost said
Dialogue: 0,0:03:24.81,0:03:28.79,Default,,0000,0000,0000,,helicopter escapes using prisons, which\Ndoesn't make any sense. But that was also
Dialogue: 0,0:03:28.79,0:03:31.83,Default,,0000,0000,0000,,a very interesting list.\NDaniel: I think we also have a category of
Dialogue: 0,0:03:31.83,0:03:35.31,Default,,0000,0000,0000,,lists of lists of lists.\NAmir: That's a page.
Dialogue: 0,0:03:35.31,0:03:39.04,Default,,0000,0000,0000,,Lucas: And every few months someone thinks\Nit's funny to redirect it to Russel's
Dialogue: 0,0:03:39.04,0:03:42.94,Default,,0000,0000,0000,,paradox or so.\NDaniel: Yeah.
Dialogue: 0,0:03:42.94,0:03:49.21,Default,,0000,0000,0000,,Amir: But also beside that, people cannot\Nread Wikipedia in Turkey or China. But
Dialogue: 0,0:03:49.21,0:03:54.45,Default,,0000,0000,0000,,three days ago, actually, the block in\NTurkey was ruled unconstitutional, but
Dialogue: 0,0:03:54.45,0:04:01.00,Default,,0000,0000,0000,,it's not lifted yet. Hopefully they will\Nlift it soon. Um, so Wikipedia, Wikimedia
Dialogue: 0,0:04:01.00,0:04:05.66,Default,,0000,0000,0000,,projects is just not Wikipedia. It's lots\Nand lots of projects. Some of them are not
Dialogue: 0,0:04:05.66,0:04:11.65,Default,,0000,0000,0000,,as successful as the Wikipedia. Um, uh,\Nlike Wikinews. But uh, for example,
Dialogue: 0,0:04:11.65,0:04:16.19,Default,,0000,0000,0000,,Wikipedia is the most successful one, and\Nthere's another one, that's Wikidata. It's
Dialogue: 0,0:04:16.19,0:04:21.68,Default,,0000,0000,0000,,being developed by Wikimedia Deutschland.\NI mean the Wikidata team, with Lucas, um,
Dialogue: 0,0:04:21.68,0:04:26.52,Default,,0000,0000,0000,,and it's being used – it's infobox – it\Nhas the data that Wikipedia or Google
Dialogue: 0,0:04:26.52,0:04:31.45,Default,,0000,0000,0000,,Knowledge Graph or Siri or Alexa uses.\NIt's basically, it's sort of a backbone of
Dialogue: 0,0:04:31.45,0:04:37.98,Default,,0000,0000,0000,,all of the data, uh, through the whole\NInternet. Um, so our infrastructure. Let
Dialogue: 0,0:04:37.98,0:04:42.91,Default,,0000,0000,0000,,me… So first of all, our infrastructure is\Nall Open Source. By principle, we never
Dialogue: 0,0:04:42.91,0:04:48.08,Default,,0000,0000,0000,,use any commercial software. Uh, we could\Nuse a lots of things. They are even
Dialogue: 0,0:04:48.08,0:04:54.33,Default,,0000,0000,0000,,sometimes were given us for free, but we\Nwere, refused to use them. Second
Dialogue: 0,0:04:54.33,0:04:59.06,Default,,0000,0000,0000,,thing is we have two primary data center\Nfor like failovers, when, for example, a
Dialogue: 0,0:04:59.06,0:05:03.96,Default,,0000,0000,0000,,whole datacenter goes offline, so we can\Nfailover to another data center. We have
Dialogue: 0,0:05:03.96,0:05:11.10,Default,,0000,0000,0000,,three caching points of presence or\NCDNs. Our CDNs are all over the world. Uh,
Dialogue: 0,0:05:11.10,0:05:15.18,Default,,0000,0000,0000,,also, we have our own CDN. We don't have,\Nwe don't use CloudFlare, because
Dialogue: 0,0:05:15.18,0:05:20.96,Default,,0000,0000,0000,,CloudFlare, we care about the privacy of\Nthe users and is very important that, for
Dialogue: 0,0:05:20.96,0:05:25.49,Default,,0000,0000,0000,,example, people edit from countries that\Nmight be, uh, dangerous for them to edit
Dialogue: 0,0:05:25.49,0:05:29.81,Default,,0000,0000,0000,,Wikipedia. So we really care to keep the\Ndata as protected as possible.
Dialogue: 0,0:05:29.81,0:05:32.40,Default,,0000,0000,0000,,{\i1}Applause{\i0}
Dialogue: 0,0:05:32.40,0:05:39.46,Default,,0000,0000,0000,,Amir: Uh, we have 17 billion page views\Nper month, and, which goes up and down
Dialogue: 0,0:05:39.46,0:05:44.35,Default,,0000,0000,0000,,based on the season and everything, we\Nhave around 100 to 200 thousand requests
Dialogue: 0,0:05:44.35,0:05:48.45,Default,,0000,0000,0000,,per second. It's different from the\Npageview because requests can be requests
Dialogue: 0,0:05:48.45,0:05:54.54,Default,,0000,0000,0000,,to the objects, can be API, can be lots of\Nthings. And we have 300,000 new editors
Dialogue: 0,0:05:54.54,0:06:03.12,Default,,0000,0000,0000,,per month and we run all of this with 1300\Nbare metal servers. So right now, Daniel
Dialogue: 0,0:06:03.12,0:06:07.01,Default,,0000,0000,0000,,is going to talk about the application\Nlayer and the inside of that
Dialogue: 0,0:06:07.01,0:06:11.83,Default,,0000,0000,0000,,infrastructure.\NDaniel: Thanks, Amir. Oh, the clicky
Dialogue: 0,0:06:11.83,0:06:20.33,Default,,0000,0000,0000,,thing. Thank you. So the application layer\Nis basically the software that actually
Dialogue: 0,0:06:20.33,0:06:25.05,Default,,0000,0000,0000,,does what a wiki does, right? It lets you\Nedit pages, create or update pages and
Dialogue: 0,0:06:25.05,0:06:29.65,Default,,0000,0000,0000,,then search the page views. {\i1}interference\Nnoise{\i0} The challenge for Wikipedia, of
Dialogue: 0,0:06:29.65,0:06:37.15,Default,,0000,0000,0000,,course, is serving all the many page views\Nthat Amir just described. The core of the
Dialogue: 0,0:06:37.15,0:06:42.69,Default,,0000,0000,0000,,application is a classic LAMP application.\N{\i1}interference noise{\i0} I have to stop
Dialogue: 0,0:06:42.69,0:06:50.13,Default,,0000,0000,0000,,moving. Yes? Is that it? It's a classic\NLAMP stack application. So it's written in
Dialogue: 0,0:06:50.13,0:06:57.08,Default,,0000,0000,0000,,PHP, it runs on an Apache server. It uses\NMySQL as a database in the backend. We
Dialogue: 0,0:06:57.08,0:07:01.63,Default,,0000,0000,0000,,used to use a HHVM instead of the… Yeah,\Nwe…
Dialogue: 0,0:07:01.63,0:07:13.83,Default,,0000,0000,0000,,Herald: Hier. Sorry. Nimm mal das hier.\NDaniel: Hello. We used to use HHVM as the
Dialogue: 0,0:07:13.83,0:07:20.81,Default,,0000,0000,0000,,PHP engine, but we just switched back to\Nthe mainstream PHP, using PHP 7.2 now,
Dialogue: 0,0:07:20.81,0:07:24.72,Default,,0000,0000,0000,,because Facebook decided that HHVM is\Ngoing to be incompatible with the standard
Dialogue: 0,0:07:24.72,0:07:35.43,Default,,0000,0000,0000,,and they were just basically developing it\Nfor, for themselves. Right. So we have
Dialogue: 0,0:07:35.43,0:07:42.74,Default,,0000,0000,0000,,separate clusters of servers for serving\Nrequests, for serving different requests,
Dialogue: 0,0:07:42.74,0:07:48.02,Default,,0000,0000,0000,,page views on the one hand, and also\Nhandling edits. Then we have a cluster for
Dialogue: 0,0:07:48.02,0:07:55.35,Default,,0000,0000,0000,,handling API calls and then we have a\Nbunch of servers set up to handle
Dialogue: 0,0:07:55.35,0:08:01.05,Default,,0000,0000,0000,,asynchronous jobs, things that happen in\Nthe background, the job runners, and…
Dialogue: 0,0:08:01.05,0:08:05.24,Default,,0000,0000,0000,,I guess video scaling is a very obvious\Nexample of that. It just takes too long to
Dialogue: 0,0:08:05.24,0:08:11.72,Default,,0000,0000,0000,,do it on the fly. But we use it for many\Nother things as well. MediaWiki, MediaWiki
Dialogue: 0,0:08:11.72,0:08:15.93,Default,,0000,0000,0000,,is kind of an amazing thing because you\Ncan just install it on your own shared-
Dialogue: 0,0:08:15.93,0:08:23.42,Default,,0000,0000,0000,,hosting, 10-bucks-a-month's webspace and\Nit will run. But you can also use it to,
Dialogue: 0,0:08:23.42,0:08:29.27,Default,,0000,0000,0000,,you know, serve half the world. And so\Nit's a very powerful and versatile system,
Dialogue: 0,0:08:29.27,0:08:34.48,Default,,0000,0000,0000,,which also… I mean, this, this wide span\Nof different applications also creates
Dialogue: 0,0:08:34.48,0:08:41.00,Default,,0000,0000,0000,,problems. That's something that I will\Ntalk about tomorrow. But for now, let's
Dialogue: 0,0:08:41.00,0:08:49.23,Default,,0000,0000,0000,,look at the fun things. So if you want to\Nserve a lot of page views, you have to do
Dialogue: 0,0:08:49.23,0:08:55.55,Default,,0000,0000,0000,,a lot of caching. And so we have a whole…\Nyeah, a whole set of different caching
Dialogue: 0,0:08:55.55,0:09:00.88,Default,,0000,0000,0000,,systems. The most important one is\Nprobably the parser cache. So as you
Dialogue: 0,0:09:00.88,0:09:07.43,Default,,0000,0000,0000,,probably know, wiki pages are created in,\Nin a markup language, Wikitext, and they
Dialogue: 0,0:09:07.43,0:09:13.29,Default,,0000,0000,0000,,need to be parsed and turned into HTML.\NAnd the result of that parsing is, of
Dialogue: 0,0:09:13.29,0:09:19.94,Default,,0000,0000,0000,,course, cached. And that cache is semi-\Npersistent, it… nothing really ever drops
Dialogue: 0,0:09:19.94,0:09:25.06,Default,,0000,0000,0000,,out of it. It's a huge thing. And it's, it\Nlives in a dedicated MySQL database
Dialogue: 0,0:09:25.06,0:09:33.49,Default,,0000,0000,0000,,system. Yeah. We use memcached a lot for\Nall kinds of miscellaneous things,
Dialogue: 0,0:09:33.49,0:09:38.93,Default,,0000,0000,0000,,anything that we need to keep around and\Nshare between server instances. And we
Dialogue: 0,0:09:38.93,0:09:43.59,Default,,0000,0000,0000,,have been using redis for a while, for\Nanything that we want to have available,
Dialogue: 0,0:09:43.59,0:09:47.56,Default,,0000,0000,0000,,not just between different servers, but\Nalso between different data centers,
Dialogue: 0,0:09:47.56,0:09:53.20,Default,,0000,0000,0000,,because redis is a bit better about\Nsynchronizing things between, between
Dialogue: 0,0:09:53.20,0:09:59.82,Default,,0000,0000,0000,,different systems, we still use it for\Nsession storage, especially, though we are
Dialogue: 0,0:09:59.82,0:10:09.60,Default,,0000,0000,0000,,about to move away from that and we'll be\Nusing Cassandra for session storage. We
Dialogue: 0,0:10:09.60,0:10:19.31,Default,,0000,0000,0000,,have a bunch of additional services\Nrunning for specialized purposes, like
Dialogue: 0,0:10:19.31,0:10:27.12,Default,,0000,0000,0000,,scaling images, rendering formulas, math\Nformulas, ORES is pretty interesting. ORES
Dialogue: 0,0:10:27.12,0:10:33.40,Default,,0000,0000,0000,,is a system for automatically detecting\Nvandalism or rating edits. So this is a
Dialogue: 0,0:10:33.40,0:10:38.12,Default,,0000,0000,0000,,machine learning based system for\Ndetecting problems and highlighting edits
Dialogue: 0,0:10:38.12,0:10:45.06,Default,,0000,0000,0000,,that may not be, may not be great and need\Nmore attention. We have some additional
Dialogue: 0,0:10:45.06,0:10:50.94,Default,,0000,0000,0000,,services that process our content for\Nconsumption on mobile devices, chopping
Dialogue: 0,0:10:50.94,0:10:56.48,Default,,0000,0000,0000,,pages up into bits and pieces that then\Ncan be consumed individually and many,
Dialogue: 0,0:10:56.48,0:11:08.20,Default,,0000,0000,0000,,many more. In the background, we also have\Nto manage events, right, we use Kafka for
Dialogue: 0,0:11:08.20,0:11:14.64,Default,,0000,0000,0000,,message queuing, and we use that to notify\Ndifferent parts of the system about
Dialogue: 0,0:11:14.64,0:11:19.98,Default,,0000,0000,0000,,changes. On the one hand, we use that to\Nfeed the job runners that I just
Dialogue: 0,0:11:19.98,0:11:27.54,Default,,0000,0000,0000,,mentioned. But we also use it, for\Ninstance, to purge the entries in the
Dialogue: 0,0:11:27.54,0:11:35.05,Default,,0000,0000,0000,,CDN when pages become updated and things\Nlike that. OK, the next session is going
Dialogue: 0,0:11:35.05,0:11:40.27,Default,,0000,0000,0000,,to be about the databases. Are there, very\Nquickly, we will have quite a bit of time
Dialogue: 0,0:11:40.27,0:11:45.23,Default,,0000,0000,0000,,for discussion afterwards. But are there\Nany questions right now about what we said
Dialogue: 0,0:11:45.23,0:11:57.12,Default,,0000,0000,0000,,so far? Everything extremely crystal\Nclear. OK, no clarity is left? I see. Oh,
Dialogue: 0,0:11:57.12,0:12:07.57,Default,,0000,0000,0000,,one question, in the back.\NQ: Can you maybe turn the volume up a
Dialogue: 0,0:12:07.57,0:12:20.22,Default,,0000,0000,0000,,little bit? Thank you.\NDaniel: Yeah, I think this is your
Dialogue: 0,0:12:20.22,0:12:27.96,Default,,0000,0000,0000,,section, right? Oh, its Amir again. Sorry.\NAmir: So I want to talk about my favorite
Dialogue: 0,0:12:27.96,0:12:32.28,Default,,0000,0000,0000,,topic, the dungeons of, dungeons of every\Nproduction system, databases. The database
Dialogue: 0,0:12:32.28,0:12:39.58,Default,,0000,0000,0000,,of Wikipedia is really interesting and\Ncomplicated on its own. We use MariaDB, we
Dialogue: 0,0:12:39.58,0:12:45.87,Default,,0000,0000,0000,,switched from MySQL in 2013 for lots of\Ncomplicated reasons. As, as I said,
Dialogue: 0,0:12:45.87,0:12:50.20,Default,,0000,0000,0000,,because we are really open source, you can\Ngo and not just check our database tree,
Dialogue: 0,0:12:50.20,0:12:55.31,Default,,0000,0000,0000,,that says, like, how it looks and what's\Nthe replicas and masters. Actually, you
Dialogue: 0,0:12:55.31,0:12:59.65,Default,,0000,0000,0000,,can even query the Wikipedia's database\Nlive when you have that, you can just go
Dialogue: 0,0:12:59.65,0:13:02.93,Default,,0000,0000,0000,,to that address and login with your\NWikipedia account and just can do whatever
Dialogue: 0,0:13:02.93,0:13:07.43,Default,,0000,0000,0000,,you want. Like, it was a funny thing that\Na couple of months ago, someone sent me a
Dialogue: 0,0:13:07.43,0:13:12.97,Default,,0000,0000,0000,,message, sent me a message like, oh, I\Nfound a security issue. You can just query
Dialogue: 0,0:13:12.97,0:13:18.00,Default,,0000,0000,0000,,Wikipedia's database. I was like, no, no,\Nit's actually, we, we let this happen.
Dialogue: 0,0:13:18.00,0:13:21.90,Default,,0000,0000,0000,,It's like, it's sanitized. We removed the\Npassword hashes and everything. But still,
Dialogue: 0,0:13:21.90,0:13:27.78,Default,,0000,0000,0000,,you can use this. And, but if you wanted\Nto say, like, how the clusters work, the
Dialogue: 0,0:13:27.78,0:13:32.03,Default,,0000,0000,0000,,database clusters, because it gets too\Nbig, they first started sharding, but now
Dialogue: 0,0:13:32.03,0:13:36.28,Default,,0000,0000,0000,,we have sections that are basically\Ndifferent clusters. Uh, really large wikis
Dialogue: 0,0:13:36.28,0:13:42.84,Default,,0000,0000,0000,,have their own section. For example,\NEnglish Wikipedia is s1. German Wikipedia
Dialogue: 0,0:13:42.84,0:13:50.82,Default,,0000,0000,0000,,with two or three other small wikis are in\Ns5. Wikidata is on s8, and so on. And
Dialogue: 0,0:13:50.82,0:13:56.25,Default,,0000,0000,0000,,each section have a master and several\Nreplicas. But one of the replicas is
Dialogue: 0,0:13:56.25,0:14:01.70,Default,,0000,0000,0000,,actually a master in another data center\Nbecause of the failover that I told you.
Dialogue: 0,0:14:01.70,0:14:08.08,Default,,0000,0000,0000,,So it is, basically two layers of\Nreplication exist. This is, what I'm
Dialogue: 0,0:14:08.08,0:14:13.07,Default,,0000,0000,0000,,telling you, is about metadata. But for\NWikitext, we also need to have a complete
Dialogue: 0,0:14:13.07,0:14:19.45,Default,,0000,0000,0000,,different set of databases. But it can be,\Nwe use consistent hashing to just scale it
Dialogue: 0,0:14:19.45,0:14:27.63,Default,,0000,0000,0000,,horizontally so we can just put more\Ndatabases on it, for that. Uh, but I don't
Dialogue: 0,0:14:27.63,0:14:32.07,Default,,0000,0000,0000,,know if you know it, but Wikipedia stores\Nevery edit. So you have the text of,
Dialogue: 0,0:14:32.07,0:14:36.93,Default,,0000,0000,0000,,Wikitext of every edit in the whole\Nhistory in the database. Uhm, also we have
Dialogue: 0,0:14:36.93,0:14:41.91,Default,,0000,0000,0000,,parser cache that Daniel explained, and\Nparser cache is also consistent hashing.
Dialogue: 0,0:14:41.91,0:14:47.00,Default,,0000,0000,0000,,So we just can horizontally scale it. But\Nfor metadata, it is slightly more
Dialogue: 0,0:14:47.00,0:14:56.44,Default,,0000,0000,0000,,complicated. Um, metadata shows and is\Nbeing used to render the page. So in order
Dialogue: 0,0:14:56.44,0:15:01.68,Default,,0000,0000,0000,,to do this, this is, for example, a very\Nshort version of the database tree that I
Dialogue: 0,0:15:01.68,0:15:07.02,Default,,0000,0000,0000,,showed you. You can even go and look for\Nother ones but this is a s1. s1 eqiad this
Dialogue: 0,0:15:07.02,0:15:12.10,Default,,0000,0000,0000,,is the main data center the master is this\Nnumber and it replicates to some of this
Dialogue: 0,0:15:12.10,0:15:16.86,Default,,0000,0000,0000,,and then this 7, the second one that this\Nwas with 2000 because it's the second data
Dialogue: 0,0:15:16.86,0:15:24.75,Default,,0000,0000,0000,,center and it's a master of the other one.\NAnd it has its own replications
Dialogue: 0,0:15:24.75,0:15:30.68,Default,,0000,0000,0000,,between cross three replications because\Nthe master, that master data center is in
Dialogue: 0,0:15:30.68,0:15:37.40,Default,,0000,0000,0000,,Ashburn, Virginia. The second data center\Nis in Dallas, Texas. So they need to have a
Dialogue: 0,0:15:37.40,0:15:43.22,Default,,0000,0000,0000,,cross DC replication and that happens\Nwith a TLS to make sure that no one starts
Dialogue: 0,0:15:43.22,0:15:49.20,Default,,0000,0000,0000,,to listen to, in between these two, and we\Nhave snapshots and even dumps of the whole
Dialogue: 0,0:15:49.20,0:15:53.44,Default,,0000,0000,0000,,history of Wikipedia. You can go to\Ndumps.wikimedia.org and download the whole
Dialogue: 0,0:15:53.44,0:15:59.13,Default,,0000,0000,0000,,reserve every wiki you want, except the\Nones that we had to remove for privacy
Dialogue: 0,0:15:59.13,0:16:04.90,Default,,0000,0000,0000,,reasons and with a lots and lots of\Nbackups. I recently realized we have lots
Dialogue: 0,0:16:04.90,0:16:15.15,Default,,0000,0000,0000,,of backups. And in total it is 570 TB of data\Nand total 150 database servers and a
Dialogue: 0,0:16:15.15,0:16:20.27,Default,,0000,0000,0000,,queries that happens to them is around\N350,000 queries per second and, in total,
Dialogue: 0,0:16:20.27,0:16:29.46,Default,,0000,0000,0000,,it requires 70 terabytes of RAM. So and\Nalso we have another storage section that
Dialogue: 0,0:16:29.46,0:16:35.00,Default,,0000,0000,0000,,called Elasticsearch which you can guess\Nit- it's being used for search, on the top
Dialogue: 0,0:16:35.00,0:16:39.05,Default,,0000,0000,0000,,right, if you're using desktop. It's\Ndifferent in mobile, I think. And also it
Dialogue: 0,0:16:39.05,0:16:44.61,Default,,0000,0000,0000,,depends on if you're rtl language as well,\Nbut also it runs by a team called search
Dialogue: 0,0:16:44.61,0:16:47.55,Default,,0000,0000,0000,,platform because none of us are from\Nsearch platform we cannot explain it this
Dialogue: 0,0:16:47.55,0:16:54.01,Default,,0000,0000,0000,,much we don't know much how it works it\Nslightly. Also we have a media storage for
Dialogue: 0,0:16:54.01,0:16:58.42,Default,,0000,0000,0000,,all of the free pictures that's being\Nuploaded to Wikimedia like, for example,
Dialogue: 0,0:16:58.42,0:17:02.40,Default,,0000,0000,0000,,if you have a category in Commons. Commons\Nis our wiki that holds all of the free
Dialogue: 0,0:17:02.40,0:17:08.13,Default,,0000,0000,0000,,media and if we have a category in Commons\Ncalled cats looking at left and you have
Dialogue: 0,0:17:08.13,0:17:15.63,Default,,0000,0000,0000,,category cats looking at right so we have\Nlots and lots of images. It's 390 terabytes
Dialogue: 0,0:17:15.63,0:17:20.62,Default,,0000,0000,0000,,of media, 1 billion object and uses Swift.\NSwift is the object is storage component
Dialogue: 0,0:17:20.62,0:17:29.19,Default,,0000,0000,0000,,of OpenStack and it has it has several\Nlayers of caching, frontend, backend.
Dialogue: 0,0:17:29.19,0:17:36.80,Default,,0000,0000,0000,,Yeah, that's mostly it. And we want to\Ntalk about traffic now and so this picture
Dialogue: 0,0:17:36.80,0:17:43.93,Default,,0000,0000,0000,,is when Sweden in 1967 moved from a left-\Ndriving from left to there driving to
Dialogue: 0,0:17:43.93,0:17:48.100,Default,,0000,0000,0000,,right. This is basically what happens in\NWikipedia infrastructure as well. So we
Dialogue: 0,0:17:48.100,0:17:54.94,Default,,0000,0000,0000,,have five caching layers and the most\Nrecent one is eqsin which is in Singapore,
Dialogue: 0,0:17:54.94,0:17:59.31,Default,,0000,0000,0000,,the three one are just CDN ulsfo, codfw,\Nesams and eqsin. Sorry, ulsfo, esams and
Dialogue: 0,0:17:59.31,0:18:06.59,Default,,0000,0000,0000,,eqsin are just CDNs. We have also two\Npoints of presence, one in Chicago and the
Dialogue: 0,0:18:06.59,0:18:15.08,Default,,0000,0000,0000,,other one is also in Amsterdam, but we\Ndon't get to that. So, we have, as I said,
Dialogue: 0,0:18:15.08,0:18:20.23,Default,,0000,0000,0000,,we have our own content delivery network\Nwith our traffic or allocation is done by
Dialogue: 0,0:18:20.23,0:18:26.86,Default,,0000,0000,0000,,GeoDNS which actually is written and\Nmaintained by one of the traffic people,
Dialogue: 0,0:18:26.86,0:18:32.14,Default,,0000,0000,0000,,and we can pool and depool DCs. It has a\Ntime to live of 10 minute- 10 minutes, so
Dialogue: 0,0:18:32.14,0:18:37.95,Default,,0000,0000,0000,,if a data center goes down. We have - it\Ntakes 10 minutes to actually propagate for
Dialogue: 0,0:18:37.95,0:18:47.11,Default,,0000,0000,0000,,being depooled and repooled again. And we\Nuse LVS as transport layer and this layer
Dialogue: 0,0:18:47.11,0:18:55.80,Default,,0000,0000,0000,,3 and 4 of the Linux load balancer for\NLinux and supports consistent hashing and
Dialogue: 0,0:18:55.80,0:19:00.68,Default,,0000,0000,0000,,also we ever got we grow so big that we\Nneeded to have something that manages the
Dialogue: 0,0:19:00.68,0:19:07.10,Default,,0000,0000,0000,,load balancer so we wrote something our\Nown system is called pybal. And also we -
Dialogue: 0,0:19:07.10,0:19:11.21,Default,,0000,0000,0000,,lots of companies actually peer with us. We\Nfor example directly connect to
Dialogue: 0,0:19:11.21,0:19:20.44,Default,,0000,0000,0000,,Amsterdam amps X. So this is how the\Ncaching works, which is, anyway, it's
Dialogue: 0,0:19:20.44,0:19:24.78,Default,,0000,0000,0000,,there is lots of reasons for this. Let's\Njust get the started. We use TLS, we
Dialogue: 0,0:19:24.78,0:19:31.08,Default,,0000,0000,0000,,support TLS 1.2 where we have K then\Nthe first layer we have nginx-. Do you
Dialogue: 0,0:19:31.08,0:19:40.05,Default,,0000,0000,0000,,know it - does anyone know what nginx-\Nmeans? And so that's related but not - not
Dialogue: 0,0:19:40.05,0:19:46.78,Default,,0000,0000,0000,,correct. So we have nginx which is the free\Nversion and we have nginx plus which is
Dialogue: 0,0:19:46.78,0:19:51.73,Default,,0000,0000,0000,,the commercial version and nginx. But we\Ndon't use nginx to do load balancing or
Dialogue: 0,0:19:51.73,0:19:56.39,Default,,0000,0000,0000,,anything so we stripped out everything\Nfrom it, and we just use it for TLS
Dialogue: 0,0:19:56.39,0:20:02.02,Default,,0000,0000,0000,,termination so we call it nginx-, is an\Ninternal joke. So and then we have Varnish
Dialogue: 0,0:20:02.02,0:20:09.81,Default,,0000,0000,0000,,frontend. Varnish also is a caching layer\Nand this is the frontend is on the memory
Dialogue: 0,0:20:09.81,0:20:15.00,Default,,0000,0000,0000,,which is very very fast and you have the\Nbackend which is on the storage and the
Dialogue: 0,0:20:15.00,0:20:22.56,Default,,0000,0000,0000,,hard disk but this is slow. The fun thing\Nis like just CDN caching layer takes 90%
Dialogue: 0,0:20:22.56,0:20:26.87,Default,,0000,0000,0000,,of our requests. Its response and 90% of\Nbecause just gets to the Varnish and just
Dialogue: 0,0:20:26.87,0:20:34.72,Default,,0000,0000,0000,,return and then with doesn't work it goes\Nthrough the application layer. The Varnish
Dialogue: 0,0:20:34.72,0:20:41.26,Default,,0000,0000,0000,,holds-- it has a TTL of 24 hours so if you\Nchange an article, it also get invalidated
Dialogue: 0,0:20:41.26,0:20:47.16,Default,,0000,0000,0000,,by the application. So if someone added the\NCDN actually purges the result. And the
Dialogue: 0,0:20:47.16,0:20:52.33,Default,,0000,0000,0000,,thing is, the frontend is shorted that can\Nspike by request so you come here load
Dialogue: 0,0:20:52.33,0:20:56.47,Default,,0000,0000,0000,,balancer just randomly sends your request\Nto a frontend but then the backend is
Dialogue: 0,0:20:56.47,0:21:00.99,Default,,0000,0000,0000,,actually, if the frontend can't find it,\Nit sends it to the backend and the backend
Dialogue: 0,0:21:00.99,0:21:09.70,Default,,0000,0000,0000,,is actually sort of - how is it called? -\Nit's a used hash by request, so, for
Dialogue: 0,0:21:09.70,0:21:15.40,Default,,0000,0000,0000,,example, article of Barack Obama is only\Nbeing served from one node in the data
Dialogue: 0,0:21:15.40,0:21:22.06,Default,,0000,0000,0000,,center in the CDN. If none of this works it\Nactually hits the other data center. So,
Dialogue: 0,0:21:22.06,0:21:29.94,Default,,0000,0000,0000,,yeah, I actually explained all of this. So\Nwe have two - two caching clusters and one
Dialogue: 0,0:21:29.94,0:21:35.82,Default,,0000,0000,0000,,is called text and the other one is called\Nupload, it's not confusing at all, and if
Dialogue: 0,0:21:35.82,0:21:42.56,Default,,0000,0000,0000,,you want to find out, you can just do mtr\Nen.wikipedia.org and you - you're - the end
Dialogue: 0,0:21:42.56,0:21:49.91,Default,,0000,0000,0000,,node is text-lb.wikimedia.org which is the\Nour text storage but if you go to
Dialogue: 0,0:21:49.91,0:21:57.79,Default,,0000,0000,0000,,upload.wikimedia.org, you get to hit the\Nupload cluster. Yeah this is so far, what
Dialogue: 0,0:21:57.79,0:22:03.67,Default,,0000,0000,0000,,is it, and it has lots of problems because\Na) varnish is open core, so the version
Dialogue: 0,0:22:03.67,0:22:09.31,Default,,0000,0000,0000,,that you use is open source we don't use\Nthe commercial one, but the open core one
Dialogue: 0,0:22:09.31,0:22:21.01,Default,,0000,0000,0000,,doesn't support TLS. What? What happened?\NOkay. No, no, no! You should I just-
Dialogue: 0,0:22:21.01,0:22:35.79,Default,,0000,0000,0000,,you're not supposed to see this. Okay,\Nsorry for the- huh? Okay, okay sorry. So
Dialogue: 0,0:22:35.79,0:22:40.12,Default,,0000,0000,0000,,Varnish has lots of problems, Varnish is\Nopen core, it doesn't support TLS
Dialogue: 0,0:22:40.12,0:22:45.22,Default,,0000,0000,0000,,termination which makes us to have this\Nnginx- their system just to do TLS
Dialogue: 0,0:22:45.22,0:22:49.54,Default,,0000,0000,0000,,termination, makes our system complicated.\NIt doesn't work very well with so if that
Dialogue: 0,0:22:49.54,0:22:55.97,Default,,0000,0000,0000,,causes us to have a cron job to restart\Nevery Varnish node twice a week. We have a
Dialogue: 0,0:22:55.97,0:23:04.33,Default,,0000,0000,0000,,cron job that this restarts every Vanish\Nnode which is embarrassing, but also, on
Dialogue: 0,0:23:04.33,0:23:08.81,Default,,0000,0000,0000,,the other hand then the end of Varnish\Nlike backend wants to talk to the
Dialogue: 0,0:23:08.81,0:23:13.01,Default,,0000,0000,0000,,application layer, it also doesn't support\Nterminate - TLS termination, so we use
Dialogue: 0,0:23:13.01,0:23:19.97,Default,,0000,0000,0000,,IPSec which is even more embarrassing, but\Nwe are changing it. So we call it, if you
Dialogue: 0,0:23:19.97,0:23:25.08,Default,,0000,0000,0000,,are using a particular fixed server which\Nis very very nice and it's also open
Dialogue: 0,0:23:25.08,0:23:31.07,Default,,0000,0000,0000,,source, a fully open source like in with\NApache Foundation, Apache does the TLS,
Dialogue: 0,0:23:31.07,0:23:37.17,Default,,0000,0000,0000,,does the TLS by termination and still\Nfor now we have a Varnish frontend that
Dialogue: 0,0:23:37.17,0:23:44.81,Default,,0000,0000,0000,,still exists but a backend is also going\Nto change to the ATS, so we call this ATS
Dialogue: 0,0:23:44.81,0:23:49.97,Default,,0000,0000,0000,,sandwich. Two ATS happening between and\Nthere the middle there's a Varnish. The
Dialogue: 0,0:23:49.97,0:23:55.27,Default,,0000,0000,0000,,good thing is that the TLS termination\Nwhen it moves to ATS, you can actually use
Dialogue: 0,0:23:55.27,0:24:01.50,Default,,0000,0000,0000,,TLS 1.3 which is more modern and more\Nsecure and even very faster so it
Dialogue: 0,0:24:01.50,0:24:05.89,Default,,0000,0000,0000,,basically drops 100 milliseconds from\Nevery request that goes to Wikipedia.
Dialogue: 0,0:24:05.89,0:24:12.35,Default,,0000,0000,0000,,That translates to centuries of our\Nusers' time every month, but ATS is going
Dialogue: 0,0:24:12.35,0:24:19.48,Default,,0000,0000,0000,,on and hopefully it will go live soon and\Nonce these are done, so this is the new
Dialogue: 0,0:24:19.48,0:24:25.67,Default,,0000,0000,0000,,version. And, as I said, the TLS and when\Nwe can do this we can actually use the
Dialogue: 0,0:24:25.67,0:24:36.52,Default,,0000,0000,0000,,more secure instead of IPSec to talk about\Nbetween data centers. Yes. And now it's
Dialogue: 0,0:24:36.52,0:24:42.26,Default,,0000,0000,0000,,time that Lucas talks about what happens\Nwhen you type in en.wikipedia.org.
Dialogue: 0,0:24:42.26,0:24:44.88,Default,,0000,0000,0000,,\NLucas: Yes, this makes sense, thank you.
Dialogue: 0,0:24:44.88,0:24:49.07,Default,,0000,0000,0000,,So, first of all, what you see on the\Nslide here as the image doesn't really
Dialogue: 0,0:24:49.07,0:24:52.30,Default,,0000,0000,0000,,have anything to do with what happens when\Nyou type in wikipedia.org because it's an
Dialogue: 0,0:24:52.30,0:24:57.25,Default,,0000,0000,0000,,offline Wikipedia reader but it's just a\Nnice image. So this is basically a summary
Dialogue: 0,0:24:57.25,0:25:02.85,Default,,0000,0000,0000,,of everything they already said, so if,\Nwhich is the most common case, you are
Dialogue: 0,0:25:02.85,0:25:10.97,Default,,0000,0000,0000,,lucky and get a URL which is cached, then,\Nso, first your computer asked for the IP
Dialogue: 0,0:25:10.97,0:25:15.62,Default,,0000,0000,0000,,address of en.wikipedia.org it reaches\Nthis whole DNS daemon and because we're at
Dialogue: 0,0:25:15.62,0:25:19.24,Default,,0000,0000,0000,,Congress here it tells you the closest\Ndata center is the one in Amsterdam, so
Dialogue: 0,0:25:19.24,0:25:25.76,Default,,0000,0000,0000,,esams and it's going to hit the edge, what\Nwe call load bouncers/router there, then
Dialogue: 0,0:25:25.76,0:25:31.93,Default,,0000,0000,0000,,going through TLS termination through\Nnginx- and then it's going to hit the
Dialogue: 0,0:25:31.93,0:25:36.81,Default,,0000,0000,0000,,Varnish caching server, either frontend or\Nbackends and then you get a response and
Dialogue: 0,0:25:36.81,0:25:40.94,Default,,0000,0000,0000,,that's already it and nothing else is ever\Nbothered again. It doesn't even reach any
Dialogue: 0,0:25:40.94,0:25:46.32,Default,,0000,0000,0000,,other data center which is very nice and\Nso that's, you said around 90% of the
Dialogue: 0,0:25:46.32,0:25:52.42,Default,,0000,0000,0000,,requests we get, and if you're unlucky and\Nthe URL you requested is not in the
Dialogue: 0,0:25:52.42,0:25:57.40,Default,,0000,0000,0000,,Varnish in the Amsterdam data center then\Nit gets forwarded to the eqiad data
Dialogue: 0,0:25:57.40,0:26:01.52,Default,,0000,0000,0000,,center, which is the primary one and there\Nit still has a chance to hit the cache and
Dialogue: 0,0:26:01.52,0:26:04.84,Default,,0000,0000,0000,,perhaps this time it's there and then the\Nresponse is going to get cached in the
Dialogue: 0,0:26:04.84,0:26:09.74,Default,,0000,0000,0000,,frontend, no, in the Amsterdam Varnish and\Nyou're also going to get a response and we
Dialogue: 0,0:26:09.74,0:26:13.64,Default,,0000,0000,0000,,still don't have to run any application\Nstuff. If we do have to hit any
Dialogue: 0,0:26:13.64,0:26:17.45,Default,,0000,0000,0000,,application stuff and then Varnish is\Ngoing to forward that, if it's
Dialogue: 0,0:26:17.45,0:26:22.97,Default,,0000,0000,0000,,upload.wikimedia.org, it goes to the media\Nstorage Swift, if it's any other domain it
Dialogue: 0,0:26:22.97,0:26:28.45,Default,,0000,0000,0000,,goes to MediaWiki and then MediaWiki does\Na ton of work to connect to the database,
Dialogue: 0,0:26:28.45,0:26:33.53,Default,,0000,0000,0000,,in this case the first shard for English\NWikipedia, get the wiki text from there,
Dialogue: 0,0:26:33.53,0:26:38.60,Default,,0000,0000,0000,,get the wiki text of all the related pages\Nand templates. No, wait I forgot
Dialogue: 0,0:26:38.60,0:26:43.52,Default,,0000,0000,0000,,something. First it checks if the HTML for\Nthis page is available in parser cache, so
Dialogue: 0,0:26:43.52,0:26:46.91,Default,,0000,0000,0000,,that's another caching layer, and this\Napplication cache - this parser cache
Dialogue: 0,0:26:46.91,0:26:53.53,Default,,0000,0000,0000,,might either be memcached or the database\Ncache behind it and if it's not there,
Dialogue: 0,0:26:53.53,0:26:57.68,Default,,0000,0000,0000,,then it has to go get the wikitext, get\Nall the related things and render that
Dialogue: 0,0:26:57.68,0:27:03.68,Default,,0000,0000,0000,,into HTML which takes a long time and goes\Nthrough some pretty ancient code and if
Dialogue: 0,0:27:03.68,0:27:07.78,Default,,0000,0000,0000,,you are doing an edit or an upload, it's\Neven worse, because then always has to go
Dialogue: 0,0:27:07.78,0:27:13.97,Default,,0000,0000,0000,,to MediaWiki and then it not only has to\Nstore this new edit, either in the media
Dialogue: 0,0:27:13.97,0:27:19.63,Default,,0000,0000,0000,,back-end or in the database, it also has\Nupdate a bunch of stuff, like, especially
Dialogue: 0,0:27:19.63,0:27:25.20,Default,,0000,0000,0000,,if you-- first of all, it has to purge the\Ncache, it has to tell all the Varnish
Dialogue: 0,0:27:25.20,0:27:28.100,Default,,0000,0000,0000,,servers that there's a new version of this\NURL available so that it doesn't take a
Dialogue: 0,0:27:28.100,0:27:33.94,Default,,0000,0000,0000,,full day until the time-to-live expires.\NIt also has to update a bunch of things,
Dialogue: 0,0:27:33.94,0:27:38.64,Default,,0000,0000,0000,,for example, if you edited a template, it\Nmight have been used in a million pages
Dialogue: 0,0:27:38.64,0:27:43.75,Default,,0000,0000,0000,,and the next time anyone requests one of\Nthose million pages, those should also
Dialogue: 0,0:27:43.75,0:27:49.02,Default,,0000,0000,0000,,actually be rendered again using the new\Nversion of the template so it has to
Dialogue: 0,0:27:49.02,0:27:54.15,Default,,0000,0000,0000,,invalidate the cache for all of those and\Nall that is deferred through the job queue
Dialogue: 0,0:27:54.15,0:28:01.44,Default,,0000,0000,0000,,and it might have to calculate thumbnails\Nif you uploaded the file or create a -
Dialogue: 0,0:28:01.44,0:28:06.61,Default,,0000,0000,0000,,retranscode media files because maybe you\Nuploaded in - what do we support? - you
Dialogue: 0,0:28:06.61,0:28:09.84,Default,,0000,0000,0000,,upload in WebM and the browser only\Nsupports some other media codec or
Dialogue: 0,0:28:09.84,0:28:12.87,Default,,0000,0000,0000,,something, we transcode that and also\Nencode it down to the different
Dialogue: 0,0:28:12.87,0:28:19.74,Default,,0000,0000,0000,,resolutions, so then it goes through that\Nwhole dance and, yeah, that was already
Dialogue: 0,0:28:19.74,0:28:23.77,Default,,0000,0000,0000,,those slides. Is Amir going to talk again\Nabout how we manage -
Dialogue: 0,0:28:23.77,0:28:29.52,Default,,0000,0000,0000,,Amir: I mean okay yeah I quickly come back\Njust for a short break to talk about
Dialogue: 0,0:28:29.52,0:28:36.69,Default,,0000,0000,0000,,managing to manage because managing 100-\N1300 bare metal hardware plus a Kubernetes
Dialogue: 0,0:28:36.69,0:28:42.70,Default,,0000,0000,0000,,cluster is not easy, so what we do is that\Nwe use Puppet for configuration
Dialogue: 0,0:28:42.70,0:28:48.22,Default,,0000,0000,0000,,management in our bare metal systems, it's\Nfun, five to 50,000 lines of Puppet code. I
Dialogue: 0,0:28:48.22,0:28:52.12,Default,,0000,0000,0000,,mean, lines of code is not a great\Nindicator but you can roughly get an
Dialogue: 0,0:28:52.12,0:28:59.15,Default,,0000,0000,0000,,estimate of how its things work and we\Nhave 100,000 lines of Ruby and we have our
Dialogue: 0,0:28:59.15,0:29:04.43,Default,,0000,0000,0000,,CI and CD cluster, we have so we don't\Nstore anything in GitHub or GitLab, we
Dialogue: 0,0:29:04.43,0:29:10.56,Default,,0000,0000,0000,,have our own system which is based on\NGerrit and for that we have a system of
Dialogue: 0,0:29:10.56,0:29:15.54,Default,,0000,0000,0000,,Jenkins and the Jenkins does all of this\Nkind of things and also because we have a
Dialogue: 0,0:29:15.54,0:29:21.96,Default,,0000,0000,0000,,Kubernetes cluster for services, some of\Nour services, if you make a merger change
Dialogue: 0,0:29:21.96,0:29:26.44,Default,,0000,0000,0000,,in the Gerrit it also builds the Docker\Nfiles and containers and push it up to the
Dialogue: 0,0:29:26.44,0:29:35.44,Default,,0000,0000,0000,,production and also in order to run remote\NSSH commands, we have cumin that's like in
Dialogue: 0,0:29:35.44,0:29:39.20,Default,,0000,0000,0000,,the house automation and we built this\Nfarm for our systems and for example you
Dialogue: 0,0:29:39.20,0:29:45.57,Default,,0000,0000,0000,,go there and say ok we pull this node or\Nrun this command in all of the data
Dialogue: 0,0:29:45.57,0:29:52.89,Default,,0000,0000,0000,,Varnish nodes that I told you like you\Nwant to restart them. And with this I get
Dialogue: 0,0:29:52.89,0:29:57.90,Default,,0000,0000,0000,,back to Lucas.\NLucas: So, I am going to talk a bit more
Dialogue: 0,0:29:57.90,0:30:01.93,Default,,0000,0000,0000,,about Wikimedia Cloud Services which is a\Nbit different in that it's not really our
Dialogue: 0,0:30:01.93,0:30:06.27,Default,,0000,0000,0000,,production stuff but it's where you\Npeople, the volunteers of the Wikimedia
Dialogue: 0,0:30:06.27,0:30:11.49,Default,,0000,0000,0000,,movement can run their own code, so you\Ncan request a project which is kind of a
Dialogue: 0,0:30:11.49,0:30:15.51,Default,,0000,0000,0000,,group of users and then you get assigned a\Npool of you have this much CPU and this
Dialogue: 0,0:30:15.51,0:30:20.100,Default,,0000,0000,0000,,much RAM and you can create virtual\Nmachines with those resources and then do
Dialogue: 0,0:30:20.100,0:30:29.12,Default,,0000,0000,0000,,stuff there and run basically whatever you\Nwant, to create and boot and shut down the
Dialogue: 0,0:30:29.12,0:30:33.36,Default,,0000,0000,0000,,VMs and stuff we use OpenStack and there's\Na Horizon frontend for that which you use
Dialogue: 0,0:30:33.36,0:30:36.41,Default,,0000,0000,0000,,through the browser and it's largely out\Nall the time but otherwise it works pretty
Dialogue: 0,0:30:36.41,0:30:42.62,Default,,0000,0000,0000,,well. Internally, ideally you manage the\NVMs using Puppet but a lot of people just
Dialogue: 0,0:30:42.62,0:30:47.86,Default,,0000,0000,0000,,SSH in and then do whatever they need to\Nset up the VM manually and it happens,
Dialogue: 0,0:30:47.86,0:30:52.76,Default,,0000,0000,0000,,well, and there's a few big projects like\NToolforge where you can run your own web-
Dialogue: 0,0:30:52.76,0:30:57.50,Default,,0000,0000,0000,,based tools or the beta cluster which is\Nbasically a copy of some of the biggest
Dialogue: 0,0:30:57.50,0:31:02.50,Default,,0000,0000,0000,,wikis like there's a beta English\NWikipedia, beta Wikidata, beta Wikimedia
Dialogue: 0,0:31:02.50,0:31:08.32,Default,,0000,0000,0000,,Commons using mostly the same\Nconfiguration as production but using the
Dialogue: 0,0:31:08.32,0:31:12.45,Default,,0000,0000,0000,,current master version of the software\Ninstead of whatever we deploy once a week so
Dialogue: 0,0:31:12.45,0:31:15.84,Default,,0000,0000,0000,,if there's a bug, we see it earlier\Nhopefully, even if we didn't catch it
Dialogue: 0,0:31:15.84,0:31:20.28,Default,,0000,0000,0000,,locally, because the beta cluster is more\Nsimilar to the production environment and
Dialogue: 0,0:31:20.28,0:31:24.23,Default,,0000,0000,0000,,also the continuous - continuous\Nintegration service run in Wikimedia Cloud
Dialogue: 0,0:31:24.23,0:31:28.98,Default,,0000,0000,0000,,Services as well. Yeah and also you have\Nto have Kubernetes somewhere on these
Dialogue: 0,0:31:28.98,0:31:33.61,Default,,0000,0000,0000,,slides right, so you can use that to\Ndistribute work between the tools in
Dialogue: 0,0:31:33.61,0:31:37.18,Default,,0000,0000,0000,,Toolforge or you can use the grid engine\Nwhich does a similar thing but it's like
Dialogue: 0,0:31:37.18,0:31:42.52,Default,,0000,0000,0000,,three decades old and through five forks\Nnow I think the current fork we use is son
Dialogue: 0,0:31:42.52,0:31:46.100,Default,,0000,0000,0000,,of grid engine and I don't know what it\Nwas called before, but that's Cloud
Dialogue: 0,0:31:46.100,0:31:54.79,Default,,0000,0000,0000,,Services.\NAmir: So in a nutshell, this is our - our
Dialogue: 0,0:31:54.79,0:32:01.09,Default,,0000,0000,0000,,systems. We have 1300 bare metal services\Nwith lots and lots of caching, like lots
Dialogue: 0,0:32:01.09,0:32:06.92,Default,,0000,0000,0000,,of layers of caching, because mostly we\Nserves read and we can just keep them as a
Dialogue: 0,0:32:06.92,0:32:12.18,Default,,0000,0000,0000,,cached version and all of this is open\Nsource, you can contribute to it, if you
Dialogue: 0,0:32:12.18,0:32:18.09,Default,,0000,0000,0000,,want to and there's a lot of configuration\Nis also open and I - this is the way I got
Dialogue: 0,0:32:18.09,0:32:21.94,Default,,0000,0000,0000,,hired like I open it started contributing\Nto the system I feel like yeah we can-
Dialogue: 0,0:32:21.94,0:32:31.55,Default,,0000,0000,0000,,come and work for us, so this is a -\NDaniel: That's actually how all of us got
Dialogue: 0,0:32:31.55,0:32:38.35,Default,,0000,0000,0000,,hired.\NAmir: So yeah, and this is the whole thing
Dialogue: 0,0:32:38.35,0:32:47.57,Default,,0000,0000,0000,,that happens in Wikimedia and if you want\Nto - no, if you want to help us, we are
Dialogue: 0,0:32:47.57,0:32:51.42,Default,,0000,0000,0000,,hiring. You can just go to jobs at\Nwikimedia.org, if you want to work for
Dialogue: 0,0:32:51.42,0:32:54.38,Default,,0000,0000,0000,,Wikimedia Foundation. If you want to work\Nwith Wikimedia Deutschland, you can go to
Dialogue: 0,0:32:54.38,0:32:59.18,Default,,0000,0000,0000,,wikimedia.de and at the bottom there's a\Nlink for jobs because the links got too
Dialogue: 0,0:32:59.18,0:33:03.47,Default,,0000,0000,0000,,long. If you can contribute, if you want\Nto contribute to us, there is so many ways
Dialogue: 0,0:33:03.47,0:33:07.93,Default,,0000,0000,0000,,to contribute, as I said, there's so many\Nbugs, we have our own graphical system,
Dialogue: 0,0:33:07.93,0:33:12.72,Default,,0000,0000,0000,,you can just look at the monitor and a\NPhabricator is our bug tracker, you can
Dialogue: 0,0:33:12.72,0:33:20.64,Default,,0000,0000,0000,,just go there and find the bug and fix\Nthings. Actually, we have one repository
Dialogue: 0,0:33:20.64,0:33:26.47,Default,,0000,0000,0000,,that is private but it only holds the\Ncertificate for as TLS and things that are
Dialogue: 0,0:33:26.47,0:33:31.50,Default,,0000,0000,0000,,really really private then we cannot\Nremove them. But also there are
Dialogue: 0,0:33:31.50,0:33:33.78,Default,,0000,0000,0000,,documentations, the documentation for\Ninfrastructure is at
Dialogue: 0,0:33:33.78,0:33:40.41,Default,,0000,0000,0000,,wikitech.wikimedia.org and documentation\Nfor configuration is at noc.wikimedia.org
Dialogue: 0,0:33:40.41,0:33:46.60,Default,,0000,0000,0000,,plus the documentation of our codebase.\NThe documentation for MediaWiki itself is
Dialogue: 0,0:33:46.60,0:33:52.99,Default,,0000,0000,0000,,at mediawiki.org and also we have a our\Nown system of URL shortener you can go to
Dialogue: 0,0:33:52.99,0:33:58.79,Default,,0000,0000,0000,,w.wiki and short and shorten any URL in\NWikimedia structure so we reserved the
Dialogue: 0,0:33:58.79,0:34:08.78,Default,,0000,0000,0000,,dollar sign for the donate site and yeah,\Nyou have any questions, please.
Dialogue: 0,0:34:08.78,0:34:16.54,Default,,0000,0000,0000,,{\i1}Applause{\i0}
Dialogue: 0,0:34:16.54,0:34:21.68,Default,,0000,0000,0000,,Daniel: It's if you know we have quite a bit of\Ntime for questions so if anything wasn't
Dialogue: 0,0:34:21.68,0:34:27.15,Default,,0000,0000,0000,,clear or they're curious about anything\Nplease, please ask.
Dialogue: 0,0:34:27.15,0:34:37.20,Default,,0000,0000,0000,,AM: So one question what is not in the\Npresentation. Do you have any efforts with
Dialogue: 0,0:34:37.20,0:34:42.46,Default,,0000,0000,0000,,hacking attacks?\NAmir: So the first rule of security issues
Dialogue: 0,0:34:42.46,0:34:49.21,Default,,0000,0000,0000,,is that we don't talk about security issues\Nbut let's say this baby has all sorts of
Dialogue: 0,0:34:49.21,0:34:56.24,Default,,0000,0000,0000,,attacks happening, we have usually we have\NDDo. Once there was happening a couple of
Dialogue: 0,0:34:56.24,0:34:59.82,Default,,0000,0000,0000,,months ago that was very successful. I\Ndon't know if you read the news about
Dialogue: 0,0:34:59.82,0:35:05.20,Default,,0000,0000,0000,,that, but we also, we have a infrastructure\Nto handle this, we have a security team
Dialogue: 0,0:35:05.20,0:35:12.74,Default,,0000,0000,0000,,that handles these cases and yes.\NAM: Hello how do you manage access to your
Dialogue: 0,0:35:12.74,0:35:20.07,Default,,0000,0000,0000,,infrastructure from your employees?\NAmir: So it's SS-- so we have a LDAP
Dialogue: 0,0:35:20.07,0:35:25.39,Default,,0000,0000,0000,,group and LDAP for the web-based\Nsystems but for SSH and for this ssh we
Dialogue: 0,0:35:25.39,0:35:30.66,Default,,0000,0000,0000,,have strict protocols and then you get a\Nprivate key and some people usually
Dialogue: 0,0:35:30.66,0:35:35.48,Default,,0000,0000,0000,,protect their private key using UV keys\Nand then you have you can SSH to the
Dialogue: 0,0:35:35.48,0:35:40.42,Default,,0000,0000,0000,,system basically.\NLucas: Yeah, well, there's some
Dialogue: 0,0:35:40.42,0:35:44.72,Default,,0000,0000,0000,,firewalling setup but there's only one\Nserver for data center that you can
Dialogue: 0,0:35:44.72,0:35:48.22,Default,,0000,0000,0000,,actually reach through SSH and then you\Nhave to tunnel through that to get to any
Dialogue: 0,0:35:48.22,0:35:51.36,Default,,0000,0000,0000,,other server.\NAmir: And also, like, we have we have a
Dialogue: 0,0:35:51.36,0:35:55.50,Default,,0000,0000,0000,,internal firewall and it's basically if\Nyou go to the inside of the production you
Dialogue: 0,0:35:55.50,0:36:01.45,Default,,0000,0000,0000,,cannot talk to the outside. You even, you\Nfor example do git clone github.org, it
Dialogue: 0,0:36:01.45,0:36:07.20,Default,,0000,0000,0000,,doesn't, github.com doesn't work. It\Nonly can access tools that are for inside
Dialogue: 0,0:36:07.20,0:36:13.39,Default,,0000,0000,0000,,Wikimedia Foundation infrastructure.\NAM: Okay, hi, you said you do TLS
Dialogue: 0,0:36:13.39,0:36:18.64,Default,,0000,0000,0000,,termination through nginx, do you still\Nallow non-HTTPS so it should be non-secure access.
Dialogue: 0,0:36:18.64,0:36:22.78,Default,,0000,0000,0000,,Amir: No we dropped it a really long\Ntime ago but also
Dialogue: 0,0:36:22.78,0:36:25.07,Default,,0000,0000,0000,,Lucas: 2013 or so\NAmir: Yeah, 2015
Dialogue: 0,0:36:25.07,0:36:28.65,Default,,0000,0000,0000,,Lucas: 2015\NAmir: 2013 started serving the most of the
Dialogue: 0,0:36:28.65,0:36:35.74,Default,,0000,0000,0000,,traffic but 15, we dropped all of the\NHTTP- non-HTTPS protocols and recently even
Dialogue: 0,0:36:35.74,0:36:43.94,Default,,0000,0000,0000,,dropped and we are not serving any SSL\Nrequests anymore and TLS 1.1 is also being
Dialogue: 0,0:36:43.94,0:36:48.46,Default,,0000,0000,0000,,phased out, so we are sending you a warning\Nto the users like you're using TLS 1.1,
Dialogue: 0,0:36:48.46,0:36:54.81,Default,,0000,0000,0000,,please migrate to these new things that\Ncame out around 10 years ago, so yeah
Dialogue: 0,0:36:54.81,0:36:59.85,Default,,0000,0000,0000,,Lucas: Yeah I think the deadline for that\Nis like February 2020 or something then
Dialogue: 0,0:36:59.85,0:37:04.71,Default,,0000,0000,0000,,we'll only have TLS 1.2\NAmir: And soon we are going to support TLS
Dialogue: 0,0:37:04.71,0:37:06.64,Default,,0000,0000,0000,,1.3\NLucas: Yeah
Dialogue: 0,0:37:06.64,0:37:12.46,Default,,0000,0000,0000,,Are there any questions?\NQ: so does read-only traffic
Dialogue: 0,0:37:12.46,0:37:18.03,Default,,0000,0000,0000,,from logged in users hit all the way\Nthrough to the parser cache or is there
Dialogue: 0,0:37:18.03,0:37:22.28,Default,,0000,0000,0000,,another layer of caching for that?\NAmir: Yes we, you bypass all of
Dialogue: 0,0:37:22.28,0:37:28.47,Default,,0000,0000,0000,,that, you can.\NDaniel: We need one more microphone. Yes,
Dialogue: 0,0:37:28.47,0:37:33.87,Default,,0000,0000,0000,,it actually does and this is a pretty big\Nproblem and something we want to look into
Dialogue: 0,0:37:33.87,0:37:38.93,Default,,0000,0000,0000,,{\i1}clears throat{\i0} but it requires quite a\Nbit of rearchitecting. If you are
Dialogue: 0,0:37:38.93,0:37:44.25,Default,,0000,0000,0000,,interested in this kind of thing, maybe\Ncome to my talk tomorrow at noon.
Dialogue: 0,0:37:44.25,0:37:48.82,Default,,0000,0000,0000,,Amir: Yeah one reason we can, we are\Nplanning to do is active active so we have
Dialogue: 0,0:37:48.82,0:37:56.50,Default,,0000,0000,0000,,two primaries and the read request gets\Nrequest - from like the users can hit
Dialogue: 0,0:37:56.50,0:37:58.46,Default,,0000,0000,0000,,their secondary data center instead of the\Nmain one.
Dialogue: 0,0:37:58.46,0:38:03.99,Default,,0000,0000,0000,,Lucas: I think there was a question way in\Nthe back there, for some time already
Dialogue: 0,0:38:03.99,0:38:13.95,Default,,0000,0000,0000,,AM: Hi, I got a question. I read on the\NWikitech that you are using karate as a
Dialogue: 0,0:38:13.95,0:38:19.04,Default,,0000,0000,0000,,validation platform for some parts, can\Nyou tell us something about this or what
Dialogue: 0,0:38:19.04,0:38:24.62,Default,,0000,0000,0000,,parts of Wikipedia or Wikimedia are hosted\Non this platform?
Dialogue: 0,0:38:24.62,0:38:29.59,Default,,0000,0000,0000,,Amir: I am I'm not oh sorry so I don't\Nknow this kind of very very sure but take
Dialogue: 0,0:38:29.59,0:38:34.39,Default,,0000,0000,0000,,it with a grain of salt but as far as I\Nknow karate is used to build a very small
Dialogue: 0,0:38:34.39,0:38:39.83,Default,,0000,0000,0000,,VMs in productions that we need for very\Nvery small micro sites that we serve to
Dialogue: 0,0:38:39.83,0:38:45.62,Default,,0000,0000,0000,,the users. So we built just one or two VMs,\Nwe don't use it very as often as I think
Dialogue: 0,0:38:45.62,0:38:54.82,Default,,0000,0000,0000,,so.\NAM: Do you also think about open hardware?
Dialogue: 0,0:38:54.82,0:39:03.95,Default,,0000,0000,0000,,Amir: I don't, you can\NDaniel: Not - not for servers. I think for
Dialogue: 0,0:39:03.95,0:39:07.50,Default,,0000,0000,0000,,the offline Reader project, but this is not\Nactually run by the Foundation, it's
Dialogue: 0,0:39:07.50,0:39:10.29,Default,,0000,0000,0000,,supported but it's not something that the\NFoundation does. They were sort of
Dialogue: 0,0:39:10.29,0:39:15.10,Default,,0000,0000,0000,,thinking about open hardware but really\Nopen hardware in practice usually means,
Dialogue: 0,0:39:15.10,0:39:19.61,Default,,0000,0000,0000,,you - you don't, you know, if you really\Nwant to go down to the chip design, it's
Dialogue: 0,0:39:19.61,0:39:25.21,Default,,0000,0000,0000,,pretty tough, so yeah, it's- it's it- it's\Nusually not practical, sadly.
Dialogue: 0,0:39:25.21,0:39:31.66,Default,,0000,0000,0000,,Amir: And one thing I can say but this is\Nthat we have a some machine - machines that
Dialogue: 0,0:39:31.66,0:39:37.15,Default,,0000,0000,0000,,are really powerful that we give to the\Nresearchers to run analysis on the between
Dialogue: 0,0:39:37.15,0:39:43.37,Default,,0000,0000,0000,,this itself and we needed to have GPUs for\Nthose but the problem was - was there
Dialogue: 0,0:39:43.37,0:39:49.11,Default,,0000,0000,0000,,wasn't any open source driver for them so\Nwe migrated and use AMD I think, but AMD
Dialogue: 0,0:39:49.11,0:39:53.61,Default,,0000,0000,0000,,didn't fit in the rack it was a quite a\Nendeavor to get it to work for our
Dialogue: 0,0:39:53.61,0:40:03.71,Default,,0000,0000,0000,,researchers to help you CPU.\NAM: I'm still impressed that you answer
Dialogue: 0,0:40:03.71,0:40:10.92,Default,,0000,0000,0000,,90% out of the cache. Do all people access\Nthe same pages or is the cache that huge?
Dialogue: 0,0:40:10.92,0:40:21.16,Default,,0000,0000,0000,,So what percentage of - of the whole\Ndatabase is in the cache then?
Dialogue: 0,0:40:21.16,0:40:29.76,Default,,0000,0000,0000,,Daniel: I don't have the exact numbers to\Nbe honest, but a large percentage of the
Dialogue: 0,0:40:29.76,0:40:36.77,Default,,0000,0000,0000,,whole database is in the cache. I mean it\Nexpires after 24 hours so really obscure
Dialogue: 0,0:40:36.77,0:40:43.43,Default,,0000,0000,0000,,stuff isn't there but I mean it's- it's a-\Nit's a- it's a power-law distribution
Dialogue: 0,0:40:43.43,0:40:47.89,Default,,0000,0000,0000,,right? You have a few pages that are\Naccessed a lot and you have many many many
Dialogue: 0,0:40:47.89,0:40:55.42,Default,,0000,0000,0000,,pages that are not actually accessed\Nat all for a week or so except maybe for a
Dialogue: 0,0:40:55.42,0:41:01.74,Default,,0000,0000,0000,,crawler, so I don't know a number. My\Nguess would be it's less than 50% that is
Dialogue: 0,0:41:01.74,0:41:06.52,Default,,0000,0000,0000,,actually cached but, you know, that still\Ncovers 90%-- it's probably the top 10% of
Dialogue: 0,0:41:06.52,0:41:11.63,Default,,0000,0000,0000,,pages would still cover 90% of the\Npageviews, but I don't-- this would be
Dialogue: 0,0:41:11.63,0:41:15.51,Default,,0000,0000,0000,,actually-- I should look this up, it would\Nbe interesting numbers to have, yes.
Dialogue: 0,0:41:15.51,0:41:20.71,Default,,0000,0000,0000,,Lucas: Do you know if this is 90% of the\Npageviews or 90% of the get requests
Dialogue: 0,0:41:20.71,0:41:24.28,Default,,0000,0000,0000,,because, like, requests for the JavaScript\Nwould also be cached more often, I assume
Dialogue: 0,0:41:24.28,0:41:27.53,Default,,0000,0000,0000,,Daniel: I would expect that for non-\Npageviews, it's even higher
Dialogue: 0,0:41:27.53,0:41:30.01,Default,,0000,0000,0000,,Lucas: Yeah\NDaniel: Yeah, because you know all the
Dialogue: 0,0:41:30.01,0:41:34.15,Default,,0000,0000,0000,,icons and- and, you know, JavaScript\Nbundles and CSS and stuff doesn't ever
Dialogue: 0,0:41:34.15,0:41:40.31,Default,,0000,0000,0000,,change\NLucas: I'm gonna say for every 180 min 90%
Dialogue: 0,0:41:40.31,0:41:50.79,Default,,0000,0000,0000,,but there's a question back there\NAM: Hey. Do your data centers run on green
Dialogue: 0,0:41:50.79,0:41:55.22,Default,,0000,0000,0000,,energy?\NAmir: Very valid question. So, the
Dialogue: 0,0:41:55.22,0:42:03.45,Default,,0000,0000,0000,,Amsterdam city n1 is a full green but the\Nother ones are partially green, partially
Dialogue: 0,0:42:03.45,0:42:10.84,Default,,0000,0000,0000,,coal and like gas. As far as I know, there\Nare some plans to make them move away from
Dialogue: 0,0:42:10.84,0:42:15.17,Default,,0000,0000,0000,,it but the other hand we realized that if\Nwe don't produce as much as a carbon
Dialogue: 0,0:42:15.17,0:42:21.35,Default,,0000,0000,0000,,emission because we don't have much servers\Nand we don't use much data, there was a
Dialogue: 0,0:42:21.35,0:42:26.79,Default,,0000,0000,0000,,summation and that we realized our carbon\Nemission is basically as the same as 200
Dialogue: 0,0:42:26.79,0:42:34.72,Default,,0000,0000,0000,,and in the datacenter plus all of their\Ntravel that all of this have to and all of
Dialogue: 0,0:42:34.72,0:42:37.88,Default,,0000,0000,0000,,the events is 250 households, it's very\Nvery small it's I think it's one
Dialogue: 0,0:42:37.88,0:42:44.89,Default,,0000,0000,0000,,thousandth of the comparable\Ntraffic with Facebook even if you just cut
Dialogue: 0,0:42:44.89,0:42:50.65,Default,,0000,0000,0000,,down with the same traffic because\NFacebook collects the data, it runs very
Dialogue: 0,0:42:50.65,0:42:54.27,Default,,0000,0000,0000,,sophisticated machine learning algorithms\Nthat's that's a real complicate, but for
Dialogue: 0,0:42:54.27,0:43:01.12,Default,,0000,0000,0000,,Wikimedia, we don't do this so we don't\Nneed much energy. Does - does the answer
Dialogue: 0,0:43:01.12,0:43:04.92,Default,,0000,0000,0000,,your question?\NHerald: Do we have any other
Dialogue: 0,0:43:04.92,0:43:15.72,Default,,0000,0000,0000,,questions left? Yeah sorry\NAM: hi how many developers do you need to
Dialogue: 0,0:43:15.72,0:43:19.79,Default,,0000,0000,0000,,maintain the whole infrastructure and how\Nmany developers or let's say head
Dialogue: 0,0:43:19.79,0:43:24.50,Default,,0000,0000,0000,,developer hours you needed to build the\Nwhole infrastructure like the question is
Dialogue: 0,0:43:24.50,0:43:29.33,Default,,0000,0000,0000,,because what I find very interesting about\Nthe talk it's a non-profit, so as an
Dialogue: 0,0:43:29.33,0:43:34.11,Default,,0000,0000,0000,,example for other nonprofits is how much\Nmoney are we talking about in order to
Dialogue: 0,0:43:34.11,0:43:38.76,Default,,0000,0000,0000,,build something like this as a digital\Ncommon.
Dialogue: 0,0:43:45.63,0:43:48.98,Default,,0000,0000,0000,,Daniel: If this is just about actually\Nrunning all this so just operations is
Dialogue: 0,0:43:48.98,0:43:53.53,Default,,0000,0000,0000,,less than 20 people I think which makes if\Nyou if you basically divide the requests
Dialogue: 0,0:43:53.53,0:43:59.87,Default,,0000,0000,0000,,per second by people you get to something\Nlike 8,000 requests per second per
Dialogue: 0,0:43:59.87,0:44:04.37,Default,,0000,0000,0000,,operations engineer which I think is a\Npretty impressive number. This is probably
Dialogue: 0,0:44:04.37,0:44:09.81,Default,,0000,0000,0000,,a lot higher I would I would really like\Nto know if there's any organization that
Dialogue: 0,0:44:09.81,0:44:17.27,Default,,0000,0000,0000,,tops that. I don't actually know the whole\Nthe the actual operations budget I know is
Dialogue: 0,0:44:17.27,0:44:24.56,Default,,0000,0000,0000,,it two two-digit millions annually. Total\Nhours for building this over the last 18
Dialogue: 0,0:44:24.56,0:44:29.07,Default,,0000,0000,0000,,years, I have no idea. For the for the\Nfirst five or so years, the people doing
Dialogue: 0,0:44:29.07,0:44:34.61,Default,,0000,0000,0000,,it were actually volunteers. We still had\Nvolunteer database administrators and
Dialogue: 0,0:44:34.61,0:44:42.16,Default,,0000,0000,0000,,stuff until maybe ten years ago, eight\Nyears ago, so yeah it's really nobody
Dialogue: 0,0:44:42.16,0:44:44.59,Default,,0000,0000,0000,,did any accounting of this I can only\Nguess.
Dialogue: 0,0:44:56.67,0:45:03.81,Default,,0000,0000,0000,,AM: Hello a tools question. I a few years\Nback I saw some interesting examples of
Dialogue: 0,0:45:03.81,0:45:09.09,Default,,0000,0000,0000,,saltstack use for Wikimedia but right now\NI see only Puppet that come in mentioned
Dialogue: 0,0:45:09.09,0:45:17.82,Default,,0000,0000,0000,,so kind of what happened with that\NAmir: I think we dished saltstack you -
Dialogue: 0,0:45:17.82,0:45:22.97,Default,,0000,0000,0000,,I don't I cannot because none of us are in\Nthe Cloud Services team and I don't think
Dialogue: 0,0:45:22.97,0:45:27.38,Default,,0000,0000,0000,,I can answer you but if you look at the\Nwikitech.wikimedia.org, it's
Dialogue: 0,0:45:27.38,0:45:30.87,Default,,0000,0000,0000,,probably if last time I checked says like\Nit's deprecated and obsolete we don't use
Dialogue: 0,0:45:30.87,0:45:32.14,Default,,0000,0000,0000,,it anymore.
Dialogue: 0,0:45:37.39,0:45:39.92,Default,,0000,0000,0000,,AM: Do you use the bat-ropes like the top
Dialogue: 0,0:45:39.92,0:45:46.13,Default,,0000,0000,0000,,runners to fill spare capacity on the web\Nserving servers or do you have dedicated
Dialogue: 0,0:45:46.13,0:45:51.59,Default,,0000,0000,0000,,servers for the roles.\NLucas: I think they're dedicated.
Dialogue: 0,0:45:51.59,0:45:56.39,Default,,0000,0000,0000,,Amir: The job runners if you're asking job runners \Nare dedicated yes they are they are I
Dialogue: 0,0:45:56.39,0:46:02.91,Default,,0000,0000,0000,,think 5 per primary data center so\NDaniel: Yeah they don't, I mean do we do we
Dialogue: 0,0:46:02.91,0:46:06.56,Default,,0000,0000,0000,,actually have any spare capacity on\Nanything? We don't have that much hardware
Dialogue: 0,0:46:06.56,0:46:08.70,Default,,0000,0000,0000,,everything is pretty much at a hundred\Npercent.
Dialogue: 0,0:46:08.70,0:46:14.11,Default,,0000,0000,0000,,Lucas: I think we still have some server\Nthat is just called misc1111 or something
Dialogue: 0,0:46:14.11,0:46:18.62,Default,,0000,0000,0000,,which run five different things at once,\Nyou can look for those on wikitech.
Dialogue: 0,0:46:18.62,0:46:25.82,Default,,0000,0000,0000,,Amir: But but we go oh sorry it's not five\Nit's 20 per data center 20 per primary
Dialogue: 0,0:46:25.82,0:46:31.44,Default,,0000,0000,0000,,data center that's our job runner and they\Nrun 700 jobs per second.
Dialogue: 0,0:46:31.44,0:46:35.69,Default,,0000,0000,0000,,Lucas: And I think that does not include\Nthe video scaler so those are separate
Dialogue: 0,0:46:35.69,0:46:38.11,Default,,0000,0000,0000,,again\NAmir: No, they merged them in like a month
Dialogue: 0,0:46:38.11,0:46:40.04,Default,,0000,0000,0000,,ago\NLucas: Okay, cool
Dialogue: 0,0:46:47.47,0:46:51.42,Default,,0000,0000,0000,,AM: Maybe a little bit off topic that can\Ntell us a little bit about decision making
Dialogue: 0,0:46:51.42,0:46:55.75,Default,,0000,0000,0000,,process for- for technical decision,\Narchitecture decisions, how does it work
Dialogue: 0,0:46:55.75,0:47:01.89,Default,,0000,0000,0000,,in an organization like this: decision\Nmaking process for architectural
Dialogue: 0,0:47:01.89,0:47:03.41,Default,,0000,0000,0000,,decisions for example.
Dialogue: 0,0:47:08.28,0:47:11.01,Default,,0000,0000,0000,,Daniel: Yeah so Wikimedia has a
Dialogue: 0,0:47:11.01,0:47:16.54,Default,,0000,0000,0000,,committee for making high-level technical\Ndecisions, it's called a Wikimedia
Dialogue: 0,0:47:16.54,0:47:23.61,Default,,0000,0000,0000,,Technical Committee, techcom and we run an\NRFC process so any decision that is a
Dialogue: 0,0:47:23.61,0:47:27.54,Default,,0000,0000,0000,,cross-cutting strategic are especially\Nhard to undo should go through this
Dialogue: 0,0:47:27.54,0:47:33.58,Default,,0000,0000,0000,,process and it's pretty informal,\Nbasically you file a ticket and start
Dialogue: 0,0:47:33.58,0:47:38.00,Default,,0000,0000,0000,,this process. It gets announced\Nin the mailing list, hopefully you get
Dialogue: 0,0:47:38.00,0:47:45.01,Default,,0000,0000,0000,,input and feedback and at some point it is\Nit's approved for implementation. We're
Dialogue: 0,0:47:45.01,0:47:48.64,Default,,0000,0000,0000,,currently looking into improving this\Nprocess, it's not- sometimes it works
Dialogue: 0,0:47:48.64,0:47:52.20,Default,,0000,0000,0000,,pretty well, sometimes things don't get\Nthat much feedback but it still it makes
Dialogue: 0,0:47:52.20,0:47:55.89,Default,,0000,0000,0000,,sure that people are aware of these high-\Nlevel decisions
Dialogue: 0,0:47:55.89,0:47:59.79,Default,,0000,0000,0000,,Amir: Daniel is the chair of that\Ncommittee
Dialogue: 0,0:48:02.16,0:48:07.84,Default,,0000,0000,0000,,Daniel: Yeah, if you want to complain\Nabout the process, please do.
Dialogue: 0,0:48:13.55,0:48:21.44,Default,,0000,0000,0000,,AM: yes regarding CI and CD across along the\Npipeline, of course with that much traffic
Dialogue: 0,0:48:21.44,0:48:27.36,Default,,0000,0000,0000,,you want to keep everything consistent\Nright. So is there any testing
Dialogue: 0,0:48:27.36,0:48:32.15,Default,,0000,0000,0000,,strategies that you have said internally,\Nlike of course unit tests integration
Dialogue: 0,0:48:32.15,0:48:35.79,Default,,0000,0000,0000,,tests but do you do something like\Ncontinuous end to end testing on beta
Dialogue: 0,0:48:35.79,0:48:40.10,Default,,0000,0000,0000,,instances?\NAmir: So if we have beta cluster but also
Dialogue: 0,0:48:40.10,0:48:44.67,Default,,0000,0000,0000,,we do deploy, we call it train and so\Nwe deploy once a week, all of the changes
Dialogue: 0,0:48:44.67,0:48:50.35,Default,,0000,0000,0000,,gets merged to one, like a branch and the\Nbranch gets cut in every Tuesday and it
Dialogue: 0,0:48:50.35,0:48:54.68,Default,,0000,0000,0000,,first goes to the test wikis and\Nthen it goes to all of the wikis that are
Dialogue: 0,0:48:54.68,0:48:59.27,Default,,0000,0000,0000,,not Wikipedia except Catalan and Hebrew\NWikipedia. So basically Hebrew and Catalan
Dialogue: 0,0:48:59.27,0:49:03.76,Default,,0000,0000,0000,,Wikipedia volunteer to be the guinea pigs\Nof the next wikis and if everything works
Dialogue: 0,0:49:03.76,0:49:07.60,Default,,0000,0000,0000,,fine usually it goes there and is like oh\Nthe fatal mater and we have a logging and
Dialogue: 0,0:49:07.60,0:49:12.58,Default,,0000,0000,0000,,then it's like okay we need to fix this\Nand we fix it immediately and then it goes
Dialogue: 0,0:49:12.58,0:49:18.69,Default,,0000,0000,0000,,live to all wikis. This is one way of\Nlooking at it well so okay yeah
Dialogue: 0,0:49:18.69,0:49:23.28,Default,,0000,0000,0000,,Daniel: So, our test coverage is not as\Ngreat as it should be and so we kind of,
Dialogue: 0,0:49:23.28,0:49:30.97,Default,,0000,0000,0000,,you know, abuse our users for this. We\Nare, of course, working to improve this
Dialogue: 0,0:49:30.97,0:49:37.23,Default,,0000,0000,0000,,and one thing that we started recently is\Na program for creating end-to-end tests
Dialogue: 0,0:49:37.23,0:49:43.46,Default,,0000,0000,0000,,for all the API modules we have, in the\Nhope that we can thereby cover pretty much
Dialogue: 0,0:49:43.46,0:49:49.85,Default,,0000,0000,0000,,all of the application logic bypassing the\Nuser interface. I mean, full end-to-end
Dialogue: 0,0:49:49.85,0:49:52.77,Default,,0000,0000,0000,,should, of course, include the user\Ninterface but user interface tests are
Dialogue: 0,0:49:52.77,0:49:58.18,Default,,0000,0000,0000,,pretty brittle and often tests you know\Nwhere things are on the screen and it just
Dialogue: 0,0:49:58.18,0:50:02.56,Default,,0000,0000,0000,,seems to us that it makes a lot of sense\Nto have more- to have tests that actually
Dialogue: 0,0:50:02.56,0:50:07.26,Default,,0000,0000,0000,,test the application logic for what the\Nsystem actually should be doing, rather
Dialogue: 0,0:50:07.26,0:50:15.91,Default,,0000,0000,0000,,than what it should look like and, yeah,\Nwe are currently working on making- so
Dialogue: 0,0:50:15.91,0:50:20.21,Default,,0000,0000,0000,,yeah, basically this has been a proof of\Nconcept and we're currently working to
Dialogue: 0,0:50:20.21,0:50:27.08,Default,,0000,0000,0000,,actually integrate it in- in CI. That\Nperhaps should land once everyone is back
Dialogue: 0,0:50:27.08,0:50:34.56,Default,,0000,0000,0000,,from the vacations and then we have to\Nwrite about a thousand or so tests, I
Dialogue: 0,0:50:34.56,0:50:37.93,Default,,0000,0000,0000,,guess.\NLucas: I think there's also a plan to move
Dialogue: 0,0:50:37.93,0:50:42.56,Default,,0000,0000,0000,,to a system where we actually deploy\Nbasically after every commit and can
Dialogue: 0,0:50:42.56,0:50:45.91,Default,,0000,0000,0000,,immediately roll back if something goes\Nwrong but that's more midterm stuff and
Dialogue: 0,0:50:45.91,0:50:48.34,Default,,0000,0000,0000,,I'm not sure what the current status of\Nthat proposal is
Dialogue: 0,0:50:48.34,0:50:50.45,Default,,0000,0000,0000,,Amir: And it will be in Kubernetes, so it\Nwill be completely different
Dialogue: 0,0:50:50.45,0:50:55.53,Default,,0000,0000,0000,,Daniel: That would be amazing\NLucas: But right now, we are on this
Dialogue: 0,0:50:55.53,0:50:59.73,Default,,0000,0000,0000,,weekly basis, if something goes wrong, we\Nroll back to the last week's version of
Dialogue: 0,0:50:59.73,0:51:06.05,Default,,0000,0000,0000,,the code\NHerald: Are there are any questions-
Dialogue: 0,0:51:06.05,0:51:18.55,Default,,0000,0000,0000,,questions left? Sorry. Yeah. Okay, um, I\Ndon't think so. So, yeah, thank you for
Dialogue: 0,0:51:18.55,0:51:25.33,Default,,0000,0000,0000,,this wonderful talk. Thank you for all\Nyour questions. Um, yeah, I hope you liked
Dialogue: 0,0:51:25.33,0:51:29.75,Default,,0000,0000,0000,,it. Um, see you around, yeah.
Dialogue: 0,0:51:29.75,0:51:33.72,Default,,0000,0000,0000,,{\i1}Applause{\i0}
Dialogue: 0,0:51:33.72,0:51:39.27,Default,,0000,0000,0000,,{\i1}Music{\i0}
Dialogue: 0,0:51:39.27,0:52:01.00,Default,,0000,0000,0000,,Subtitles created by c3subtitles.de\Nin the year 2021. Join, and help us!