36C3 Wikipaka WG: Wikimedia Cloud Services introduction

Music

Herald: Hello everybody, we are ready to
get started we have Lucas and Amir here

and they want to give us a quick
introduction of a project from the

wikimedia foundation called "cloud
services" and how it might be may be

useful to all of us. So let's give a round
of welcoming applause to Lucas and Amir.

Applause

Lucas: Thanks! yea, hello. So "wikimedia
cloud services" is basically this big

collection of all kinds of different
things which are useful if you want to do

taking your things in the wikimedia
universe like with wikipedia or other

projects and you get them free of charge
or you can just use them and the only

requirement is that you use them for
something that's a kind of relevant to the

mission of wikimedia of promoting free
knowledge and that kind of stuff and it's

kind of split into the things that you can
do with your regular wikimedia account

which any registered user can do and then
there's also things you need a special

account for on a different system called
wiki tech and Amir is going to talk more

about those later but first let's just
look into some of the things you can do

with your regular wikimedia account. And
if you want to follow any of these links

there's a shortcut here. I was about to
switch the next tab, so let's just stay

here for a few seconds yeah. So the first
thing is the API sandbox which is if you

want to use the MediaWiki API to figure
out what you have on a page or to make

edits or any kind of stuff. The API
sandbox is a special page that's really

useful to find out how to use the API for
example here's all the different actions I

can use that say query is the kind of
general catch-all action that's here and

then I get down here a list of all the
parameters I can use with queries such as:

I want to have all the user info and what
kind of user info do I want? I want

options, blablabla. I would like to have
some different format versions. So it

gives you all these nice inputs for
figuring out exactly how to use the API

what's valid what's not valid and then you
can make the API request and there you get

a response and we can't read anything
because it's zoomed in way too much. But

it's very helpful when trying to use the
API and then in the end here you can see

what you need to do in your own code to
make the same API request. And for

anything that you can't do with the normal
API - so if you want to do some kind of

more expensive analysis - you can often do
that with Quarry, which is a tool that

lets you write SQL queries against
databases that are almost like the ones in

production like you don't have user
passwords and stuff but you'll have all

the database tables with page metadata and
connections between them and the logs and

all kinds of stuff and you can just write
your SQL here send it and you get the

results for example here's the number of
lexemes published a days so it's some kind

of selecting from the page where the
namespace is the lexeme namespace and

grouping that by the date and then we get
something like all the way down to

September which is apparently when I ran
this query there were here there were 116

lexemes created in this day. Or here
someone had a list of edits to JavaScript

and CSS pages on Hungarian Wikipedia so
you can run these queries against any Wiki

you like, like this here in wikipedia one.
And if you can't get by with just SQL what

you also have is this thing called Paws,
which gives you a Jupiter(?) instance if

you've heard of that you can basically
write your own Python code here and do it

in a very convenient way because there's
all kinds of auto-completion and helpful

things. So i can just try to copy this and
run the code (then I needed a new cell

below it… there we go, Thanks!) and if I
type item I should get helpful hints what

I can do with the item (if it's not
hanging or something or the tab control

space no oh there we go yeah) and it's
also a very useful way to work with py-

wiki-bot or you can also directly get
normal shell here. And one thing (oops did

I click and wrong thing? I would like to
have oh no I don't want a bash notebook I

want a new terminal that's what I want).
And here you have for example database

dumps in (where was it?) public/dumps/
something public again… So if you want to

do some kind of analysis here on the data
dumps you can get them here and then have

all the computing that you want I guess to
analyze the wiki more thoroughly and all

of this is hosted in the Wikimedia Cloud
for you and you don't need your own server

or anything. Oh yeah I had two more
examples of that, for example here: I use

that too so there were a lot of items on
Wikipedia where there was some encoding

error, this should be an apostrophe like
down here and instead it was this kind of

I with an accent and I hacked together
some ugly Java/Python code to make all of

these edits and it was already logged in
as well I didn't need to worry about

logging in or having a password or
anything. So it's a very convenient way to

make edits as well. Or you can build
something nicer here you can insert like

markdown cells to explain what you're
doing and how the code works and build

nice notebooks like that, which are almost
self-explanatory. And those are some of

the things you can do just with your
Wikimedia account and now Amir is going to

talk about some other things.
Amir: Thanks Lucas! So the thing that we

can do is that maybe some of you like me
think that doing thing in browser is for

kids I need to do things in terminal I
need to do connected system and then you

can access for a wiki tech account which
you can just make a wiki tech account in

this place called wiki tech. (where is the
li… no no but I do'… the main thing, the

main list. yeah okay) And so in here so
and then you make a wiki tech account and

it gets approved quickly and then you get
the shell and then you can just quickly go

there (where is yer…) and you can go to
this shell and just log in and then you

have access to day a big set of nodes in
the cloud and you can just do whatever you

want. Also you have access to the core
dumps and you have access to the replica

database. Let me show it to you.
[mumbling] So for example you can go to LS

/public/dumps/public/wikidatawiki/ and
then you get - oh there's like all sorts

of time and everything that you want to,
but if you also… you can do something else

is that you can just do SQL wikidatawiki
and then you go inside the wikidata's

database, I mean it does you don't have
the rights you can you cannot write to

their replica because it's a replica and
also it's sanitized so it doesn't have

their like hash of user password and stuff
like that but still you can do just select

varies from recent changes limits five and
yeah and then you get all of the things

that you want you cannot even describe
anything you want to directly into their

system and then there is also we have
something called the job grid so you can

just put a crown and anything that you
want to or just it's something run

something directly and you goes to the a
big note of cloud kubernetes and then just

runs everything that you want to in its
here there's a more information about it

in here there's a like a long help that it
says like oh I used to run this job and

then job of what it does and you can get
this so you just need to it's a bash

command you can run any bash command and
send it okay return me this output to this

place and the other places one thing that
you can do is also there's a web server

that you can access everything directly so
you can just put a PHP file there and into

the Apache and then yet for example this
is this is an example that we built

together I think two two Christmases ago
but this was like you can just see this is

a piece before the source code is
available and you just copy pasted that

source code into like a directory and it
was there and every time we click on it

and you get most of the edits that happen
on description wiki data that might be

vandalism and we can fix it also a this is
not just the only thing that you can do

with this is that you can also put a
Python flask application is this the file

implants and then this can be just a
Python application and you can just have

the file there and also know JSON Java
there's so many of them also you can have

own database like I have something that
has its own database for example quick

categories in here has jobs that are here
this is this tool for its own built-in

database inside our select cloud services
and its uses it just fine you can do that

as well and also there's a cloud VPS that
it doesn't do any kubernetes it just you

can make a VPS of your own and then do
whatever you want with it so for example

and you get a project and you get the
quota it's a slightly more limited but

also you have access to the whole VPS you
have sudo rights on it you can do whatever

you feel like about it so we have like for
example this project in here and it's

called tools and then there's proxies and
you can for example go into that instance

and reboot it and do whatever you want and
you can make new instance and look at your

culture and look at everything else there
and also you can also make it even a wiki

on one of those cloud VPS systems which is
for example we did in here in here if you

look at it it's just a wiki and the
difference is that for other ones for

example for the vandalism dashboard you
have tools that wmf labs org and then

slash WD w VD which is the tool itself but
in here we get our own subdomain so which

will be wiki data - like seam that flew
out the wmf labs org and you can even put

all sorts of add subdomains for the wmf
labs or as long it's not taken so you can

build a media week instance instance or
you can just complete a new software

anything you can put a word processor who
cares and then you can use it it's very

simple your own thing and you can help
lots of experience. Anything else?

Lucas: I don't think so. Most

important I would say is tool Forge to run
your websites or if that's not enough for

you cloud VPS and then you get your own
VMware you can do absolutely anything you

want as long as it matches those rules and
stuff and I think that's it are there any questions?

Herald: Hello thank you very much for
the talk that was very quick so maybe

anybody has a question here I'll give you
my microphone to ask it. I don't see any

hands nope okay I don't think we have
questions but if you're just too shy to

ask I think these guys always hanging
around here around the wikipaka wiki so

if you have anything you want to talk
about you'll find them later okay then

give a round of applause again
for Lucas and Amir.

Applause

Music