WEBVTT

00:00:00.000 --> 00:00:12.180
<i>rC3 preroll music</i>

00:00:12.180 --> 00:00:17.311
ysf: Hello and welcome to the
infrastructure review of the rC3 this

00:00:17.311 --> 00:00:22.730
year, 2020. What the hell happened? How
could it happen? I'm not alone this year.

00:00:22.730 --> 00:00:28.410
With me is lindworm who will help me with
the slides and everything else I'm going

00:00:28.410 --> 00:00:35.586
to say before. And this is going to be a
great fuck up like last year, maybe. We

00:00:35.586 --> 00:00:40.683
have more teams, more people, more
streams, more of everything. And the first

00:00:40.683 --> 00:00:44.934
team and lindworm who I'm going to introduce 
is the SHOC. Are you there with me?

00:00:44.934 --> 00:00:52.140
Lindworm: Oh, yeah, so I got to go to the
SHOC. Yeah, it's kind of a stress this

00:00:52.140 --> 00:01:00.101
year. We only had about 18 heralds for the
main talks rC1 and rC2. And we have

00:01:00.101 --> 00:01:05.089
introduced about 51 talks with that.
Everybody from this home setup, which was

00:01:05.089 --> 00:01:10.020
a very, very hard struggle. So we all had
a metric ton of adrenaline and excitement

00:01:10.020 --> 00:01:15.750
without… within us. So here you can see
what you have seen, how a herald looks

00:01:15.750 --> 00:01:25.210
from the front. And so it does look in the
background. Oof. That was hard, really

00:01:25.210 --> 00:01:32.240
hard for us. So you see all our different
set ups here, do we have? And we are very,

00:01:32.240 --> 00:01:38.240
very pleased to also have set up a
completely new operation center: the

00:01:38.240 --> 00:01:46.280
Herald News Show, which I really, really
like you to review on YouTube. This was

00:01:46.280 --> 00:01:53.909
such a struggle. And we have about, oh,
wait a second, so as we said, we're a

00:01:53.909 --> 00:01:58.950
little bit unprepared here, I need to have
my notes up. There were 20 members that

00:01:58.950 --> 00:02:05.690
formed a new team on the first day. They
made 23 shows, 10 hours of video

00:02:05.690 --> 00:02:13.029
recording, 20 times the pizza man rung at
the door. And 23 mate bottles had been

00:02:13.029 --> 00:02:19.019
drunk during the preps because all of
those people needed to be online the

00:02:19.019 --> 00:02:25.090
complete time. So I really applaud to
them. That was really awesome, what they

00:02:25.090 --> 00:02:30.489
brought over the team and what they
brought over the stream. And this is an

00:02:30.489 --> 00:02:39.069
awesome team I hope we see more of. ysf,
would you take it over? <i>ysf is muted</i>

00:02:39.069 --> 00:02:46.319
Oh, no. My, my bad. So is the heaven
ready? We need to go to the heaven and

00:02:46.319 --> 00:02:51.239
would have an infrastructure review of the
heaven.

00:02:51.239 --> 00:03:29.109
raziel: OK. Du hörst mich noch? Ja, hallo?
Ich bin der raziel aus dem Heaven und ehm…

00:03:29.109 --> 00:03:38.809
Yeah, heaven is ready, so welcome,
everybody. I'm raziel from heaven, and I

00:03:38.809 --> 00:03:48.109
will present you the infrastructure review
from the heaven team. We had some angel

00:03:48.109 --> 00:03:55.620
statistics scrapped out a few hours ago.
And on this year, we have not so much

00:03:55.620 --> 00:04:04.809
angels like last year, because we had a
remote event, but we had a total of 1487

00:04:04.809 --> 00:04:17.410
total angels from which 710 arrived and
even more of 300 angels that at least

00:04:17.410 --> 00:04:28.120
still did one shift. And in total the
recorded work done to that point was

00:04:28.120 --> 00:04:41.479
roughly 17 and 75 weeks of done working
hours, and for the rC3 world we also

00:04:41.479 --> 00:04:51.210
prepared a few goodies so people could
come visit us. And so we provided them a

00:04:51.210 --> 00:05:01.439
few badges there. And every angel that,
for example, found our extinguished…

00:05:01.439 --> 00:05:09.040
expired extinguisher and also extinguished
fire in heaven. The first batch was

00:05:09.040 --> 00:05:21.610
achieved from 232 of our angels and even
less. But still a good number of 125

00:05:21.610 --> 00:05:27.879
angels accomplished to help us and
extinguish the fire that broke out during

00:05:27.879 --> 00:05:38.090
an event. And with that numbers checked,
we also will jump into our heaven. So I

00:05:38.090 --> 00:05:45.870
would like to show you some expressions
and impressions from it. We had quite the

00:05:45.870 --> 00:05:52.830
team working to do exactly what the heaven
could do: manage its people so we needed

00:05:52.830 --> 00:06:01.320
our heaven office. And we also did this
with respect to your privacy, so. We

00:06:01.320 --> 00:06:07.259
painted our color… our clouds white as
ever, so we cannot see your nicknames, and

00:06:07.259 --> 00:06:12.979
you could do your angel work but not be
bothered with us asking for your names.

00:06:12.979 --> 00:06:22.539
And also, we had prepared some secret
passage to our back office. And every time

00:06:22.539 --> 00:06:30.330
on the real event, it would happen that
some adventurers would find their way into

00:06:30.330 --> 00:06:35.050
our back office. And so we needed to
provide that opportunity as well, as you

00:06:35.050 --> 00:06:42.699
can see here. And let me say that some
adventurers tried to find the way in our

00:06:42.699 --> 00:06:49.600
sacred digital back office, but only a few
were successful. So we hope everyone found

00:06:49.600 --> 00:06:58.169
its way back into the real world from our
labyrinth. And we also did not spare any

00:06:58.169 --> 00:07:07.760
expenses to do some additional update for
our angels as well. As you can see, we

00:07:07.760 --> 00:07:13.349
tried to do some multi-instance support.
So some of our angels also accomplished to

00:07:13.349 --> 00:07:21.409
split up and serve more than one angel at
a time. And that was quite awesome. And so

00:07:21.409 --> 00:07:29.030
we tried to provide the same things we
would do on Congress, but now from our

00:07:29.030 --> 00:07:39.409
remote offices. And one last thing that
doesn't… normally doesn't need to be said.

00:07:39.409 --> 00:07:48.060
But I think in this year and with this
different kind of event, I think it's

00:07:48.060 --> 00:07:55.099
necessary that the heaven as a
representative, mostly for people trying

00:07:55.099 --> 00:08:05.029
to help make this event awesome. And I
think it's time to say the things we do

00:08:05.029 --> 00:08:11.610
take for granted. And that is thank you
for all your help. Thank you for all the

00:08:11.610 --> 00:08:20.259
entities, all the teams, all the
participants that achieved the goal to

00:08:20.259 --> 00:08:27.289
bring our real Congress that many, many
entities missed this year into a new

00:08:27.289 --> 00:08:33.570
stage. We tried that online. It had its
ups and downs. But I still think it was an

00:08:33.570 --> 00:08:40.360
awesome adventure for everyone. And from
the Heaven team I can only say thank you

00:08:40.360 --> 00:08:47.870
and I hope to see you all again in the
future on a real event. Bye! And have a

00:08:47.870 --> 00:09:06.519
nice New Year.
lindworm: Hello, hello, back again. So we

00:09:06.519 --> 00:09:17.500
now are switching over to the Signal
Angels. Are the signal angels ready?

00:09:17.500 --> 00:09:24.260
Hello!
trilader: Yeah, hello, uhm, welcome to the

00:09:24.260 --> 00:09:30.350
infrastructure review for the Signal
Angels, I have prepared some stuff for

00:09:30.350 --> 00:09:36.180
you. This was for us… slides, please? This
was for us the first time running a fully

00:09:36.180 --> 00:09:49.579
remote Q&amp;A session set, I guess? We had
some experience with DiVOC and had gotten

00:09:49.579 --> 00:09:54.160
some help from there on how to do this,
but just to compare, our usual procedure

00:09:54.160 --> 00:09:59.260
is to have a signal angel in the room.
They collect the question on their laptop

00:09:59.260 --> 00:10:06.000
there and they communicate with the Herald
on stage and they have a microphone like

00:10:06.000 --> 00:10:09.509
I'm wearing a headset. But in there we
have a studio microphone and we speak

00:10:09.509 --> 00:10:17.500
questions into it. Yeah, but remotely we
really can't do that. Next slide. Because,

00:10:17.500 --> 00:10:22.730
well, it would be quite a lot of hassle
for everyone to set up good audio setups.

00:10:22.730 --> 00:10:29.550
So we needed a new remote procedure. So we
figured out that with a signal Angel and

00:10:29.550 --> 00:10:34.230
the Herald could communicate via
a pad and we could also collect the

00:10:34.230 --> 00:10:39.060
question in there. And the Herald will
read the question to the speaker and

00:10:39.060 --> 00:10:52.779
collect feedback and stuff. So we had 175.
No, 157 shifts, and sadly we couldn't fill

00:10:52.779 --> 00:11:02.980
five of them in the beginning because
there was not enough people already there.

00:11:02.980 --> 00:11:07.930
And also technically it was more than five
unfilled shifts because for some reasons

00:11:07.930 --> 00:11:16.800
there were DJ sets and other things that
aren't talks and also don't have Q&amp;A. We

00:11:16.800 --> 00:11:21.589
had 61 angels coordinated by four
supporters, so me and three other people,

00:11:21.589 --> 00:11:26.110
and we had a 60 additional angels that
in theory wanted to do signal angel work

00:11:26.110 --> 00:11:34.660
but didn't show up to the introduction
meeting. Next! For, as I've said for each

00:11:34.660 --> 00:11:40.290
session, each talk, we created a pad where
we put in the questions from IRC,

00:11:40.290 --> 00:11:47.240
Mastodon, and Twitter and. Well, we have a
bit more pads than talks we actually

00:11:47.240 --> 00:11:53.970
handled, and I have some statistics about
an estimated number of questions per talk.

00:11:53.970 --> 00:11:59.310
What we usually assume is that there's a
question per line, but some questions are

00:11:59.310 --> 00:12:03.180
really long and have to split over
multiple lines. There are some structured

00:12:03.180 --> 00:12:08.089
questions with headings and paragraphs
some heralds or signal angels removed

00:12:08.089 --> 00:12:12.730
questions after they were done. And also
there were some chat and other

00:12:12.730 --> 00:12:19.029
communication in there. So next slide, we
took a Python script, download all the pad

00:12:19.029 --> 00:12:23.220
contents, read them, counted the number of
lines, remove the size of the static

00:12:23.220 --> 00:12:36.699
header. And in the end we had 179 pads and
1,627 lines if we discount the static

00:12:36.699 --> 00:12:42.870
header of nine lines per pad. So that in
theory leads to about nine questions in

00:12:42.870 --> 00:12:47.610
quotation marks because it's not really
questions but lines. But it's an estimate,

00:12:47.610 --> 00:12:55.620
per talk. Thank you.
ysf: ... talk and what I've learned is

00:12:55.620 --> 00:13:03.470
never miss the introduction. So the next
in line are the line producers ha ha ha ha

00:13:03.470 --> 00:13:32.550
stb are you there?
stb: I am here, in fact, so <i>singing</i>. So

00:13:32.550 --> 00:13:39.170
the people a bit older might recognize
this melody badly sung by yours truly and

00:13:39.170 --> 00:13:46.029
other members of the line producers team,
and I'll get to why that is relevant to

00:13:46.029 --> 00:13:53.040
what we've been doing at this particular
event. So what does, what do line

00:13:53.040 --> 00:13:57.709
producers do? What does an,
Aufnahmeleitung actually perform? It's

00:13:57.709 --> 00:14:01.339
basically communication between everybody
who's involved in the production, the

00:14:01.339 --> 00:14:06.529
people behind the camera and also in front
of the camera. And so our work started

00:14:06.529 --> 00:14:14.009
really early, basically at the beginning
of November, taking on like prepping

00:14:14.009 --> 00:14:18.449
speakers in a technical setup and
rehearsing with them a little bit and then

00:14:18.449 --> 00:14:25.089
enabling the studios to allow them to
actually do the production coordination on

00:14:25.089 --> 00:14:29.490
an organizational side. The technical side
was handled by the VOC, and we'll get to

00:14:29.490 --> 00:14:36.910
hear about that in a minute. But getting
all these people synced up and working

00:14:36.910 --> 00:14:42.730
together well, that was quite a challenge.
And that took a lot of Mumbles with a lot

00:14:42.730 --> 00:14:50.820
of people in them. We only worked on the
two main channels. There's quite a few

00:14:50.820 --> 00:14:57.690
more channels that are run independently
of kind of the central organization. And

00:14:57.690 --> 00:15:02.410
again, we'll get to hear about the details
of that in a minute. And so we provided

00:15:02.410 --> 00:15:06.860
information. We tried to fill wiki pages
with relevant information for everybody

00:15:06.860 --> 00:15:15.440
involved. So that was our main task. So
what does that mean specifically, the

00:15:15.440 --> 00:15:24.649
production set up? We had 25 studios,
mainly in Germany, also one in

00:15:24.649 --> 00:15:32.170
Switzerland. These did produce recordings
ahead of time for some speakers, and many

00:15:32.170 --> 00:15:39.370
did live set ups for their own channels
and also for the two main channels. And

00:15:39.370 --> 00:15:43.850
I've listed everybody involved in the live
production here. And there were 19

00:15:43.850 --> 00:15:49.959
channels in total. So a lot of stuff
happening. 25 studios, 19 channels that

00:15:49.959 --> 00:15:54.620
broadcast content produced by these
studios. So that's kind of the Eurovision

00:15:54.620 --> 00:15:59.529
kind of thing, where you have different
studios producing content and trying to

00:15:59.529 --> 00:16:05.610
mix it all together. Again, the VOC took
care of the technical side of things very

00:16:05.610 --> 00:16:11.040
admirably, but getting everybody on the
same page to actually do this was not

00:16:11.040 --> 00:16:18.709
easy. For the talk program, we had over
350 talks in total, 53 in the main channels

00:16:18.709 --> 00:16:24.680
And so handling all that, making
sure everybody has the speaker information

00:16:24.680 --> 00:16:33.149
they need and all these organizational
stuff, that was a lot of work. So we

00:16:33.149 --> 00:16:38.300
didn't have a studio for the main
channels, the 25 studios or the nine, the

00:16:38.300 --> 00:16:43.709
live channels, the 12, they actually did
provide the production facilities for the

00:16:43.709 --> 00:16:49.259
speakers so we can look at the next slide.
There's a couple more numbers and of

00:16:49.259 --> 00:16:56.029
course, a couple pictures from us working
basically from today. We had 53 channel...

00:16:56.029 --> 00:17:05.010
53 talks in the main channel. 18 of them
were prerecorded and played out. We had 3

00:17:05.010 --> 00:17:10.020
where people were actually on location in
a studio and gave their talk from there.

00:17:10.020 --> 00:17:16.310
And we had 32 that were streamed live like
I am speaking to you now with various

00:17:16.310 --> 00:17:21.700
technical bits that again the VOC will go
into in a minute. And we did a lot of

00:17:21.700 --> 00:17:26.070
Q&amp;As, I don't have the numbers how many
talks actually had Q&amp;As, but most of them

00:17:26.070 --> 00:17:33.261
did, and those were always like. We had a
total of 63 speakers we did prepare, at

00:17:33.261 --> 00:17:38.750
least the live Q&amp;A session for and helped
them set up, we helped them record their

00:17:38.750 --> 00:17:44.670
talks if they wanted to prerecord them. So
we spent anywhere between one and two

00:17:44.670 --> 00:17:49.550
hours with every speaker to make sure they
would appear correctly and in good quality

00:17:49.550 --> 00:17:55.900
on the screen. And then during the four
days, we, of course, helped coordinate

00:17:55.900 --> 00:18:00.350
between the master control room and the
twelve live studios to make sure that the

00:18:00.350 --> 00:18:03.940
speakers were where they were supposed to
be and any technical glitches could be

00:18:03.940 --> 00:18:09.390
worked out and decide on the spot. If, for
example, the line producers made a mistake

00:18:09.390 --> 00:18:14.718
and a talk couldn't happen as we had
planned because we forgot something. So we

00:18:14.718 --> 00:18:19.697
rescheduled and found a new spot for the
speakers. So apologies again for that. And

00:18:19.697 --> 00:18:24.560
thank you for your understanding and
helping us bring you on screen on day two

00:18:24.560 --> 00:18:31.473
and not day one. But I'm very glad that
that we could work that out. And that's

00:18:31.473 --> 00:18:40.130
pretty much it from the line producers, I
think. Next up is the VOC.

00:18:40.130 --> 00:18:44.923
ysf: Thank you stb. Yes, you're right, the
next are the VOC and kunsi and

00:18:44.923 --> 00:18:53.870
JW2CAlex are waiting for us.
Franzi: ... is Franzi from the VOC. 2020

00:18:53.870 --> 00:19:05.150
was the year... Hm? Hi, this is Franzi
from the... from VOC. 2020 was the year of

00:19:05.150 --> 00:19:12.420
distributed conferences. We had 2 DiVOCs
and the FrOSCon to learn how we are going

00:19:12.420 --> 00:19:17.380
to produce remote talks. We learned a lot
of stuff on organization, Big Blue Button

00:19:17.380 --> 00:19:23.760
and Jitsi recording. We had a lot of other
events which was just streaming like

00:19:23.760 --> 00:19:33.020
business as usual. So for rC3, we extended
the streaming CDN with two new locations,

00:19:33.020 --> 00:19:41.740
now 7 in total, with a total bandwidth of
about 80 gigabits per second. We have two

00:19:41.740 --> 00:19:51.380
new mirrors for media.ccc.de and are now
also distributing the front end. We got

00:19:51.380 --> 00:19:57.860
two new transcoder machines, Erfas and
Enhanced cir setup we now have 10 Erfas

00:19:57.860 --> 00:20:07.152
with own productions on media.ccc.de. So
the question is, will it scale? On the

00:20:07.152 --> 00:20:10.070
next slide...
Alex: Yeah, next slide.

00:20:10.070 --> 00:20:21.435
Franzi: ... we will see that it did
scale. We did produce content for 25

00:20:21.435 --> 00:20:28.560
studios and 19 channels, so we got lots of
lots of recordings which will be published

00:20:28.560 --> 00:20:36.190
on media.ccc.de in the next days and
weeks. Some have already been published,

00:20:36.190 --> 00:20:43.180
so there's a lot of content for you to
watch. And now Alex will tell us something

00:20:43.180 --> 00:20:47.920
about the technical part.
Alex: My name is Alex, Pronouns it/its. I

00:20:47.920 --> 00:20:52.300
will now tell you the technical part
first, but more of the organization. I was

00:20:52.300 --> 00:20:56.907
between the VOC and the line producing
team. And now a bit how it worked. So we

00:20:56.907 --> 00:21:02.290
had those two main channels, rc-one and
rc-two. Those channels have been produced

00:21:02.290 --> 00:21:07.740
by the various studios distributed around
the whole country. And those streams,

00:21:07.740 --> 00:21:12.210
this is now the upper path in the picture,
went to our ingest relay, to the FEM, to

00:21:12.210 --> 00:21:15.990
the master control room. In Ilmenau there
were a team of people adding the

00:21:15.990 --> 00:21:20.830
translations, making the mix, making the
mixdown, making records and then

00:21:20.830 --> 00:21:25.977
publishing it back to the streaming
relays. All the other studios produced to

00:21:25.977 --> 00:21:30.532
channels. Those channels took the also the
signals from different studios, make a

00:21:30.532 --> 00:21:36.381
mixdown, etc. publish to our CDN and
relays and we publish to the studio

00:21:36.381 --> 00:21:40.901
channels. As you can see, this is not the
typical setup we had in the last year in

00:21:40.901 --> 00:21:47.148
the presence. So, our next slide, we can
see where this leads: Lots of

00:21:47.148 --> 00:21:53.080
communication. We had the line producing
team, we had some production in Ilmenau

00:21:53.080 --> 00:21:57.060
that has to be coordinated. We have the
studios, we have the local studio helping

00:21:57.060 --> 00:22:02.120
Angels. We have some Mumbles there, some
RocketChat here, some CDN people some web

00:22:02.120 --> 00:22:07.032
where something happens. We have some
documentation that should be. And then we

00:22:07.032 --> 00:22:12.640
started to plot down the communication
paths. Next slide, please. If you plotted

00:22:12.640 --> 00:22:16.510
all of them, it really looks like the
world, but this is actually the world, but

00:22:16.510 --> 00:22:20.490
sometimes it feels like they're just
getting lost in different paths. Who you

00:22:20.490 --> 00:22:25.107
have to ask, who do you have to call?
Where are you? What's the shortest path to

00:22:25.107 --> 00:22:33.120
communicate? But let's have a look at the
studios. First going to ChaosWest.

00:22:33.120 --> 00:22:39.610
Franzi: Yes, on the next slide, you will
see the studio set up at ChaosWest TV. So

00:22:39.610 --> 00:22:46.782
thank you, ChaosWest for producing your
channel.

00:22:46.782 --> 00:22:51.130
Alex: At the next slide, you see the
Wikipaka television and fernseh-streamen

00:22:51.130 --> 00:22:54.922
(WTF) who have the internal motto:
"Absolut nicht sendefähig - chaos of

00:22:54.922 --> 00:23:00.531
recording". But even then, at some
studios, you look more like studios, so

00:23:00.531 --> 00:23:08.330
this time at the next slide at the hacc.
Franzi: Yeah, at hacc, you will also see

00:23:08.330 --> 00:23:14.920
some of the bloopers we had to deal with.
So, for example, here you can see there

00:23:14.920 --> 00:23:25.270
was a cat in the camera view, so, yeah.
And Alex, tell us about the open

00:23:25.270 --> 00:23:28.390
infrastructure orbit.
Alex: The open infrastructure orbit

00:23:28.390 --> 00:23:31.720
showed. In this picture, you can see it's
really hard to see how you can make a

00:23:31.720 --> 00:23:34.930
studio look really nice, even if you're
alone, feeling a bit comfier, more

00:23:34.930 --> 00:23:39.988
hackish. But you have also those normal
productions as in the next slide. The

00:23:39.988 --> 00:23:45.679
Chaosstudio Hamburg
Franzi: Yeah, at Chaosstudio Hamburg, we

00:23:45.679 --> 00:23:52.555
had two regular work cases like, you know,
from all the other conferences, and they

00:23:52.555 --> 00:24:02.400
were producing, onsite in a regular studio
set up. And last but not least, we got

00:24:02.400 --> 00:24:08.220
some impressions from ChaosZone TV.
Alex: As you can see here, also quite

00:24:08.220 --> 00:24:12.680
regular studio setup, quite regular. No.
There was some Corona virus ongoing, and

00:24:12.680 --> 00:24:16.530
this is we had a lot of distancing,
wearing mask and all the stuff that

00:24:16.530 --> 00:24:22.580
everyone is safe but c3yellow (c3gelb)
will tell you some facts about it. But

00:24:22.580 --> 00:24:28.670
let's look at the nice things. For
example, the minor issue: On the second

00:24:28.670 --> 00:24:33.864
day, we were sitting there looking at our
nice Grafana. Oh, we got a lot of more

00:24:33.864 --> 00:24:38.580
connections. The server load's increasing.
The first question was: Have we enabled

00:24:38.580 --> 00:24:44.570
our cache?". We don't know. But the number
of connections is growing that people are

00:24:44.570 --> 00:24:50.320
watching our streams, the interest goes
up. And we were, well, at least the people

00:24:50.320 --> 00:24:56.630
are watching the streams. If there is a
website, who cares, the interest works.

00:24:56.630 --> 00:25:01.840
But then we suddenly get the relations.
Well, something did not really scale that

00:25:01.840 --> 00:25:09.820
good. And then using the next slide, this
view. This switched pretty fast from after

00:25:09.820 --> 00:25:14.595
looking at this traffic graph. "Well,
that's interesting" into "Well, we should

00:25:14.595 --> 00:25:18.951
investigate". We get thousands of messages
on Twitter DMs. We got thousands of

00:25:18.951 --> 00:25:23.470
messages in RocketChat, IRC, and suddenly
we had a lot of connections to handle; a

00:25:23.470 --> 00:25:27.716
lot of inquiries to handle, a lot of phone
calls, etc. to handle. And we have to

00:25:27.716 --> 00:25:31.110
prioritize for us the hardware then the
communication, because otherwise the

00:25:31.110 --> 00:25:39.280
information won't stop. On the next slide
you can see what our minor issue was. So

00:25:39.280 --> 00:25:43.150
at first, we get a lot of connections to
our streaming web pages, then to load

00:25:43.150 --> 00:25:48.755
balancers, and finally to our DNS servers.
A lot of them were quite malformed. It

00:25:48.755 --> 00:25:53.906
looked like a storm. But the more
important thing we had to deal was all

00:25:53.906 --> 00:25:59.694
those passive aggressive messages from,
from different persons who said: "Well,

00:25:59.694 --> 00:26:04.050
you can't even handle streaming. What are
you doing here?" And we worked together

00:26:04.050 --> 00:26:08.280
with the c3infra team, thanks for that, how 
to scale and decentralize a bit more just to

00:26:08.280 --> 00:26:13.920
provide the people the connection power
they need. So I think in the last years,

00:26:13.920 --> 00:26:18.870
we don't need to use more bandwith. We
showed we can provide even more bandwith

00:26:18.870 --> 00:26:27.170
if we need it. And then, noting everything
down…

00:26:27.170 --> 00:26:35.990
Franzi: So is it time to shut everything
down? No, we won't shut everything down.

00:26:35.990 --> 00:26:42.910
The studios can keep their endpoints, can
continue to stream on their endpoints as

00:26:42.910 --> 00:26:48.030
they wish. We want to keep in touch with
you and the studios, produce content with

00:26:48.030 --> 00:26:56.520
you, improve our software stack, improve
other things like the ISDN, the Internet

00:26:56.520 --> 00:27:05.540
Streaming Digital Node, the project for
small camera recording setups for sending

00:27:05.540 --> 00:27:12.872
to speakers needs developers for the
software. Also, KEVIN needs developers and

00:27:12.872 --> 00:27:20.530
testers. What's KEVIN? Oh, we have
prepared another slide or the next slide.

00:27:20.530 --> 00:27:28.450
KEVIN is short for Killer Experimental
Video Internet Noise, because we initially

00:27:28.450 --> 00:27:36.260
wanted to use OBS.Ninja, but there are a
couple of licensing issues. There is not

00:27:36.260 --> 00:27:45.190
everything in OBS.Ninja is open source
like we wanted, so we decided to code our

00:27:45.190 --> 00:27:52.910
own OBS.Ninja-style software. So if you
are interested in doing so, please get

00:27:52.910 --> 00:28:01.710
into contact with us or visit the wiki. So
that's all from the VOC. And we are now

00:28:01.710 --> 00:28:10.960
heading over to c3lingo.
ysf: Exactly. c3lingo oskar should be

00:28:10.960 --> 00:28:23.130
waiting Studio 2, aren't you?

00:28:23.130 --> 00:28:28.190
oskar: Yeah, hallo. Hi, yeah, I'm oskar

00:28:28.190 --> 00:28:41.210
from c3lingo. We will jump straight into
the stats on our slides. As you can see

00:28:41.210 --> 00:28:47.920
here, we translated 138 talks this time,
as you can see, it's also way less

00:28:47.920 --> 00:28:54.430
languages than in the other chaos events
that we had since our second languages

00:28:54.430 --> 00:28:57.950
team that does everything that is not
English and German was only five people

00:28:57.950 --> 00:29:02.880
strong this time. So we only managed to do
five talks into French and three talks

00:29:02.880 --> 00:29:12.730
into Brazilian Portuguese. And then on the
next slide… We are looking at our coverage

00:29:12.730 --> 00:29:17.320
for the talks and we can see that on the
main talks we managed to cover all talks

00:29:17.320 --> 00:29:22.480
that were happening from English to German
and German to English, depending on what

00:29:22.480 --> 00:29:30.350
the source language was. And then, on the
other languages track, we only managed to

00:29:30.350 --> 00:29:35.151
do 15 percent of the talks from the main
channels. And then on the further

00:29:35.151 --> 00:29:39.240
channels, which is a couple of others that
also were provided to us in the

00:29:39.240 --> 00:29:46.410
translation team, we managed to do 68% of
the talks, but none of them were

00:29:46.410 --> 00:29:52.530
translated into other languages than
English and German. On the next slide,

00:29:52.530 --> 00:29:58.710
some global stats. We have 36
interpreters, which in total managed to

00:29:58.710 --> 00:30:06.454
translate 106 hours and 7 minutes of talks
into another language simultaneously. And

00:30:06.454 --> 00:30:11.480
the maximum number of hours one person did
was 16 hours and the minimum number of

00:30:11.480 --> 00:30:16.770
hours, the average number of hours people
did was around 3 hours of translation

00:30:16.770 --> 00:30:26.970
across the entire event. All right. Then I
also have some anecdotes to tell and some

00:30:26.970 --> 00:30:31.190
some mentions I want to do. We had two new
interpreters that we want to say "hi" to,

00:30:31.190 --> 00:30:35.980
and we had a couple of issues with the
digital thing that didn't have before with

00:30:35.980 --> 00:30:41.100
regular events where people were present.
For example, the issue of sometimes when

00:30:41.100 --> 00:30:46.160
two people are translating one person's
starts interpreting something on wrong

00:30:46.160 --> 00:30:50.230
stream. Maybe they were watching the wrong
one. And then the partner just thinks they

00:30:50.230 --> 00:30:54.230
have more delay or something. Or, for
example, a partner having a smaller delay

00:30:54.230 --> 00:30:58.530
and then thinking the partner can suddenly
read minds because they can translate

00:30:58.530 --> 00:31:02.506
faster than the other person is actually
seeing the stream. Those are issues that

00:31:02.506 --> 00:31:09.180
we usually didn't have with the regular
stream, but only with the regular events,

00:31:09.180 --> 00:31:17.620
not with remote events. And yeah, some
hurdles to overcome. Another thing was,

00:31:17.620 --> 00:31:24.390
for example, when on the r3s stage, the
audio cut out sometimes for us and but

00:31:24.390 --> 00:31:27.851
because one of our translators had also
already translated the talk twice, at

00:31:27.851 --> 00:31:33.160
least partially to because and it was
already canceled after those, they

00:31:33.160 --> 00:31:37.630
basically knew most of the content and
could basically do a Powerpoint Karaoke

00:31:37.630 --> 00:31:43.360
translation and was able to do most of the
talk just from the slides without any

00:31:43.360 --> 00:31:54.370
audio. Yeah, and then there also was...
The last thing I want to say is actually I

00:31:54.370 --> 00:31:58.990
wanted to say, give a big shout out to the
two of our team members that weren't able

00:31:58.990 --> 00:32:02.380
to interpret with us this time because
they put their heart and soul into this

00:32:02.380 --> 00:32:06.750
event happening. And that's stb and katti,
and that's basically everything from

00:32:06.750 --> 00:32:16.492
c3lingo. Thanks.

00:32:16.492 --> 00:32:29.200
ysf: <i>muted</i>

00:32:29.200 --> 00:32:37.275
Hello, c3subtitles is it now. td will show
the right text to his slides you already

00:32:37.275 --> 00:32:48.353
saw a minute ago.
td: OK. OK, hi, so I'm td from the

00:32:48.353 --> 00:32:54.590
c3subtitles team. And next slide, please.
So just to quickly let you know how we get

00:32:54.590 --> 00:32:59.690
from the recorded talks to the released
subtitles. Well we take the recording

00:32:59.690 --> 00:33:04.756
videos and apply speech recognition
software to get a raw transcript. And then

00:33:04.756 --> 00:33:07.860
Angels work on that transcript to correct
all the mistakes that the speech

00:33:07.860 --> 00:33:12.620
recognition software makes. And we again
apply some autotiming magic to to get some

00:33:12.620 --> 00:33:18.070
raw subtitles. And then again Angels do
quality control on these tracks to get

00:33:18.070 --> 00:33:24.340
released subtitles. Next slide, please. So
as you can see, we have various subtitle

00:33:24.340 --> 00:33:30.360
tracks in different stages of completion.
And these are seconds of material that we

00:33:30.360 --> 00:33:35.101
have can see all the numbers are going up
and to the right as they should be. So

00:33:35.101 --> 00:33:42.280
next slide, please. In total, we had 68
distinct angels that worked 4 shifts on

00:33:42.280 --> 00:33:47.290
average. 83 percent of our angels returned
for a second shift. 10 percent of our

00:33:47.290 --> 00:33:55.110
angels worked 12 or more shifts. And in
sum we had 382 hours of angel work for 47

00:33:55.110 --> 00:34:01.267
hours of material. So far we've had two
releases for rc3 and hopefully more yet to

00:34:01.267 --> 00:34:05.590
come, and 37 releases for all the
congresses, mostly on the first few days

00:34:05.590 --> 00:34:10.590
where we didn't have many recordings. We
have 41 hours still in the transcribing

00:34:10.590 --> 00:34:16.530
stage of material, 26 hours of material in
the timing stage and 51 hours material in

00:34:16.530 --> 00:34:21.509
the quality control stage. So there's
still lots of work to be done. Next slide,

00:34:21.509 --> 00:34:26.206
please. When you have transcripts, you can
do fun stuff with them. For example, you

00:34:26.206 --> 00:34:31.679
can see that important to people in this
talk are "people". We are working on other

00:34:31.679 --> 00:34:37.217
cool features that are yet to come. Stay
tuned for that. Next slide, please. So to

00:34:37.217 --> 00:34:42.250
keep track of all these tasks, we've been
using a state-of-the-art high-performance

00:34:42.250 --> 00:34:46.737
lock-free NoSQL columnar data store,
a.k.a. a kanban board in the previous

00:34:46.737 --> 00:34:50.960
years. And because we don't have any
windows in the CCL building anymore, we

00:34:50.960 --> 00:34:55.799
had to virtualize that. So we're using
kanban software now. At this point, I

00:34:55.799 --> 00:34:59.861
would like to thank all our hard-working
angels for the work. And next slide

00:34:59.861 --> 00:35:04.959
please. If you're feeling bored between
congresses then you can work on some

00:35:04.959 --> 00:35:08.940
transcripts. Just go to c3subtitles.de. If
you're interested in our work, follow us

00:35:08.940 --> 00:35:14.972
on Twitter. And there's also a link to the
release subtitles here. So that's all.

00:35:14.972 --> 00:35:23.379
Thank you.
ysf: Thank you, td. And before we go into

00:35:23.379 --> 00:35:28.990
the POC, where Drake is waiting, I'm sure
everyone is asking why are those guys

00:35:28.990 --> 00:35:37.869
saying "next slide"? So wait. In
the end, we have the infrastructure review

00:35:37.869 --> 00:35:44.319
of the infrastructure review meeting going
on. So be patient. Now, Drake, are you

00:35:44.319 --> 00:35:56.399
ready in Studio 1?

00:35:56.399 --> 00:36:00.935
Drake: OK. Hello, I'm Drake from the Phone
Operations Center, and

00:36:00.935 --> 00:36:04.480
I like to present to you our 
numbers and maybe some

00:36:04.480 --> 00:36:12.293
anecdotes at the end of our part. So
please switch to the next slide. Let's get

00:36:12.293 --> 00:36:21.028
into the numbers first. So first off,
first off, you registered about 1950 ...

00:36:21.028 --> 00:36:29.154
5195 sip extensions, which is about 500
more than you registered on the last

00:36:29.154 --> 00:36:38.325
congress. Also, you did about 21 000
calls, a little bit less than on the last

00:36:38.325 --> 00:36:44.367
congress. But, yeah, we are still quite
proud of what you have used our system

00:36:44.367 --> 00:36:50.440
with. And yeah, it ran quite stable. And
as you may notice on the bottom, we also

00:36:50.440 --> 00:36:56.957
had about 23 DECT antennas at the congress
or at this event. So please switch to the

00:36:56.957 --> 00:37:06.800
next slide. And this is our new feature,
it's called the... next slide ..., it

00:37:06.800 --> 00:37:11.750
is called the eventphone decentralized
DECT infrastructure, which we especially

00:37:11.750 --> 00:37:18.839
prepared for this event, the EPDDI. So we
had about 23 RFPs online throughout

00:37:18.839 --> 00:37:29.510
Germany with 68 DECT telephones of which
is up to it. But it's not only the the

00:37:29.510 --> 00:37:36.140
German part that we covered. We actually
had one mobile station walking out through

00:37:36.140 --> 00:37:41.900
Austria, through Passau, I think. So
indeed we had an European Eventphone DECT

00:37:41.900 --> 00:37:52.119
decentralized infrastructure. Next slide
please. We also have some anecdotes, so

00:37:52.119 --> 00:37:57.170
maybe some of you have noticed that we had
a public phone, a working public phone in

00:37:57.170 --> 00:38:03.759
the RC World where you could call other
people on the SIP telephone system and

00:38:03.759 --> 00:38:10.200
also other people started to play with our
system. And I think about yesterday

00:38:10.200 --> 00:38:18.759
someone started to introduce c3fire so you
could actually control a flame thrower

00:38:18.759 --> 00:38:25.111
through our telephone system. And I like
to present here a video. Next slide

00:38:25.111 --> 00:38:34.650
please. Maybe you can play it. I have
quite a delay in waiting for the video to

00:38:34.650 --> 00:38:42.670
play. So what you can see here is the
c3fire system actually controlled by a

00:38:42.670 --> 00:38:54.090
DECT telephone somewhere in Germany. So
next slide please. We also provided you

00:38:54.090 --> 00:39:04.220
with SSTV servers via the phone
number 229, where you could receive some

00:39:04.220 --> 00:39:10.220
pictures from event phone, like a postcard
basically. So basically you could call the

00:39:10.220 --> 00:39:18.961
number and receive a picture or some other
pictures, some more pictures. And next

00:39:18.961 --> 00:39:28.325
slide please. Yeah basically, that's all
from the Eventphone and with that we say

00:39:28.325 --> 00:39:34.420
thank you all for the nice and awesome
event and yeah, bye from the first

00:39:34.420 --> 00:39:43.769
certified assembly POC. Bye.
ysf: Thank you, POC, and hello GSM Lynxes

00:39:43.769 --> 00:39:51.480
is waiting for us.

00:39:51.480 --> 00:39:55.890
lynxes: Yeah, hallo, I'm lynxes, I'm from

00:39:55.890 --> 00:40:02.830
the GSM team. This year was quite
different as you can imagine. However,

00:40:02.830 --> 00:40:11.180
next slide please. So but we managed to
get a small network running and also a

00:40:11.180 --> 00:40:19.619
couple of SIM cards registering, so where are
we now. So next slide please. As you can

00:40:19.619 --> 00:40:24.420
see, we are just there in the red dot.
There's not even a single line for our

00:40:24.420 --> 00:40:32.007
five extensions but we managed 130 calls
over five extensions. And next slide

00:40:32.007 --> 00:40:44.289
please. So we got, so we got five
extensions registered with four SIM cards

00:40:44.289 --> 00:40:51.599
and three locations with mixed
technologies also two users so far sadly.

00:40:51.599 --> 00:40:57.880
And one network with more or less zero
problems. And so let's take a look on the

00:40:57.880 --> 00:41:06.230
coverage. So next slide please. So we are
quite lucky that we managed to get an

00:41:06.230 --> 00:41:14.579
international network running. So we got
two base stations in Berlin. One in the

00:41:14.579 --> 00:41:19.980
hackerspace AfRA, another one north of
Berlin. And yeah one of our members is

00:41:19.980 --> 00:41:33.200
currently in Mexico. And he's providing
the remote chaos networks there. Yes, so

00:41:33.200 --> 00:41:43.359
that's basically our network. So before we
going to the next slide, we have what we

00:41:43.359 --> 00:41:51.490
have done so far is, we are just two
people instead of 10 to 20 and had some

00:41:51.490 --> 00:41:59.501
fun with improving our network and
preparing for the next congress. And next

00:41:59.501 --> 00:42:05.813
slide please. And yeah, now I'm closing
with the EDGE computing. We improved our

00:42:05.813 --> 00:42:15.390
EDGE capabilities and yeah, I wish you a
hopefully better year and yeah maybe see

00:42:15.390 --> 00:42:22.040
you next year remote or in person. Have
fun.

00:42:22.040 --> 00:42:30.793
ysf: Thanks and I give a hand to Iindworm
for doing the "slide DJ" all the time, and

00:42:30.793 --> 00:42:37.020
he now switch to the Haecksen who are
next and they bring an image and melzai is

00:42:37.020 --> 00:42:46.850
waiting for us in Studio 3.

00:42:46.850 --> 00:42:49.720
melzai: Hello, what's phones without
people?

00:42:49.720 --> 00:42:53.150
So I'm giving now an introduction
over here. How many people we needed to

00:42:53.150 --> 00:42:58.630
run the whole Haecksen assembly. We had
around 20 organizing haecksen and we had

00:42:58.630 --> 00:43:03.660
around 20 speakers in our events. And we
had in total around 40 events, but I'm

00:43:03.660 --> 00:43:09.545
pretty sure that I even don`t know all of
these. As you realize, the world is pretty

00:43:09.545 --> 00:43:14.999
large. So we needed around seven million
pixels to display the whole Haecksen

00:43:14.999 --> 00:43:22.785
world. And that needed around 400 commits
at our github corner of the internet.

00:43:22.785 --> 00:43:28.680
Around 130 people receive the fireplace
badge in our case. And around 100 people

00:43:28.680 --> 00:43:35.806
tested our swimming pool and received that
badge. So great a year for non ???. Also

00:43:35.806 --> 00:43:42.930
around 49 people showed some very deep
dedication and checked on all memorials at

00:43:42.930 --> 00:43:47.329
our Haecksen assembly. Congratulations for
that. There were quite a many of these

00:43:47.329 --> 00:43:53.499
ones. Our events are run on our BigBlueButton
from the Congress and so we had

00:43:53.499 --> 00:44:00.349
starting from day 0 no lags and we're able
to host up to 133 people in one session.

00:44:00.349 --> 00:44:04.748
And that was quite stable. We also
introduced four new members around 13 new

00:44:04.748 --> 00:44:10.579
Haecksen joinded just for the Congress.
And we increased about to the size of 440

00:44:10.579 --> 00:44:16.650
Haecksen overall. Also somewhat, we got new
Twitter accounts supporting us, so we have

00:44:16.650 --> 00:44:22.411
added over 200 more Twitter accounts. And
so, you know, our messages are getting

00:44:22.411 --> 00:44:28.202
heard. But besides the ritual, we also did
some quite physical things. First of all,

00:44:28.202 --> 00:44:32.990
we distributed over 50 physical goodie
bags to the people with microcontrollers

00:44:32.990 --> 00:44:38.637
and self-sewed masks in it, as you can see
on the picture. And also sadly, we shopped

00:44:38.637 --> 00:44:44.003
so many rC3 Haecksen-themed trunks that
they are now out of stock. But they will

00:44:44.003 --> 00:44:53.770
be back in January. Thank you.
ysf: No, thank you. And I'm going to send

00:44:53.770 --> 00:45:00.200
thanks to the Choaspatinnen…
Chaospat*innen… who are waiting in Studio

00:45:00.200 --> 00:45:11.010
One.
Mike: Hi, all this is Mike from the

00:45:11.010 --> 00:45:15.740
Chaospat*innen team. We've been welcoming
new attendees and underrepresented

00:45:15.740 --> 00:45:20.730
minorities to the chaos community for over
eight years. We match up our mentees with

00:45:20.730 --> 00:45:25.400
experienced chaos mentors. These mentors
help their mentees navigate our world of

00:45:25.400 --> 00:45:30.250
chaos events. DiVOC was our first remote
event and it was a good proof of concept

00:45:30.250 --> 00:45:37.559
for rc3. This year, we had 65 amazing
mentees and mentors, two in-world

00:45:37.559 --> 00:45:43.249
mentee/mentor matchup sessions, one great
assembly event hosted by two of our new

00:45:43.249 --> 00:45:49.640
mentees, and a wonderful world map
assembly built with more than 1337

00:45:49.640 --> 00:45:58.490
kilograms of multicolor pixels. Next
slide, please. And here's a small part of

00:45:58.490 --> 00:46:03.519
our assembly with our signature propeller
hat tables. And thank you to the amazing

00:46:03.519 --> 00:46:09.091
Chaospat*innen team: fragilant, jali,
azriel and lilafish. And to our great

00:46:09.091 --> 00:46:13.730
mentees and mentors. We're looking forward
to meeting all of the new mentees at the

00:46:13.730 --> 00:46:26.340
next chaos event.

00:46:26.340 --> 00:46:33.210
lindworm: Yeah, I think that was my call.

00:46:33.210 --> 00:46:49.860
So next up, we'll have the, let me see,
the c3adventure! Are you ready?

00:46:49.860 --> 00:46:53.569
Roang: Hello, my name is Roang
Mewp: and I'm Mewp

00:46:53.569 --> 00:46:59.380
Roang: and we will talk about the
c3adventure, the 2D world, and what we did

00:46:59.380 --> 00:47:11.480
to bring it all online. Next slide please.
OK, so when we started out, we looked into

00:47:11.480 --> 00:47:20.374
how we could bring a Congress-like
adventure to the remote experience. And on

00:47:20.374 --> 00:47:29.680
October we started with the development
and we had some trouble in that we had

00:47:29.680 --> 00:47:35.819
multiple upstream merges that gave us some
problems. And also due to just Congress

00:47:35.819 --> 00:47:40.769
being Congress, or remote experience being
a remote experience, we needed to

00:47:40.769 --> 00:47:49.044
introduce features a bit late or add
features on the first day. So auth was

00:47:49.044 --> 00:47:57.819
merged just 4:40 AM in the first day. And
on the second day, we finally fixed the

00:47:57.819 --> 00:48:03.630
instance jumps – you know, when you walk
from one map to the next – we had some

00:48:03.630 --> 00:48:08.279
problems there. But on the second day it
all went up. And I hope you have all

00:48:08.279 --> 00:48:14.809
enjoyed the badges that have finally been
updated and brought into the world today.

00:48:14.809 --> 00:48:23.059
What does that all mean? Since we started
implementing, there have been 400 git

00:48:23.059 --> 00:48:28.782
commits in our repository all-in-all,
including the upstream merges. But I think

00:48:28.782 --> 00:48:35.920
the more interesting stuff is what has
been done since the whole thing went live.

00:48:35.920 --> 00:48:42.140
We had 200 additional commits, fixing
stuff and making the experience better for

00:48:42.140 --> 00:48:52.339
you. Next slide. In order to bring this
all online, we not only had to think about

00:48:52.339 --> 00:48:57.289
the product itself, not only think about
the world itself, but we also had to think

00:48:57.289 --> 00:49:03.395
about the deployment. The first commit on
the deployer, it's a background service

00:49:03.395 --> 00:49:10.459
that brings the experience to you, has
been done on 26th of November. We started

00:49:10.459 --> 00:49:16.279
the first instance, the first clone of the
work adventure through this deployer on

00:49:16.279 --> 00:49:23.163
8th of December and a couple of days
beforehand, I was getting a bit swamped. I

00:49:23.163 --> 00:49:26.863
couldn't do all of the work anymore,
because I had to coordinate both of the

00:49:26.863 --> 00:49:32.280
projects. And so my colleague took over
for me, and helped me out a lot. So I'll

00:49:32.280 --> 00:49:38.609
give over to him to explain what he did.
Mewp: Yeah. So imagine that on Day -5 I

00:49:38.609 --> 00:49:46.581
get a message from a friend that, "Hey,
help is needed!" So I say, "OK, let's do

00:49:46.581 --> 00:49:55.950
it." And Roang tells me that, "OK, so we
can spawn a instance and to scale it

00:49:55.950 --> 00:50:03.589
somehow and do that." And I spawned the
deployer and my music stops. I streamed

00:50:03.589 --> 00:50:08.920
music from the internet, and I wondered
why did it stop? And I have noticed that,

00:50:08.920 --> 00:50:16.150
oh, there are a lot of logs now. Like, a
lot. And I have finally Day -4 noticed

00:50:16.150 --> 00:50:26.507
that the deployer was spawning copies of
itself each few seconds in the log. So

00:50:26.507 --> 00:50:32.762
that was the state back then. Since Day -4
until Day 1, we have basically written the

00:50:32.762 --> 00:50:44.793
thing. And that's, well… Day 1 we were
ready. Well, almost ready. I mean, we have

00:50:44.793 --> 00:50:51.319
like 14 instances deployed. And I forgot
to mention that, when we were about to

00:50:51.319 --> 00:51:00.450
deploy 200 ones at once, it wouldn't work
because all of the things would time out.

00:51:00.450 --> 00:51:09.210
So we patched things quickly, and 13
o'clock we had our first deployment. This

00:51:09.210 --> 00:51:16.600
worked, and everything was fine, and…
wait… Why is everybody on one instance?

00:51:16.600 --> 00:51:24.259
So, it turns out that we had a bug, not in
the deployer, in the app that would move

00:51:24.259 --> 00:51:31.470
you from the lobby to the lobby on a
different map. So during the first day, we

00:51:31.470 --> 00:51:36.319
have we've had a lot of issues of people
not seeing each other because they were on

00:51:36.319 --> 00:51:45.329
different instances of the lobby. So we
are working hard, and… next slide, please,

00:51:45.329 --> 00:51:55.630
so we can see that… we are working hard to
reconfigure that to bring you together in

00:51:55.630 --> 00:52:01.671
the assembly. I think we have succeeded.
You can see the population graph on this

00:52:01.671 --> 00:52:09.869
slide. The first day was our almost most
popular one. And the next day it would

00:52:09.869 --> 00:52:23.600
seem, that's OK, not as popular, but we
have hit the peak of 1600 users that day.

00:52:23.600 --> 00:52:30.309
What else about this? The most popular
instance was lobby, of course. The second

00:52:30.309 --> 00:52:37.930
most popular instance was hardware hacking
area for a while. Then the third, I think.

00:52:37.930 --> 00:52:51.279
Next slide please. We have counted, well,
first of all, we've had in total about 205

00:52:51.279 --> 00:52:57.159
assemblies. The number was increase day-
by-day, because people, through the whole

00:52:57.159 --> 00:53:04.880
congress, they were working on their maps.
For a while, CERT had over a thousand maps

00:53:04.880 --> 00:53:11.328
active in their assembly. Which led to the
map server crashing. Some of you might

00:53:11.328 --> 00:53:19.072
have noticed that. It stopped working
quite a few times during Day 3. And they

00:53:19.072 --> 00:53:29.200
have reduced the number of maps to 255.
And that was fine. At the end of Day 3, I

00:53:29.200 --> 00:53:41.779
have counted for about 628 maps, and this
is less than the, if, than was available

00:53:41.779 --> 00:53:49.799
in reality, because it was the middle of
the night (as always), and it was it

00:53:49.799 --> 00:53:56.044
wasn't trivial to count them. But in the
maps I have found, we have found over two

00:53:56.044 --> 00:54:01.800
million used tiles. So that's something
you can really explore. I wish I could

00:54:01.800 --> 00:54:12.150
have, but deploying this was also fun.
Next slide, please. And what… Yeah?

00:54:12.150 --> 00:54:17.610
Roang: Just a quick interject. I really
want to thank everyone that has put work

00:54:17.610 --> 00:54:23.218
into their maps and made this whole
experience work. We, we provided the

00:54:23.218 --> 00:54:28.599
infrastructure, but you provided the fun.
And so I really want to thank everyone.

00:54:28.599 --> 00:54:34.369
Mewp: Yeah, the more things happen on the
infrastructure, the more fun we have. We

00:54:34.369 --> 00:54:42.819
especially don't like to sleep. So we
didn't. I basically exchanged with Roang

00:54:42.819 --> 00:54:50.422
the way that I slept five hours and during
the night and he slept five hours in the

00:54:50.422 --> 00:54:57.441
day. And the rest of the time, we were up.
The record, though, is incorrect. Roang is

00:54:57.441 --> 00:55:05.350
now 30 hours up straight, because the
budgets were too important to bring to you

00:55:05.350 --> 00:55:14.400
to go to sleep. The thing you see on this
graph is undeployed instances. We were

00:55:14.400 --> 00:55:19.522
redeploying things constantly. Usually in
the form of redeploying half of the

00:55:19.522 --> 00:55:24.200
infrastructure at any given time. The way
it was developed, you wouldn't have

00:55:24.200 --> 00:55:29.150
noticed that. You wouldn't be kicked off
your instances, but for a brief period of

00:55:29.150 --> 00:55:40.349
time you wouldn't be able to enter any
one. But… Next slide. I have been joking

00:55:40.349 --> 00:55:46.214
for a few days at the Congress that they
have been implementing a sort of

00:55:46.214 --> 00:55:50.323
Kubernetes thing, because it's
automatically deploy things, and manage

00:55:50.323 --> 00:55:57.160
things, and so on. And I have noticed by
Day 3 that I have achieved true

00:55:57.160 --> 00:56:04.695
enlightenment and true automation, because
we have decided to deploy everything at

00:56:04.695 --> 00:56:11.455
once at some point. The reason was that we
are being DDOSed, and we had to change

00:56:11.455 --> 00:56:21.479
something to mitigate that. And so we
did that, and everything was fine. But we

00:56:21.479 --> 00:56:27.012
made a typo. We made a typo and the
deployment failed. And one the deployment

00:56:27.012 --> 00:56:39.070
failed, it deleted all the servers. So,
yeah, 405 servers got deleted by what I'm

00:56:39.070 --> 00:56:47.990
remembering was a single line. So it was
brought out automatically, and that wasn't

00:56:47.990 --> 00:56:55.195
a problem. It was all fine, but well, to
err is human, to automate mistakes is

00:56:55.195 --> 00:57:04.073
devops. Next slide? What's important is
that these 405 servers were provided by

00:57:04.073 --> 00:57:08.410
Hetzner. We couldn't have done that
without their infrastructure, without

00:57:08.410 --> 00:57:15.720
their cloud. The reason we got up so
quickly after this was that the servers

00:57:15.720 --> 00:57:21.479
were deleted, but they could have been
reprovisioned almost instantly. So the

00:57:21.479 --> 00:57:28.190
whole thing took like 10 minutes to get it
back up. And, next slide. That's all.

00:57:28.190 --> 00:57:38.686
Thank you all for testing our
infrastructure, and see you next year.

00:57:38.686 --> 00:57:45.930
ysf: Thank you, c3adventure! So this was
clearly the first conference that didn't

00:57:45.930 --> 00:57:54.358
clap for falling mate bottles! If that's
not the thing, maybe we try next year? The

00:57:54.358 --> 00:58:03.369
Lounge. And I know I have to ask for the
next slide too. The rc3 Lounge artists.

00:58:03.369 --> 00:58:09.239
And I was asked to read every country
where someone is in, because everyone had

00:58:09.239 --> 00:58:17.491
to make the Lounge what it was: an awesome
experience. So there were: Berlin, Mexico City

00:58:17.491 --> 00:58:26.269
Honduras, London, Zürich, Stockholm,
Amsterdam, Rostock, Glasgow, Leipzig,

00:58:26.269 --> 00:58:35.635
Santiago de Chile, Prag, Hamburg,
Mallorca, Krakow, Tokyo, Philadelphia.

00:58:35.635 --> 00:58:45.170
Frankfurt am Main, Köln, Moscow, Taipei
Taiwan, Hannover, Shanghai, Seoul… Seoul,

00:58:45.170 --> 00:58:54.885
I think, sorry. Vienna, Hong Kong,
Karlsruhe and Guatamala. Thank you guys

00:58:54.885 --> 00:59:03.260
for making the Lounge. So the next is the
Hub and they should be waiting in

00:59:03.260 --> 00:59:32.493
Studio Two.
<i>audible echo</i>

00:59:32.493 --> 00:59:35.190
XXX: …software is based on Django. And

00:59:35.190 --> 00:59:41.329
it's intended to be used for the next
event. The problem is it was a new

00:59:41.329 --> 00:59:52.859
software. We had to do a lot of
integrations, yeah, live during Day 0.

00:59:52.859 --> 01:00:13.650
Well, OK. No. OK, yeah, hi. I'm presenting
the Hub, which is a software we wrote for

01:00:13.650 --> 01:00:20.310
this conference. Yeah. It's based on
different components, all of them are

01:00:20.310 --> 01:00:28.170
based on Django. It's intended to be used
on future events as well. Our main problem

01:00:28.170 --> 01:00:34.171
was it's a new software. We wrote it and,
yeah, a lot of the integrations were only

01:00:34.171 --> 01:00:41.790
possible on Day 0 or Day 1. And yeah. So
even still today on Day 4, we did a lot of

01:00:41.790 --> 01:00:47.220
updates, commits to the repository, and
even that numbers on the screens are

01:00:47.220 --> 01:00:56.019
already outdated again. But yeah, as you
could possibly see, we have a lot of

01:00:56.019 --> 01:01:02.346
commits all day, night, or all night long.
Only a small digit, 6 AM. I am sorry for

01:01:02.346 --> 01:01:10.289
that. Next slide, please. And yeah,
because the numbers you're quite busy

01:01:10.289 --> 01:01:15.020
using the platform, some of these numbers
on the screen are already outdated again.

01:01:15.020 --> 01:01:24.935
Out of the 360 assemblies which were
registered, only 300 got accepted. Most of

01:01:24.935 --> 01:01:32.788
them were, yeah, event or people wanting
to do a workshop and trying to register an

01:01:32.788 --> 01:01:39.730
assembly. Or, duplicates. So, please
organize yourself. Events, currently we have over

01:01:39.730 --> 01:01:46.529
940 in the system. You're still clicking
events, nice. Thanks for that. The events

01:01:46.529 --> 01:01:52.970
are coordinating with the studios, so we
are integrating all of the events of all

01:01:52.970 --> 01:01:59.156
the studios, and the individual ones, and
the self organized sessions. All of them. A new

01:01:59.156 --> 01:02:08.180
feature, the badges. Currently you have
created 411. And, yeah, from these badges

01:02:08.180 --> 01:02:17.830
redeemed, we have 9269 achievements and
19 000 stickers. Documentation, sadly, was

01:02:17.830 --> 01:02:26.569
a 404, because yeah. We were really busy
doing stuff. Some documentation has

01:02:26.569 --> 01:02:33.489
already been written, but yeah. More
documentation is, will become available

01:02:33.489 --> 01:02:39.859
later. We will open source the whole thing
of course, but right now we're still in

01:02:39.859 --> 01:02:45.990
production and cleaning up things. And
yeah. Finally, for some numbers. Total

01:02:45.990 --> 01:02:54.219
requests per second were about 400. In the
night, when the world was redeploying,

01:02:54.219 --> 01:03:01.359
then we only had about 50 requests per
second, but it maxed up to 700 requests

01:03:01.359 --> 01:03:08.530
per second. And the authentication for the
world, for the 2D adventure, it was about

01:03:08.530 --> 01:03:16.640
220 requests per second. More or less
stable due to some bugs and due to some

01:03:16.640 --> 01:03:23.460
heavy usage. So, yeah, we appreciate that
you used the platform, used the new Hub,

01:03:23.460 --> 01:03:34.939
and hope to see you on the next event.
Thanks.

01:03:34.939 --> 01:03:41.744
ysf: Hello Hub. Thank you Hub. And the
next is betalars waiting for us. He's from

01:03:41.744 --> 01:03:53.380
the c3auti team, and he will tell us what
he does and his team did this year.

01:03:53.380 --> 01:04:04.119
betalars: Hi, I'm betalars from c3auti,
and we've been really busy this year as

01:04:04.119 --> 01:04:15.193
you can probably see by the numbers on my
next slide. We have 37 confirmed Auti-Angles

01:04:15.193 --> 01:04:24.519
and today we surpassed the 200
hours mark. We have 10 Orga Mumbles

01:04:24.519 --> 01:04:30.170
leading up to the event and there are
almost five million unique pixels in our

01:04:30.170 --> 01:04:37.400
repository. I'm pretty convinced we've
managed to create the smallest Fairydust

01:04:37.400 --> 01:04:45.140
of rC3, provided by an actual space
engineer. And the Tree of Solitude is not

01:04:45.140 --> 01:04:52.150
the only thing we've managed to create,
contribute to this wonderful experience.

01:04:52.150 --> 01:05:01.849
On our next slide, you can see that we
also contributed six panel sessions for

01:05:01.849 --> 01:05:08.260
autistic creatures to discuss their
experiences and five Play sessions for

01:05:08.260 --> 01:05:18.249
them to socialize. We helped to contribute
a talk, a podcast, and an external panel

01:05:18.249 --> 01:05:26.099
to the big streams. And on our own panels,
we've had up to 80 participants that need

01:05:26.099 --> 01:05:32.650
to be split up to five breakout rooms so
they could all have a meaningful

01:05:32.650 --> 01:05:45.390
discussion. And all their ideas and thoughts 
were anonymized and stored on more than 1000

01:05:45.390 --> 01:05:54.780
lines of markdown documentation that you can
find on the Internet. But 1000 lines of

01:05:54.780 --> 01:06:00.950
markdown wouldn't be enough for me to
express the gratitude I have towards all

01:06:00.950 --> 01:06:08.870
the amazing creatures that helped us make
this experience happen and for all the

01:06:08.870 --> 01:06:17.170
amazing teams that worked with us. I'm so
happy to see you again soon, but now I

01:06:17.170 --> 01:06:25.639
think I will need some solitude for
myself.

01:06:25.639 --> 01:06:32.125
ysf: Thank you betalars. So, lindworm, are
you ready? The next one is the video, as

01:06:32.125 --> 01:06:46.116
far as I know. It's from the C3 Inclusion
Operation Center. I don't know the short

01:06:46.116 --> 01:06:55.428
name; C3IOC? And it's counting down three
two one go.

01:06:55.428 --> 01:07:18.921
<i>video without audio</i>

01:07:18.921 --> 01:07:26.009
So, video is like a very difficult thing to play in
those days, because we only used to do

01:07:26.009 --> 01:07:33.146
stuff live. Live means a lot of pixels and
traffic is done from this here, from this

01:07:33.146 --> 01:07:40.047
glass, to all the wires and cables and
back to the glass of your screen. And this

01:07:40.047 --> 01:07:46.986
is like magic to me, somehow. Although, I.
am only. being. a robot. to talk.

01:07:46.986 --> 01:07:57.809
synchronistically. with all the.... It's
been around enough time, I think, to

01:07:57.809 --> 01:08:04.641
switch back to Lindy with the video. I
tell you what we are you going to…

01:08:04.641 --> 01:08:17.660
<i>video without audio</i>

01:08:17.660 --> 01:08:23.322
nwng: Hello everyone, I'm nwng from the
new C3 Inclusion Operation Center. This

01:08:23.322 --> 01:08:27.440
year, we've been working on accessibility
guides that help the organizing teams and

01:08:27.440 --> 01:08:32.930
assemblies improve the event for everyone,
and especially people with disabilities.

01:08:32.930 --> 01:08:36.390
We have also worked with other teams
individually to figure out what can still

01:08:36.390 --> 01:08:40.470
be improved in their specific range of
functions - but there are still a lot to

01:08:40.470 --> 01:08:45.424
catch up on! Additionally, we have
published a completely free and accessible

01:08:45.424 --> 01:08:50.800
CSS design template that features dark
mode and an accessible font selection. And

01:08:50.800 --> 01:08:56.273
it still looks good without Javascript.
100 Internet points for that! For you

01:08:56.273 --> 01:08:59.940
visitors, we have been collecting your
feedback through mail or twitter – and

01:08:59.940 --> 01:09:03.934
won't stop after the Congress! If you
stumbled across some barriers, please get

01:09:03.934 --> 01:09:11.262
in touch via c3ioc.de or @c3inclusion on
twitter to tell us about your findings!

01:09:11.262 --> 01:09:19.180
Thanks a lot for having us.
ysf: Thank you for the video. Finally,

01:09:19.180 --> 01:09:27.670
technical's working! We should… does
someone know computers? Maybe? Kritis is

01:09:27.670 --> 01:09:33.113
one of them, and he is waiting in Studio
One to tell us something about C3 Yellow

01:09:33.113 --> 01:09:44.880
or c3gelb wie wir hier sagen.

01:09:44.880 --> 01:09:46.540
Kritis: Yeah, welcome. I'm still looking

01:09:46.540 --> 01:09:50.470
at this hard drive. Maybe you remember
this from the very beginning? It has to be

01:09:50.470 --> 01:09:55.750
disinfected really thoroughly, and I guess
I can take it out by the end of the event.

01:09:55.750 --> 01:10:05.010
And for… the next slide with the words,
please. We did found roughly 0777 hands

01:10:05.010 --> 01:10:12.770
wash options and 0x3FF waste disposal
possibilities. We checked the correct date

01:10:12.770 --> 01:10:22.076
on almost all of the 175 disinfectant
options you had around here. And because

01:10:22.076 --> 01:10:27.390
at a certain point of time, people from
CERT were not reachable in the CERT room

01:10:27.390 --> 01:10:30.654
because they were running around
everywhere else in this great 2D world. We

01:10:30.654 --> 01:10:33.996
had the chance to bypass and channel all
the information because there were two

01:10:33.996 --> 01:10:39.610
digital cats on a digital tree. And so we
got the right help to the right option.

01:10:39.610 --> 01:10:45.210
Next slide, please. We have a couple of
options ongoing. A lot of work had been

01:10:45.210 --> 01:10:51.180
done before. We had all the studios with
all the corona things going on before, but

01:10:51.180 --> 01:10:58.070
now we think we should really watch into
an angel disinfectant swimming basin for

01:10:58.070 --> 01:11:04.140
the next time, to have there the maximum
option of cleanliness. And we will talk

01:11:04.140 --> 01:11:10.540
with the BOC. If we can maybe achieve to
use this Globuli maxi-cubes for the

01:11:10.540 --> 01:11:17.370
Tschunk in the upcoming time. Apart from
that, in order to get more Bachblüten and

01:11:17.370 --> 01:11:24.330
everything else, we need someone who is
able to help us with the Potenzieren

01:11:24.330 --> 01:11:31.961
for homoeopathic substances. So if you
feel welcome with that, please just drop

01:11:31.961 --> 01:11:40.173
us a line to: info@c3gelb.de. Thank you
very much and good luck.

01:11:40.173 --> 01:11:44.890
ysf: Thank you Kritis. Finally happy to
hear your voice. I only know you from

01:11:44.890 --> 01:11:50.700
Twitter, where we treat our stuff
together, or I yours and you, mine, don't.

01:11:50.700 --> 01:11:56.885
Maybe you're going to change it… please?
And, talking about messages. Chaos Post

01:11:56.885 --> 01:12:06.150
was here too, and trilader, whom we
already heard earlier, has more to say.

01:12:06.150 --> 01:12:11.350
trilader: OK, welcome. It's me again. I've
changed outfits a bit. I'm not here for

01:12:11.350 --> 01:12:16.050
the Signal Angels anymore, but for Chaos
Post. So, yeah. We had an online office

01:12:16.050 --> 01:12:22.540
this year again, as we had with the DiVOCs
before. And I've got some mail numbers for

01:12:22.566 --> 01:12:28.984
you that should be on the screen right
now. If it's not, if it's on the title

01:12:28.984 --> 01:12:37.982
page, please switch to the first one where
it lists a lot of numbers. We had 576

01:12:37.982 --> 01:12:46.370
messages delivered total. This is numbers
from around half to six. And 12 of them we

01:12:46.370 --> 01:12:51.210
weren't able to deliver because, well,
non-existent mailboxes or full mailboxes

01:12:51.210 --> 01:12:58.990
mostly. We delivered mail to 43 TLDs, the
most going to Germany, to .de domains,

01:12:58.990 --> 01:13:06.060
followed by .com, .org, .net, and to
Austria with .at; We had a couple of

01:13:06.060 --> 01:13:11.810
motifs you could choose from, the most
popular one was "Fairydust at Sunset", 95

01:13:11.810 --> 01:13:18.050
people selected that. Next slide. About
our service quality. We had a minimum

01:13:18.050 --> 01:13:25.300
delay from the message coming in, us
checking it, and it going out for about a

01:13:25.300 --> 01:13:29.643
bit more than four seconds. The maximum
delay was about seven hours. That was

01:13:29.643 --> 01:13:36.220
overnight, when no agents were ready, or
they were all asleep, or having… being

01:13:36.220 --> 01:13:40.660
busy with, I don't know, the Lounge or
something? And on average a message took

01:13:40.660 --> 01:13:47.090
you, took us 33 minutes from you putting
it into our mailbox to it getting out.

01:13:47.090 --> 01:13:52.620
Some fun facts: We had issues delivering
to T-Online at the first two days, but we

01:13:52.620 --> 01:13:57.690
managed to get that fixed. A different
mail provider refused our mail because it

01:13:57.690 --> 01:14:05.170
contained the string c3world, the domain
in the mail text. And apparently new

01:14:05.170 --> 01:14:08.896
domains are scary, and you can't trust
them or something. We had created a ticket

01:14:08.896 --> 01:14:14.833
with them, they fixed it, and it was super
fast, super nice service. Yeah. Also, some

01:14:14.833 --> 01:14:21.090
people tried to sent digital postcards to
Mastodon accounts because they looked like

01:14:21.090 --> 01:14:26.090
email addresses or something. Another
thing that's not on a slide is we had

01:14:26.090 --> 01:14:31.870
another new feature this time. That was
our named recipients. So you could, for

01:14:31.870 --> 01:14:39.850
example, send mail to CERT without knowing
their address. And they also have a really

01:14:39.850 --> 01:14:44.041
nice postcard wall, where you can see all
the postcards you sent them. The link for

01:14:44.041 --> 01:14:55.871
that is on our Twitter. Thank you.
ysf: Thank you Chaos Post. lindworm, are

01:14:55.871 --> 01:14:59.579
you there?
lindworm: Ja, ja. Ich bin da, Ich bin da.

01:14:59.579 --> 01:15:04.920
Hallo, you're hearing me?
ysf: I hear you.

01:15:04.920 --> 01:15:12.795
lindworm: So I have to switch some more.
It's kind of stressy for me, really.

01:15:12.795 --> 01:15:21.370
ysf: You're doing an awesome job. Thank
you for doing it. So just out of

01:15:21.370 --> 01:15:27.760
curiosity, and did you have a problem
accepting any cookies or so?

01:15:27.760 --> 01:15:35.500
lindworm: No, not really.
ysf: I heard somewhere. That some really

01:15:35.500 --> 01:15:39.030
smart people had problems using the site
because of cookies.

01:15:39.030 --> 01:15:44.800
lindworm: Oh, no, that was not my problem.
I only couldn't use the site because of

01:15:44.800 --> 01:15:54.190
overcrowding. That was often one of my my
little problems. And please, I hope you

01:15:54.190 --> 01:15:58.750
don't see what I'm doing right now in the
background with starting our pets and so

01:15:58.750 --> 01:16:11.820
on. And what I wanted to say to all of
you, this was the first Congress where we

01:16:11.820 --> 01:16:18.850
have so many women and so many non-cis
people running that show and being up

01:16:18.850 --> 01:16:25.343
front the camera and making everything up.
I would really thank you all. Thank you,

01:16:25.343 --> 01:16:30.701
that you made that possible. And thank you
that we get more and more diverse, year by

01:16:30.701 --> 01:16:38.580
year.
ysf: I can only second that. And now we

01:16:38.580 --> 01:16:43.100
are switching to C3 Infrastructure.
lindworm: Yeah, we need to.

01:16:43.100 --> 01:16:50.000
ysf: I'm sure a lot of questions will be
answered by them.

01:16:50.000 --> 01:16:58.210
lindworm: And I try to make up the slides
for that, but I do not find them right

01:16:58.210 --> 01:17:02.680
now.
patrick: Look mom, I'm on TV.

01:17:02.680 --> 01:17:11.044
thies: Yeah. Welcome to the infrastructure
review of the Team Infrastructure. I'm not

01:17:11.044 --> 01:17:15.960
quite sure if we have the newest revision
of the slides, cause my version of the

01:17:15.960 --> 01:17:21.890
stream isn't loading right now. So maybe
lindworm, is it possible to press

01:17:21.890 --> 01:17:30.380
control-R? And you're seeing a burning
computer, then we have the actual slides.

01:17:30.380 --> 01:17:35.470
Patrick: Let's just Powerpoint Karaoke
without the background music.

01:17:35.470 --> 01:17:44.000
thies: Yeah, and without the PowerPoint
presentation in realtime. Now I'm seeing

01:17:44.000 --> 01:17:48.070
me. Let's wait a few seconds until we see
a slide.

01:17:48.070 --> 01:17:51.800
Patrick: We want to wait the entire stream
delay.

01:17:51.800 --> 01:18:00.190
thies: It's just about 30 to one minute.
Patrick: Well done.

01:18:00.190 --> 01:18:09.960
thies: Yeah, I'm thies and I'm waiting.
And this is Patrick, and he's waiting too.

01:18:09.960 --> 01:18:19.850
Yeah, but that's in the middle of the
slides. Can we go… OK. Yeah. I'm now

01:18:19.850 --> 01:18:26.799
seeing something in the middle of the
slides, but it seems fine. OK, yeah. We

01:18:26.799 --> 01:18:36.640
are the team C3 Infra. rC3 Infra. We are
creating the infrastructure. Next slide.

01:18:36.640 --> 01:18:50.230
We had about nine terabytes of RAM and
1,700 CPU cores. The whole event there's

01:18:50.230 --> 01:18:58.023
only one dead SSD that died because
everything's broken. We had five dead RAID

01:18:58.023 --> 01:19:02.520
controllers, and didn't bother to replace
the RAID controllers, just replaced them

01:19:02.520 --> 01:19:14.120
with new servers. And 100 percent uptime.
Next slide. We looked about 42 hours on

01:19:14.120 --> 01:19:22.980
starting screens of enterprise servers. 20
minutes max is what HP delivered. And we

01:19:22.980 --> 01:19:32.330
are now certified enterprise observers. We
had only 27%-ish of visitors using IPv6.

01:19:32.330 --> 01:19:39.540
So that's even less than Google publishes.
And even though we had almost full IPv6

01:19:39.540 --> 01:19:48.010
coverage – except some really, really shady
out-of-band management networks – we're

01:19:48.010 --> 01:19:55.370
still not at the IPv6 coverage that we are
hoping for. I'm not quite sure if that's

01:19:55.370 --> 01:20:05.020
the right slides. But I'm not quite sure
where we are in the text. Yeah, Patrick.

01:20:05.020 --> 01:20:11.290
Patrick: Yeah, so before the Congress
there was one prediction: there's no way

01:20:11.290 --> 01:20:17.737
it cannot be not DNS. And while it was DNS
at least once, so we checked that box. And

01:20:17.737 --> 01:20:27.090
let's go over to the next topic, OS. We
provisioned about 300 nodes, and it was an

01:20:27.090 --> 01:20:33.181
Ansible-powered madness. So, yeah, there
was full disk encryption on all nodes. No

01:20:33.181 --> 01:20:37.621
IP logged in the access logs, we took
extra care of that. And we configured

01:20:37.621 --> 01:20:43.470
minimal logging wherever possible, so the
case of some problems we only had WARNINGs

01:20:43.470 --> 01:20:50.533
available. And there are no INFO logs, no
DEBUG logs; just the minimal logging

01:20:50.533 --> 01:20:55.850
configuration. And with some software, we
had to pipe logs to /dev/null because the

01:20:55.850 --> 01:21:01.080
software just wouldn't stop logging IP's,
and we didn't want that. So no personal

01:21:01.080 --> 01:21:06.897
data in logs, so no GDPR headache, and
your data is safe with us. The Ansible

01:21:06.897 --> 01:21:12.030
madness I've talked about was a magical
deployment that deep bootstrapped into the

01:21:12.030 --> 01:21:18.370
live system and assimilated into the rC3
infrastructure while it's still running.

01:21:18.370 --> 01:21:27.430
So if you didn't boot your machine then
what? They're just running. When a OS

01:21:27.430 --> 01:21:31.529
deployment was broken, it was almost
always due to a network or routing. At

01:21:31.529 --> 01:21:37.320
least the OS team claims that, and this
claim is disputed by the network team of

01:21:37.320 --> 01:21:42.723
course. One time, the deployment broke
because of a trigger happy infra angel.

01:21:42.723 --> 01:21:52.450
But let's not talk about that. Of course,
at this point, we want to announce our

01:21:52.450 --> 01:21:58.370
great cooperation with our gold sponsor
ddos24.net, who provided an excellent

01:21:58.370 --> 01:22:05.750
service of handcrafted request to our
infrastructure. That was a great demand or

01:22:05.750 --> 01:22:14.080
great public demand, with a million
requests per second for a while. But even

01:22:14.080 --> 01:22:21.540
during the highest or peak demand, we were
able to serve most of these services. We

01:22:21.540 --> 01:22:27.920
provide the infrastructure to the VOC, and
they quickly made use of the provided

01:22:27.920 --> 01:22:35.616
infrastructure deployed there. Overall, an
amazing time to market. We had six

01:22:35.616 --> 01:22:41.254
locations, and those six locations where
some wildly different, special snowflakes

01:22:41.254 --> 01:22:49.381
overall. So we had Düsseldorf, 816 CPU
cores there, two terabytes of RAM, and we

01:22:49.381 --> 01:22:55.200
had 10 gigabits per second interconnect.
There was also a 1 terabit per second

01:22:55.200 --> 01:22:59.670
Infiniband available, but sadly, we
couldn't use that. It would have been

01:22:59.670 --> 01:23:05.351
nice. The machines that had a weird and
ancient IPMI, which made it hard to deploy

01:23:05.351 --> 01:23:10.459
there. And the admin on location never
deployed bare metal hardware to a

01:23:10.459 --> 01:23:15.330
datacenter, so there were also some
learning experience there. Fun fact about

01:23:15.330 --> 01:23:20.737
Düsseldorf, this was the data center with
the maximum heat. One server, seven units,

01:23:20.737 --> 01:23:29.770
over 9000 watts of power. 11.6 to be
exact. Which is why they had some to take

01:23:29.770 --> 01:23:39.882
some creative heat management solutions.
Next was Frankfurt, there we had 620

01:23:39.882 --> 01:23:47.890
gigabits of total uplink capacity, and we
actually only used 22 gigabit during peak

01:23:47.890 --> 01:23:54.430
demand. Again, by our premium sponsor:
ddos24.net. There was zero network

01:23:54.430 --> 01:24:02.690
congestion and 1.5 gigabits per second
were IPv6. So there was no real traffic

01:24:02.690 --> 01:24:08.980
challenge. For the network engineers of
you, it was a full Layer 3 architecture

01:24:08.980 --> 01:24:15.960
with MPLS between the WAN routers. And
there was a night shift on the 26the and

01:24:15.960 --> 01:24:25.400
27th for more servers, because some
shipments didn't arrive yet. The fun fact

01:24:25.400 --> 01:24:30.346
about this datacenter was the maximum
bandwidth. Some servers there had 50

01:24:30.346 --> 01:24:36.727
gigabit uplink on the server configured.
It was the data center with the maximum

01:24:36.727 --> 01:24:41.520
manual intervention. Of course, we had the
most infrastructure there and it wasn't

01:24:41.520 --> 01:24:47.927
oversubscribed at any point. We had some
hardware in Stuttgart, which was basically

01:24:47.927 --> 01:24:53.080
the easiest deployment. There were also
some night shifts, but the thanks to

01:24:53.080 --> 01:24:58.710
neuner and team this was a really easy
deployment. It was also the most silent

01:24:58.710 --> 01:25:06.590
DC, so no incident from Day -5 until now.
So if you're currently watching from

01:25:06.590 --> 01:25:13.841
Stuttgart now, you can create some issues
because now we said it. Wolfsberg was the

01:25:13.841 --> 01:25:18.420
smallest DC. We only had three servers and
we managed to kill one hardware RAID

01:25:18.420 --> 01:25:25.826
controller, so we only could use two
servers there. So, yeah. And then Hamburg

01:25:25.826 --> 01:25:30.982
was the data center with the minimum
uptime. We never could deploy to this data

01:25:30.982 --> 01:25:35.490
center because there was a broken netboot
and we couldn't provision anything there.

01:25:35.490 --> 01:25:42.210
And of course, the sixth data center was
the Hetzler Cloud, where we deployed it on

01:25:42.210 --> 01:25:48.220
all locations. Deployment fun facts: we
received a covid warning from the data

01:25:48.220 --> 01:25:52.790
center. Luckily, it didn't affect us. It
was at another location. But thanks for

01:25:52.790 --> 01:25:59.710
the heads-up and the warning. The team
leader of a sponsor needed to install

01:25:59.710 --> 01:26:07.003
Proxmox in a DC with no knowledge, without
any clue what they were doing. We

01:26:07.003 --> 01:26:11.400
installed Proxmox in the Hamburg DC, and
no server actually wanted to talk to us,

01:26:11.400 --> 01:26:17.000
so we had to give up on that. And there
had to be a lorry relocated before we

01:26:17.000 --> 01:26:27.456
could deploy other servers. So that's that
was standing in the way there. Now, let's

01:26:27.456 --> 01:26:34.877
get to Jitsi. Our peak count were 1,105
users at the same time, on the same

01:26:34.877 --> 01:26:41.670
cluster. I don't know if it was at the
same time as the peak user count, but the

01:26:41.670 --> 01:26:44.500
peak conference count was 204 conferences.

01:26:44.500 --> 01:26:49.850
I hope we can still beat
that today, but this is data from

01:26:49.850 --> 01:26:58.810
yesterday. The peak conference size was 94
participants in a single conference. And

01:26:58.810 --> 01:27:06.780
let me give condolences to your computer,
because that must have been hard on it.

01:27:06.780 --> 01:27:14.171
Our peak outgoing video traffic on the
Jitsi video bridges was 1.3 gigabits per

01:27:14.171 --> 01:27:23.715
second. And we had about three quarters of
the participants were streaming video and

01:27:23.715 --> 01:27:31.750
one quarter of them had video disabled.
Interesting ratio. Our Jitsi deployment

01:27:31.750 --> 01:27:38.140
was completely automated with Ansible, so
it was zero to Jitsi in 15 minutes. We

01:27:38.140 --> 01:27:43.390
broke up the Jitsi cluster into four
shards to have better scalability and

01:27:43.390 --> 01:27:48.360
resilience. So if one shard went down, it
would only affect part of the conferences

01:27:48.360 --> 01:27:53.380
and not all of them. Because there are
some infrastructure components that you

01:27:53.380 --> 01:28:00.495
can't really scale or cluster, so we went
with with the sharding route. Our Jitsi

01:28:00.495 --> 01:28:07.640
video bridges were at about 42% peak usage
– excluding our smallest video bridge,

01:28:07.640 --> 01:28:11.400
which was only eight cores and eight
gigabytes, which we added in the beginning

01:28:11.400 --> 01:28:17.120
to test some stuff out, and it remained in
there. And yes, we overprovisioned a bit.

01:28:17.120 --> 01:28:21.780
There will also be a blog post on our
Jitsi Meet deployment coming in the

01:28:21.780 --> 01:28:31.110
future. And for the next time we, for the
upcoming days, we will enable 4K streaming

01:28:31.110 --> 01:28:40.740
on there. So why not use that? And we want
to say thanks to the FFMEET Projekt, who

01:28:40.740 --> 01:28:46.400
contacted us after our initial load test
and gave us some tips to handle load

01:28:46.400 --> 01:28:58.190
effectively and so on. We also tried
making DECT call-out working. Spent 48

01:28:58.190 --> 01:29:06.760
hours trying to get it to work, but there
were some troubles there. So sadly, no

01:29:06.760 --> 01:29:14.610
adding DECT participants to your Jitsi
conferences for now. jitsi.rc3.world will

01:29:14.610 --> 01:29:22.980
be running over New Year. So you can use
that to get together with your friends and

01:29:22.980 --> 01:29:27.829
so on over the New Year. Stay separate,
don't visit each other please. Don't

01:29:27.829 --> 01:29:35.920
contribute to covid-19 spread. You've got
the alternative there. Now let's go over

01:29:35.920 --> 01:29:40.960
to monitoring. thies.
thies: Yeah, thanks. First of all, it's

01:29:40.960 --> 01:29:46.660
really funny how you edit this page, but
reveal.js doesn't work that way until

01:29:46.660 --> 01:29:51.824
lindworm reloads the page, which hopefully
doesn't do right now. Everything's fine,

01:29:51.824 --> 01:29:58.410
so you can leave it to be. Yeah,
monitoring. We had to Prometheus and

01:29:58.410 --> 01:30:04.920
Alertmanager set up completely driven out
of our solemnly one and only source of

01:30:04.920 --> 01:30:14.347
truth: our Netbox. We received about
34 858 critical alerts. It's – looking at

01:30:14.347 --> 01:30:21.340
my mobile phone – it's definitely more 
right now. And about 13,070 warnings. Also

01:30:21.340 --> 01:30:30.456
definitely more right now. And we tended
about 100 of them. The rest was kind of

01:30:30.456 --> 01:30:42.494
useless. Next slide, please. As it's
important to have an abuse hotline and an

01:30:42.494 --> 01:30:48.133
abuse contact, we received two network
abuse messages, both from Hetzner – one of

01:30:48.133 --> 01:30:51.950
our providers – letting us know that
someone doesn't like our infrastructure as

01:30:51.950 --> 01:31:01.730
much as we do. Props to ddos24.net. And we
got one call it our abuse hotline, and it

01:31:01.730 --> 01:31:09.190
was one person who wanted to buy a ticket
from us – Sadly, we were out of tickets.

01:31:09.190 --> 01:31:16.020
Next slide, please. Some other stuff. We
got a premium Ansible deployment brought

01:31:16.020 --> 01:31:26.050
to you by turing-complete YAML. That sounds
scary. And we had about 130k DNS updates

01:31:26.050 --> 01:31:32.291
thanks to the World team. At this point
they're really stressing our DNS API with

01:31:32.291 --> 01:31:39.210
the re-deployments. And also our DNS,
Prometheus, and Grafana are deployed on

01:31:39.210 --> 01:31:48.025
and by NixOS thanks to flüpke and head
over to flüpkes interweb thingy. He wrote

01:31:48.025 --> 01:31:54.706
some blog posts about how to deploy stuff
with his NixOS. And the next slide,

01:31:54.706 --> 01:32:02.020
please. And the last slide from the team
is the list of our sponsors. Huge thanks

01:32:02.020 --> 01:32:08.210
to all of them. It won't be possible to
create such a huge event and such loads of

01:32:08.210 --> 01:32:15.490
infrastructure without them. And that's
everything we have.

01:32:15.490 --> 01:32:25.707
ysf: Amazing. Thank you for all you've
done. Truly incredible, and showing

01:32:25.707 --> 01:32:30.980
everything to the public. So I promised
that there will be a kind of behind the

01:32:30.980 --> 01:32:37.130
scenes look of this infrastructure talk or
review. And I really have nothing to do

01:32:37.130 --> 01:32:41.000
with it. Everything was done by completely
different people. I'm only a Herald,

01:32:41.000 --> 01:32:47.040
somehow lost and tumbled into this stream.
And so I'm just going to say switch to

01:32:47.040 --> 01:33:03.722
wherever. Show us the magic.
Karlsruhe: Three hours ago, I got the

01:33:03.722 --> 01:33:10.592
call… Hello and welcome from the last part
of the infrastructure review and greetings

01:33:10.592 --> 01:33:15.973
from Karlsruhe. So three hours ago, I got
a call from lindworm and he asked me, how

01:33:15.973 --> 01:33:23.485
is it with this last talk we have? It may
be a bit complicated. And he told me, OK,

01:33:23.485 --> 01:33:28.950
we have a speaker. I'm the Herald. Oh,
that's always so. And then we realized,

01:33:28.950 --> 01:33:35.110
yeah, we don't have only one speaker, we
have 24. And so that's why we called

01:33:35.110 --> 01:33:41.780
ChaosWest and built up an infrastructure
which dampfkatze will explain you now in a

01:33:41.780 --> 01:33:48.200
short minute. I think so.
dampfkatze: Thank you. Yes. Oh, I lost the

01:33:48.200 --> 01:33:57.710
sticker. OK, after we called ChaosWest, we
came up with this monstrosity of the video

01:33:57.710 --> 01:34:08.777
cluster. And we start here. The teams
streamed via OBS.Ninja onto three

01:34:08.777 --> 01:34:19.611
ChoasWest studios. They were brought
together via RTMP on our Mix1 local

01:34:19.611 --> 01:34:31.030
studio, and then we pumped that into Mix2,
which pumped it further to the VOC. The

01:34:31.030 --> 01:34:37.730
slides were brought in via another
OBS.Ninja directly onto Mix2. They came

01:34:37.730 --> 01:34:43.746
from lindworm. Also, the closing you will
see shortly hopefully will also come from

01:34:43.746 --> 01:34:52.790
there. And ysf and lindworm were directly
connected via OBS.Ninja onto our Mix1

01:34:52.790 --> 01:35:02.709
computer. And Mix2 also has the studio
camera you're watching right now. And for

01:35:02.709 --> 01:35:09.767
the background communication, we had a
Mumble connected with our audio matrix.

01:35:09.767 --> 01:35:17.750
And lindworm, ysf, and the teams, and we
in the studio locally could all talk

01:35:17.750 --> 01:35:24.164
together. And now back to the closing
with… No, to the Herald News Show, I

01:35:24.164 --> 01:35:32.790
think. lindworm will introduce it to you.
lindworm is live.

01:35:32.790 --> 01:35:51.640
lindworm: Is ysf still there? Or do you
come with me? So it will take a second or

01:35:51.640 --> 01:36:02.412
billions of years. So thank you very much
for this review. It was as chaotic as the

01:36:02.412 --> 01:36:04.800
Congress.

01:36:04.800 --> 01:36:16.850
<i>postroll music</i>

01:36:16.850 --> 01:36:27.780
Subtitles created by c3subtitles.de
in the year 2021. Join, and help us!