#rC3 - Infrastructure Review

rC3 preroll music

ysf: Hello and welcome to the
infrastructure review of the rC3 this

year, 2020. What the hell happened? How
could it happen? I'm not alone this year.

With me is lindworm who will help me with
the slides and everything else I'm going

to say before. And this is going to be a
great fuck up like last year, maybe. We

have more teams, more people, more
streams, more of everything. And the first

team and lindworm who I'm going to introduce
is the SHOC. Are you there with me?

Lindworm: Oh, yeah, so I got to go to the
SHOC. Yeah, it's kind of a stress this

year. We only had about 18 heralds for the
main talks rC1 and rC2. And we have

introduced about 51 talks with that.
Everybody from this home setup, which was

a very, very hard struggle. So we all had
a metric ton of adrenaline and excitement

without… within us. So here you can see
what you have seen, how a herald looks

from the front. And so it does look in the
background. Oof. That was hard, really

hard for us. So you see all our different
set ups here, do we have? And we are very,

very pleased to also have set up a
completely new operation center: the

Herald News Show, which I really, really
like you to review on YouTube. This was

such a struggle. And we have about, oh,
wait a second, so as we said, we're a

little bit unprepared here, I need to have
my notes up. There were 20 members that

formed a new team on the first day. They
made 23 shows, 10 hours of video

recording, 20 times the pizza man rung at
the door. And 23 mate bottles had been

drunk during the preps because all of
those people needed to be online the

complete time. So I really applaud to
them. That was really awesome, what they

brought over the team and what they
brought over the stream. And this is an

awesome team I hope we see more of. ysf,
would you take it over? ysf is muted

Oh, no. My, my bad. So is the heaven
ready? We need to go to the heaven and

would have an infrastructure review of the
heaven.

raziel: OK. Du hörst mich noch? Ja, hallo?
Ich bin der raziel aus dem Heaven und ehm…

Yeah, heaven is ready, so welcome,
everybody. I'm raziel from heaven, and I

will present you the infrastructure review
from the heaven team. We had some angel

statistics scrapped out a few hours ago.
And on this year, we have not so much

angels like last year, because we had a
remote event, but we had a total of 1487

total angels from which 710 arrived and
even more of 300 angels that at least

still did one shift. And in total the
recorded work done to that point was

roughly 17 and 75 weeks of done working
hours, and for the rC3 world we also

prepared a few goodies so people could
come visit us. And so we provided them a

few badges there. And every angel that,
for example, found our extinguished…

expired extinguisher and also extinguished
fire in heaven. The first batch was

achieved from 232 of our angels and even
less. But still a good number of 125

angels accomplished to help us and
extinguish the fire that broke out during

an event. And with that numbers checked,
we also will jump into our heaven. So I

would like to show you some expressions
and impressions from it. We had quite the

team working to do exactly what the heaven
could do: manage its people so we needed

our heaven office. And we also did this
with respect to your privacy, so. We

painted our color… our clouds white as
ever, so we cannot see your nicknames, and

you could do your angel work but not be
bothered with us asking for your names.

And also, we had prepared some secret
passage to our back office. And every time

on the real event, it would happen that
some adventurers would find their way into

our back office. And so we needed to
provide that opportunity as well, as you

can see here. And let me say that some
adventurers tried to find the way in our

sacred digital back office, but only a few
were successful. So we hope everyone found

its way back into the real world from our
labyrinth. And we also did not spare any

expenses to do some additional update for
our angels as well. As you can see, we

tried to do some multi-instance support.
So some of our angels also accomplished to

split up and serve more than one angel at
a time. And that was quite awesome. And so

we tried to provide the same things we
would do on Congress, but now from our

remote offices. And one last thing that
doesn't… normally doesn't need to be said.

But I think in this year and with this
different kind of event, I think it's

necessary that the heaven as a
representative, mostly for people trying

to help make this event awesome. And I
think it's time to say the things we do

take for granted. And that is thank you
for all your help. Thank you for all the

entities, all the teams, all the
participants that achieved the goal to

bring our real Congress that many, many
entities missed this year into a new

stage. We tried that online. It had its
ups and downs. But I still think it was an

awesome adventure for everyone. And from
the Heaven team I can only say thank you

and I hope to see you all again in the
future on a real event. Bye! And have a

nice New Year.
lindworm: Hello, hello, back again. So we

now are switching over to the Signal
Angels. Are the signal angels ready?

Hello!
trilader: Yeah, hello, uhm, welcome to the

infrastructure review for the Signal
Angels, I have prepared some stuff for

you. This was for us… slides, please? This
was for us the first time running a fully

remote Q&A session set, I guess? We had
some experience with DiVOC and had gotten

some help from there on how to do this,
but just to compare, our usual procedure

is to have a signal angel in the room.
They collect the question on their laptop

there and they communicate with the Herald
on stage and they have a microphone like

I'm wearing a headset. But in there we
have a studio microphone and we speak

questions into it. Yeah, but remotely we
really can't do that. Next slide. Because,

well, it would be quite a lot of hassle
for everyone to set up good audio setups.

So we needed a new remote procedure. So we
figured out that with a signal Angel and

the Herald could communicate via
a pad and we could also collect the

question in there. And the Herald will
read the question to the speaker and

collect feedback and stuff. So we had 175.
No, 157 shifts, and sadly we couldn't fill

five of them in the beginning because
there was not enough people already there.

And also technically it was more than five
unfilled shifts because for some reasons

there were DJ sets and other things that
aren't talks and also don't have Q&A. We

had 61 angels coordinated by four
supporters, so me and three other people,

and we had a 60 additional angels that
in theory wanted to do signal angel work

but didn't show up to the introduction
meeting. Next! For, as I've said for each

session, each talk, we created a pad where
we put in the questions from IRC,

Mastodon, and Twitter and. Well, we have a
bit more pads than talks we actually

handled, and I have some statistics about
an estimated number of questions per talk.

What we usually assume is that there's a
question per line, but some questions are

really long and have to split over
multiple lines. There are some structured

questions with headings and paragraphs
some heralds or signal angels removed

questions after they were done. And also
there were some chat and other

communication in there. So next slide, we
took a Python script, download all the pad

contents, read them, counted the number of
lines, remove the size of the static

header. And in the end we had 179 pads and
1,627 lines if we discount the static

header of nine lines per pad. So that in
theory leads to about nine questions in

quotation marks because it's not really
questions but lines. But it's an estimate,

per talk. Thank you.
ysf: ... talk and what I've learned is

never miss the introduction. So the next
in line are the line producers ha ha ha ha

stb are you there?
stb: I am here, in fact, so singing. So

the people a bit older might recognize
this melody badly sung by yours truly and

other members of the line producers team,
and I'll get to why that is relevant to

what we've been doing at this particular
event. So what does, what do line

producers do? What does an,
Aufnahmeleitung actually perform? It's

basically communication between everybody
who's involved in the production, the

people behind the camera and also in front
of the camera. And so our work started

really early, basically at the beginning
of November, taking on like prepping

speakers in a technical setup and
rehearsing with them a little bit and then

enabling the studios to allow them to
actually do the production coordination on

an organizational side. The technical side
was handled by the VOC, and we'll get to

hear about that in a minute. But getting
all these people synced up and working

together well, that was quite a challenge.
And that took a lot of Mumbles with a lot

of people in them. We only worked on the
two main channels. There's quite a few

more channels that are run independently
of kind of the central organization. And

again, we'll get to hear about the details
of that in a minute. And so we provided

information. We tried to fill wiki pages
with relevant information for everybody

involved. So that was our main task. So
what does that mean specifically, the

production set up? We had 25 studios,
mainly in Germany, also one in

Switzerland. These did produce recordings
ahead of time for some speakers, and many

did live set ups for their own channels
and also for the two main channels. And

I've listed everybody involved in the live
production here. And there were 19

channels in total. So a lot of stuff
happening. 25 studios, 19 channels that

broadcast content produced by these
studios. So that's kind of the Eurovision

kind of thing, where you have different
studios producing content and trying to

mix it all together. Again, the VOC took
care of the technical side of things very

admirably, but getting everybody on the
same page to actually do this was not

easy. For the talk program, we had over
350 talks in total, 53 in the main channels

And so handling all that, making
sure everybody has the speaker information

they need and all these organizational
stuff, that was a lot of work. So we

didn't have a studio for the main
channels, the 25 studios or the nine, the

live channels, the 12, they actually did
provide the production facilities for the

speakers so we can look at the next slide.
There's a couple more numbers and of

course, a couple pictures from us working
basically from today. We had 53 channel...

53 talks in the main channel. 18 of them
were prerecorded and played out. We had 3

where people were actually on location in
a studio and gave their talk from there.

And we had 32 that were streamed live like
I am speaking to you now with various

technical bits that again the VOC will go
into in a minute. And we did a lot of

Q&As, I don't have the numbers how many
talks actually had Q&As, but most of them

did, and those were always like. We had a
total of 63 speakers we did prepare, at

least the live Q&A session for and helped
them set up, we helped them record their

talks if they wanted to prerecord them. So
we spent anywhere between one and two

hours with every speaker to make sure they
would appear correctly and in good quality

on the screen. And then during the four
days, we, of course, helped coordinate

between the master control room and the
twelve live studios to make sure that the

speakers were where they were supposed to
be and any technical glitches could be

worked out and decide on the spot. If, for
example, the line producers made a mistake

and a talk couldn't happen as we had
planned because we forgot something. So we

rescheduled and found a new spot for the
speakers. So apologies again for that. And

thank you for your understanding and
helping us bring you on screen on day two

and not day one. But I'm very glad that
that we could work that out. And that's

pretty much it from the line producers, I
think. Next up is the VOC.

ysf: Thank you stb. Yes, you're right, the
next are the VOC and kunsi and

JW2CAlex are waiting for us.
Franzi: ... is Franzi from the VOC. 2020

was the year... Hm? Hi, this is Franzi
from the... from VOC. 2020 was the year of

distributed conferences. We had 2 DiVOCs
and the FrOSCon to learn how we are going

to produce remote talks. We learned a lot
of stuff on organization, Big Blue Button

and Jitsi recording. We had a lot of other
events which was just streaming like

business as usual. So for rC3, we extended
the streaming CDN with two new locations,

now 7 in total, with a total bandwidth of
about 80 gigabits per second. We have two

new mirrors for media.ccc.de and are now
also distributing the front end. We got

two new transcoder machines, Erfas and
Enhanced cir setup we now have 10 Erfas

with own productions on media.ccc.de. So
the question is, will it scale? On the

next slide...
Alex: Yeah, next slide.

Franzi: ... we will see that it did
scale. We did produce content for 25

studios and 19 channels, so we got lots of
lots of recordings which will be published

on media.ccc.de in the next days and
weeks. Some have already been published,

so there's a lot of content for you to
watch. And now Alex will tell us something

about the technical part.
Alex: My name is Alex, Pronouns it/its. I

will now tell you the technical part
first, but more of the organization. I was

between the VOC and the line producing
team. And now a bit how it worked. So we

had those two main channels, rc-one and
rc-two. Those channels have been produced

by the various studios distributed around
the whole country. And those streams,

this is now the upper path in the picture,
went to our ingest relay, to the FEM, to

the master control room. In Ilmenau there
were a team of people adding the

translations, making the mix, making the
mixdown, making records and then

publishing it back to the streaming
relays. All the other studios produced to

channels. Those channels took the also the
signals from different studios, make a

mixdown, etc. publish to our CDN and
relays and we publish to the studio

channels. As you can see, this is not the
typical setup we had in the last year in

the presence. So, our next slide, we can
see where this leads: Lots of

communication. We had the line producing
team, we had some production in Ilmenau

that has to be coordinated. We have the
studios, we have the local studio helping

Angels. We have some Mumbles there, some
RocketChat here, some CDN people some web

where something happens. We have some
documentation that should be. And then we

started to plot down the communication
paths. Next slide, please. If you plotted

all of them, it really looks like the
world, but this is actually the world, but

sometimes it feels like they're just
getting lost in different paths. Who you

have to ask, who do you have to call?
Where are you? What's the shortest path to

communicate? But let's have a look at the
studios. First going to ChaosWest.

Franzi: Yes, on the next slide, you will
see the studio set up at ChaosWest TV. So

thank you, ChaosWest for producing your
channel.

Alex: At the next slide, you see the
Wikipaka television and fernseh-streamen

(WTF) who have the internal motto:
"Absolut nicht sendefähig - chaos of

recording". But even then, at some
studios, you look more like studios, so

this time at the next slide at the hacc.
Franzi: Yeah, at hacc, you will also see

some of the bloopers we had to deal with.
So, for example, here you can see there

was a cat in the camera view, so, yeah.
And Alex, tell us about the open

infrastructure orbit.
Alex: The open infrastructure orbit

showed. In this picture, you can see it's
really hard to see how you can make a

studio look really nice, even if you're
alone, feeling a bit comfier, more

hackish. But you have also those normal
productions as in the next slide. The

Chaosstudio Hamburg
Franzi: Yeah, at Chaosstudio Hamburg, we

had two regular work cases like, you know,
from all the other conferences, and they

were producing, onsite in a regular studio
set up. And last but not least, we got

some impressions from ChaosZone TV.
Alex: As you can see here, also quite

regular studio setup, quite regular. No.
There was some Corona virus ongoing, and

this is we had a lot of distancing,
wearing mask and all the stuff that

everyone is safe but c3yellow (c3gelb)
will tell you some facts about it. But

let's look at the nice things. For
example, the minor issue: On the second

day, we were sitting there looking at our
nice Grafana. Oh, we got a lot of more

connections. The server load's increasing.
The first question was: Have we enabled

our cache?". We don't know. But the number
of connections is growing that people are

watching our streams, the interest goes
up. And we were, well, at least the people

are watching the streams. If there is a
website, who cares, the interest works.

But then we suddenly get the relations.
Well, something did not really scale that

good. And then using the next slide, this
view. This switched pretty fast from after

looking at this traffic graph. "Well,
that's interesting" into "Well, we should

investigate". We get thousands of messages
on Twitter DMs. We got thousands of

messages in RocketChat, IRC, and suddenly
we had a lot of connections to handle; a

lot of inquiries to handle, a lot of phone
calls, etc. to handle. And we have to

prioritize for us the hardware then the
communication, because otherwise the

information won't stop. On the next slide
you can see what our minor issue was. So

at first, we get a lot of connections to
our streaming web pages, then to load

balancers, and finally to our DNS servers.
A lot of them were quite malformed. It

looked like a storm. But the more
important thing we had to deal was all

those passive aggressive messages from,
from different persons who said: "Well,

you can't even handle streaming. What are
you doing here?" And we worked together

with the c3infra team, thanks for that, how
to scale and decentralize a bit more just to

provide the people the connection power
they need. So I think in the last years,

we don't need to use more bandwith. We
showed we can provide even more bandwith

if we need it. And then, noting everything
down…

Franzi: So is it time to shut everything
down? No, we won't shut everything down.

The studios can keep their endpoints, can
continue to stream on their endpoints as

they wish. We want to keep in touch with
you and the studios, produce content with

you, improve our software stack, improve
other things like the ISDN, the Internet

Streaming Digital Node, the project for
small camera recording setups for sending

to speakers needs developers for the
software. Also, KEVIN needs developers and

testers. What's KEVIN? Oh, we have
prepared another slide or the next slide.

KEVIN is short for Killer Experimental
Video Internet Noise, because we initially

wanted to use OBS.Ninja, but there are a
couple of licensing issues. There is not

everything in OBS.Ninja is open source
like we wanted, so we decided to code our

own OBS.Ninja-style software. So if you
are interested in doing so, please get

into contact with us or visit the wiki. So
that's all from the VOC. And we are now

heading over to c3lingo.
ysf: Exactly. c3lingo oskar should be

waiting Studio 2, aren't you?

oskar: Yeah, hallo. Hi, yeah, I'm oskar

from c3lingo. We will jump straight into
the stats on our slides. As you can see

here, we translated 138 talks this time,
as you can see, it's also way less

languages than in the other chaos events
that we had since our second languages

team that does everything that is not
English and German was only five people

strong this time. So we only managed to do
five talks into French and three talks

into Brazilian Portuguese. And then on the
next slide… We are looking at our coverage

for the talks and we can see that on the
main talks we managed to cover all talks

that were happening from English to German
and German to English, depending on what

the source language was. And then, on the
other languages track, we only managed to

do 15 percent of the talks from the main
channels. And then on the further

channels, which is a couple of others that
also were provided to us in the

translation team, we managed to do 68% of
the talks, but none of them were

translated into other languages than
English and German. On the next slide,

some global stats. We have 36
interpreters, which in total managed to

translate 106 hours and 7 minutes of talks
into another language simultaneously. And

the maximum number of hours one person did
was 16 hours and the minimum number of

hours, the average number of hours people
did was around 3 hours of translation

across the entire event. All right. Then I
also have some anecdotes to tell and some

some mentions I want to do. We had two new
interpreters that we want to say "hi" to,

and we had a couple of issues with the
digital thing that didn't have before with

regular events where people were present.
For example, the issue of sometimes when

two people are translating one person's
starts interpreting something on wrong

stream. Maybe they were watching the wrong
one. And then the partner just thinks they

have more delay or something. Or, for
example, a partner having a smaller delay

and then thinking the partner can suddenly
read minds because they can translate

faster than the other person is actually
seeing the stream. Those are issues that

we usually didn't have with the regular
stream, but only with the regular events,

not with remote events. And yeah, some
hurdles to overcome. Another thing was,

for example, when on the r3s stage, the
audio cut out sometimes for us and but

because one of our translators had also
already translated the talk twice, at

least partially to because and it was
already canceled after those, they

basically knew most of the content and
could basically do a Powerpoint Karaoke

translation and was able to do most of the
talk just from the slides without any

audio. Yeah, and then there also was...
The last thing I want to say is actually I

wanted to say, give a big shout out to the
two of our team members that weren't able

to interpret with us this time because
they put their heart and soul into this

event happening. And that's stb and katti,
and that's basically everything from

c3lingo. Thanks.

ysf: muted

Hello, c3subtitles is it now. td will show
the right text to his slides you already

saw a minute ago.
td: OK. OK, hi, so I'm td from the

c3subtitles team. And next slide, please.
So just to quickly let you know how we get

from the recorded talks to the released
subtitles. Well we take the recording

videos and apply speech recognition
software to get a raw transcript. And then

Angels work on that transcript to correct
all the mistakes that the speech

recognition software makes. And we again
apply some autotiming magic to to get some

raw subtitles. And then again Angels do
quality control on these tracks to get

released subtitles. Next slide, please. So
as you can see, we have various subtitle

tracks in different stages of completion.
And these are seconds of material that we

have can see all the numbers are going up
and to the right as they should be. So

next slide, please. In total, we had 68
distinct angels that worked 4 shifts on

average. 83 percent of our angels returned
for a second shift. 10 percent of our

angels worked 12 or more shifts. And in
sum we had 382 hours of angel work for 47

hours of material. So far we've had two
releases for rc3 and hopefully more yet to

come, and 37 releases for all the
congresses, mostly on the first few days

where we didn't have many recordings. We
have 41 hours still in the transcribing

stage of material, 26 hours of material in
the timing stage and 51 hours material in

the quality control stage. So there's
still lots of work to be done. Next slide,

please. When you have transcripts, you can
do fun stuff with them. For example, you

can see that important to people in this
talk are "people". We are working on other

cool features that are yet to come. Stay
tuned for that. Next slide, please. So to

keep track of all these tasks, we've been
using a state-of-the-art high-performance

lock-free NoSQL columnar data store,
a.k.a. a kanban board in the previous

years. And because we don't have any
windows in the CCL building anymore, we

had to virtualize that. So we're using
kanban software now. At this point, I

would like to thank all our hard-working
angels for the work. And next slide

please. If you're feeling bored between
congresses then you can work on some

transcripts. Just go to c3subtitles.de. If
you're interested in our work, follow us

on Twitter. And there's also a link to the
release subtitles here. So that's all.

Thank you.
ysf: Thank you, td. And before we go into

the POC, where Drake is waiting, I'm sure
everyone is asking why are those guys

saying "next slide"? So wait. In
the end, we have the infrastructure review

of the infrastructure review meeting going
on. So be patient. Now, Drake, are you

ready in Studio 1?

Drake: OK. Hello, I'm Drake from the Phone
Operations Center, and

I like to present to you our
numbers and maybe some

anecdotes at the end of our part. So
please switch to the next slide. Let's get

into the numbers first. So first off,
first off, you registered about 1950 ...

5195 sip extensions, which is about 500
more than you registered on the last

congress. Also, you did about 21 000
calls, a little bit less than on the last

congress. But, yeah, we are still quite
proud of what you have used our system

with. And yeah, it ran quite stable. And
as you may notice on the bottom, we also

had about 23 DECT antennas at the congress
or at this event. So please switch to the

next slide. And this is our new feature,
it's called the... next slide ..., it

is called the eventphone decentralized
DECT infrastructure, which we especially

prepared for this event, the EPDDI. So we
had about 23 RFPs online throughout

Germany with 68 DECT telephones of which
is up to it. But it's not only the the

German part that we covered. We actually
had one mobile station walking out through

Austria, through Passau, I think. So
indeed we had an European Eventphone DECT

decentralized infrastructure. Next slide
please. We also have some anecdotes, so

maybe some of you have noticed that we had
a public phone, a working public phone in

the RC World where you could call other
people on the SIP telephone system and

also other people started to play with our
system. And I think about yesterday

someone started to introduce c3fire so you
could actually control a flame thrower

through our telephone system. And I like
to present here a video. Next slide

please. Maybe you can play it. I have
quite a delay in waiting for the video to

play. So what you can see here is the
c3fire system actually controlled by a

DECT telephone somewhere in Germany. So
next slide please. We also provided you

with SSTV servers via the phone
number 229, where you could receive some

pictures from event phone, like a postcard
basically. So basically you could call the

number and receive a picture or some other
pictures, some more pictures. And next

slide please. Yeah basically, that's all
from the Eventphone and with that we say

thank you all for the nice and awesome
event and yeah, bye from the first

certified assembly POC. Bye.
ysf: Thank you, POC, and hello GSM Lynxes

is waiting for us.

lynxes: Yeah, hallo, I'm lynxes, I'm from

the GSM team. This year was quite
different as you can imagine. However,

next slide please. So but we managed to
get a small network running and also a

couple of SIM cards registering, so where are
we now. So next slide please. As you can

see, we are just there in the red dot.
There's not even a single line for our

five extensions but we managed 130 calls
over five extensions. And next slide

please. So we got, so we got five
extensions registered with four SIM cards

and three locations with mixed
technologies also two users so far sadly.

And one network with more or less zero
problems. And so let's take a look on the

coverage. So next slide please. So we are
quite lucky that we managed to get an

international network running. So we got
two base stations in Berlin. One in the

hackerspace AfRA, another one north of
Berlin. And yeah one of our members is

currently in Mexico. And he's providing
the remote chaos networks there. Yes, so

that's basically our network. So before we
going to the next slide, we have what we

have done so far is, we are just two
people instead of 10 to 20 and had some

fun with improving our network and
preparing for the next congress. And next

slide please. And yeah, now I'm closing
with the EDGE computing. We improved our

EDGE capabilities and yeah, I wish you a
hopefully better year and yeah maybe see

you next year remote or in person. Have
fun.

ysf: Thanks and I give a hand to Iindworm
for doing the "slide DJ" all the time, and

he now switch to the Haecksen who are
next and they bring an image and melzai is

waiting for us in Studio 3.

melzai: Hello, what's phones without
people?

So I'm giving now an introduction
over here. How many people we needed to

run the whole Haecksen assembly. We had
around 20 organizing haecksen and we had

around 20 speakers in our events. And we
had in total around 40 events, but I'm

pretty sure that I even don`t know all of
these. As you realize, the world is pretty

large. So we needed around seven million
pixels to display the whole Haecksen

world. And that needed around 400 commits
at our github corner of the internet.

Around 130 people receive the fireplace
badge in our case. And around 100 people

tested our swimming pool and received that
badge. So great a year for non ???. Also

around 49 people showed some very deep
dedication and checked on all memorials at

our Haecksen assembly. Congratulations for
that. There were quite a many of these

ones. Our events are run on our BigBlueButton
from the Congress and so we had

starting from day 0 no lags and we're able
to host up to 133 people in one session.

And that was quite stable. We also
introduced four new members around 13 new

Haecksen joinded just for the Congress.
And we increased about to the size of 440

Haecksen overall. Also somewhat, we got new
Twitter accounts supporting us, so we have

added over 200 more Twitter accounts. And
so, you know, our messages are getting

heard. But besides the ritual, we also did
some quite physical things. First of all,

we distributed over 50 physical goodie
bags to the people with microcontrollers

and self-sewed masks in it, as you can see
on the picture. And also sadly, we shopped

so many rC3 Haecksen-themed trunks that
they are now out of stock. But they will

be back in January. Thank you.
ysf: No, thank you. And I'm going to send

thanks to the Choaspatinnen…
Chaospat*innen… who are waiting in Studio

One.
Mike: Hi, all this is Mike from the

Chaospat*innen team. We've been welcoming
new attendees and underrepresented

minorities to the chaos community for over
eight years. We match up our mentees with

experienced chaos mentors. These mentors
help their mentees navigate our world of

chaos events. DiVOC was our first remote
event and it was a good proof of concept

for rc3. This year, we had 65 amazing
mentees and mentors, two in-world

mentee/mentor matchup sessions, one great
assembly event hosted by two of our new

mentees, and a wonderful world map
assembly built with more than 1337

kilograms of multicolor pixels. Next
slide, please. And here's a small part of

our assembly with our signature propeller
hat tables. And thank you to the amazing

Chaospat*innen team: fragilant, jali,
azriel and lilafish. And to our great

mentees and mentors. We're looking forward
to meeting all of the new mentees at the

next chaos event.

lindworm: Yeah, I think that was my call.

So next up, we'll have the, let me see,
the c3adventure! Are you ready?

Roang: Hello, my name is Roang
Mewp: and I'm Mewp

Roang: and we will talk about the
c3adventure, the 2D world, and what we did

to bring it all online. Next slide please.
OK, so when we started out, we looked into

how we could bring a Congress-like
adventure to the remote experience. And on

October we started with the development
and we had some trouble in that we had

multiple upstream merges that gave us some
problems. And also due to just Congress

being Congress, or remote experience being
a remote experience, we needed to

introduce features a bit late or add
features on the first day. So auth was

merged just 4:40 AM in the first day. And
on the second day, we finally fixed the

instance jumps – you know, when you walk
from one map to the next – we had some

problems there. But on the second day it
all went up. And I hope you have all

enjoyed the badges that have finally been
updated and brought into the world today.

What does that all mean? Since we started
implementing, there have been 400 git

commits in our repository all-in-all,
including the upstream merges. But I think

the more interesting stuff is what has
been done since the whole thing went live.

We had 200 additional commits, fixing
stuff and making the experience better for

you. Next slide. In order to bring this
all online, we not only had to think about

the product itself, not only think about
the world itself, but we also had to think

about the deployment. The first commit on
the deployer, it's a background service

that brings the experience to you, has
been done on 26th of November. We started

the first instance, the first clone of the
work adventure through this deployer on

8th of December and a couple of days
beforehand, I was getting a bit swamped. I

couldn't do all of the work anymore,
because I had to coordinate both of the

projects. And so my colleague took over
for me, and helped me out a lot. So I'll

give over to him to explain what he did.
Mewp: Yeah. So imagine that on Day -5 I

get a message from a friend that, "Hey,
help is needed!" So I say, "OK, let's do

it." And Roang tells me that, "OK, so we
can spawn a instance and to scale it

somehow and do that." And I spawned the
deployer and my music stops. I streamed

music from the internet, and I wondered
why did it stop? And I have noticed that,

oh, there are a lot of logs now. Like, a
lot. And I have finally Day -4 noticed

that the deployer was spawning copies of
itself each few seconds in the log. So

that was the state back then. Since Day -4
until Day 1, we have basically written the

thing. And that's, well… Day 1 we were
ready. Well, almost ready. I mean, we have

like 14 instances deployed. And I forgot
to mention that, when we were about to

deploy 200 ones at once, it wouldn't work
because all of the things would time out.

So we patched things quickly, and 13
o'clock we had our first deployment. This

worked, and everything was fine, and…
wait… Why is everybody on one instance?

So, it turns out that we had a bug, not in
the deployer, in the app that would move

you from the lobby to the lobby on a
different map. So during the first day, we

have we've had a lot of issues of people
not seeing each other because they were on

different instances of the lobby. So we
are working hard, and… next slide, please,

so we can see that… we are working hard to
reconfigure that to bring you together in

the assembly. I think we have succeeded.
You can see the population graph on this

slide. The first day was our almost most
popular one. And the next day it would

seem, that's OK, not as popular, but we
have hit the peak of 1600 users that day.

What else about this? The most popular
instance was lobby, of course. The second

most popular instance was hardware hacking
area for a while. Then the third, I think.

Next slide please. We have counted, well,
first of all, we've had in total about 205

assemblies. The number was increase day-
by-day, because people, through the whole

congress, they were working on their maps.
For a while, CERT had over a thousand maps

active in their assembly. Which led to the
map server crashing. Some of you might

have noticed that. It stopped working
quite a few times during Day 3. And they

have reduced the number of maps to 255.
And that was fine. At the end of Day 3, I

have counted for about 628 maps, and this
is less than the, if, than was available

in reality, because it was the middle of
the night (as always), and it was it

wasn't trivial to count them. But in the
maps I have found, we have found over two

million used tiles. So that's something
you can really explore. I wish I could

have, but deploying this was also fun.
Next slide, please. And what… Yeah?

Roang: Just a quick interject. I really
want to thank everyone that has put work

into their maps and made this whole
experience work. We, we provided the

infrastructure, but you provided the fun.
And so I really want to thank everyone.

Mewp: Yeah, the more things happen on the
infrastructure, the more fun we have. We

especially don't like to sleep. So we
didn't. I basically exchanged with Roang

the way that I slept five hours and during
the night and he slept five hours in the

day. And the rest of the time, we were up.
The record, though, is incorrect. Roang is

now 30 hours up straight, because the
budgets were too important to bring to you

to go to sleep. The thing you see on this
graph is undeployed instances. We were

redeploying things constantly. Usually in
the form of redeploying half of the

infrastructure at any given time. The way
it was developed, you wouldn't have

noticed that. You wouldn't be kicked off
your instances, but for a brief period of

time you wouldn't be able to enter any
one. But… Next slide. I have been joking

for a few days at the Congress that they
have been implementing a sort of

Kubernetes thing, because it's
automatically deploy things, and manage

things, and so on. And I have noticed by
Day 3 that I have achieved true

enlightenment and true automation, because
we have decided to deploy everything at

once at some point. The reason was that we
are being DDOSed, and we had to change

something to mitigate that. And so we
did that, and everything was fine. But we

made a typo. We made a typo and the
deployment failed. And one the deployment

failed, it deleted all the servers. So,
yeah, 405 servers got deleted by what I'm

remembering was a single line. So it was
brought out automatically, and that wasn't

a problem. It was all fine, but well, to
err is human, to automate mistakes is

devops. Next slide? What's important is
that these 405 servers were provided by

Hetzner. We couldn't have done that
without their infrastructure, without

their cloud. The reason we got up so
quickly after this was that the servers

were deleted, but they could have been
reprovisioned almost instantly. So the

whole thing took like 10 minutes to get it
back up. And, next slide. That's all.

Thank you all for testing our
infrastructure, and see you next year.

ysf: Thank you, c3adventure! So this was
clearly the first conference that didn't

clap for falling mate bottles! If that's
not the thing, maybe we try next year? The

Lounge. And I know I have to ask for the
next slide too. The rc3 Lounge artists.

And I was asked to read every country
where someone is in, because everyone had

to make the Lounge what it was: an awesome
experience. So there were: Berlin, Mexico City

Honduras, London, Zürich, Stockholm,
Amsterdam, Rostock, Glasgow, Leipzig,

Santiago de Chile, Prag, Hamburg,
Mallorca, Krakow, Tokyo, Philadelphia.

Frankfurt am Main, Köln, Moscow, Taipei
Taiwan, Hannover, Shanghai, Seoul… Seoul,

I think, sorry. Vienna, Hong Kong,
Karlsruhe and Guatamala. Thank you guys

for making the Lounge. So the next is the
Hub and they should be waiting in

Studio Two.
audible echo

XXX: …software is based on Django. And

it's intended to be used for the next
event. The problem is it was a new

software. We had to do a lot of
integrations, yeah, live during Day 0.

Well, OK. No. OK, yeah, hi. I'm presenting
the Hub, which is a software we wrote for

this conference. Yeah. It's based on
different components, all of them are

based on Django. It's intended to be used
on future events as well. Our main problem

was it's a new software. We wrote it and,
yeah, a lot of the integrations were only

possible on Day 0 or Day 1. And yeah. So
even still today on Day 4, we did a lot of

updates, commits to the repository, and
even that numbers on the screens are

already outdated again. But yeah, as you
could possibly see, we have a lot of

commits all day, night, or all night long.
Only a small digit, 6 AM. I am sorry for

that. Next slide, please. And yeah,
because the numbers you're quite busy

using the platform, some of these numbers
on the screen are already outdated again.

Out of the 360 assemblies which were
registered, only 300 got accepted. Most of

them were, yeah, event or people wanting
to do a workshop and trying to register an

assembly. Or, duplicates. So, please
organize yourself. Events, currently we have over

940 in the system. You're still clicking
events, nice. Thanks for that. The events

are coordinating with the studios, so we
are integrating all of the events of all

the studios, and the individual ones, and
the self organized sessions. All of them. A new

feature, the badges. Currently you have
created 411. And, yeah, from these badges

redeemed, we have 9269 achievements and
19 000 stickers. Documentation, sadly, was

a 404, because yeah. We were really busy
doing stuff. Some documentation has

already been written, but yeah. More
documentation is, will become available

later. We will open source the whole thing
of course, but right now we're still in

production and cleaning up things. And
yeah. Finally, for some numbers. Total

requests per second were about 400. In the
night, when the world was redeploying,

then we only had about 50 requests per
second, but it maxed up to 700 requests

per second. And the authentication for the
world, for the 2D adventure, it was about

220 requests per second. More or less
stable due to some bugs and due to some

heavy usage. So, yeah, we appreciate that
you used the platform, used the new Hub,

and hope to see you on the next event.
Thanks.

ysf: Hello Hub. Thank you Hub. And the
next is betalars waiting for us. He's from

the c3auti team, and he will tell us what
he does and his team did this year.

betalars: Hi, I'm betalars from c3auti,
and we've been really busy this year as

you can probably see by the numbers on my
next slide. We have 37 confirmed Auti-Angles

and today we surpassed the 200
hours mark. We have 10 Orga Mumbles

leading up to the event and there are
almost five million unique pixels in our

repository. I'm pretty convinced we've
managed to create the smallest Fairydust

of rC3, provided by an actual space
engineer. And the Tree of Solitude is not

the only thing we've managed to create,
contribute to this wonderful experience.

On our next slide, you can see that we
also contributed six panel sessions for

autistic creatures to discuss their
experiences and five Play sessions for

them to socialize. We helped to contribute
a talk, a podcast, and an external panel

to the big streams. And on our own panels,
we've had up to 80 participants that need

to be split up to five breakout rooms so
they could all have a meaningful

discussion. And all their ideas and thoughts
were anonymized and stored on more than 1000

lines of markdown documentation that you can
find on the Internet. But 1000 lines of

markdown wouldn't be enough for me to
express the gratitude I have towards all

the amazing creatures that helped us make
this experience happen and for all the

amazing teams that worked with us. I'm so
happy to see you again soon, but now I

think I will need some solitude for
myself.

ysf: Thank you betalars. So, lindworm, are
you ready? The next one is the video, as

far as I know. It's from the C3 Inclusion
Operation Center. I don't know the short

name; C3IOC? And it's counting down three
two one go.

video without audio

So, video is like a very difficult thing to play in
those days, because we only used to do

stuff live. Live means a lot of pixels and
traffic is done from this here, from this

glass, to all the wires and cables and
back to the glass of your screen. And this

is like magic to me, somehow. Although, I.
am only. being. a robot. to talk.

synchronistically. with all the.... It's
been around enough time, I think, to

switch back to Lindy with the video. I
tell you what we are you going to…

video without audio

nwng: Hello everyone, I'm nwng from the
new C3 Inclusion Operation Center. This

year, we've been working on accessibility
guides that help the organizing teams and

assemblies improve the event for everyone,
and especially people with disabilities.

We have also worked with other teams
individually to figure out what can still

be improved in their specific range of
functions - but there are still a lot to

catch up on! Additionally, we have
published a completely free and accessible

CSS design template that features dark
mode and an accessible font selection. And

it still looks good without Javascript.
100 Internet points for that! For you

visitors, we have been collecting your
feedback through mail or twitter – and

won't stop after the Congress! If you
stumbled across some barriers, please get

in touch via c3ioc.de or @c3inclusion on
twitter to tell us about your findings!

Thanks a lot for having us.
ysf: Thank you for the video. Finally,

technical's working! We should… does
someone know computers? Maybe? Kritis is

one of them, and he is waiting in Studio
One to tell us something about C3 Yellow

or c3gelb wie wir hier sagen.

Kritis: Yeah, welcome. I'm still looking

at this hard drive. Maybe you remember
this from the very beginning? It has to be

disinfected really thoroughly, and I guess
I can take it out by the end of the event.

And for… the next slide with the words,
please. We did found roughly 0777 hands

wash options and 0x3FF waste disposal
possibilities. We checked the correct date

on almost all of the 175 disinfectant
options you had around here. And because

at a certain point of time, people from
CERT were not reachable in the CERT room

because they were running around
everywhere else in this great 2D world. We

had the chance to bypass and channel all
the information because there were two

digital cats on a digital tree. And so we
got the right help to the right option.

Next slide, please. We have a couple of
options ongoing. A lot of work had been

done before. We had all the studios with
all the corona things going on before, but

now we think we should really watch into
an angel disinfectant swimming basin for

the next time, to have there the maximum
option of cleanliness. And we will talk

with the BOC. If we can maybe achieve to
use this Globuli maxi-cubes for the

Tschunk in the upcoming time. Apart from
that, in order to get more Bachblüten and

everything else, we need someone who is
able to help us with the Potenzieren

for homoeopathic substances. So if you
feel welcome with that, please just drop

us a line to: [e-mail]. Thank you
very much and good luck.

ysf: Thank you Kritis. Finally happy to
hear your voice. I only know you from

Twitter, where we treat our stuff
together, or I yours and you, mine, don't.

Maybe you're going to change it… please?
And, talking about messages. Chaos Post

was here too, and trilader, whom we
already heard earlier, has more to say.

trilader: OK, welcome. It's me again. I've
changed outfits a bit. I'm not here for

the Signal Angels anymore, but for Chaos
Post. So, yeah. We had an online office

this year again, as we had with the DiVOCs
before. And I've got some mail numbers for

you that should be on the screen right
now. If it's not, if it's on the title

page, please switch to the first one where
it lists a lot of numbers. We had 576

messages delivered total. This is numbers
from around half to six. And 12 of them we

weren't able to deliver because, well,
non-existent mailboxes or full mailboxes

mostly. We delivered mail to 43 TLDs, the
most going to Germany, to .de domains,

followed by .com, .org, .net, and to
Austria with .at; We had a couple of

motifs you could choose from, the most
popular one was "Fairydust at Sunset", 95

people selected that. Next slide. About
our service quality. We had a minimum

delay from the message coming in, us
checking it, and it going out for about a

bit more than four seconds. The maximum
delay was about seven hours. That was

overnight, when no agents were ready, or
they were all asleep, or having… being

busy with, I don't know, the Lounge or
something? And on average a message took

you, took us 33 minutes from you putting
it into our mailbox to it getting out.

Some fun facts: We had issues delivering
to T-Online at the first two days, but we

managed to get that fixed. A different
mail provider refused our mail because it

contained the string c3world, the domain
in the mail text. And apparently new

domains are scary, and you can't trust
them or something. We had created a ticket

with them, they fixed it, and it was super
fast, super nice service. Yeah. Also, some

people tried to sent digital postcards to
Mastodon accounts because they looked like

email addresses or something. Another
thing that's not on a slide is we had

another new feature this time. That was
our named recipients. So you could, for

example, send mail to CERT without knowing
their address. And they also have a really

nice postcard wall, where you can see all
the postcards you sent them. The link for

that is on our Twitter. Thank you.
ysf: Thank you Chaos Post. lindworm, are

you there?
lindworm: Ja, ja. Ich bin da, Ich bin da.

Hallo, you're hearing me?
ysf: I hear you.

lindworm: So I have to switch some more.
It's kind of stressy for me, really.

ysf: You're doing an awesome job. Thank
you for doing it. So just out of

curiosity, and did you have a problem
accepting any cookies or so?

lindworm: No, not really.
ysf: I heard somewhere. That some really

smart people had problems using the site
because of cookies.

lindworm: Oh, no, that was not my problem.
I only couldn't use the site because of

overcrowding. That was often one of my my
little problems. And please, I hope you

don't see what I'm doing right now in the
background with starting our pets and so

on. And what I wanted to say to all of
you, this was the first Congress where we

have so many women and so many non-cis
people running that show and being up

front the camera and making everything up.
I would really thank you all. Thank you,

that you made that possible. And thank you
that we get more and more diverse, year by

year.
ysf: I can only second that. And now we

are switching to C3 Infrastructure.
lindworm: Yeah, we need to.

ysf: I'm sure a lot of questions will be
answered by them.

lindworm: And I try to make up the slides
for that, but I do not find them right

now.
patrick: Look mom, I'm on TV.

thies: Yeah. Welcome to the infrastructure
review of the Team Infrastructure. I'm not

quite sure if we have the newest revision
of the slides, cause my version of the

stream isn't loading right now. So maybe
lindworm, is it possible to press

control-R? And you're seeing a burning
computer, then we have the actual slides.

Patrick: Let's just Powerpoint Karaoke
without the background music.

thies: Yeah, and without the PowerPoint
presentation in realtime. Now I'm seeing

me. Let's wait a few seconds until we see
a slide.

Patrick: We want to wait the entire stream
delay.

thies: It's just about 30 to one minute.
Patrick: Well done.

thies: Yeah, I'm thies and I'm waiting.
And this is Patrick, and he's waiting too.

Yeah, but that's in the middle of the
slides. Can we go… OK. Yeah. I'm now

seeing something in the middle of the
slides, but it seems fine. OK, yeah. We

are the team C3 Infra. rC3 Infra. We are
creating the infrastructure. Next slide.

We had about nine terabytes of RAM and
1,700 CPU cores. The whole event there's

only one dead SSD that died because
everything's broken. We had five dead RAID

controllers, and didn't bother to replace
the RAID controllers, just replaced them

with new servers. And 100 percent uptime.
Next slide. We looked about 42 hours on

starting screens of enterprise servers. 20
minutes max is what HP delivered. And we

are now certified enterprise observers. We
had only 27%-ish of visitors using IPv6.

So that's even less than Google publishes.
And even though we had almost full IPv6

coverage – except some really, really shady
out-of-band management networks – we're

still not at the IPv6 coverage that we are
hoping for. I'm not quite sure if that's

the right slides. But I'm not quite sure
where we are in the text. Yeah, Patrick.

Patrick: Yeah, so before the Congress
there was one prediction: there's no way

it cannot be not DNS. And while it was DNS
at least once, so we checked that box. And

let's go over to the next topic, OS. We
provisioned about 300 nodes, and it was an

Ansible-powered madness. So, yeah, there
was full disk encryption on all nodes. No

IP logged in the access logs, we took
extra care of that. And we configured

minimal logging wherever possible, so the
case of some problems we only had WARNINGs

available. And there are no INFO logs, no
DEBUG logs; just the minimal logging

configuration. And with some software, we
had to pipe logs to /dev/null because the

software just wouldn't stop logging IP's,
and we didn't want that. So no personal

data in logs, so no GDPR headache, and
your data is safe with us. The Ansible

madness I've talked about was a magical
deployment that deep bootstrapped into the

live system and assimilated into the rC3
infrastructure while it's still running.

So if you didn't boot your machine then
what? They're just running. When a OS

deployment was broken, it was almost
always due to a network or routing. At

least the OS team claims that, and this
claim is disputed by the network team of

course. One time, the deployment broke
because of a trigger happy infra angel.

But let's not talk about that. Of course,
at this point, we want to announce our

great cooperation with our gold sponsor
ddos24.net, who provided an excellent

service of handcrafted request to our
infrastructure. That was a great demand or

great public demand, with a million
requests per second for a while. But even

during the highest or peak demand, we were
able to serve most of these services. We

provide the infrastructure to the VOC, and
they quickly made use of the provided

infrastructure deployed there. Overall, an
amazing time to market. We had six

locations, and those six locations where
some wildly different, special snowflakes

overall. So we had Düsseldorf, 816 CPU
cores there, two terabytes of RAM, and we

had 10 gigabits per second interconnect.
There was also a 1 terabit per second

Infiniband available, but sadly, we
couldn't use that. It would have been

nice. The machines that had a weird and
ancient IPMI, which made it hard to deploy

there. And the admin on location never
deployed bare metal hardware to a

datacenter, so there were also some
learning experience there. Fun fact about

Düsseldorf, this was the data center with
the maximum heat. One server, seven units,

over 9000 watts of power. 11.6 to be
exact. Which is why they had some to take

some creative heat management solutions.
Next was Frankfurt, there we had 620

gigabits of total uplink capacity, and we
actually only used 22 gigabit during peak

demand. Again, by our premium sponsor:
ddos24.net. There was zero network

congestion and 1.5 gigabits per second
were IPv6. So there was no real traffic

challenge. For the network engineers of
you, it was a full Layer 3 architecture

with MPLS between the WAN routers. And
there was a night shift on the 26the and

27th for more servers, because some
shipments didn't arrive yet. The fun fact

about this datacenter was the maximum
bandwidth. Some servers there had 50

gigabit uplink on the server configured.
It was the data center with the maximum

manual intervention. Of course, we had the
most infrastructure there and it wasn't

oversubscribed at any point. We had some
hardware in Stuttgart, which was basically

the easiest deployment. There were also
some night shifts, but the thanks to

neuner and team this was a really easy
deployment. It was also the most silent

DC, so no incident from Day -5 until now.
So if you're currently watching from

Stuttgart now, you can create some issues
because now we said it. Wolfsberg was the

smallest DC. We only had three servers and
we managed to kill one hardware RAID

controller, so we only could use two
servers there. So, yeah. And then Hamburg

was the data center with the minimum
uptime. We never could deploy to this data

center because there was a broken netboot
and we couldn't provision anything there.

And of course, the sixth data center was
the Hetzler Cloud, where we deployed it on

all locations. Deployment fun facts: we
received a covid warning from the data

center. Luckily, it didn't affect us. It
was at another location. But thanks for

the heads-up and the warning. The team
leader of a sponsor needed to install

Proxmox in a DC with no knowledge, without
any clue what they were doing. We

installed Proxmox in the Hamburg DC, and
no server actually wanted to talk to us,

so we had to give up on that. And there
had to be a lorry relocated before we

could deploy other servers. So that's that
was standing in the way there. Now, let's

get to Jitsi. Our peak count were 1,105
users at the same time, on the same

cluster. I don't know if it was at the
same time as the peak user count, but the

peak conference count was 204 conferences.

I hope we can still beat
that today, but this is data from

yesterday. The peak conference size was 94
participants in a single conference. And

let me give condolences to your computer,
because that must have been hard on it.

Our peak outgoing video traffic on the
Jitsi video bridges was 1.3 gigabits per

second. And we had about three quarters of
the participants were streaming video and

one quarter of them had video disabled.
Interesting ratio. Our Jitsi deployment

was completely automated with Ansible, so
it was zero to Jitsi in 15 minutes. We

broke up the Jitsi cluster into four
shards to have better scalability and

resilience. So if one shard went down, it
would only affect part of the conferences

and not all of them. Because there are
some infrastructure components that you

can't really scale or cluster, so we went
with with the sharding route. Our Jitsi

video bridges were at about 42% peak usage
– excluding our smallest video bridge,

which was only eight cores and eight
gigabytes, which we added in the beginning

to test some stuff out, and it remained in
there. And yes, we overprovisioned a bit.

There will also be a blog post on our
Jitsi Meet deployment coming in the

future. And for the next time we, for the
upcoming days, we will enable 4K streaming

on there. So why not use that? And we want
to say thanks to the FFMEET Projekt, who

contacted us after our initial load test
and gave us some tips to handle load

effectively and so on. We also tried
making DECT call-out working. Spent 48

hours trying to get it to work, but there
were some troubles there. So sadly, no

adding DECT participants to your Jitsi
conferences for now. jitsi.rc3.world will

be running over New Year. So you can use
that to get together with your friends and

so on over the New Year. Stay separate,
don't visit each other please. Don't

contribute to covid-19 spread. You've got
the alternative there. Now let's go over

to monitoring. thies.
thies: Yeah, thanks. First of all, it's

really funny how you edit this page, but
reveal.js doesn't work that way until

lindworm reloads the page, which hopefully
doesn't do right now. Everything's fine,

so you can leave it to be. Yeah,
monitoring. We had to Prometheus and

Alertmanager set up completely driven out
of our solemnly one and only source of

truth: our Netbox. We received about
34 858 critical alerts. It's – looking at

my mobile phone – it's definitely more
right now. And about 13,070 warnings. Also

definitely more right now. And we tended
about 100 of them. The rest was kind of

useless. Next slide, please. As it's
important to have an abuse hotline and an

abuse contact, we received two network
abuse messages, both from Hetzner – one of

our providers – letting us know that
someone doesn't like our infrastructure as

much as we do. Props to ddos24.net. And we
got one call it our abuse hotline, and it

was one person who wanted to buy a ticket
from us – Sadly, we were out of tickets.

Next slide, please. Some other stuff. We
got a premium Ansible deployment brought

to you by turing-complete YAML. That sounds
scary. And we had about 130k DNS updates

thanks to the World team. At this point
they're really stressing our DNS API with

the re-deployments. And also our DNS,
Prometheus, and Grafana are deployed on

and by NixOS thanks to flüpke and head
over to flüpkes interweb thingy. He wrote

some blog posts about how to deploy stuff
with his NixOS. And the next slide,

please. And the last slide from the team
is the list of our sponsors. Huge thanks

to all of them. It won't be possible to
create such a huge event and such loads of

infrastructure without them. And that's
everything we have.

ysf: Amazing. Thank you for all you've
done. Truly incredible, and showing

everything to the public. So I promised
that there will be a kind of behind the

scenes look of this infrastructure talk or
review. And I really have nothing to do

with it. Everything was done by completely
different people. I'm only a Herald,

somehow lost and tumbled into this stream.
And so I'm just going to say switch to

wherever. Show us the magic.
Karlsruhe: Three hours ago, I got the

call… Hello and welcome from the last part
of the infrastructure review and greetings

from Karlsruhe. So three hours ago, I got
a call from lindworm and he asked me, how

is it with this last talk we have? It may
be a bit complicated. And he told me, OK,

we have a speaker. I'm the Herald. Oh,
that's always so. And then we realized,

yeah, we don't have only one speaker, we
have 24. And so that's why we called

ChaosWest and built up an infrastructure
which dampfkatze will explain you now in a

short minute. I think so.
dampfkatze: Thank you. Yes. Oh, I lost the

sticker. OK, after we called ChaosWest, we
came up with this monstrosity of the video

cluster. And we start here. The teams
streamed via OBS.Ninja onto three

ChoasWest studios. They were brought
together via RTMP on our Mix1 local

studio, and then we pumped that into Mix2,
which pumped it further to the VOC. The

slides were brought in via another
OBS.Ninja directly onto Mix2. They came

from lindworm. Also, the closing you will
see shortly hopefully will also come from

there. And ysf and lindworm were directly
connected via OBS.Ninja onto our Mix1

computer. And Mix2 also has the studio
camera you're watching right now. And for

the background communication, we had a
Mumble connected with our audio matrix.

And lindworm, ysf, and the teams, and we
in the studio locally could all talk

together. And now back to the closing
with… No, to the Herald News Show, I

think. lindworm will introduce it to you.
lindworm is live.

lindworm: Is ysf still there? Or do you
come with me? So it will take a second or

billions of years. So thank you very much
for this review. It was as chaotic as the

Congress.

postroll music

Subtitles created by c3subtitles.de
in the year 2021. Join, and help us!