36C3 - 36C3 Infrastructure Review

Herald: Hello and welcome to
Infrastructure Review. This review is

being translated into a lot of languages
and we don't know yet which one, but the

c3lingo team will be on stage and will
tell us how and what it did. I'd like to

start, as always and every year, with the
NOC, right. So please give the NOC a hand.

applause
[Filler, please remove me in amara]

Momo: All right, everyone, welcome to the
State of the Internet manufacture report.

This is JC, I'm Momo and we're going to
talk to you about what we did this year

for the network. So obviously, organizing
Congress is a quite tedious task. Took us

about six months of pre-planning. We came
in on the 15th, did a fiber day and then

it took us from the 18th and will take us
till tomorrow to tear everything down with

like 20 to 40 people. And we'll be busy
wiping every device because that is

actually what we do every year. We delete
everything. There are no logs leaving this

building and this will take us probably
the next 24 hours. So yeah, for the usual

numbers. Edge capacity: this year because
you didn't use all the Internet last year,

we only brought you 300 gigs, but that was
fine as well, I suppose. We got 100 gig

from HLKomm here in Leipzig, 100 gig from
Deutsche Telekom and as well as 100 gig

from BCIX which we got via DWDM wave to
Berlin. In the core we used Juniper

MX960s, MX480s, MX204s and QFX10002 in the
yolocolo. Basically all the halls were

connected via a 200 gig link and yolocolo
had three times 100 gig. As probably the

last five or so years, we're using IS-IS
and BGP for our protocols of choice. And

this year we also first off rejected RPKI
invalid routes and secondly applied for

the first time at congress BCP38 ingress
filtering to be a good internet citizen

and not to allow you to spoof IP
addresses. So yeah, that was nice. As

Congress keeps on growing, we have 2500
tables somewhere around the building in

all the assemblies, but we only have 300
switches. So sorry if you had to bring a

long cable and if you have switches to
spare with 10 gig uplinks and POE+, feel

free to donate them to us. Access and Wi-
Fi. We had like 300 access switches. We

are obviously again running Aruba Wi-Fi
controllers. This year, like at camp we

had a few 802.1x access points, more on
that later. We tried to use Juniper vMX to

route the Wi-Fi traffic. And had quite a
shitload of switches, most of them from

Juniper, some Cisco 2960s, some Brocades
which are new to us, and some crappy old

HPE stuff which is basically configured
for us to work like a brick you get from

like eBay or whatnot. We had a few
incidents this year we'd like to talk

about. First off, we had, I'm not sure if
any of you noticed, quite a lot of packet

loss and missing router advertisements on
the Wi-Fi. This was caused by some weird

Juniper vMX behavior. We couldn't figure
out what it was. So we had them running in

a redundant VRRP setup. We shut down one
of them and then it worked. So yeah, fuck

redundancy. There was a pixelflut client
which somehow messed up his IP address and

caused a broadcast storm which took down
most parts of Hall 2. We found them, shut

it down and deployed storm control to all
our access switches. Yeah, to the Congress

motto resource exhaustion: someone was
running aggressive zmap scanning over the

whole internet, came by our Wi-Fi access
controller and caused a state table

exhaustion. And that brought it down. We
null-routed the source and yeah, there was

this issue. So thank you to whoever was
that. And in the morning of day 3, we had

another issue with Juniper vMX where it
forgot it had a network card. We rebooted

it and everything was fine again. So yeah,
some numbers. You actually managed to use

more bandwidth, thank you.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

M: But it's still only 20% of our uplink
capacity. So use more. 20% of that was

IPv6, which is good, but could be more. We
had like 11000 clients into Wi-Fi. 86% 5

GHz, 96% in a peak.
We had eleven 802.11ax clients. Our

favorite one was obviously the one with
the lovely hostname ILOVETHENOC. So yeah,

about that number we have, 96% 5 GHz
obviously shows us that we are finally at

the point where we can say: thank you 2.4
GHz, it was nice. Goodbye.

applause
[Filler, please remove me in amara]

M: Also, obviously, thank you to our
sponsors. We couldn't do this if we would

not get like 10 millions, of list price
obviously, of equipment and loan and quite

a lot of services. So give them a round of
applause as well. Thank you.

applause
[Filler, please remove me in amara]

M: And obviously NOC not only stands for
Network Operation Center, but if you

extend it, it is No CO2. So we believe in
green power and clean traffic and

therefore we obviously see that
sustainability is a great part of our

role. This is why we even use old crappy
HP switches to cut our lines for our

tchunk to serve your cheese boards,
whatever you need.

laughter
[Filler, please remove me in amara]

M: Also...
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

M: Also, we somehow estimated what our
network will run us in CO2 and that was

about 11 tons. We're not very good with
mass and not very good with CO2 emissions,

but this was roughly what we came up with.
And to make Congress or the world a bit

better place, we actually offset all our
CO2 and bought eleven tons of CO2

emissions. And now Frederick is gonna tell
you how we got all those numbers.

Frederick: Yeah, as a good internet
manufacture, we also do monitoring a lot

and we run our own Prometheus server
inside. You probably know the dashboard

that we propagate all over the internet
and that's powered by Prometheus. We have

an internal Grafana that is part of this
whole ecosystem. And if you are a little

bit of a nerd, you might have clicked on
the dashboard sections and seen that there

are more dashboards than this. We fill our
Prometheus with lots of different sources:

we get SNMP data from, and screw SNMP, but
it does a quite good job at getting all

the insights we need from all the network
equipment. We have node_exporter, influx

and all that, but we got a decent amount
of data from everyone in the Congress

ecosystem and we had that at camp as well
where we got the water pressure of the

showers. And we get the colo power, which
also helps with estimating the CO2

footprint. And everything is being
configured by Netbox, which is a tool that

is an asset database. And as I said, we
have lots of dashboards and graphs. Of

course, the public one where you can see
lots of different things from everyone.

This is only part of it. If you scroll
down on the dashboard, you see a lot more.

But this helps everyone to have a good
understanding of what is happening

currently. And we even draw a nice little
Christmas trees on the Wi-Fi traffic for

you. That is mostly because it's a router-
on-a-stick and we cannot measure it

correctly. We have an internal dashboard
which gives us a little bit of a status

for build-up mostly: which switches and
routers are up? And that gives us a very

quick sight of all the devices that are
out there. What's broken? What's not

broken? We improved it a little and now
have alarms so someone can look at stuff

and see if things are broken, run out
there and fix it. We also built

weathermaps. As you can see that's a
little bit of a mess. But we couldn't do

it better because the graphing library
doesn't allow us to do it better. If

someone has a good idea to do it better in
Grafana or anywhere else with sources from

Prometheus, please come to us, we're happy
to talk. But this shows our core and all

the links between it and how much capacity
is being used. Red indicates that it's

used more heavily. We also have that for
the yolocolo. And all the traffic around

it as well. And, yeah, teardown starts
now. Please don't touch our equipment. And

if you want to come and help, please come
to Hall 4 and get in touch. We always need

helping hands. But please in an organized
way don't disassemble switches or access

points. We have lists and everything. We
need to account for everything. So please

come to Hall 4 if you want to help. And
yes, use more bandwidth and offset more

CO2. Thank you.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

H: So thanks a lot. Actually, because
you're clearly the backbone or one of

those many backbones of the conference, is
there any Q? Let's do a Q&A for like one

or two questions.
Is anyone having a question right now?

Someone is standing up. Right, microphone
number one, please.

Q: Hey, we've absolutely don't ...
[Filler, please remove me in amara]

H: Nearer, nearer.
[Filler, please remove me in amara]

Q: We've absolutely done bcp38 in
previous years, by the way.

M: Sorry for what I said then, I'm sorry.
Q: So I wanted to correct the record.

We've been good netizens in previous years
as well.

H: Oh, you're right. Thank you. Was there
another question or something else to

correct them? Because they clearly don't
know what they're doing.

laughter
[Filler, please remove me in amara]

H: Yeah. Feel free. Microphone number two.
Q: So if 2.4GHz is over, what's going to

happen to all of the ESP32 in various IoT
devices? Are they going to have a home

here next year?
M: They'll definitely have a home. But as

usual, we cannot support it as good as we
can on 5GHz because obviously this band is

overused and not even remotely suited for
that amount of clients we put on it.

Q: A follow-up for the ESP32. How exactly
can you locate them through the wireless

if they are lost?
M: Well, we can't. We can basically look

at which access point they are, and then
if someone really would want to, we could

start triangulation, but we've never done
that before. So yeah, we can just pin them

roughly to an access point.
H: Maybe we can ask c3nav next time. So

please give the NOC a hand.
Thank you.

applause
[Filler, please remove me in amara]

H: So, the next team up is the POC. Do we
have to click this? Use more bandwidth,

I'm going to try. Ah! Thank you. Your
stage.

Garvin: Thanks.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Maria: Hi, my name is Maria.
[Filler, please remove me in amara]

G: Hi, and I'm Garvin and we're from Phone
Operation Center and we want to talk a bit

about the phone infrastructure at this
event.

M: Yeah. So we arrived at day -6 and
planned on hacking some things

and socializing and we planned a team
event, but then everything was different.

G: Yeah. When we arrived, I went into the
NOC office and they said to me, "Ja, CCL

is up, internet is up, everything is just
working nicely. You can start hook up your

telephony system right now." And I was
like, whoa. So thanks a lot NOC. Really

great performance this year, we were
amazed. Nobody expected that it works so

well.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

M: Yeah. So we put up the first antennas
and then we decided to have our team event

anyway. And yeah, so we did a lot of
things. So we handed out 150 orga loan

DECTs and we deployed 51 SIP telephones.
We also deployed 67 antennas and we had a

POC party on day three until 7:30 a.m.
G: So you can see almost everything is

done. I guess the remaining things are not
that important. So this is the overview of

DECT coverage at the event. Only level
zero, because otherwise I think it would

be overcrowded to show. Just so that you
can get a rough impression on how many

antennas we deployed in order to give you
this DECT coverage. That you can be

reached almost everywhere in the event and
that you can see how our tooling looks

like, where we see how good the antennas
see each other, and that we can see that

seamless handover work so that you can
start at our desk, walk through the area,

into the lounge and just continue talking.
And oh, there are also some antennas that

are outside of the building. What could
that be?

M: That is our hotel DECT. And you can see
a typical hotel DECT installation on the

photo. And people got really confused
about it because we also had DECT coverage

at main station.
G: Yeah. So I got a call roughly at 4:00

p.m. in the morning.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

G: And somebody told me. Oh, my DECT rang.
Why? Now I need to turn it off at the

night. Why did you do this?
M: Yeah. So we also had problems. We have

a new feature since Camp where you can see
your DECT devices and can assign them to

your number before you even arrive, and
then everything is set up and you don't

have to call your token anymore. And
people get really confused because they

would call their token anyway and it calls
it invalid. So we had to explain a lot of

this unexpected simplicity to them.
G: And then we had a battery issue this

year. We had not enough batteries and, you
know, batteries are always empty in the

phones when it's the most important. So we
were thinking what what can we do about

that?
M: So we build a new device. That's our

microwave and it can also charge devices.
So, many thanks to C3Power, because they

helped us with tooling and actually they
have expertise to put power cables on

devices like this. Thank you very much.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

G: In the last years we were often asked
"how expensive is your service actually?".

So we decided that we now provide invoices
so you can see how expensive our services

are and we send out a lot of invoices. And
we got paid some money. But as you can see

on the invoices, most is sponsored by CCC.
M: Yeah. And people also paid with Mate

which is really awesome.
G: Yeah. And also people have invoices on

a fraction of a cent and they got quite
creative on how they can pay us.

laughter
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

M: So here are some more stats. We have
7473 registered extensions.

G: Oh, we didn't remove this. So we were
thinking on how to compare this with

things and we were looking at villages in
Saarland and then we thought, this is a

stupid comparison, but we didn't remove
this.

M: So there were 5021 attached DECT phones
and 3251 concurrent DECT phones. Which is

about more than 1000 more than last year.
G: So thanks a lot for using DECT.

M: Yeah. We had...
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

M: There were also max 120 calls in
parallel, and we had more than 300

thousand calls in total. That's also
really, really much. We had five eating

meetings at heaven, the angel eating
place, and there were an average of 42

eating meeting live viewers. We had two
lectures. You can see them on media.ccc.de

and we had 23 super fast charged phones in
our microwave.

applause
[Filler, please remove me in amara]

G: Like the NOC, we also had to deal a bit
with issues during the event, and actually

there were some DDOS attack on our account
system and somebody configured over 4000

extensions with really stupid names. And
it took us quite a while to get rid of

them again cleanly from the system because
the synchronization turned out to be

really slow. So you can see it took us a
while to get them removed again. So we can

only say, you all know it's a hacker
congress, but it's kind of stupid to hack

your own infrastructure.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

G: So as a consequence of this, we only
allow now only 50 extensions per account

and per event. If you think you need more,
feel free to contact us if you have a

valid use case.
[Filler, please remove me in amara]

M: So every device has an IPEI that is
like a MAC address and we ask the

responsible institution to give us the
manufacturer. But it's really secret. So

they don't give it to us. So we ask you
for help. Please enter the models of your

devices for your phones. And then we can
match to the IPEI and get some data to

build more awesome features for you.
[Filler, please remove me in amara]

G: This would really help us. And the only
way for us is to crowdsource it because it

seems to be super secret. Whatever.
[Filler, please remove me in amara]

M: You can find it in guru3 on the device
page and there's this little pen. And if

you click on it, then you can enter the
model.

G: Thanks upfront. And yep, that's from
us, and I guess now we have a little time

for you to ask questions.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

H: Great. So, any questions for the POC? I
don't see... were they correct or do we

have to...? Ah, someone is getting up. So
microphone number 1, please.

Q: Yes, a few years ago there was a
translation service via DECT. Is the

capacity enough to service also this
crowd?

G: The problem is that we switched the
phone system a while ago last year at the

Congress. And the old phone system had a
way how we can do the translation via one

channel and the problem is that the new
system doesn't support this. Let me say

the new system doesn't support it yet. So
have a look at our talk and then you can

see that there is some potential.
H: I see someone at microphone number 3,

please. This would be the last question
because we have to hurry a bit.

Q: Can I know a little bit more about the
super charging microwave? I'm confused.

applause
[Filler, please remove me in amara]

M: Sure, you can come to our POC desk and
then we answer all your questions.

H: Ooh... mystery. Please, give the POC a
hand.

So next team, is it from the GSM crew?
Someone there? I think we have two. I see

some Chaos Post, you have to wait one
round. So, GSM guy thank you. I am good

with computers, I think. Yeah, I am. Have
fun. If it's working or not. Look at this

smile then looks better. Try this. Yeah.
Right. Try this.

- 423.
- Maybe you have to use GSM.

- 23 test.
- What happened?

- Hello. Hello.
- Aah.

GSM Person 1: So as every year we ran our
own mobile phone network at the Congress

using osmocom open source software for 2G
and 3G, and open5gs interfacing with the

osmocom HLR. And all you need to take part
is a SIM card that you can buy from the

POC and for 5 euros you get a flat rate.
The price increases because we have less

SIM cards every year. We need to
manufacture new ones. You can even call

outside like you can with DECT phones.
Lynxis: Hello. I'm Lynxis. So as every

year for the GSM team, the first problem
is the license. That's the first step

usually. Because in Germany, you have to
get the official form, get a license but

... Where do you get it? What can you ask
for frequencies? Because, for example, the

POC for DECT or Wi-Fi, you just place it
and you can use it. You're fine.

But for GSM, they didn't think about it or
for 3G or 4G. So yeah, this year we

usually get the license middle of
december, maybe start of december. So it's

already late. So this year we didn't get
all our licenses. But we get some. We got

850 MHz, which is not assigned in Europe
because it's usually in the US only. But

we have a small hole. This year we got a
4G license instead of a 2G license with 10

Mhz from Telefonica. So thanks Telefonica
for borrowing us spectrum.

applause
[Filler, please remove me in amara]

L: Just a short example of how the
spectrum looks like. The yellow stuff is

usually somewhere behind actually. By the
way, this microphone, it might share the

same frequency with us. But so far, we
haven't found any interference together

with the VOC. Down there you can see the
small hole which we are using. Because we

didn't get the 2G license there, we
thought, OK. Let's take a look. Can we fit

them both in the same frequencies? It's
not good. But you see the spikes, this

nice antenna on the right. That's the GSM,
which is sending on the same frequency. It

works because they are using different
codings. But I have heard from people who

know more about it, this is not the way
you use it.

laughing
[Filler, please remove me in amara]

L: So we took some photos of our base
stations. This is actually the fairy dust

from the 2G base station, so we have a
idea what we are using here. We have even

more fairy dust in our 3G femtocells and
our 4G cells. They are looking like small

toasters. They are taking actually 90
watts via POE, they have special POE

adapters. So maybe we could ask if
somebody can do a similar adapter to get

even running a toaster on the line.
So basically for the 4G setup this year,

we weren't sure if it's stable enough or
we lose all our phones to the LTE and they

don't like to come back to the 2G and 3G
setup where we have voice, because on LTE

we don't have yet voice so you have to
select specifically to join the LTE

network. That worked quite fine if you
change it. So your phone will register.

Everything fine there.
GSM1: So the rollout this year: we had the

voice working on day 0 which is new.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

GSM1: Someone even noticed on Mastodon, I
saw it, too. We already had LTE at the CCC

camp this year. Yeah, but unfortunately we
lost crypto password, so LTE roll out took

a bit longer this time. Sorry. So some
numbers. In total we saw just about 1100

people doing a location updating on our
network and 845 eventphone tokens were

dialed on the GSM. That is 2G or 3G. And
there were roughly 200 phones actively

subscribed on the network at all times.
And even though we basically only deployed

3G nano stations in all the halls and only
had two 2G BTS in the glass halle, there

were roughly more than half of all the
phones were still subscribed on 2G instead

of 3G. We had like 18 3G stations and only
two plus one in the GSM room 2G stations.

So that's a bit surprising. And SIM cards:
starting from the bottom, we sold about

700 SIM cards, but only saw half of them
activated, which is curious. And luckily,

most or some people bring old SIM cards
from previous years. And it's not so easy

to get cards manufactured. So we are very
glad for everyone who brings old SIM cards

from previous events. We might even
consider introducing charging phone calls

only for SIM cards that are newly bought.
So if you want to continue calling for

free, rather bring your old SIM cards.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

GSM1: So you see the numbers, I'm not
going to go through them. And you can also

get the ADM, the admin keys for your SIM
cards if you like to write to them.

L: Or if you have seen the talk from
LaForge, you might want to play with the

SIM cards, we give out all the keys you
want to have to play with it.

GSM1: So operation was mostly smooth,
except iPhones, for unknown reasons. And

except the data service, which might even
be related because maybe Apple is a bit

more strict on whether data service is
working reliably. Yeah, we still had some

problems in the SGs instance introducing
3G changing between the radio access

technologies. It's a whole new ballgame so
there are still some bugs in there. And as

you see, we had many more tickets than the
POC. This is actually reversed from the

POC, the "done" is on the left. So this
whole bunch of stuff is done and there's

some backlog and canceled and fantasy
tasks. It worked pretty nicely. Are you

taking over? Oh, yeah, no, this is still
mine. And this year we actually had also a

denial of service attack. The code was the
same as previous years and we never saw

this before. But this year we got an
invalid mobile identity which managed to

crash our mobile switching center. And
thank you very much for uncovering this

bug and thanks to fixeria for fixing it on
day 2. Ever since the mobile switching

center for Voice and SMS and subscription
has been running stable.

applause
[Filler, please remove me in amara]

L: Maybe some interference. We recovered
the old phones again. Last year we

couldn't support them. But we managed to
implement the missing parts. Old phones

could work if they support the
frequencies. So that's really nice. Maybe

next time. Since camp, we also did a nice
angel helpdesk. And it was really

impressive to see that we even had to add
more shifts in our shifts. We had so many

motivated angels. Thanks to everybody who
helped us, it was really great.

applause
[Filler, please remove me in amara]

H: Unfortunately, we don't have enough
time for a Q&A. So please give them a hand

for everything they did.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

H: Thank you. There are quite a lot of
teams next. I count at least like eight,

maybe nine. So we need to speed up a bit.
Our next team for now will be... We don't

have working microphones.
Chaos post 1: Sorry, we need to interrupt

you anyway.
H: OK. So tell them, chaos post! Chaos

post!
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Chaos post 1: Sorry, guys, we interrupt...
[Filler, please remove me in amara]

H: Let's get rid of the broken one because
it's not working anyway.

Chaos post 1: All right. Sorry, we are
interrupting for a few minutes only. We

would like to deliver a few statistics as
well. So thanks of all, we had multiple

chaos deliverers working throughout the
whole Congress 24/7 basically, delivering

at the speed of chaos, as our mission
statement clearly states. So thank you

therefore, first of all thank you very
much to all of you who contributed to

that.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Chaos post 2: So let's have some numbers,
because you all love numbers, right? So we

delivered about 3000 external postcards.
So that means with like outside chaos, so

to the real world or the default world, as
we call it. We delivered those to over 42

countries all over the world. So you guys
are really good connected internationally.

And also, we don't have exact numbers, but
we estimate around 3500, no, 35000,

internal postcards. And you also use our
online office for a total of 789 times. So

that is only 15 less than camp, and that
was a longer event. So, hey, you guys

write a lot of postcards!
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Chaos post 1: As you might already know
from camp, we also do have a few special

services. As already on camp, we had the
serving proposal, basically a pre-

assembled text, you just had to cross what
you want. The postbox certification and of

course, the bi-directional chaos, in
Germany also known as "Einschreiben mit

Rückschein".
H: Wow.

Chaos post 2: And this time we also
offered some new services. We had like

sang telegrams, gesunges Telegram in
Deutsch. We had a forever alone box for

people wanting to write postcards with or
exchange postcards with no idea who to

write it to. We had love letters, so we
had some nice pre-assembled texts and also

a really nice selection of perfumes for
scented postcards. You could write some

secret messages. We had some UV pens and
also we had some, let's say call it

security, or rather temper evidence,
because we had some scratch off stickers

for you. And also we had over 30 new
postcard designs that you could use for

postcards.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Chaos post 1: Short remark for the love
letters and the perfumes. Well, that was

kind of not really thought through. It was
a bit... it was fun sorting them out, and

stamping all of that, smelling all the
perfume all the time.

Chaos post 2: And then brushing your hands
really thoroughly because, well, that

stuff gets on your hands when you do that.
Chaos post 1: Also for UV pens. Just a

little remark. It's not a good idea to use
it for addressing and the message.

laughing
[Filler, please remove me in amara]

Chaos post 1: You can take the risk. All
right, then let me close up. We also

supported mail this year and
we had 130 letters for activists in

prison, which I find really great. I think
that's something we can all support.

Amazing.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

So thank you all for this amazing event,
and have some fun for the rest of the

Congress.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

H: Chaos post! So the next team is the
VOC. Yes, you can have it. Here, the VOC.

We only have 20 minutes left in total for
every team. So you know what to do.

VOC1: Hurry up. Welcome, guys. So. Yeah.
We don't have as many statistics as usual.

But we have some great stories too. We'll
hurry up. So this year in total, we

covered 10 stages apart from the 5 stages
that we do usually for Congress. We had

streams from the critical decentralisation
cluster, Sendezentrum, Wikipaka WG, Open

Infrastructure Orbit, Chaos West. And in
total, we served 255 hours and 35 minutes

of total talk time. So you know what to do
until the next Congress.

applause
[Filler, please remove me in amara]

VOC1: And of course, sustainability was a
big topic during this Congress. So part of

what we have to do is stream reencoding so
you can watch it with the VP5 codec or use

it at lower resolution. And so far we've
been using 4 Xeon-based machines and 2

desktop machines. And thanks to hardware-
based encoding, we now replace this with a

single laptop. As you can maybe read, this
is critical infrastructure now. And for

all the streams, for 30 reencoding
streams, we are on a 45 watt power budget

now. And as an added benefit, because we
also encode the master slides with

hardware encoders, and hardware encoders
can generally use a higher profile that

allows for better quality in real time,
you now get a better picture as well. So,

yeah.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

VOC1: And of course, we had some minor
fuckups this year. We thought the audio

setup was a bit less complex. But when we
checked rooms, we had buzzing everywhere.

So we replaced some SDI lines with fiber
and turns out buzzing goes away. Then the

PA sound console had a buggy DANTE card.
After you reboot them, the auxiliary out

to the VOC cameras was muted. So that
messed up the particular talk that has

been redone today, but other than that, we
are rather happy. And we figured that 3 of

our audio mixers actually had broken
outputs. I don't know how we did that, but

it clearly shows that they've been used on
one event or another during the last

years, which is actually a good thing we
may think. On the Wikipaka stage, we did

not use Ansible because there was sort of
a playground for us. But if you don't do

things properly, well, then you run into
edge cases with things. And yeah. So I

need to hurry up. And one virtualization
host suddenly started leaking memory. And

so if you were affected by that during the
main talk season in the evening, we are

very sorry about that. Updating the kernel
helped and we have no idea what happened.

Yeah, and Icecast got stuck as well. And
some relive... so if you want to see talks

later, that may not be possible because we
ran temporarily out of space. But if you

watch this on media.ccc.de, and the talks
were not yet released, we have relive

integration, so the talks show up in
media.ccc.de even though there is no

proper release yet, for your convenience.
And main track and assemblies are now

integrated in all events, so you don't
have to click through 4 separate events to

find your favorite talk. And now I pass to
Pat to talk about VOCTOMIX 2.

Pat: Yeah. Thank you. Okay. I have now 20
seconds, I think. I made a redesign of

VOCTOMIX, it's now called VOC2MIX. And
Peter was doing that meme some weeks ago,

because we had to switch to VOCTOMIX 2 and
we wanted to try it in 2 rooms. And in the

night from day 0 to 1 we decided to do it
for every stage because the old solution

was not working anymore. That was a little
bit hot, but it worked. And, the redesign

caused the new UI, complete with some new
base features. We have now transitions

where pictures are moving, and we have
insertions for blending text into the

picture and we have a new audio mixing and
we are now proper A/V Sync in every case,

I think.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Pat: And some mixer angels were exploring
the software and they found some bonus

features like random video distortion in
some cases, which I have to fix, I think.

And the party mode where you click some
buttons and they are clicking without any

doing of the user and everything is
flickering and they called it party mode.

This is what the current pipeline looks
like. We have now over 200 gstreamer

elements doing all this stuff to get your
pictures, which you are having on the

stream and in the recordings. That's it.
VOC1: So, of course, there were some

issues. I mean VOCTOMIX2 is essentially a
rewrite.

H: You have to speed up a bit. Actually we
are stealing time from the other teams. I

might call it a quit.
VOC1: OK OK OK. So...

H: Pressure!
VOC1: Yes, I know. But you're not making

things better. So one thing to mention: we
had to deploy a sweaty finger fix after

the first talk started. Ok, here are your
stats, read them now, read them read them

read them!
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

H: Please give VOC a hand. Thank you. So
what's next? Oh, the Stage manager and

Herald Operation Center. I'm part of it.
And we have 36 heralds, every one of them

very eloquent and good-looking. Then we
have 70 stage managers and stage

supporters. We had 150 shows on official
stages and the assemblies on top of it. We

have one stage fright council yet for the
speakers who took care of at least six

talks. And then we threw away over 100
hosting cards on day 1 only. So clearly,

give a hand for the SHOC.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

H: So next, c3power. Your time.
Arif: So my name is Arif Guy. I am from

the power team and from the radio
operation center. We make the power. We

load five double trucks of shit out on
Sunday 15th. We deploy a lot of power

boxes and many cables. So we have teardown
today so every help is needed. We have

only one day to bring us back. The main
thing, we had the power meter that we made

on the camp this year. We have two nice
setup on room H. You see the power factor

is very bad. And the other one was the
Waffle Operation Center so you see they

have a nice power factor. So please use
more ohmic devices like heaters, waffle

iron or something. We only have 5 Seaview
to monitor Yolocolo. You have on

c3power.top a Grafana link that also links
to the main Grafana to the NOC. Another

nice thing we have on the lounge, I have a
video here. It started now. You can see

the current on all 3 phases to the audio
and you can see the audio from the lounge.

recorded music from the lounge
[Filler, please remove me in amara]

Arif: It's very nice.
[Filler, please remove me in amara]

recorded music from the lounge
[Filler, please remove me in amara]

Arif: As you see, the only thing is...
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Arif: Use more bass, or make more current.
Later we can show you another slide. Go to

next slide please. So for the radio team
we have 120 portable radios. We have

updated them to a new firmware, and a new
programming software with a new feature

that is a lot slower than the last
version. Very nice. 50 bring-your-own-

device radios. 5 dead radios from the
camp. 2 dead from 36c3. 2 dead repeaters

from the camp. 1 dead on arrival on this
stage. 3 rental repeaters from 2

companies, one we picked up in Hannover
just before the Congress, and even the

windows PC crashed down dead on arrival.
Business as usual. So, next team!

H: Thank you! So, the next is c3subtitles.
I think it's Amy and Julia. Amy. They are

not. C3SOS. S.O.S. It's you. Okay. Okay.
Sorry. Oh, I'm sorry. Not the subtitles.

It's well, sustainability. Your stage, go
for it. Here, feel free.

Amy: Hello. Okay, hi. My name is Amy, I'm
part of the c3sustainability team. I only

have four slides, so I'll be very quick.
You can see, the first of our biggest

projects was the drinking water
dispensers. So for angels, we had drinking

water from a dispenser rather than from
bottles and they could refill their

bottles. We go through some stats there.
So when we started planning how difficult

it was to implement it, the locations, and
how many volunteers, satisfaction was very

high for this one. So thank you to c3geld
who really helped organize the water

dispensers with us. We're very happy. They
were really great. Please give them a

round of applause.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Amy: Thank you. The next one was a give
and take electronic box. So we wanted to

encourage people not to throw away their
electronics. Of course, they can be

recycled. So we have deployed 10 boxes.
You can find them in the sticker stations.

And we did this in collaboration with the
hardware hacking area. Thank you. And

other assemblies. Oh, wow, you are really
fast.

H: Yes I am.
Amy: Okay!

H: Others are waiting!
Amy: Okay. So the last one I wanted to go

through. We have two initiatives on this
one. I'm sorry for the trashy picture

there. We have organic bins in the halls.
It was very, very difficult to do this.

But actually it was quite satisfactory.
But I would say there is room for

improvement there. And there was also an
initiative for recycling cigaret butts. So

we actually had two people go round,
collect your cigaret butts and they will

be recycled into cool ashtrays. What a
success! Thank you so much to everyone who

collaborated with us. We couldn't do out
without your help.

H: Thank you. Your applause, please.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

H: So unfortunately, the c3sign is not
coming. So just take a look at those

pictures while I go through them. And now
the next one is C3 Assemblies. Here, take

this one. Thank you. Be quick.
Pingu: Hi, my name is Pingu. I'm here for

the assembly team. And I just want to ask,
give a hand who hasn't found his assembly

on day zero or day one? So then give a big
applause to c3nav because they really

helped us a lot. Because without them, it
wasn't able to do this event just for us.

Because for some figures we had 419
assemblies to place on this area which is

about 35000 square meters. We had 3000
tables all over, 2500 in the assemblies,

with 6000 chairs. And here, please give a
big applause to the C3Möbelhaus or IKEA as

you call it.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Pingu: Because the Möbelhaus basically
placed all the tables in a magic night on

basically day -2. And they will disappear
with the help of C3Möbelhaus today. We

assigned the last assembly on day zero at
22:00 and we started our work in mid-

October with weekly meetings. And yeah, as
you can see, it was a lot of work and...

Oh.
H: Thank you.

Pingu: Thank you. And just one thing, for
teardown, for tearing down the assemblies,

please stack the chairs on the assembly
but leave every chair and every table on

the assembly, we will get rid of them.
H: Thanks a lot. Your applause. So the

next ones are... Les prochains, ce sont
les gens de c3lingo, voilà, vos tours!

c3lingo 1: Hallo! Schön euch mal (...) Wir
sind übersetzet. (...)

German>
Oh, sorry. Well, I'm also fine with that.

We translated all the German talks into
English. All the English talks into

German. All in all, fifteen thousand
minutes in seven different target

languages. And here is from the second
language team, who talk all the non-

English and non-German things.
c3lingo 2: Okay. So basically we did have

about 1/3 of the talks were translated to
French. Yeah. You can read the rest. And

we even have currently, exactly, right
now, another one which is translated into

Swabian. So if you want to listen into it.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

c3lingo 2: Which means that if you were
listening to a talk, there is 2 chance out

of 3 that it was translated not only once,
but twice. Into either French, Spanish,

Russian, or Polish. One special mention
for the Russian and Polish teams, that was

their first time this year. So one big
round of applause for them, please.

applause
[Filler, please remove me in amara]

c3lingo 2: And one more big thank about
for the guys who brought the cough candies

and cough drops in the booth. That's a
lifesaver. Thanks.

H: Thank you. Merci beaucoup. So we have
some heart operation going on here. You

have to switch it or not? Oh, you're good
with computers.

laughing
[Filler, please remove me in amara]

H: He just mimicks something. So
c3infrastructure from the subtitles. Your

stage. Yeah. My microphone. Ha ha.
td: So, thanks. Just a quick look into the

subtitles. So what do we actually do when
we subtitle a talk? Well, first of all, we

take the video from the C3VOC and put it
through speech recognition just to get a

rough transcript that we can then give to
angels to actually correct, because, well,

machine speech recognition doesn't work so
good at all. And then once we have a

working transcript that humans have looked
at, we put that through auto-timing, which

just takes the transcript and aligns that
with the audio, and that usually works

pretty well. And, well, once that is
finished, then we actually have working

subtitles, but we give them to angels for
another round of review just to fix any

mistakes that got overlooked. And maybe
sometimes the timing needs to be adjusted.

And then, when that is done, well, the
subtitle is released. And actually one of

our angels did a nice chart about all this
process that you can see here. It all

sounds better, so thank you for that.
Well, no presentation without graphs. As

you can see, the important thing is
really: everything goes up and to the

right. On the bottom here we have finished
seconds of transcribed talks. So this is

really completed subtitles here. And it
starts already quite high because it

includes all the Congresses before. Then
we have stuff that has been reviewed in

orange. Stuff that has been timed but not
reviewed yet in yellow. And transcribed

but not yet timed stuff in green. All in
all, we had 144 distinct angels. Yeah, I

need to hurry up. 71% of which took 2
shifts and 10% took 7 or more shifts. So

433 hours of work for 126 hours of
material. And so far we've had 6 releases

from this Congress and then lots of hours
worked. All of these numbers are at least

as high as last year's numbers, so good,
thanks. When you have transcripts, you can

do cool stuff with that. So, for example,
generate word clouds to see what people

like. And this case, people seem to like
people and questions and time, which we

don't have any here. So. Well, how do we
actually keep track of all this

complicated stuff? Well, we use a state-
of-the-art NoSQL lock-free columnar data

store, like many of the other teams also
do. And well, thank you all of our angels

for your hard work. And also thanks for
the Heaven for supporting us. And then,

well, if you if you feel bored between
Congresses, you can still continue to work

on transcripts. You have all the
informations here, these slides will be

online, follow us on Twitter. And thank
you.

applause
[Filler, please remove me in amara]

H: Please do your thing again, like your
thing again, like do *mimicks

something*. Take your laptop.
[Filler, please remove me in amara]

laughing
[Filler, please remove me in amara]

H: So this was the last one. I think I
will try my best to do something like this

too. Actually I'm not good with computers,
but I know someone who is and who takes

very great care of everyone of us. So one
of my highlights of every Congress, feel

free, the LOC!
LOC: Hello. I'm the stand-in for LOC. As

with all good projects, they're too busy
for documentation. They're packing. So LOC

doesn't have anything. I'm more of the
Department of Health and Safety again. So

for everyone: we have the message from
CERT that there were no work-related

accidents that caused real harm. The odd
broken Mate bottle maybe. But thank you

for having built a city again safely and
orderly, even with all the chaos. For the

people who are driving, please make sure
that the drugs wear off and that you get

some sleep and for all people riding
along, please keep the guys awake. That

would be greatly appreciated. Go home
safely. Thank you very much.

applause
[Filler, please remove me in amara]

H: So. Wow, ah, OK. So I'm very sorry to
have to rush some teams and I'm very sorry

that we don't have any time left. We are 1
minute over and I promised the teardown

crew to not do overtime. So please, please
give all the teams their respect and clap

and tramp as loud as you can for now to
finish!

standing ovation
[Filler, please remove me in amara]

H: Everything! They did! For us! We did!
For them! Why are you still sitting? You

have to leave. Thank you. From the
c3infrastructure Review. Goodbye.

Subtitles created by many many volunteers and
the c3subtitles.de team. Join us, and help us!