Herald: Hello and welcome to
Infrastructure Review. This review is
being translated into a lot of languages
and we don't know yet which one, but the
c3lingo team will be on stage and will
tell us how and what it did. I'd like to
start, as always and every year, with the
NOC, right. So please give the NOC a hand.
*applause*
[Filler, please remove me in amara]
Momo: All right, everyone, welcome to the
State of the Internet manufacture report.
This is JC, I'm Momo and we're going to
talk to you about what we did this year
for the network. So obviously, organizing
Congress is a quite tedious task. Took us
about six months of pre-planning. We came
in on the 15th, did a fiber day and then
it took us from the 18th and will take us
till tomorrow to tear everything down with
like 20 to 40 people. And we'll be busy
wiping every device because that is
actually what we do every year. We delete
everything. There are no logs leaving this
building and this will take us probably
the next 24 hours. So yeah, for the usual
numbers. Edge capacity: this year because
you didn't use all the Internet last year,
we only brought you 300 gigs, but that was
fine as well, I suppose. We got 100 gig
from HLKomm here in Leipzig, 100 gig from
Deutsche Telekom and as well as 100 gig
from BCIX which we got via DWDM wave to
Berlin. In the core we used Juniper
MX960s, MX480s, MX204s and QFX10002 in the
yolocolo. Basically all the halls were
connected via a 200 gig link and yolocolo
had three times 100 gig. As probably the
last five or so years, we're using IS-IS
and BGP for our protocols of choice. And
this year we also first off rejected RPKI
invalid routes and secondly applied for
the first time at congress BCP38 ingress
filtering to be a good internet citizen
and not to allow you to spoof IP
addresses. So yeah, that was nice. As
Congress keeps on growing, we have 2500
tables somewhere around the building in
all the assemblies, but we only have 300
switches. So sorry if you had to bring a
long cable and if you have switches to
spare with 10 gig uplinks and POE+, feel
free to donate them to us. Access and Wi-
Fi. We had like 300 access switches. We
are obviously again running Aruba Wi-Fi
controllers. This year, like at camp we
had a few 802.1x access points, more on
that later. We tried to use Juniper vMX to
route the Wi-Fi traffic. And had quite a
shitload of switches, most of them from
Juniper, some Cisco 2960s, some Brocades
which are new to us, and some crappy old
HPE stuff which is basically configured
for us to work like a brick you get from
like eBay or whatnot. We had a few
incidents this year we'd like to talk
about. First off, we had, I'm not sure if
any of you noticed, quite a lot of packet
loss and missing router advertisements on
the Wi-Fi. This was caused by some weird
Juniper vMX behavior. We couldn't figure
out what it was. So we had them running in
a redundant VRRP setup. We shut down one
of them and then it worked. So yeah, fuck
redundancy. There was a pixelflut client
which somehow messed up his IP address and
caused a broadcast storm which took down
most parts of Hall 2. We found them, shut
it down and deployed storm control to all
our access switches. Yeah, to the Congress
motto resource exhaustion: someone was
running aggressive zmap scanning over the
whole internet, came by our Wi-Fi access
controller and caused a state table
exhaustion. And that brought it down. We
null-routed the source and yeah, there was
this issue. So thank you to whoever was
that. And in the morning of day 3, we had
another issue with Juniper vMX where it
forgot it had a network card. We rebooted
it and everything was fine again. So yeah,
some numbers. You actually managed to use
more bandwidth, thank you.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
M: But it's still only 20% of our uplink
capacity. So use more. 20% of that was
IPv6, which is good, but could be more. We
had like 11000 clients into Wi-Fi. 86% 5
GHz, 96% in a peak.
We had eleven 802.11ax clients. Our
favorite one was obviously the one with
the lovely hostname ILOVETHENOC. So yeah,
about that number we have, 96% 5 GHz
obviously shows us that we are finally at
the point where we can say: thank you 2.4
GHz, it was nice. Goodbye.
*applause*
[Filler, please remove me in amara]
M: Also, obviously, thank you to our
sponsors. We couldn't do this if we would
not get like 10 millions, of list price
obviously, of equipment and loan and quite
a lot of services. So give them a round of
applause as well. Thank you.
*applause*
[Filler, please remove me in amara]
M: And obviously NOC not only stands for
Network Operation Center, but if you
extend it, it is No CO2. So we believe in
green power and clean traffic and
therefore we obviously see that
sustainability is a great part of our
role. This is why we even use old crappy
HP switches to cut our lines for our
tchunk to serve your cheese boards,
whatever you need.
*laughter*
[Filler, please remove me in amara]
M: Also...
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
M: Also, we somehow estimated what our
network will run us in CO2 and that was
about 11 tons. We're not very good with
mass and not very good with CO2 emissions,
but this was roughly what we came up with.
And to make Congress or the world a bit
better place, we actually offset all our
CO2 and bought eleven tons of CO2
emissions. And now Frederick is gonna tell
you how we got all those numbers.
Frederick: Yeah, as a good internet
manufacture, we also do monitoring a lot
and we run our own Prometheus server
inside. You probably know the dashboard
that we propagate all over the internet
and that's powered by Prometheus. We have
an internal Grafana that is part of this
whole ecosystem. And if you are a little
bit of a nerd, you might have clicked on
the dashboard sections and seen that there
are more dashboards than this. We fill our
Prometheus with lots of different sources:
we get SNMP data from, and screw SNMP, but
it does a quite good job at getting all
the insights we need from all the network
equipment. We have node_exporter, influx
and all that, but we got a decent amount
of data from everyone in the Congress
ecosystem and we had that at camp as well
where we got the water pressure of the
showers. And we get the colo power, which
also helps with estimating the CO2
footprint. And everything is being
configured by Netbox, which is a tool that
is an asset database. And as I said, we
have lots of dashboards and graphs. Of
course, the public one where you can see
lots of different things from everyone.
This is only part of it. If you scroll
down on the dashboard, you see a lot more.
But this helps everyone to have a good
understanding of what is happening
currently. And we even draw a nice little
Christmas trees on the Wi-Fi traffic for
you. That is mostly because it's a router-
on-a-stick and we cannot measure it
correctly. We have an internal dashboard
which gives us a little bit of a status
for build-up mostly: which switches and
routers are up? And that gives us a very
quick sight of all the devices that are
out there. What's broken? What's not
broken? We improved it a little and now
have alarms so someone can look at stuff
and see if things are broken, run out
there and fix it. We also built
weathermaps. As you can see that's a
little bit of a mess. But we couldn't do
it better because the graphing library
doesn't allow us to do it better. If
someone has a good idea to do it better in
Grafana or anywhere else with sources from
Prometheus, please come to us, we're happy
to talk. But this shows our core and all
the links between it and how much capacity
is being used. Red indicates that it's
used more heavily. We also have that for
the yolocolo. And all the traffic around
it as well. And, yeah, teardown starts
now. Please don't touch our equipment. And
if you want to come and help, please come
to Hall 4 and get in touch. We always need
helping hands. But please in an organized
way don't disassemble switches or access
points. We have lists and everything. We
need to account for everything. So please
come to Hall 4 if you want to help. And
yes, use more bandwidth and offset more
CO2. Thank you.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
H: So thanks a lot. Actually, because
you're clearly the backbone or one of
those many backbones of the conference, is
there any Q? Let's do a Q&A for like one
or two questions.
Is anyone having a question right now?
Someone is standing up. Right, microphone
number one, please.
Q: Hey, we've absolutely don't ...
[Filler, please remove me in amara]
H: Nearer, nearer.
[Filler, please remove me in amara]
Q: We've absolutely done bcp38 in
previous years, by the way.
M: Sorry for what I said then, I'm sorry.
Q: So I wanted to correct the record.
We've been good netizens in previous years
as well.
H: Oh, you're right. Thank you. Was there
another question or something else to
correct them? Because they clearly don't
know what they're doing.
*laughter*
[Filler, please remove me in amara]
H: Yeah. Feel free. Microphone number two.
Q: So if 2.4GHz is over, what's going to
happen to all of the ESP32 in various IoT
devices? Are they going to have a home
here next year?
M: They'll definitely have a home. But as
usual, we cannot support it as good as we
can on 5GHz because obviously this band is
overused and not even remotely suited for
that amount of clients we put on it.
Q: A follow-up for the ESP32. How exactly
can you locate them through the wireless
if they are lost?
M: Well, we can't. We can basically look
at which access point they are, and then
if someone really would want to, we could
start triangulation, but we've never done
that before. So yeah, we can just pin them
roughly to an access point.
H: Maybe we can ask c3nav next time. So
please give the NOC a hand.
Thank you.
*applause*
[Filler, please remove me in amara]
H: So, the next team up is the POC. Do we
have to click this? Use more bandwidth,
I'm going to try. Ah! Thank you. Your
stage.
Garvin: Thanks.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
Maria: Hi, my name is Maria.
[Filler, please remove me in amara]
G: Hi, and I'm Garvin and we're from Phone
Operation Center and we want to talk a bit
about the phone infrastructure at this
event.
M: Yeah. So we arrived at day -6 and
planned on hacking some things
and socializing and we planned a team
event, but then everything was different.
G: Yeah. When we arrived, I went into the
NOC office and they said to me, "Ja, CCL
is up, internet is up, everything is just
working nicely. You can start hook up your
telephony system right now." And I was
like, whoa. So thanks a lot NOC. Really
great performance this year, we were
amazed. Nobody expected that it works so
well.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
M: Yeah. So we put up the first antennas
and then we decided to have our team event
anyway. And yeah, so we did a lot of
things. So we handed out 150 orga loan
DECTs and we deployed 51 SIP telephones.
We also deployed 67 antennas and we had a
POC party on day three until 7:30 a.m.
G: So you can see almost everything is
done. I guess the remaining things are not
that important. So this is the overview of
DECT coverage at the event. Only level
zero, because otherwise I think it would
be overcrowded to show. Just so that you
can get a rough impression on how many
antennas we deployed in order to give you
this DECT coverage. That you can be
reached almost everywhere in the event and
that you can see how our tooling looks
like, where we see how good the antennas
see each other, and that we can see that
seamless handover work so that you can
start at our desk, walk through the area,
into the lounge and just continue talking.
And oh, there are also some antennas that
are outside of the building. What could
that be?
M: That is our hotel DECT. And you can see
a typical hotel DECT installation on the
photo. And people got really confused
about it because we also had DECT coverage
at main station.
G: Yeah. So I got a call roughly at 4:00
p.m. in the morning.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
G: And somebody told me. Oh, my DECT rang.
Why? Now I need to turn it off at the
night. Why did you do this?
M: Yeah. So we also had problems. We have
a new feature since Camp where you can see
your DECT devices and can assign them to
your number before you even arrive, and
then everything is set up and you don't
have to call your token anymore. And
people get really confused because they
would call their token anyway and it calls
it invalid. So we had to explain a lot of
this unexpected simplicity to them.
G: And then we had a battery issue this
year. We had not enough batteries and, you
know, batteries are always empty in the
phones when it's the most important. So we
were thinking what what can we do about
that?
M: So we build a new device. That's our
microwave and it can also charge devices.
So, many thanks to C3Power, because they
helped us with tooling and actually they
have expertise to put power cables on
devices like this. Thank you very much.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
G: In the last years we were often asked
"how expensive is your service actually?".
So we decided that we now provide invoices
so you can see how expensive our services
are and we send out a lot of invoices. And
we got paid some money. But as you can see
on the invoices, most is sponsored by CCC.
M: Yeah. And people also paid with Mate
which is really awesome.
G: Yeah. And also people have invoices on
a fraction of a cent and they got quite
creative on how they can pay us.
*laughter*
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
M: So here are some more stats. We have
7473 registered extensions.
G: Oh, we didn't remove this. So we were
thinking on how to compare this with
things and we were looking at villages in
Saarland and then we thought, this is a
stupid comparison, but we didn't remove
this.
M: So there were 5021 attached DECT phones
and 3251 concurrent DECT phones. Which is
about more than 1000 more than last year.
G: So thanks a lot for using DECT.
M: Yeah. We had...
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
M: There were also max 120 calls in
parallel, and we had more than 300
thousand calls in total. That's also
really, really much. We had five eating
meetings at heaven, the angel eating
place, and there were an average of 42
eating meeting live viewers. We had two
lectures. You can see them on media.ccc.de
and we had 23 super fast charged phones in
our microwave.
*applause*
[Filler, please remove me in amara]
G: Like the NOC, we also had to deal a bit
with issues during the event, and actually
there were some DDOS attack on our account
system and somebody configured over 4000
extensions with really stupid names. And
it took us quite a while to get rid of
them again cleanly from the system because
the synchronization turned out to be
really slow. So you can see it took us a
while to get them removed again. So we can
only say, you all know it's a hacker
congress, but it's kind of stupid to hack
your own infrastructure.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
G: So as a consequence of this, we only
allow now only 50 extensions per account
and per event. If you think you need more,
feel free to contact us if you have a
valid use case.
[Filler, please remove me in amara]
M: So every device has an IPEI that is
like a MAC address and we ask the
responsible institution to give us the
manufacturer. But it's really secret. So
they don't give it to us. So we ask you
for help. Please enter the models of your
devices for your phones. And then we can
match to the IPEI and get some data to
build more awesome features for you.
[Filler, please remove me in amara]
G: This would really help us. And the only
way for us is to crowdsource it because it
seems to be super secret. Whatever.
[Filler, please remove me in amara]
M: You can find it in guru3 on the device
page and there's this little pen. And if
you click on it, then you can enter the
model.
G: Thanks upfront. And yep, that's from
us, and I guess now we have a little time
for you to ask questions.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
H: Great. So, any questions for the POC? I
don't see... were they correct or do we
have to...? Ah, someone is getting up. So
microphone number 1, please.
Q: Yes, a few years ago there was a
translation service via DECT. Is the
capacity enough to service also this
crowd?
G: The problem is that we switched the
phone system a while ago last year at the
Congress. And the old phone system had a
way how we can do the translation via one
channel and the problem is that the new
system doesn't support this. Let me say
the new system doesn't support it yet. So
have a look at our talk and then you can
see that there is some potential.
H: I see someone at microphone number 3,
please. This would be the last question
because we have to hurry a bit.
Q: Can I know a little bit more about the
super charging microwave? I'm confused.
*applause*
[Filler, please remove me in amara]
M: Sure, you can come to our POC desk and
then we answer all your questions.
H: Ooh... mystery. Please, give the POC a
hand.
So next team, is it from the GSM crew?
Someone there? I think we have two. I see
some Chaos Post, you have to wait one
round. So, GSM guy thank you. I am good
with computers, I think. Yeah, I am. Have
fun. If it's working or not. Look at this
smile then looks better. Try this. Yeah.
Right. Try this.
- 423.
- Maybe you have to use GSM.
- 23 test.
- What happened?
- Hello. Hello.
- Aah.
GSM Person 1: So as every year we ran our
own mobile phone network at the Congress
using osmocom open source software for 2G
and 3G, and open5gs interfacing with the
osmocom HLR. And all you need to take part
is a SIM card that you can buy from the
POC and for 5 euros you get a flat rate.
The price increases because we have less
SIM cards every year. We need to
manufacture new ones. You can even call
outside like you can with DECT phones.
Lynxis: Hello. I'm Lynxis. So as every
year for the GSM team, the first problem
is the license. That's the first step
usually. Because in Germany, you have to
get the official form, get a license but
... Where do you get it? What can you ask
for frequencies? Because, for example, the
POC for DECT or Wi-Fi, you just place it
and you can use it. You're fine.
But for GSM, they didn't think about it or
for 3G or 4G. So yeah, this year we
usually get the license middle of
december, maybe start of december. So it's
already late. So this year we didn't get
all our licenses. But we get some. We got
850 MHz, which is not assigned in Europe
because it's usually in the US only. But
we have a small hole. This year we got a
4G license instead of a 2G license with 10
Mhz from Telefonica. So thanks Telefonica
for borrowing us spectrum.
*applause*
[Filler, please remove me in amara]
L: Just a short example of how the
spectrum looks like. The yellow stuff is
usually somewhere behind actually. By the
way, this microphone, it might share the
same frequency with us. But so far, we
haven't found any interference together
with the VOC. Down there you can see the
small hole which we are using. Because we
didn't get the 2G license there, we
thought, OK. Let's take a look. Can we fit
them both in the same frequencies? It's
not good. But you see the spikes, this
nice antenna on the right. That's the GSM,
which is sending on the same frequency. It
works because they are using different
codings. But I have heard from people who
know more about it, this is not the way
you use it.
*laughing*
[Filler, please remove me in amara]
L: So we took some photos of our base
stations. This is actually the fairy dust
from the 2G base station, so we have a
idea what we are using here. We have even
more fairy dust in our 3G femtocells and
our 4G cells. They are looking like small
toasters. They are taking actually 90
watts via POE, they have special POE
adapters. So maybe we could ask if
somebody can do a similar adapter to get
even running a toaster on the line.
So basically for the 4G setup this year,
we weren't sure if it's stable enough or
we lose all our phones to the LTE and they
don't like to come back to the 2G and 3G
setup where we have voice, because on LTE
we don't have yet voice so you have to
select specifically to join the LTE
network. That worked quite fine if you
change it. So your phone will register.
Everything fine there.
GSM1: So the rollout this year: we had the
voice working on day 0 which is new.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
GSM1: Someone even noticed on Mastodon, I
saw it, too. We already had LTE at the CCC
camp this year. Yeah, but unfortunately we
lost crypto password, so LTE roll out took
a bit longer this time. Sorry. So some
numbers. In total we saw just about 1100
people doing a location updating on our
network and 845 eventphone tokens were
dialed on the GSM. That is 2G or 3G. And
there were roughly 200 phones actively
subscribed on the network at all times.
And even though we basically only deployed
3G nano stations in all the halls and only
had two 2G BTS in the glass halle, there
were roughly more than half of all the
phones were still subscribed on 2G instead
of 3G. We had like 18 3G stations and only
two plus one in the GSM room 2G stations.
So that's a bit surprising. And SIM cards:
starting from the bottom, we sold about
700 SIM cards, but only saw half of them
activated, which is curious. And luckily,
most or some people bring old SIM cards
from previous years. And it's not so easy
to get cards manufactured. So we are very
glad for everyone who brings old SIM cards
from previous events. We might even
consider introducing charging phone calls
only for SIM cards that are newly bought.
So if you want to continue calling for
free, rather bring your old SIM cards.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
GSM1: So you see the numbers, I'm not
going to go through them. And you can also
get the ADM, the admin keys for your SIM
cards if you like to write to them.
L: Or if you have seen the talk from
LaForge, you might want to play with the
SIM cards, we give out all the keys you
want to have to play with it.
GSM1: So operation was mostly smooth,
except iPhones, for unknown reasons. And
except the data service, which might even
be related because maybe Apple is a bit
more strict on whether data service is
working reliably. Yeah, we still had some
problems in the SGs instance introducing
3G changing between the radio access
technologies. It's a whole new ballgame so
there are still some bugs in there. And as
you see, we had many more tickets than the
POC. This is actually reversed from the
POC, the "done" is on the left. So this
whole bunch of stuff is done and there's
some backlog and canceled and fantasy
tasks. It worked pretty nicely. Are you
taking over? Oh, yeah, no, this is still
mine. And this year we actually had also a
denial of service attack. The code was the
same as previous years and we never saw
this before. But this year we got an
invalid mobile identity which managed to
crash our mobile switching center. And
thank you very much for uncovering this
bug and thanks to fixeria for fixing it on
day 2. Ever since the mobile switching
center for Voice and SMS and subscription
has been running stable.
*applause*
[Filler, please remove me in amara]
L: Maybe some interference. We recovered
the old phones again. Last year we
couldn't support them. But we managed to
implement the missing parts. Old phones
could work if they support the
frequencies. So that's really nice. Maybe
next time. Since camp, we also did a nice
angel helpdesk. And it was really
impressive to see that we even had to add
more shifts in our shifts. We had so many
motivated angels. Thanks to everybody who
helped us, it was really great.
*applause*
[Filler, please remove me in amara]
H: Unfortunately, we don't have enough
time for a Q&A. So please give them a hand
for everything they did.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
H: Thank you. There are quite a lot of
teams next. I count at least like eight,
maybe nine. So we need to speed up a bit.
Our next team for now will be... We don't
have working microphones.
Chaos post 1: Sorry, we need to interrupt
you anyway.
H: OK. So tell them, chaos post! Chaos
post!
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
Chaos post 1: Sorry, guys, we interrupt...
[Filler, please remove me in amara]
H: Let's get rid of the broken one because
it's not working anyway.
Chaos post 1: All right. Sorry, we are
interrupting for a few minutes only. We
would like to deliver a few statistics as
well. So thanks of all, we had multiple
chaos deliverers working throughout the
whole Congress 24/7 basically, delivering
at the speed of chaos, as our mission
statement clearly states. So thank you
therefore, first of all thank you very
much to all of you who contributed to
that.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
Chaos post 2: So let's have some numbers,
because you all love numbers, right? So we
delivered about 3000 external postcards.
So that means with like outside chaos, so
to the real world or the default world, as
we call it. We delivered those to over 42
countries all over the world. So you guys
are really good connected internationally.
And also, we don't have exact numbers, but
we estimate around 3500, no, 35000,
internal postcards. And you also use our
online office for a total of 789 times. So
that is only 15 less than camp, and that
was a longer event. So, hey, you guys
write a lot of postcards!
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
Chaos post 1: As you might already know
from camp, we also do have a few special
services. As already on camp, we had the
serving proposal, basically a pre-
assembled text, you just had to cross what
you want. The postbox certification and of
course, the bi-directional chaos, in
Germany also known as "Einschreiben mit
Rückschein".
H: Wow.
Chaos post 2: And this time we also
offered some new services. We had like
sang telegrams, gesunges Telegram in
Deutsch. We had a forever alone box for
people wanting to write postcards with or
exchange postcards with no idea who to
write it to. We had love letters, so we
had some nice pre-assembled texts and also
a really nice selection of perfumes for
scented postcards. You could write some
secret messages. We had some UV pens and
also we had some, let's say call it
security, or rather temper evidence,
because we had some scratch off stickers
for you. And also we had over 30 new
postcard designs that you could use for
postcards.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
Chaos post 1: Short remark for the love
letters and the perfumes. Well, that was
kind of not really thought through. It was
a bit... it was fun sorting them out, and
stamping all of that, smelling all the
perfume all the time.
Chaos post 2: And then brushing your hands
really thoroughly because, well, that
stuff gets on your hands when you do that.
Chaos post 1: Also for UV pens. Just a
little remark. It's not a good idea to use
it for addressing and the message.
*laughing*
[Filler, please remove me in amara]
Chaos post 1: You can take the risk. All
right, then let me close up. We also
supported mail this year and
we had 130 letters for activists in
prison, which I find really great. I think
that's something we can all support.
Amazing.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
So thank you all for this amazing event,
and have some fun for the rest of the
Congress.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
H: Chaos post! So the next team is the
VOC. Yes, you can have it. Here, the VOC.
We only have 20 minutes left in total for
every team. So you know what to do.
VOC1: Hurry up. Welcome, guys. So. Yeah.
We don't have as many statistics as usual.
But we have some great stories too. We'll
hurry up. So this year in total, we
covered 10 stages apart from the 5 stages
that we do usually for Congress. We had
streams from the critical decentralisation
cluster, Sendezentrum, Wikipaka WG, Open
Infrastructure Orbit, Chaos West. And in
total, we served 255 hours and 35 minutes
of total talk time. So you know what to do
until the next Congress.
*applause*
[Filler, please remove me in amara]
VOC1: And of course, sustainability was a
big topic during this Congress. So part of
what we have to do is stream reencoding so
you can watch it with the VP5 codec or use
it at lower resolution. And so far we've
been using 4 Xeon-based machines and 2
desktop machines. And thanks to hardware-
based encoding, we now replace this with a
single laptop. As you can maybe read, this
is critical infrastructure now. And for
all the streams, for 30 reencoding
streams, we are on a 45 watt power budget
now. And as an added benefit, because we
also encode the master slides with
hardware encoders, and hardware encoders
can generally use a higher profile that
allows for better quality in real time,
you now get a better picture as well. So,
yeah.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
VOC1: And of course, we had some minor
fuckups this year. We thought the audio
setup was a bit less complex. But when we
checked rooms, we had buzzing everywhere.
So we replaced some SDI lines with fiber
and turns out buzzing goes away. Then the
PA sound console had a buggy DANTE card.
After you reboot them, the auxiliary out
to the VOC cameras was muted. So that
messed up the particular talk that has
been redone today, but other than that, we
are rather happy. And we figured that 3 of
our audio mixers actually had broken
outputs. I don't know how we did that, but
it clearly shows that they've been used on
one event or another during the last
years, which is actually a good thing we
may think. On the Wikipaka stage, we did
not use Ansible because there was sort of
a playground for us. But if you don't do
things properly, well, then you run into
edge cases with things. And yeah. So I
need to hurry up. And one virtualization
host suddenly started leaking memory. And
so if you were affected by that during the
main talk season in the evening, we are
very sorry about that. Updating the kernel
helped and we have no idea what happened.
Yeah, and Icecast got stuck as well. And
some relive... so if you want to see talks
later, that may not be possible because we
ran temporarily out of space. But if you
watch this on media.ccc.de, and the talks
were not yet released, we have relive
integration, so the talks show up in
media.ccc.de even though there is no
proper release yet, for your convenience.
And main track and assemblies are now
integrated in all events, so you don't
have to click through 4 separate events to
find your favorite talk. And now I pass to
Pat to talk about VOCTOMIX 2.
Pat: Yeah. Thank you. Okay. I have now 20
seconds, I think. I made a redesign of
VOCTOMIX, it's now called VOC2MIX. And
Peter was doing that meme some weeks ago,
because we had to switch to VOCTOMIX 2 and
we wanted to try it in 2 rooms. And in the
night from day 0 to 1 we decided to do it
for every stage because the old solution
was not working anymore. That was a little
bit hot, but it worked. And, the redesign
caused the new UI, complete with some new
base features. We have now transitions
where pictures are moving, and we have
insertions for blending text into the
picture and we have a new audio mixing and
we are now proper A/V Sync in every case,
I think.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
Pat: And some mixer angels were exploring
the software and they found some bonus
features like random video distortion in
some cases, which I have to fix, I think.
And the party mode where you click some
buttons and they are clicking without any
doing of the user and everything is
flickering and they called it party mode.
This is what the current pipeline looks
like. We have now over 200 gstreamer
elements doing all this stuff to get your
pictures, which you are having on the
stream and in the recordings. That's it.
VOC1: So, of course, there were some
issues. I mean VOCTOMIX2 is essentially a
rewrite.
H: You have to speed up a bit. Actually we
are stealing time from the other teams. I
might call it a quit.
VOC1: OK OK OK. So...
H: Pressure!
VOC1: Yes, I know. But you're not making
things better. So one thing to mention: we
had to deploy a sweaty finger fix after
the first talk started. Ok, here are your
stats, read them now, read them read them
read them!
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
H: Please give VOC a hand. Thank you. So
what's next? Oh, the Stage manager and
Herald Operation Center. I'm part of it.
And we have 36 heralds, every one of them
very eloquent and good-looking. Then we
have 70 stage managers and stage
supporters. We had 150 shows on official
stages and the assemblies on top of it. We
have one stage fright council yet for the
speakers who took care of at least six
talks. And then we threw away over 100
hosting cards on day 1 only. So clearly,
give a hand for the SHOC.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
H: So next, c3power. Your time.
Arif: So my name is Arif Guy. I am from
the power team and from the radio
operation center. We make the power. We
load five double trucks of shit out on
Sunday 15th. We deploy a lot of power
boxes and many cables. So we have teardown
today so every help is needed. We have
only one day to bring us back. The main
thing, we had the power meter that we made
on the camp this year. We have two nice
setup on room H. You see the power factor
is very bad. And the other one was the
Waffle Operation Center so you see they
have a nice power factor. So please use
more ohmic devices like heaters, waffle
iron or something. We only have 5 Seaview
to monitor Yolocolo. You have on
c3power.top a Grafana link that also links
to the main Grafana to the NOC. Another
nice thing we have on the lounge, I have a
video here. It started now. You can see
the current on all 3 phases to the audio
and you can see the audio from the lounge.
*recorded music from the lounge*
[Filler, please remove me in amara]
Arif: It's very nice.
[Filler, please remove me in amara]
*recorded music from the lounge*
[Filler, please remove me in amara]
Arif: As you see, the only thing is...
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
Arif: Use more bass, or make more current.
Later we can show you another slide. Go to
next slide please. So for the radio team
we have 120 portable radios. We have
updated them to a new firmware, and a new
programming software with a new feature
that is a lot slower than the last
version. Very nice. 50 bring-your-own-
device radios. 5 dead radios from the
camp. 2 dead from 36c3. 2 dead repeaters
from the camp. 1 dead on arrival on this
stage. 3 rental repeaters from 2
companies, one we picked up in Hannover
just before the Congress, and even the
windows PC crashed down dead on arrival.
Business as usual. So, next team!
H: Thank you! So, the next is c3subtitles.
I think it's Amy and Julia. Amy. They are
not. C3SOS. S.O.S. It's you. Okay. Okay.
Sorry. Oh, I'm sorry. Not the subtitles.
It's well, sustainability. Your stage, go
for it. Here, feel free.
Amy: Hello. Okay, hi. My name is Amy, I'm
part of the c3sustainability team. I only
have four slides, so I'll be very quick.
You can see, the first of our biggest
projects was the drinking water
dispensers. So for angels, we had drinking
water from a dispenser rather than from
bottles and they could refill their
bottles. We go through some stats there.
So when we started planning how difficult
it was to implement it, the locations, and
how many volunteers, satisfaction was very
high for this one. So thank you to c3geld
who really helped organize the water
dispensers with us. We're very happy. They
were really great. Please give them a
round of applause.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
Amy: Thank you. The next one was a give
and take electronic box. So we wanted to
encourage people not to throw away their
electronics. Of course, they can be
recycled. So we have deployed 10 boxes.
You can find them in the sticker stations.
And we did this in collaboration with the
hardware hacking area. Thank you. And
other assemblies. Oh, wow, you are really
fast.
H: Yes I am.
Amy: Okay!
H: Others are waiting!
Amy: Okay. So the last one I wanted to go
through. We have two initiatives on this
one. I'm sorry for the trashy picture
there. We have organic bins in the halls.
It was very, very difficult to do this.
But actually it was quite satisfactory.
But I would say there is room for
improvement there. And there was also an
initiative for recycling cigaret butts. So
we actually had two people go round,
collect your cigaret butts and they will
be recycled into cool ashtrays. What a
success! Thank you so much to everyone who
collaborated with us. We couldn't do out
without your help.
H: Thank you. Your applause, please.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
H: So unfortunately, the c3sign is not
coming. So just take a look at those
pictures while I go through them. And now
the next one is C3 Assemblies. Here, take
this one. Thank you. Be quick.
Pingu: Hi, my name is Pingu. I'm here for
the assembly team. And I just want to ask,
give a hand who hasn't found his assembly
on day zero or day one? So then give a big
applause to c3nav because they really
helped us a lot. Because without them, it
wasn't able to do this event just for us.
Because for some figures we had 419
assemblies to place on this area which is
about 35000 square meters. We had 3000
tables all over, 2500 in the assemblies,
with 6000 chairs. And here, please give a
big applause to the C3Möbelhaus or IKEA as
you call it.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
Pingu: Because the Möbelhaus basically
placed all the tables in a magic night on
basically day -2. And they will disappear
with the help of C3Möbelhaus today. We
assigned the last assembly on day zero at
22:00 and we started our work in mid-
October with weekly meetings. And yeah, as
you can see, it was a lot of work and...
Oh.
H: Thank you.
Pingu: Thank you. And just one thing, for
teardown, for tearing down the assemblies,
please stack the chairs on the assembly
but leave every chair and every table on
the assembly, we will get rid of them.
H: Thanks a lot. Your applause. So the
next ones are... Les prochains, ce sont
les gens de c3lingo, voilà, vos tours!
c3lingo 1: Hallo! Schön euch mal (...) Wir
sind übersetzet. (...)
German>
Oh, sorry. Well, I'm also fine with that.
We translated all the German talks into
English. All the English talks into
German. All in all, fifteen thousand
minutes in seven different target
languages. And here is from the second
language team, who talk all the non-
English and non-German things.
c3lingo 2: Okay. So basically we did have
about 1/3 of the talks were translated to
French. Yeah. You can read the rest. And
we even have currently, exactly, right
now, another one which is translated into
Swabian. So if you want to listen into it.
[Filler, please remove me in amara]
*applause*
[Filler, please remove me in amara]
c3lingo 2: Which means that if you were
listening to a talk, there is 2 chance out
of 3 that it was translated not only once,
but twice. Into either French, Spanish,
Russian, or Polish. One special mention
for the Russian and Polish teams, that was
their first time this year. So one big
round of applause for them, please.
*applause*
[Filler, please remove me in amara]
c3lingo 2: And one more big thank about
for the guys who brought the cough candies
and cough drops in the booth. That's a
lifesaver. Thanks.
H: Thank you. Merci beaucoup. So we have
some heart operation going on here. You
have to switch it or not? Oh, you're good
with computers.
*laughing*
[Filler, please remove me in amara]
H: He just *mimicks something*. So
c3infrastructure from the subtitles. Your
stage. Yeah. My microphone. Ha ha.
td: So, thanks. Just a quick look into the
subtitles. So what do we actually do when
we subtitle a talk? Well, first of all, we
take the video from the C3VOC and put it
through speech recognition just to get a
rough transcript that we can then give to
angels to actually correct, because, well,
machine speech recognition doesn't work so
good at all. And then once we have a
working transcript that humans have looked
at, we put that through auto-timing, which
just takes the transcript and aligns that
with the audio, and that usually works
pretty well. And, well, once that is
finished, then we actually have working
subtitles, but we give them to angels for
another round of review just to fix any
mistakes that got overlooked. And maybe
sometimes the timing needs to be adjusted.
And then, when that is done, well, the
subtitle is released. And actually one of
our angels did a nice chart about all this
process that you can see here. It all
sounds better, so thank you for that.
Well, no presentation without graphs. As
you can see, the important thing is
really: everything goes up and to the
right. On the bottom here we have finished
seconds of transcribed talks. So this is
really completed subtitles here. And it
starts already quite high because it
includes all the Congresses before. Then
we have stuff that has been reviewed in
orange. Stuff that has been timed but not
reviewed yet in yellow. And transcribed
but not yet timed stuff in green. All in
all, we had 144 distinct angels. Yeah, I
need to hurry up. 71% of which took 2
shifts and 10% took 7 or more shifts. So
433 hours of work for 126 hours of
material. And so far we've had 6 releases
from this Congress and then lots of hours
worked. All of these numbers are at least
as high as last year's numbers, so good,
thanks. When you have transcripts, you can
do cool stuff with that. So, for example,
generate word clouds to see what people
like. And this case, people seem to like
people and questions and time, which we
don't have any here. So. Well, how do we
actually keep track of all this
complicated stuff? Well, we use a state-
of-the-art NoSQL lock-free columnar data
store, like many of the other teams also
do. And well, thank you all of our angels
for your hard work. And also thanks for
the Heaven for supporting us. And then,
well, if you if you feel bored between
Congresses, you can still continue to work
on transcripts. You have all the
informations here, these slides will be
online, follow us on Twitter. And thank
you.
*applause*
[Filler, please remove me in amara]
H: Please do your thing again, like your
thing again, like do *mimicks
something*. Take your laptop.
[Filler, please remove me in amara]
*laughing*
[Filler, please remove me in amara]
H: So this was the last one. I think I
will try my best to do something like this
too. Actually I'm not good with computers,
but I know someone who is and who takes
very great care of everyone of us. So one
of my highlights of every Congress, feel
free, the LOC!
LOC: Hello. I'm the stand-in for LOC. As
with all good projects, they're too busy
for documentation. They're packing. So LOC
doesn't have anything. I'm more of the
Department of Health and Safety again. So
for everyone: we have the message from
CERT that there were no work-related
accidents that caused real harm. The odd
broken Mate bottle maybe. But thank you
for having built a city again safely and
orderly, even with all the chaos. For the
people who are driving, please make sure
that the drugs wear off and that you get
some sleep and for all people riding
along, please keep the guys awake. That
would be greatly appreciated. Go home
safely. Thank you very much.
*applause*
[Filler, please remove me in amara]
H: So. Wow, ah, OK. So I'm very sorry to
have to rush some teams and I'm very sorry
that we don't have any time left. We are 1
minute over and I promised the teardown
crew to not do overtime. So please, please
give all the teams their respect and clap
and tramp as loud as you can for now to
finish!
*standing ovation*
[Filler, please remove me in amara]
H: Everything! They did! For us! We did!
For them! Why are you still sitting? You
have to leave. Thank you. From the
c3infrastructure Review. Goodbye.
Subtitles created by many many volunteers and
the c3subtitles.de team. Join us, and help us!