36C3 - 36C3 Infrastructure Review

Herald: Hello and welcome to Infrastructure Review. This review is being translated into a lot of languages and we don't know yet which one, but the c3lingo team will be on stage and will tell us how and what it did. I'd like to start, as always and every year, with the NOC, right. So please give the NOC a hand.

applause
[Filler, please remove me in amara]

Momo: All right, everyone, welcome to the State of the Internet manufacture report. This is JC, I'm Momo and we're going to talk to you about what we did this year for the network. So obviously, organizing Congress is a quite tedious task. Took us about six months of pre-planning. We came in on the 15th, did a fiber day and then it took us from the 18th and will take us till tomorrow to tear everything down with like 20 to 40 people. And we'll be busy wiping every device because that is actually what we do every year. We delete everything. There are no logs leaving this building and this will take us probably the next 24 hours. So yeah, for the usual numbers. Edge capacity: this year because you didn't use all the Internet last year, we only brought you 300 gigs, but that was fine as well, I suppose. We got 100 gig from HLKomm here in Leipzig, 100 gig from Deutsche Telekom and as well as 100 gig from BCIX which we got via DWDM wave to Berlin. In the core we used Juniper MX960s, MX480s, MX204s and QFX10002 in the yolocolo. Basically all the halls were connected via a 200 gig link and yolocolo had three times 100 gig. As probably the last five or so years, we're using IS-IS and BGP for our protocols of choice. And this year we also first off rejected RPKI invalid routes and secondly applied for the first time at congress BCP38 ingress filtering to be a good internet citizen and not to allow you to spoof IP addresses. So yeah, that was nice. As Congress keeps on growing, we have 2500 tables somewhere around the building in all the assemblies, but we only have 300 switches. So sorry if you had to bring a long cable and if you have switches to spare with 10 gig uplinks and POE+, feel free to donate them to us. Access and Wi-Fi. We had like 300 access switches. We are obviously again running Aruba Wi-Fi controllers. This year, like at camp we had a few 802.1x access points, more on that later. We tried to use Juniper vMX to route the Wi-Fi traffic. And had quite a shitload of switches, most of them from Juniper, some Cisco 2960s, some Brocades which are new to us, and some crappy old HPE stuff which is basically configured for us to work like a brick you get from like eBay or whatnot. We had a few incidents this year we'd like to talk about. First off, we had, I'm not sure if any of you noticed, quite a lot of packet loss and missing router advertisements on the Wi-Fi. This was caused by some weird Juniper vMX behavior. We couldn't figure out what it was. So we had them running in a redundant VRRP setup. We shut down one of them and then it worked. So yeah, fuck redundancy. There was a pixelflut client which somehow messed up his IP address and caused a broadcast storm which took down most parts of Hall 2. We found them, shut it down and deployed storm control to all our access switches. Yeah, to the Congress motto resource exhaustion: someone was running aggressive zmap scanning over the whole internet, came by our Wi-Fi access controller and caused a state table exhaustion. And that brought it down. We null-routed the source and yeah, there was this issue. So thank you to whoever was that. And in the morning of day 3, we had another issue with Juniper vMX where it forgot it had a network card. We rebooted it and everything was fine again. So yeah, some numbers. You actually managed to use more bandwidth, thank you.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

M: But it's still only 20% of our uplink capacity. So use more. 20% of that was IPv6, which is good, but could be more. We had like 11000 clients into Wi-Fi. 86% 5 GHz, 96% in a peak.
We had eleven 802.11ax clients. Our favorite one was obviously the one with the lovely hostname ILOVETHENOC. So yeah, about that number we have, 96% 5 GHz obviously shows us that we are finally at the point where we can say: thank you 2.4 GHz, it was nice. Goodbye.

applause
[Filler, please remove me in amara]

M: Also, obviously, thank you to our sponsors. We couldn't do this if we would not get like 10 millions, of list price obviously, of equipment and loan and quite a lot of services. So give them a round of applause as well. Thank you.

applause
[Filler, please remove me in amara]

M: And obviously NOC not only stands for Network Operation Center, but if you extend it, it is No CO2. So we believe in green power and clean traffic and therefore we obviously see that sustainability is a great part of our role. This is why we even use old crappy HP switches to cut our lines for our tchunk to serve your cheese boards, whatever you need.

laughter
[Filler, please remove me in amara]

M: Also...
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

M: Also, we somehow estimated what our network will run us in CO2 and that was about 11 tons. We're not very good with mass and not very good with CO2 emissions, but this was roughly what we came up with. And to make Congress or the world a bit better place, we actually offset all our CO2 and bought eleven tons of CO2 emissions. And now Frederick is gonna tell you how we got all those numbers.

Frederick: Yeah, as a good internet manufacture, we also do monitoring a lot and we run our own Prometheus server inside. You probably know the dashboard that we propagate all over the internet and that's powered by Prometheus. We have an internal Grafana that is part of this whole ecosystem. And if you are a little bit of a nerd, you might have clicked on the dashboard sections and seen that there are more dashboards than this. We fill our Prometheus with lots of different sources: we get SNMP data from, and screw SNMP, but it does a quite good job at getting all the insights we need from all the network equipment. We have node_exporter, influx and all that, but we got a decent amount of data from everyone in the Congress ecosystem and we had that at camp as well where we got the water pressure of the showers. And we get the colo power, which also helps with estimating the CO2 footprint. And everything is being configured by Netbox, which is a tool that is an asset database. And as I said, we have lots of dashboards and graphs. Of course, the public one where you can see lots of different things from everyone. This is only part of it. If you scroll down on the dashboard, you see a lot more. But this helps everyone to have a good understanding of what is happening currently. And we even draw a nice little Christmas trees on the Wi-Fi traffic for you. That is mostly because it's a router-on-a-stick and we cannot measure it correctly. We have an internal dashboard which gives us a little bit of a status for build-up mostly: which switches and routers are up? And that gives us a very quick sight of all the devices that are out there. What's broken? What's not broken? We improved it a little and now have alarms so someone can look at stuff and see if things are broken, run out there and fix it. We also built weathermaps. As you can see that's a little bit of a mess. But we couldn't do it better because the graphing library doesn't allow us to do it better. If someone has a good idea to do it better in Grafana or anywhere else with sources from Prometheus, please come to us, we're happy to talk. But this shows our core and all the links between it and how much capacity is being used. Red indicates that it's used more heavily. We also have that for the yolocolo. And all the traffic around it as well. And, yeah, teardown starts now. Please don't touch our equipment. And if you want to come and help, please come to Hall 4 and get in touch. We always need helping hands. But please in an organized way don't disassemble switches or access points. We have lists and everything. We need to account for everything. So please come to Hall 4 if you want to help. And yes, use more bandwidth and offset more CO2. Thank you.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

H: So thanks a lot. Actually, because you're clearly the backbone or one of those many backbones of the conference, is there any Q? Let's do a Q&A for like one or two questions.
Is anyone having a question right now? Someone is standing up. Right, microphone number one, please.

Q: Hey, we've absolutely don't ...
[Filler, please remove me in amara]

H: Nearer, nearer.
[Filler, please remove me in amara]

Q: We've absolutely done bcp38 in previous years, by the way.

M: Sorry for what I said then, I'm sorry.

Q: So I wanted to correct the record. We've been good netizens in previous years as well.

H: Oh, you're right. Thank you. Was there another question or something else to correct them? Because they clearly don't know what they're doing.

laughter
[Filler, please remove me in amara]

H: Yeah. Feel free. Microphone number two.

Q: So if 2.4GHz is over, what's going to happen to all of the ESP32 in various IoT devices? Are they going to have a home here next year?

M: They'll definitely have a home. But as usual, we cannot support it as good as we can on 5GHz because obviously this band is overused and not even remotely suited for that amount of clients we put on it.

Q: A follow-up for the ESP32. How exactly can you locate them through the wireless if they are lost?

M: Well, we can't. We can basically look at which access point they are, and then if someone really would want to, we could start triangulation, but we've never done that before. So yeah, we can just pin them roughly to an access point.

H: Maybe we can ask c3nav next time. So please give the NOC a hand.
Thank you.

applause
[Filler, please remove me in amara]

H: So, the next team up is the POC. Do we have to click this? Use more bandwidth, I'm going to try. Ah! Thank you. Your stage.

Garvin: Thanks.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Maria: Hi, my name is Maria.
[Filler, please remove me in amara]

G: Hi, and I'm Garvin and we're from Phone Operation Center and we want to talk a bit about the phone infrastructure at this event.

M: Yeah. So we arrived at day -6 and planned on hacking some things
and socializing and we planned a team event, but then everything was different.

G: Yeah. When we arrived, I went into the NOC office and they said to me, "Ja, CCL is up, internet is up, everything is just working nicely. You can start hook up your telephony system right now." And I was like, whoa. So thanks a lot NOC. Really great performance this year, we were amazed. Nobody expected that it works so well.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

M: Yeah. So we put up the first antennas and then we decided to have our team event anyway. And yeah, so we did a lot of things. So we handed out 150 orga loan DECTs and we deployed 51 SIP telephones. We also deployed 67 antennas and we had a POC party on day three until 7:30 a.m.

G: So you can see almost everything is done. I guess the remaining things are not that important. So this is the overview of DECT coverage at the event. Only level zero, because otherwise I think it would be overcrowded to show. Just so that you can get a rough impression on how many antennas we deployed in order to give you this DECT coverage. That you can be reached almost everywhere in the event and that you can see how our tooling looks like, where we see how good the antennas see each other, and that we can see that seamless handover work so that you can start at our desk, walk through the area, into the lounge and just continue talking. And oh, there are also some antennas that are outside of the building. What could that be?

M: That is our hotel DECT. And you can see a typical hotel DECT installation on the photo. And people got really confused about it because we also had DECT coverage at main station.

G: Yeah. So I got a call roughly at 4:00 p.m. in the morning.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

G: And somebody told me. Oh, my DECT rang. Why? Now I need to turn it off at the night. Why did you do this?

M: Yeah. So we also had problems. We have a new feature since Camp where you can see your DECT devices and can assign them to your number before you even arrive, and
then everything is set up and you don't have to call your token anymore. And people get really confused because they would call their token anyway and it calls it invalid. So we had to explain a lot of this unexpected simplicity to them.

G: And then we had a battery issue this year. We had not enough batteries and, you know, batteries are always empty in the phones when it's the most important. So we were thinking what what can we do about that?

M: So we build a new device. That's our microwave and it can also charge devices.
So, many thanks to C3Power, because they helped us with tooling and actually they have expertise to put power cables on devices like this. Thank you very much.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

G: In the last years we were often asked "how expensive is your service actually?". So we decided that we now provide invoices so you can see how expensive our services are and we send out a lot of invoices. And we got paid some money. But as you can see on the invoices, most is sponsored by CCC.

M: Yeah. And people also paid with Mate which is really awesome.

G: Yeah. And also people have invoices on a fraction of a cent and they got quite creative on how they can pay us.

laughter
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

M: So here are some more stats. We have 7473 registered extensions.

G: Oh, we didn't remove this. So we were thinking on how to compare this with things and we were looking at villages in Saarland and then we thought, this is a stupid comparison, but we didn't remove this.

M: So there were 5021 attached DECT phones and 3251 concurrent DECT phones. Which is about more than 1000 more than last year.

G: So thanks a lot for using DECT.

M: Yeah. We had...
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

M: There were also max 120 calls in parallel, and we had more than 300 thousand calls in total. That's also really, really much. We had five eating meetings at heaven, the angel eating place, and there were an average of 42 eating meeting live viewers. We had two lectures. You can see them on media.ccc.de and we had 23 super fast charged phones in our microwave.

applause
[Filler, please remove me in amara]

G: Like the NOC, we also had to deal a bit with issues during the event, and actually there were some DDOS attack on our account system and somebody configured over 4000 extensions with really stupid names. And it took us quite a while to get rid of them again cleanly from the system because the synchronization turned out to be really slow. So you can see it took us a while to get them removed again. So we can only say, you all know it's a hacker congress, but it's kind of stupid to hack your own infrastructure.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

G: So as a consequence of this, we only allow now only 50 extensions per account and per event. If you think you need more, feel free to contact us if you have a valid use case.
[Filler, please remove me in amara]

M: So every device has an IPEI that is like a MAC address and we ask the responsible institution to give us the manufacturer. But it's really secret. So they don't give it to us. So we ask you for help. Please enter the models of your devices for your phones. And then we can match to the IPEI and get some data to build more awesome features for you.
[Filler, please remove me in amara]

G: This would really help us. And the only way for us is to crowdsource it because it seems to be super secret. Whatever.
[Filler, please remove me in amara]

M: You can find it in guru3 on the device page and there's this little pen. And if you click on it, then you can enter the model.

G: Thanks upfront. And yep, that's from us, and I guess now we have a little time for you to ask questions.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

H: Great. So, any questions for the POC? I don't see... were they correct or do we have to...? Ah, someone is getting up. So microphone number 1, please.

Q: Yes, a few years ago there was a translation service via DECT. Is the capacity enough to service also this crowd?

G: The problem is that we switched the phone system a while ago last year at the Congress. And the old phone system had a way how we can do the translation via one channel and the problem is that the new system doesn't support this. Let me say the new system doesn't support it yet. So have a look at our talk and then you can see that there is some potential.

H: I see someone at microphone number 3, please. This would be the last question because we have to hurry a bit.

Q: Can I know a little bit more about the super charging microwave? I'm confused.

applause
[Filler, please remove me in amara]

M: Sure, you can come to our POC desk and then we answer all your questions.

H: Ooh... mystery. Please, give the POC a hand.
So next team, is it from the GSM crew? Someone there? I think we have two. I see some Chaos Post, you have to wait one round. So, GSM guy thank you. I am good with computers, I think. Yeah, I am. Have fun. If it's working or not. Look at this smile then looks better. Try this. Yeah. Right. Try this.
- 423.
- Maybe you have to use GSM.
- 23 test.
- What happened?
- Hello. Hello.
- Aah.

GSM Person 1: So as every year we ran our own mobile phone network at the Congress using osmocom open source software for 2G and 3G, and open5gs interfacing with the osmocom HLR. And all you need to take part is a SIM card that you can buy from the POC and for 5 euros you get a flat rate. The price increases because we have less SIM cards every year. We need to manufacture new ones. You can even call outside like you can with DECT phones.

Lynxis: Hello. I'm Lynxis. So as every year for the GSM team, the first problem is the license. That's the first step usually. Because in Germany, you have to get the official form, get a license but ... Where do you get it? What can you ask for frequencies? Because, for example, the POC for DECT or Wi-Fi, you just place it and you can use it. You're fine.
But for GSM, they didn't think about it or for 3G or 4G. So yeah, this year we usually get the license middle of december, maybe start of december. So it's already late. So this year we didn't get all our licenses. But we get some. We got 850 MHz, which is not assigned in Europe because it's usually in the US only. But we have a small hole. This year we got a 4G license instead of a 2G license with 10 Mhz from Telefonica. So thanks Telefonica for borrowing us spectrum.

applause
[Filler, please remove me in amara]

L: Just a short example of how the spectrum looks like. The yellow stuff is usually somewhere behind actually. By the way, this microphone, it might share the same frequency with us. But so far, we haven't found any interference together with the VOC. Down there you can see the small hole which we are using. Because we didn't get the 2G license there, we thought, OK. Let's take a look. Can we fit them both in the same frequencies? It's not good. But you see the spikes, this nice antenna on the right. That's the GSM, which is sending on the same frequency. It works because they are using different codings. But I have heard from people who know more about it, this is not the way you use it.

laughing
[Filler, please remove me in amara]

L: So we took some photos of our base stations. This is actually the fairy dust from the 2G base station, so we have a idea what we are using here. We have even more fairy dust in our 3G femtocells and our 4G cells. They are looking like small toasters. They are taking actually 90 watts via POE, they have special POE adapters. So maybe we could ask if somebody can do a similar adapter to get even running a toaster on the line.
So basically for the 4G setup this year, we weren't sure if it's stable enough or we lose all our phones to the LTE and they don't like to come back to the 2G and 3G setup where we have voice, because on LTE we don't have yet voice so you have to select specifically to join the LTE network. That worked quite fine if you change it. So your phone will register. Everything fine there.

GSM1: So the rollout this year: we had the voice working on day 0 which is new.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

GSM1: Someone even noticed on Mastodon, I saw it, too. We already had LTE at the CCC camp this year. Yeah, but unfortunately we lost crypto password, so LTE roll out took a bit longer this time. Sorry. So some numbers. In total we saw just about 1100 people doing a location updating on our network and 845 eventphone tokens were dialed on the GSM. That is 2G or 3G. And there were roughly 200 phones actively subscribed on the network at all times. And even though we basically only deployed 3G nano stations in all the halls and only had two 2G BTS in the glass halle, there were roughly more than half of all the phones were still subscribed on 2G instead of 3G. We had like 18 3G stations and only two plus one in the GSM room 2G stations. So that's a bit surprising. And SIM cards: starting from the bottom, we sold about 700 SIM cards, but only saw half of them activated, which is curious. And luckily, most or some people bring old SIM cards from previous years. And it's not so easy to get cards manufactured. So we are very glad for everyone who brings old SIM cards from previous events. We might even consider introducing charging phone calls only for SIM cards that are newly bought. So if you want to continue calling for free, rather bring your old SIM cards.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

GSM1: So you see the numbers, I'm not going to go through them. And you can also get the ADM, the admin keys for your SIM cards if you like to write to them.

L: Or if you have seen the talk from LaForge, you might want to play with the SIM cards, we give out all the keys you want to have to play with it.

GSM1: So operation was mostly smooth, except iPhones, for unknown reasons. And except the data service, which might even be related because maybe Apple is a bit more strict on whether data service is working reliably. Yeah, we still had some problems in the SGs instance introducing 3G changing between the radio access technologies. It's a whole new ballgame so there are still some bugs in there. And as you see, we had many more tickets than the POC. This is actually reversed from the POC, the "done" is on the left. So this whole bunch of stuff is done and there's some backlog and canceled and fantasy tasks. It worked pretty nicely. Are you taking over? Oh, yeah, no, this is still mine. And this year we actually had also a denial of service attack. The code was the same as previous years and we never saw this before. But this year we got an invalid mobile identity which managed to crash our mobile switching center. And thank you very much for uncovering this bug and thanks to fixeria for fixing it on day 2. Ever since the mobile switching center for Voice and SMS and subscription has been running stable.

applause
[Filler, please remove me in amara]

L: Maybe some interference. We recovered the old phones again. Last year we couldn't support them. But we managed to implement the missing parts. Old phones could work if they support the frequencies. So that's really nice. Maybe next time. Since camp, we also did a nice angel helpdesk. And it was really impressive to see that we even had to add more shifts in our shifts. We had so many motivated angels. Thanks to everybody who helped us, it was really great.

applause
[Filler, please remove me in amara]

H: Unfortunately, we don't have enough time for a Q&A. So please give them a hand for everything they did.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

H: Thank you. There are quite a lot of teams next. I count at least like eight, maybe nine. So we need to speed up a bit. Our next team for now will be... We don't have working microphones.

Chaos post 1: Sorry, we need to interrupt you anyway.

H: OK. So tell them, chaos post! Chaos post!
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Chaos post 1: Sorry, guys, we interrupt...
[Filler, please remove me in amara]

H: Let's get rid of the broken one because it's not working anyway.

Chaos post 1: All right. Sorry, we are interrupting for a few minutes only. We would like to deliver a few statistics as well. So thanks of all, we had multiple chaos deliverers working throughout the whole Congress 24/7 basically, delivering at the speed of chaos, as our mission statement clearly states. So thank you therefore, first of all thank you very much to all of you who contributed to that.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Chaos post 2: So let's have some numbers, because you all love numbers, right? So we delivered about 3000 external postcards. So that means with like outside chaos, so to the real world or the default world, as we call it. We delivered those to over 42 countries all over the world. So you guys are really good connected internationally. And also, we don't have exact numbers, but we estimate around 3500, no, 35000, internal postcards. And you also use our online office for a total of 789 times. So that is only 15 less than camp, and that was a longer event. So, hey, you guys write a lot of postcards!
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Chaos post 1: As you might already know from camp, we also do have a few special services. As already on camp, we had the serving proposal, basically a pre-assembled text, you just had to cross what you want. The postbox certification and of course, the bi-directional chaos, in Germany also known as "Einschreiben mit Rückschein".

H: Wow.

Chaos post 2: And this time we also offered some new services. We had like sang telegrams, gesunges Telegram in Deutsch. We had a forever alone box for people wanting to write postcards with or exchange postcards with no idea who to write it to. We had love letters, so we had some nice pre-assembled texts and also a really nice selection of perfumes for scented postcards. You could write some secret messages. We had some UV pens and also we had some, let's say call it security, or rather temper evidence, because we had some scratch off stickers for you. And also we had over 30 new postcard designs that you could use for postcards.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Chaos post 1: Short remark for the love letters and the perfumes. Well, that was kind of not really thought through. It was a bit... it was fun sorting them out, and stamping all of that, smelling all the perfume all the time.

Chaos post 2: And then brushing your hands really thoroughly because, well, that stuff gets on your hands when you do that.

Chaos post 1: Also for UV pens. Just a little remark. It's not a good idea to use it for addressing and the message.

laughing
[Filler, please remove me in amara]

Chaos post 1: You can take the risk. All right, then let me close up. We also supported mail this year and we had 130 letters for activists in prison, which I find really great. I think that's something we can all support. Amazing.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

So thank you all for this amazing event, and have some fun for the rest of the Congress.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

H: Chaos post! So the next team is the VOC. Yes, you can have it. Here, the VOC. We only have 20 minutes left in total for every team. So you know what to do.

VOC1: Hurry up. Welcome, guys. So. Yeah. We don't have as many statistics as usual. But we have some great stories too. We'll hurry up. So this year in total, we covered 10 stages apart from the 5 stages that we do usually for Congress. We had streams from the critical decentralisation cluster, Sendezentrum, Wikipaka WG, Open Infrastructure Orbit, Chaos West. And in total, we served 255 hours and 35 minutes of total talk time. So you know what to do until the next Congress.

applause
[Filler, please remove me in amara]

VOC1: And of course, sustainability was a big topic during this Congress. So part of what we have to do is stream reencoding so you can watch it with the VP5 codec or use it at lower resolution. And so far we've been using 4 Xeon-based machines and 2 desktop machines. And thanks to hardware-based encoding, we now replace this with a single laptop. As you can maybe read, this is critical infrastructure now. And for all the streams, for 30 reencoding streams, we are on a 45 watt power budget now. And as an added benefit, because we also encode the master slides with hardware encoders, and hardware encoders can generally use a higher profile that allows for better quality in real time, you now get a better picture as well. So, yeah.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

VOC1: And of course, we had some minor fuckups this year. We thought the audio setup was a bit less complex. But when we checked rooms, we had buzzing everywhere. So we replaced some SDI lines with fiber and turns out buzzing goes away. Then the PA sound console had a buggy DANTE card. After you reboot them, the auxiliary out to the VOC cameras was muted. So that messed up the particular talk that has been redone today, but other than that, we are rather happy. And we figured that 3 of our audio mixers actually had broken outputs. I don't know how we did that, but it clearly shows that they've been used on one event or another during the last years, which is actually a good thing we may think. On the Wikipaka stage, we did not use Ansible because there was sort of a playground for us. But if you don't do things properly, well, then you run into edge cases with things. And yeah. So I need to hurry up. And one virtualization host suddenly started leaking memory. And so if you were affected by that during the main talk season in the evening, we are very sorry about that. Updating the kernel helped and we have no idea what happened. Yeah, and Icecast got stuck as well. And some relive... so if you want to see talks later, that may not be possible because we ran temporarily out of space. But if you watch this on media.ccc.de, and the talks were not yet released, we have relive integration, so the talks show up in media.ccc.de even though there is no proper release yet, for your convenience. And main track and assemblies are now integrated in all events, so you don't have to click through 4 separate events to find your favorite talk. And now I pass to Pat to talk about VOCTOMIX 2.

Pat: Yeah. Thank you. Okay. I have now 20 seconds, I think. I made a redesign of VOCTOMIX, it's now called VOC2MIX. And Peter was doing that meme some weeks ago, because we had to switch to VOCTOMIX 2 and we wanted to try it in 2 rooms. And in the night from day 0 to 1 we decided to do it for every stage because the old solution was not working anymore. That was a little bit hot, but it worked. And, the redesign caused the new UI, complete with some new base features. We have now transitions where pictures are moving, and we have insertions for blending text into the picture and we have a new audio mixing and we are now proper A/V Sync in every case, I think.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Pat: And some mixer angels were exploring the software and they found some bonus features like random video distortion in some cases, which I have to fix, I think. And the party mode where you click some buttons and they are clicking without any doing of the user and everything is flickering and they called it party mode. This is what the current pipeline looks like. We have now over 200 gstreamer elements doing all this stuff to get your pictures, which you are having on the stream and in the recordings. That's it.

VOC1: So, of course, there were some issues. I mean VOCTOMIX2 is essentially a rewrite.

H: You have to speed up a bit. Actually we are stealing time from the other teams. I might call it a quit.

VOC1: OK OK OK. So...

H: Pressure!

VOC1: Yes, I know. But you're not making things better. So one thing to mention: we had to deploy a sweaty finger fix after the first talk started. Ok, here are your stats, read them now, read them read them read them!
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

H: Please give VOC a hand. Thank you. So what's next? Oh, the Stage manager and Herald Operation Center. I'm part of it. And we have 36 heralds, every one of them very eloquent and good-looking. Then we have 70 stage managers and stage supporters. We had 150 shows on official stages and the assemblies on top of it. We have one stage fright council yet for the speakers who took care of at least six talks. And then we threw away over 100 hosting cards on day 1 only. So clearly, give a hand for the SHOC.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

H: So next, c3power. Your time.

Arif: So my name is Arif Guy. I am from the power team and from the radio operation center. We make the power. We load five double trucks of shit out on Sunday 15th. We deploy a lot of power boxes and many cables. So we have teardown today so every help is needed. We have only one day to bring us back. The main thing, we had the power meter that we made on the camp this year. We have two nice setup on room H. You see the power factor is very bad. And the other one was the Waffle Operation Center so you see they have a nice power factor. So please use more ohmic devices like heaters, waffle iron or something. We only have 5 Seaview to monitor Yolocolo. You have on c3power.top a Grafana link that also links to the main Grafana to the NOC. Another nice thing we have on the lounge, I have a video here. It started now. You can see the current on all 3 phases to the audio and you can see the audio from the lounge.

recorded music from the lounge
[Filler, please remove me in amara]

Arif: It's very nice.
[Filler, please remove me in amara]

recorded music from the lounge
[Filler, please remove me in amara]

Arif: As you see, the only thing is...
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Arif: Use more bass, or make more current. Later we can show you another slide. Go to next slide please. So for the radio team we have 120 portable radios. We have updated them to a new firmware, and a new programming software with a new feature that is a lot slower than the last version. Very nice. 50 bring-your-own-device radios. 5 dead radios from the camp. 2 dead from 36c3. 2 dead repeaters from the camp. 1 dead on arrival on this stage. 3 rental repeaters from 2 companies, one we picked up in Hannover just before the Congress, and even the windows PC crashed down dead on arrival. Business as usual. So, next team!

H: Thank you! So, the next is c3subtitles. I think it's Amy and Julia. Amy. They are not. C3SOS. S.O.S. It's you. Okay. Okay. Sorry. Oh, I'm sorry. Not the subtitles. It's well, sustainability. Your stage, go for it. Here, feel free.

Amy: Hello. Okay, hi. My name is Amy, I'm part of the c3sustainability team. I only have four slides, so I'll be very quick. You can see, the first of our biggest projects was the drinking water dispensers. So for angels, we had drinking water from a dispenser rather than from bottles and they could refill their bottles. We go through some stats there. So when we started planning how difficult it was to implement it, the locations, and how many volunteers, satisfaction was very high for this one. So thank you to c3geld who really helped organize the water dispensers with us. We're very happy. They were really great. Please give them a round of applause.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Amy: Thank you. The next one was a give and take electronic box. So we wanted to encourage people not to throw away their electronics. Of course, they can be recycled. So we have deployed 10 boxes. You can find them in the sticker stations. And we did this in collaboration with the hardware hacking area. Thank you. And other assemblies. Oh, wow, you are really fast.

H: Yes I am.

Amy: Okay!

H: Others are waiting!

Amy: Okay. So the last one I wanted to go through. We have two initiatives on this one. I'm sorry for the trashy picture there. We have organic bins in the halls. It was very, very difficult to do this. But actually it was quite satisfactory. But I would say there is room for improvement there. And there was also an initiative for recycling cigaret butts. So we actually had two people go round, collect your cigaret butts and they will be recycled into cool ashtrays. What a success! Thank you so much to everyone who collaborated with us. We couldn't do out without your help.

H: Thank you. Your applause, please.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

H: So unfortunately, the c3sign is not coming. So just take a look at those pictures while I go through them. And now the next one is C3 Assemblies. Here, take this one. Thank you. Be quick.

Pingu: Hi, my name is Pingu. I'm here for the assembly team. And I just want to ask, give a hand who hasn't found his assembly on day zero or day one? So then give a big applause to c3nav because they really helped us a lot. Because without them, it wasn't able to do this event just for us. Because for some figures we had 419 assemblies to place on this area which is about 35000 square meters. We had 3000 tables all over, 2500 in the assemblies, with 6000 chairs. And here, please give a big applause to the C3Möbelhaus or IKEA as you call it.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

Pingu: Because the Möbelhaus basically placed all the tables in a magic night on basically day -2. And they will disappear with the help of C3Möbelhaus today. We assigned the last assembly on day zero at 22:00 and we started our work in mid-October with weekly meetings. And yeah, as you can see, it was a lot of work and... Oh.

H: Thank you.

Pingu: Thank you. And just one thing, for teardown, for tearing down the assemblies, please stack the chairs on the assembly but leave every chair and every table on the assembly, we will get rid of them.

H: Thanks a lot. Your applause. So the next ones are... Les prochains, ce sont les gens de c3lingo, voilà, vos tours!

c3lingo 1: Hallo! Schön euch mal (...) Wir sind übersetzet. (...)
Oh, sorry. Well, I'm also fine with that. We translated all the German talks into English. All the English talks into German. All in all, fifteen thousand minutes in seven different target languages. And here is from the second language team, who talk all the non-English and non-German things.

c3lingo 2: Okay. So basically we did have about 1/3 of the talks were translated to French. Yeah. You can read the rest. And we even have currently, exactly, right now, another one which is translated into Swabian. So if you want to listen into it.
[Filler, please remove me in amara]

applause
[Filler, please remove me in amara]

c3lingo 2: Which means that if you were listening to a talk, there is 2 chance out of 3 that it was translated not only once, but twice. Into either French, Spanish, Russian, or Polish. One special mention for the Russian and Polish teams, that was their first time this year. So one big round of applause for them, please.

applause
[Filler, please remove me in amara]

c3lingo 2: And one more big thank about for the guys who brought the cough candies and cough drops in the booth. That's a lifesaver. Thanks.

H: Thank you. Merci beaucoup. So we have some heart operation going on here. You have to switch it or not? Oh, you're good with computers.

laughing
[Filler, please remove me in amara]

H: He just mimicks something. So c3infrastructure from the subtitles. Your stage. Yeah. My microphone. Ha ha.

td: So, thanks. Just a quick look into the subtitles. So what do we actually do when we subtitle a talk? Well, first of all, we take the video from the C3VOC and put it through speech recognition just to get a rough transcript that we can then give to angels to actually correct, because, well, machine speech recognition doesn't work so good at all. And then once we have a working transcript that humans have looked at, we put that through auto-timing, which just takes the transcript and aligns that with the audio, and that usually works pretty well. And, well, once that is finished, then we actually have working subtitles, but we give them to angels for another round of review just to fix any mistakes that got overlooked. And maybe sometimes the timing needs to be adjusted. And then, when that is done, well, the subtitle is released. And actually one of our angels did a nice chart about all this process that you can see here. It all sounds better, so thank you for that. Well, no presentation without graphs. As you can see, the important thing is really: everything goes up and to the right. On the bottom here we have finished seconds of transcribed talks. So this is really completed subtitles here. And it starts already quite high because it includes all the Congresses before. Then we have stuff that has been reviewed in orange. Stuff that has been timed but not reviewed yet in yellow. And transcribed but not yet timed stuff in green. All in all, we had 144 distinct angels. Yeah, I need to hurry up. 71% of which took 2 shifts and 10% took 7 or more shifts. So 433 hours of work for 126 hours of material. And so far we've had 6 releases from this Congress and then lots of hours worked. All of these numbers are at least as high as last year's numbers, so good, thanks. When you have transcripts, you can do cool stuff with that. So, for example, generate word clouds to see what people like. And this case, people seem to like people and questions and time, which we don't have any here. So. Well, how do we actually keep track of all this complicated stuff? Well, we use a state-of-the-art NoSQL lock-free columnar data store, like many of the other teams also do. And well, thank you all of our angels for your hard work. And also thanks for the Heaven for supporting us. And then, well, if you if you feel bored between Congresses, you can still continue to work on transcripts. You have all the informations here, these slides will be online, follow us on Twitter. And thank you.

applause
[Filler, please remove me in amara]

H: Please do your thing again, like your thing again, like do mimicks something. Take your laptop.
[Filler, please remove me in amara]

laughing
[Filler, please remove me in amara]

H: So this was the last one. I think I will try my best to do something like this too. Actually I'm not good with computers, but I know someone who is and who takes very great care of everyone of us. So one of my highlights of every Congress, feel free, the LOC!

LOC: Hello. I'm the stand-in for LOC. As with all good projects, they're too busy for documentation. They're packing. So LOC doesn't have anything. I'm more of the Department of Health and Safety again. So for everyone: we have the message from CERT that there were no work-related accidents that caused real harm. The odd broken Mate bottle maybe. But thank you for having built a city again safely and orderly, even with all the chaos. For the people who are driving, please make sure that the drugs wear off and that you get some sleep and for all people riding along, please keep the guys awake. That would be greatly appreciated. Go home safely. Thank you very much.

applause
[Filler, please remove me in amara]

H: So. Wow, ah, OK. So I'm very sorry to have to rush some teams and I'm very sorry that we don't have any time left. We are 1 minute over and I promised the teardown crew to not do overtime. So please, please give all the teams their respect and clap and tramp as loud as you can for now to finish!

standing ovation
[Filler, please remove me in amara]

H: Everything! They did! For us! We did! For them! Why are you still sitting? You have to leave. Thank you. From the c3infrastructure Review. Goodbye.

Subtitles created by many many volunteers and
the c3subtitles.de team. Join us, and help us!