[Talkmeister]: Welcome, our next talk will be about the Debian Long Term support team and the speaker is Raphaël Hertzog. [Raphaël Hertzog]: Hello. Today I will speak a bit about Debian long term support. I guess most of you already know about it. Are there some people who have no idea what this is about? No, good. I will make my talk in 3 parts. First I will present the team, how it works I will give some facts about how it evolved over the first years. I took some time to collect statistics and believe they are rather interesting I will also speak about the future but there will be a separate discussion about this in a BoF later this week. I will show you how to help because, like any other team in Debian it is open to anyone. We welcome help. At the end I will answer your questions. What is LTS about? The idea is really simple. We wanted to extend the support period of all Debian releases. Currently it is basically for 1 year after the next stable release comes out. We wanted to extend this to 5 years to match, at least, Ubuntu's offering. which is not our competitor, but for the companies that are making choices it is one of the important criteria. So we wanted to do as well. Since we publish new stable releases every 2 years it is roughly 3 years. A nice side benefit is that the user can skip a full release. We don't officially support dist-upgrade over going from Debian 6 to 8 but you can do 2 dist-upgrades at the same time, limiting the downtime to once every 5 years. By the way, in practice, in simple server configurations, dist-upgrades tend to work rather well even across 2 releases. Keeping a distribution secure for 5 years is a real challenge. It is hard work that not everybody is willing to do. In Debian all the work is done by volunteers who do the work they enjoy. Generally we enjoy working on new features on top of latest releases and not really on backporting patches to crud that was written 5 years ago. The security team has limited resources so we could not just ask the security team to now do twice the work. But they were still really interested in the project and wanted to support the idea and really helped to get it bootstrapped. The security team has a slightly larger scope. They support all architectures, which means you have lots of problems of coordination when security updates do not compile and stuff like that. What did we do? We restricted the scope by picking the 2 most popular architectures that most users care about. We have had some demand for ARM architectures but up to now we only support amd64 and i386. We also excluded some packages from security support. Either because they are taking too much time, like a security issue every 2 weeks or that upstream is not cooperative enough to be able to support it. This list was basically made by the current security team based on their experience of doing security support. If you look at the list there are some important restrictions. There's no xen, no kvm, no rails, no browser. It sucks a bit. But it's a way to get it started. I think we can do better for wheezy. Basically there is no virtualization support on the host, only on the guest. The security team helped to bootstrap the LTS team but it is not the same team. Obviously there are members of the security team who also work on the LTS team. They work in close collaboration. We have regular contact and they watch our mailing lists etc. But the policies are different and the infrastructure is separate, which is a problem but I will talk about that later. We have a dedicated mailing list debian-lts@lists.debian.org and a dedicated IRC channel as well. You are welcome to subscribe and to join. It's a new team which means new people and new members. Where do they come from? The first idea was to get help from people in various companies who are already doing such in-house support. We had contact with EDF, and still have, but they were one of the first companies who were pushing for this because they basically said we are doing this already and we can share with other companies. The idea was to pool security support of multiple companies. We sent a press release asking companies to join. We had a few responses. But I'll come back later to how it evolved It's not really satisfying. The other thing that we did is that we offered companies the option to fund the project to bring money and use this to pay the work of actual Debian contributors to do the security updates that we need. We have wiki pages listing all the ways that companies can help with money. In practice, most of the (wanting to be) paid contributors joined together under a single offer managed by Freexian SARL which is my own company. I'll quickly explain how this works. Most companies don't want to bother bringing human resources They buy long term support contracts from Freexian. We have a rate. When you give €85 you fund 1 hour of LTS work. This is the current list of sponsors. Top level gold sponsors sponsoring more than 1 day of work per month. On the other side we have Debian contributors that are doing the work and Freexian is paying them. There is a small difference between the rate to cover administration costs because I have to handle the invoices and some customers are using Paypal which is taking a cut. We ask contributors to follow some rules. There is a requirement to publish a monthly report on work done on paid time. So they won't get paid until they have published a report. So everybody can know how the money has been spent. Currently we have 7 Debian contributors and about 30 sponsors. Some figures. Who uploaded packages? How has it evolved since June last year? How is the funding evolving? I just updated those figures a few days ago. I used this talk before at the mini DebConf in Lyon in March, but I updated it again. The number of uploads is roughly over one year since we started last year. Over 300 uploads so it is not so much but it is almost 1 per day. So it is significant work. I have given a state here of who paid for the work and who did it on the left The sponsors of Freexian are paying for most of the uploads. None is a separate category grouping all Debian maintainers. There are maintainers who are taking care of their own packages in LTS. Security team is members of the security team who also work within the LTS team. EDF is Électricité de France Individuals are Debian developers that have listed themselves as members of the LTS team and did uploads for packages of other maintainers not their own. Credativ is a German company that you probably know. They have a booth here if you want a job. Toshiba, Univention, Catalyst etc are other lower figures. On the right are people. The top 5 people are paid by Freexian. Raphaël Geissert is working for EDF. Thijs is a member of the security team. Kurt is openssl maintainer so he maintains his own packages. He has a lot of work. Mike Gabriel is also paid by Freexian. Christoph Bieldl is mainly maintaining the debian-security-support in Squeeze LTS Nguyen Cong is employed by Toshiba. Christoph Berg is employed by credativ doing PostGreSQL maintainance. How did it evolve over the year? Again it is by affiliation. The big blue part is paid contributors You don't see it very well but the part about maintainers is this one [points] It tends to do better over the months because here we started to contact maintainers every time that we have a new upload coming up and ask them first 'do you want to handle it yourself' so it slightly increased. but the contribution of other companies has not really increased over time. Rather it has disappeared. It is unfortunate but it looks like paid contributors are more productive than others. In particular with EDF, they do the work, but with some lag and we are faster so they just reuse what we have done. I want to talk to Raphaël to see how we can do better towards this. How did the sponsorship level evolve? We have a steady increase, which is rather nice. It is not a huge amount but it is significant because we fund almost 80 hours per month. It is close to our first goal. We wanted that amount to be able to sustain ourselves. If you look at the sponsors, we have a few big ones, possibly one very big We can't give the name officially yet so I won't.It will be a big jump in the graph A few gold and many small sponsors. I don't want to be dependent too much on one big sponsor. I really prefer many sponsors who are doing small donations but donations which are sustainable year after year because we are not here for 1 year or 2. We want to do it over the long term. We have some figures about how many hours have been funded since the start Feel free to interrupt me if you have any questions. I can take them at any time That's it for evolution. Now, the future. What do we expect for the future? First keep doing what we have been up to Keep supporting the current set of packages. But for wheezy long term support we would really like to have more supported packages. A browser would be nice for desktop deployment. Virtualization support is also important for many companies so we should be able to support something here. Also we want to avoid some pitfalls that we had with squeeze LTS. As you know LTS users are currently required to add a separate source list entry with squeeze-lts repository. The security.debian.org squeeze repository is unused. It should be possible for the LTS team to continue using the same repository as the security team once it no longer use it. This will be the topic of a BoF next week on Tue at 1800. What's the problem with supporting the current set of packages? For example, we have MySQL right now in Squeeze LTS in version 5.1 which is no longer supported by Oracle. We don't even know if it's affected by security issues because Oracle doesn't give info on non supported releases. This is a problem, we should consider switching to a newer version, but newer versions involve library transition, which is not really nice in Long Term Support releases. There is some work to do here if we want to do something serious. And we have other problems, other packages which are similar. From time to time, we backport, we take newer upstream versions. We did that for wireshark for example. This is what I said before, the limited scope sucks, we want to be able to support more packages and we need more support for this. [Q] Speaking of wireshark, we switched to Wheezy's version, so it's solved. [Raphael] Yes, exactly, that's what I just said. It used to be a problem. That's a part I did no update since March. Additionally, the problem with a separate repository is that there is a propagation delay to the mirrors that we don't have with security.debian.org. I will speak a bit of how the team works. Basically, the first step is triaging new security issues. We have a list of CVEs that comes in and there are added to a text file data/CVE/list. Someone dispatches those on source packages and then someone else reviews status in each release. Then the LTS team reviews the status in Squeeze and members of security team review status in Wheezy and Jessie. And then, we decide what we must do with the package. Depending on the analysis, either we say "We need to prepare an update", so we add it to data/dla-needed.txt and someone will have to take care of preparing the update. Or we say "The issue does not apply, does not affect the current version" or we ignore it because the package is unsupported or because the issue is really minor, not severe enough. Sometimes, it can be that the issue is already fixed in Debian due to some maintainer choices. When we do this classification, we contact the maintainer to keep him in the loop and offer him the possibility to help us. Here's what such a text file looks like. The bold line is the line that we are adding when we have decided what we're doing with the packages. This is all in the Subversion repository on Alioth. Then, someone has to prepare the security update. Basically, looking for a patch, it's often useful to be able to look up at RedHat or Ubuntu or upstream. Best case is upstream because there are nice upstream who are providing patches also for older versions. Usually there are already patches available, not always for the good version. Sometimes we have to backport it. Then we prepare an upload with this +deb6uX suffix that we're using now for security updates, stable updates. This is a rather known territory for package maintainers. This is the hardest part sometimes, testing the update, making sure the issue is fixed. When tools have test suites, it's nice, but sometimes we have to set it up ourselves. Sometimes it is too hard, so we tend to, out of safety measure, to mail the mailing list and say "Ok, I've done my best, but please double check" It doesn't happen often that we have testers but some lts users are willing to test it before ??? It would be really nice if we had more of those. And if we don't get any negative feedback, then we upload. Then, you have to send an announce. We call that DLA for Debian LTS Advisory. We have a little script ./bin/gen-DLA that generates the template mail and you have to add a little bit of description and basically send it to debian-lts-announce with a GPG signature of someone in the DD or DM keyring. The process is rather open to all maintainers even the write rights to the secure testing repository where we have this data/DLA/list file is writable by all Debian Developpers on Alioth so, really, the entrance barrier is rather low for Debian Maintainers and Debian Developpers. There's a file which is automatically updated when you generate this template and which will update the tracker on the web pages because all this data is nicely browsable on security-tracker.debian.org. And it's all linked from the package tracker. That's it. If you have a few questions, I'm here to answer you. [inaudible question] … library and operating server, so that you have same API for clients, or same ABI for clients and a newer server that actually gets security patches. [RH] In my head, yes, but we did not look further yet. I really want to use the BOF that is upcoming to discuss it As I said, the amount of funding we have allows us right now to keep up with security update but we do not have many spare cycles for dealing with such things. But it's slowly coming up, as I said there's potentially a new platinum sponsor. We will have a few hours free to look into this and I think it's probably the best solution but I don't know if it works in practice, I have not tried it. And since LTS, the end of life of Squeeze happens in February, it's not too far away so it's possibly a nice solution because upgrading to a newer upstream version is harder. [Q] I've heard ??? Freexian ??? you have enough contributors but you could do more work but you don't have the funding or is it, like, you also need more developpers? [RH] A bit of both, actually, the web page has always been very clear on the rules That means any Debian Developper who has made security uploads on its own already can apply and join the program and be funded. Fortunately, I did not have too many because it would have been hard, but it has been steadily growing over the months. Last year, there was only Thorsten, Holger and me, but in the meantime we got Ben Hutchings who is helping us on the kernel and a few more. I'm always looking, trying to make sure that we don't give too much money to a single person because it's Debian, there has been ??? in the past. I've been involved, I know it. I don't want anyone to have their life depend on LTS work, really. And I don't want the opposite way also, LTS to depend on a single person that can go away. I try to limit it to maximum 20 ??? per month for each person and right now, we're well below that so it's good. By the way, most Debian Developpers are currently european, the fact that everything is euro-based is fine but one developer is american and it's no problem to pay in dollars So the amount will vary with the rates but it's not a problem. And even more, I'm surprised to have no contributors in the countries where the rate of labor is well bellow. I mean, if we have south american contributors, or african or I don't know, I don't want to stigmatize anyone, but if there are people who could live for much more, if they would be payed 10 hours and have made their month, they can still join, it's not a problem, it's not a high rate because we want only europeans, it's because we want to allow everybody and if they can be payed for 10 hours by Freexian and then spend the rest of the month doing free work on Debian, the better. If you want to join the program, get in touch, there are details on the webpages and the more people funded the better. [Q] How do you or other developers motivate yourself to do this kind of LTS work, because obviously it's important for companies to have stable releases, but it also, to me at least, doesn't seem very exciting from a technology standpoint [RH] Do you think it's exciting or not exciting? I'm not sure I understood. Not exciting, ok. I agree, it's not exciting. [laughter] [inaudible question] [RH] A bit of both. I want Debian to succeed and I know that LTS support is important for Debian to succeed so I want to make sure the project is working well and alive but clearly, I'm doing it because… Let's say, the security support work, providing security update, backporting code, it's for money and because I need to live. But organizing the LTS project is something I do for free because I want it for Debian because Debian needs it. [Q] I have a question. Who announces the public relation, the marketing stuff of the Debian Long Term Support? Do you do it yourself or do you have some professional help? [RH] I do not have any professional help. Basically, the way we… Well, we could do much better, I mean, in number of sponsors. There are almost no US-based sponsors, there are lots of US companies that could help. It's true, I'm fighting between my time spending free time on Debian, doing LTS Debian. I contact a few companies that people suggest me to contact, but I'm not doing much things by myself. [Q] So, you would benefit a lot from some professional help in the marketing department or [RH] Yes [Q] public relation [RH] Yes, but I don't have money to fund this or not much. The 10€ difference is for administrative cost. Maybe a bit for this kind of stuff but it doesn't look like a lot for this. [Q] I just wanted to ask to the previous questioner about finding motivation for providing patches for LTS packages and in my view it's not very different from providing security patches for stable. Sometimes it's a bit harder because the software is a bit older but we overcame this situation for wireshark for example by updating wireshark because it was quite impossible to backport older security fixes. I think it's part of the maintainance. If you maintain a project, you should maintain the security issues in stable and if you care a bit more and if you have a little more time, you can care about the package in LTS as well. I did it for wireshark for free, because it's easy for me. [RH] I'm grateful, thank you. I want to point out that it's a good thing that LTS and security are different, because we have different policies like we said, importing a new upstream version is something we can do when it doesn't break too much. The command line interface had not changed so badly between both versions so it was possible. It's not possible for all projects, but we are not so strict on new upstream versions And if you look at what others have been doing, RedHat, they too import new upstream version quite often, in fact, even on libraries like nss. [Q] Due to lack of browsers in LTS support and Xen support was it an issue with companies you are working on, who are sponsoring Freexian? [RH] I'm not sure I understood. [Q] The lack of browsers support and Xen support in the LTS version, was that an issue with the companies who you work with, who are sponsoring your work? [RH] Browsers, not so much. Most sponsors are hosters and stuff like that but they were annoyed by the lack of qemu or Xen support. They did upgrade their host machines but not their guest machines for their customers. Obviously, that would be the first priority to fix. Browser is nice, but it's also one of those cases where we just need to integrate newer upstream versions ??? doing it. And I was hoping for Raphael Geissert to do it because he does it for Électricité de France. I'll catch him. Convince him. I know him quite well, he's working in Lyon near me, not far away. [Q] Having sponsors for doing the LTS work is a signal of the need for LTS versions, but do you have additional statistics on the usage of LTS? [RH] Not really. Since LTS is not hosted in security mirrors but on all mirrors, we don't have access to all the statistics. But I know from mails, thank mails and stuff like that that it's really appreciated and used quite a lot, even from end users. There are users who like the antiquated GNOME stuff. I'm fine with GNOME 3, by the way. Yes, it's really widely used and widely appreciated. [Rhonda] Thanks for your attention. [Applause]