9:59:59.000,9:59:59.000 [Talkmeister]: Welcome, our next talk will[br]be about the Debian Long Term support 9:59:59.000,9:59:59.000 team and the speaker is[br]Raphaël Hertzog. 9:59:59.000,9:59:59.000 [Raphaël Hertzog]: Hello. 9:59:59.000,9:59:59.000 Today I will speak a bit about Debian[br]long term support. 9:59:59.000,9:59:59.000 I guess most of you already know about[br]it. 9:59:59.000,9:59:59.000 Are there some people who have no[br]idea what this is about? 9:59:59.000,9:59:59.000 No, good. 9:59:59.000,9:59:59.000 I will make my talk in 3 parts. 9:59:59.000,9:59:59.000 First I will present the team, how it[br]works 9:59:59.000,9:59:59.000 I will give some facts about how it[br]evolved over the first years. 9:59:59.000,9:59:59.000 I took some time to collect statistics[br]and believe they are rather interesting 9:59:59.000,9:59:59.000 I will also speak about the future 9:59:59.000,9:59:59.000 but there will be a separate discussion[br]about this in a BoF later this week. 9:59:59.000,9:59:59.000 I will show you how to help because, like[br]any other team in Debian it is open 9:59:59.000,9:59:59.000 to anyone. We welcome help. 9:59:59.000,9:59:59.000 At the end I will answer your questions. 9:59:59.000,9:59:59.000 What is LTS about? 9:59:59.000,9:59:59.000 The idea is really simple. 9:59:59.000,9:59:59.000 We wanted to extend the support period[br]of all Debian releases. 9:59:59.000,9:59:59.000 Currently it is basically for 1 year after[br]the next stable release comes out. 9:59:59.000,9:59:59.000 We wanted to extend this to 5 years to[br]match, at least, Ubuntu's offering. 9:59:59.000,9:59:59.000 which is not our competitor, but for the[br]companies that are making choices 9:59:59.000,9:59:59.000 it is one of the important criteria.[br]So we wanted to do as well. 9:59:59.000,9:59:59.000 Since we publish new stable releases[br]every 2 years it is roughly 3 years. 9:59:59.000,9:59:59.000 A nice side benefit is that the user can[br]skip a full release. 9:59:59.000,9:59:59.000 We don't officially support dist-upgrade[br]over going from Debian 6 to 8 9:59:59.000,9:59:59.000 but you can do 2 dist-upgrades at[br]the same time, limiting the downtime 9:59:59.000,9:59:59.000 to once every 5 years. 9:59:59.000,9:59:59.000 By the way, in practice, in simple server[br]configurations, dist-upgrades tend to 9:59:59.000,9:59:59.000 work rather well even across 2 releases. 9:59:59.000,9:59:59.000 Keeping a distribution secure for 5 years[br]is a real challenge. 9:59:59.000,9:59:59.000 It is hard work that not everybody is[br]willing to do. 9:59:59.000,9:59:59.000 In Debian all the work is done by[br]volunteers who do the work they enjoy. 9:59:59.000,9:59:59.000 Generally we enjoy working on new[br]features on top of latest releases 9:59:59.000,9:59:59.000 and not really on backporting patches to[br]crud that was written 5 years ago. 9:59:59.000,9:59:59.000 The security team has limited resources[br]so we could not just ask the security 9:59:59.000,9:59:59.000 team to now do twice the work. 9:59:59.000,9:59:59.000 But they were still really interested in[br]the project and wanted to support the idea 9:59:59.000,9:59:59.000 and really helped to get it bootstrapped. 9:59:59.000,9:59:59.000 The security team has a slightly larger[br]scope. 9:59:59.000,9:59:59.000 They support all architectures, which[br]means you have lots of problems of 9:59:59.000,9:59:59.000 coordination when security updates do[br]not compile and stuff like that. 9:59:59.000,9:59:59.000 What did we do? 9:59:59.000,9:59:59.000 We restricted the scope by picking[br]the 2 most popular architectures 9:59:59.000,9:59:59.000 that most users care about.[br] 9:59:59.000,9:59:59.000 We have had some demand for ARM[br]architectures but up to now we only 9:59:59.000,9:59:59.000 support amd64 and i386. 9:59:59.000,9:59:59.000 We also excluded some packages from[br]security support. 9:59:59.000,9:59:59.000 Either because they are taking too much[br]time, like a security issue every 2 weeks 9:59:59.000,9:59:59.000 or that upstream is not cooperative[br]enough to be able to support it. 9:59:59.000,9:59:59.000 This list was basically made by the[br]current security team based on their 9:59:59.000,9:59:59.000 experience of doing security support. 9:59:59.000,9:59:59.000 If you look at the list there are some[br]important restrictions. 9:59:59.000,9:59:59.000 There's no xen, no kvm, no rails,[br]no browser. It sucks a bit. 9:59:59.000,9:59:59.000 But it's a way to get it started. 9:59:59.000,9:59:59.000 I think we can do better for wheezy. 9:59:59.000,9:59:59.000 Basically there is no virtualization[br]support on the host, only on the guest. 9:59:59.000,9:59:59.000 The security team helped to bootstrap[br]the LTS team but it is not the same team. 9:59:59.000,9:59:59.000 Obviously there are members of the[br]security team who also work on the LTS 9:59:59.000,9:59:59.000 team. They work in close collaboration. 9:59:59.000,9:59:59.000 We have regular contact and they watch our[br]mailing lists etc. 9:59:59.000,9:59:59.000 But the policies are different and the[br]infrastructure is separate, 9:59:59.000,9:59:59.000 which is a problem but I will talk about[br]that later. 9:59:59.000,9:59:59.000 We have a dedicated mailing list 9:59:59.000,9:59:59.000 and a dedicated IRC channel as well. 9:59:59.000,9:59:59.000 You are welcome to subscribe and to[br]join. 9:59:59.000,9:59:59.000 It's a new team which means new people[br]and new members. 9:59:59.000,9:59:59.000 Where do they come from? 9:59:59.000,9:59:59.000 The first idea was to get help from[br]people in various companies 9:59:59.000,9:59:59.000 who are already doing such in-house[br]support. 9:59:59.000,9:59:59.000 We had contact with EDF, and still have,[br]but they were one of the first 9:59:59.000,9:59:59.000 companies who were pushing for this[br]because they basically said 9:59:59.000,9:59:59.000 we are doing this already and we can[br]share with other companies. 9:59:59.000,9:59:59.000 The idea was to pool security support of[br]multiple companies. 9:59:59.000,9:59:59.000 We sent a press release asking[br]companies to join. 9:59:59.000,9:59:59.000 We had a few responses. 9:59:59.000,9:59:59.000 But I'll come back later to how it evolved[br]It's not really satisfying. 9:59:59.000,9:59:59.000 The other thing that we did is that we[br]offered companies the option to 9:59:59.000,9:59:59.000 fund the project to bring money and use[br]this to pay the work of 9:59:59.000,9:59:59.000 actual Debian contributors to do the[br]security updates that we need. 9:59:59.000,9:59:59.000 We have wiki pages listing all the ways[br]that companies can help with money. 9:59:59.000,9:59:59.000 In practice, most of the (wanting to be)[br]paid contributors joined together 9:59:59.000,9:59:59.000 under a single offer managed by[br]Freexian SARL which is my own company. 9:59:59.000,9:59:59.000 I'll quickly explain how this works. 9:59:59.000,9:59:59.000 Most companies don't want to bother[br]bringing human resources ??? (08:25) 9:59:59.000,9:59:59.000 They buy long term support contracts[br]from Freexian. 9:59:59.000,9:59:59.000 We have a rate. When you give €85 you[br]fund 1 hour of LTS work. 9:59:59.000,9:59:59.000 This is the current list of sponsors. 9:59:59.000,9:59:59.000 Top level gold sponsors sponsoring[br]more than 1 day of work per month. 9:59:59.000,9:59:59.000 On the other side we have Debian [br]contributors that are doing the work 9:59:59.000,9:59:59.000 and Freexian is paying them. There is a[br]small difference between the rate 9:59:59.000,9:59:59.000 to cover administration costs because I[br]have to handle the invoices 9:59:59.000,9:59:59.000 and some customers are using Paypal[br]which is taking a cut. 9:59:59.000,9:59:59.000 We ask contributors to follow some rules. 9:59:59.000,9:59:59.000 There is a requirement to publish a[br]monthly report on work done 9:59:59.000,9:59:59.000 on paid time. So they won't get paid until[br]they have published a report. 9:59:59.000,9:59:59.000 So everybody can know how the money[br]has been spent. 9:59:59.000,9:59:59.000 Currently we have 7 Debian contributors[br]and about 30 sponsors. 9:59:59.000,9:59:59.000 Some figures. 9:59:59.000,9:59:59.000 Who uploaded packages?[br]How has it evolved since June last year? 9:59:59.000,9:59:59.000 How is the funding evolving? 9:59:59.000,9:59:59.000 I just updated those figures a few[br]days ago. 9:59:59.000,9:59:59.000 I used this talk before at the mini[br]DebConf in Lyon in March, 9:59:59.000,9:59:59.000 but I updated it again. 9:59:59.000,9:59:59.000 The number of uploads is roughly over[br]one year since we started last year. 9:59:59.000,9:59:59.000 Over 300 uploads so it is not so much[br]but it is almost 1 per day. 9:59:59.000,9:59:59.000 So it is significant work. 9:59:59.000,9:59:59.000 I have given a state here of who paid[br]for the work and who did it on the left 9:59:59.000,9:59:59.000 The sponsors of Freexian are paying for[br]most of the uploads. ??? 9:59:59.000,9:59:59.000 None is a separate category grouping all[br]Debian maintainers. 9:59:59.000,9:59:59.000 There are maintainers who are taking[br]care of their own packages in LTS. 9:59:59.000,9:59:59.000 Security team is members of the security[br]team who also work within the LTS team. 9:59:59.000,9:59:59.000 EDF is Électricité de France 9:59:59.000,9:59:59.000 Individuals are Debian developers that[br]have listed themselves as members of 9:59:59.000,9:59:59.000 the LTS team and did uploads for packages[br]of other maintainers not their own. 9:59:59.000,9:59:59.000 Credativ is a German company that you[br]probably know. 9:59:59.000,9:59:59.000 They have a booth here if you want a[br]job. 9:59:59.000,9:59:59.000 Toshiba, Univention, Catalyst etc[br]are other lower ??? 9:59:59.000,9:59:59.000 On the right are people. The top 5 people[br]are paid by Freexian. 9:59:59.000,9:59:59.000 Raphaël Geissert is working for EDF. 9:59:59.000,9:59:59.000 Thijs is a member of the security team. 9:59:59.000,9:59:59.000 Kurt is openssl maintainer ??? 9:59:59.000,9:59:59.000 Mike Gabriel is also paid by Freexian. 9:59:59.000,9:59:59.000 Christoph Bieldl is mainly maintaining[br]the debian-security-support in Squeeze LTS 9:59:59.000,9:59:59.000 Nguyen Cong is employed by Toshiba. 9:59:59.000,9:59:59.000 Christoph Berg is employed by creditv[br]doing postgresql maintainence. 9:59:59.000,9:59:59.000 How did it evolve over the year? 9:59:59.000,9:59:59.000 Again it is by affiliation. 9:59:59.000,9:59:59.000 The big blue part is paid contributors 9:59:59.000,9:59:59.000 You don't see it very well but the part[br]about maintainers is this one [points] 9:59:59.000,9:59:59.000 It tends to do better over the months[br]because here we started to 9:59:59.000,9:59:59.000 contact maintainers every time that we[br]have a new upload coming up 9:59:59.000,9:59:59.000 and ask them first 'do you want to handle[br]it yourself' so it slightly increased. 9:59:59.000,9:59:59.000 but the contribution of other companies[br]has not really increased over time. 9:59:59.000,9:59:59.000 Rather it has disappeared. It is[br]unfortunate but it looks like paid 9:59:59.000,9:59:59.000 contributors are more productive than[br]others. 9:59:59.000,9:59:59.000 In particular with EDF, they do the work,[br]but with some lag and we are faster 9:59:59.000,9:59:59.000 so they just reuse what we have done. I[br]want to talk to Raphaël to see 9:59:59.000,9:59:59.000 how we can do better towards this. 9:59:59.000,9:59:59.000 How did the sponsorship level evolve? 9:59:59.000,9:59:59.000 We have a steady increase, which is[br]rather nice. It is not a huge amount but 9:59:59.000,9:59:59.000 it is significant because we fund almost[br]80 hours per month. 9:59:59.000,9:59:59.000 It is close to our first goal. We wanted[br]that amount to be able to sustain 9:59:59.000,9:59:59.000 ourselves. 9:59:59.000,9:59:59.000 If you look at the sponsors, we have a[br]few big ones, possibly one very big 9:59:59.000,9:59:59.000 We can't give the name officially yet so[br]I won't.It will be a big jump in the graph 9:59:59.000,9:59:59.000 A few gold and many small sponsors. 9:59:59.000,9:59:59.000 I don't want to be dependent too much on[br]one big sponsor. I really prefer many 9:59:59.000,9:59:59.000 sponsors who are doing small donations[br]but donations which are sustainable 9:59:59.000,9:59:59.000 year after year because we are not here[br]for 1 year or 2. 9:59:59.000,9:59:59.000 We want to do it over the long term. 9:59:59.000,9:59:59.000 We have some figures about how many[br]hours have been funded since the start 9:59:59.000,9:59:59.000 Feel free to interrupt me if you have any[br]questions. I can take them at any time 9:59:59.000,9:59:59.000 That's it for evolution. Now, the future. 9:59:59.000,9:59:59.000 What do we expect for the future? 9:59:59.000,9:59:59.000 First keep doing what we have been up[br]to[br] 9:59:59.000,9:59:59.000 Keep supporting the current set of packages. 9:59:59.000,9:59:59.000 But for wheezy long term support we[br]would really like to have more 9:59:59.000,9:59:59.000 supported packages. A browser would[br]be nice for desktop deployment. 9:59:59.000,9:59:59.000 Virtualization support is also important[br]for many companies so we should be 9:59:59.000,9:59:59.000 able to support something here. 9:59:59.000,9:59:59.000 Also we want to avoid some pitfalls that[br]we had with squeeze LTS. 9:59:59.000,9:59:59.000 As you know LTS users are currently[br]required to add a separate source 9:59:59.000,9:59:59.000 list entry with squeeze-lts repository. The[br]security.debian.org squeeze 9:59:59.000,9:59:59.000 repository is unused. It should be[br]possible for the LTS team to 9:59:59.000,9:59:59.000 continue using the same repository as[br]the security team once it no longer 9:59:59.000,9:59:59.000 use it. This will be the topic of a BoF[br]next week on Tue at 1800. 9:59:59.000,9:59:59.000 What's the problem with supporting the[br]current set of packages? 9:59:59.000,9:59:59.000 For example, we have MySQL right now in[br]Squeeze LTS in version 5.1 9:59:59.000,9:59:59.000 which is no longer supported by Oracle. 9:59:59.000,9:59:59.000 We don't even know if it's affected by[br]security issues because 9:59:59.000,9:59:59.000 Oracle doesn't give info on[br]??? release. 9:59:59.000,9:59:59.000 This is a problem, we should consider[br]switching to a newer version, 9:59:59.000,9:59:59.000 but newer versions involve library[br]transition, which is not really nice 9:59:59.000,9:59:59.000 in Long Term Support releases. 9:59:59.000,9:59:59.000 There is some work to do here if we want[br]to do something serious. 9:59:59.000,9:59:59.000 And we have other problems, other[br]packages which are similar. 9:59:59.000,9:59:59.000 From time to time, we backport, we take[br]newer upstream versions. 9:59:59.000,9:59:59.000 We did that for wireshark for example. 9:59:59.000,9:59:59.000 This is what I said before, the limited[br]scope sucks, we want to be able to 9:59:59.000,9:59:59.000 support more packages and we need more[br]support for this. 9:59:59.000,9:59:59.000 [Q] Speaking of wireshark, we switched to[br]Wheezy's version, so it's solved. 9:59:59.000,9:59:59.000 [Raphael] Yes, exactly, that's what I just[br]said. 9:59:59.000,9:59:59.000 It used to be a problem. 9:59:59.000,9:59:59.000 That's a part I did no update since March. 9:59:59.000,9:59:59.000 Additionally, the problem with a separate[br]repository is that 9:59:59.000,9:59:59.000 there is a propagation delay to the[br]mirrors 9:59:59.000,9:59:59.000 that we don't have with[br]security.debian.org. 9:59:59.000,9:59:59.000 I will speak a bit of how the team works. 9:59:59.000,9:59:59.000 Basically, the first step is triaging new[br]security issues. 9:59:59.000,9:59:59.000 We have a list of CVEs that comes in and[br]there are added to a text file 9:59:59.000,9:59:59.000 data/CVE/list. 9:59:59.000,9:59:59.000 Some dispatches those on source packages[br]and then someone else reviews 9:59:59.000,9:59:59.000 status in each release. 9:59:59.000,9:59:59.000 Then the LTS team reviews the status in[br]Squeeze and members of security team 9:59:59.000,9:59:59.000 review status in Wheezy and Jessie. 9:59:59.000,9:59:59.000 And then, we decide what we must do with[br]the package. 9:59:59.000,9:59:59.000 Depending on the analysis, either we say 9:59:59.000,9:59:59.000 "We need to prepare an update", so we add[br]it to data/dla-needed.txt 9:59:59.000,9:59:59.000 and someone will have to take care of[br]preparing the update. 9:59:59.000,9:59:59.000 Or we say "The issue does not apply, does[br]not affect the current version" 9:59:59.000,9:59:59.000 or we ignore it because the package is[br]unsupported or because 9:59:59.000,9:59:59.000 the issue is really minor, not severe[br]enough. 9:59:59.000,9:59:59.000 Sometimes, it can be that the issue is[br]already fixed in Debian 9:59:59.000,9:59:59.000 due to some maintainer choices. 9:59:59.000,9:59:59.000 When we do this classification, we contact[br]the maintainer to keep him in the loop 9:59:59.000,9:59:59.000 and offer him the possibility to help us. 9:59:59.000,9:59:59.000 Here's what such a text file looks like. 9:59:59.000,9:59:59.000 The bold line is the line that we are[br]adding when we have decided 9:59:59.000,9:59:59.000 what we're doing with the packages. 9:59:59.000,9:59:59.000 This is all in the Subversion repository[br]on Alioth. 9:59:59.000,9:59:59.000 Then, someone has to prepare the security[br]update. 9:59:59.000,9:59:59.000 Basically, looking for a patch, it's often[br]useful to be able to look up at 9:59:59.000,9:59:59.000 RedHat or Ubuntu or upstream. 9:59:59.000,9:59:59.000 Best case is upstream because there are[br]nice upstream who are providing 9:59:59.000,9:59:59.000 patches also for older versions. 9:59:59.000,9:59:59.000 Usually there are already patches[br]available, 9:59:59.000,9:59:59.000 not always for the good version. 9:59:59.000,9:59:59.000 Sometimes we have to backport it. 9:59:59.000,9:59:59.000 Then we prepare an upload with this[br]+deb6uX suffix that we're using now 9:59:59.000,9:59:59.000 for security updates, stable updates. 9:59:59.000,9:59:59.000 This is a rather known territory for[br]package maintainers. 9:59:59.000,9:59:59.000 This is the hardest part sometimes, 9:59:59.000,9:59:59.000 testing the update, making sure the issue[br]is fixed. 9:59:59.000,9:59:59.000 When tools have test suites, it's nice, 9:59:59.000,9:59:59.000 but sometimes we have to set it up[br]ourselves. 9:59:59.000,9:59:59.000 Sometimes it is too hard, so we tend to,[br]out of safety measure, 9:59:59.000,9:59:59.000 to mail the mailing list and say 9:59:59.000,9:59:59.000 "Ok, I've done my best, but please double[br]check" 9:59:59.000,9:59:59.000 It doesn't happen often that we have[br]testers 9:59:59.000,9:59:59.000 but some lts users are willing to test it[br]before ??? 9:59:59.000,9:59:59.000 It would be really nice if we had[br]more of those. 9:59:59.000,9:59:59.000 And if we don't get any negative feedback, 9:59:59.000,9:59:59.000 then we upload.