[Talkmeister]: Welcome, our next talk will
be about the Debian Long Term support
team and the speaker is
Raphaël Hertzog.
[Raphaël Hertzog]: Hello.
Today I will speak a bit about Debian
long term support.
I guess most of you already know about
it.
Are there some people who have no
idea what this is about?
No, good.
I will make my talk in 3 parts.
First I will present the team, how it
works
I will give some facts about how it
evolved over the first years.
I took some time to collect statistics
and believe they are rather interesting
I will also speak about the future
but there will be a separate discussion
about this in a BoF later this week.
I will show you how to help because, like
any other team in Debian it is open
to anyone. We welcome help.
At the end I will answer your questions.
What is LTS about?
The idea is really simple.
We wanted to extend the support period
of all Debian releases.
Currently it is basically for 1 year after
the next stable release comes out.
We wanted to extend this to 5 years to
match, at least, Ubuntu's offering.
which is not our competitor, but for the
companies that are making choices
it is one of the important criteria.
So we wanted to do as well.
Since we publish new stable releases
every 2 years it is roughly 3 years.
A nice side benefit is that the user can
skip a full release.
We don't officially support dist-upgrade
over going from Debian 6 to 8
but you can do 2 dist-upgrades at
the same time, limiting the downtime
to once every 5 years.
By the way, in practice, in simple server
configurations, dist-upgrades tend to
work rather well even across 2 releases.
Keeping a distribution secure for 5 years
is a real challenge.
It is hard work that not everybody is
willing to do.
In Debian all the work is done by
volunteers who do the work they enjoy.
Generally we enjoy working on new
features on top of latest releases
and not really on backporting patches to
crud that was written 5 years ago.
The security team has limited resources
so we could not just ask the security
team to now do twice the work.
But they were still really interested in
the project and wanted to support the idea
and really helped to get it bootstrapped.
The security team has a slightly larger
scope.
They support all architectures, which
means you have lots of problems of
coordination when security updates do
not compile and stuff like that.
What did we do?
We restricted the scope by picking
the 2 most popular architectures
that most users care about.
We have had some demand for ARM
architectures but up to now we only
support amd64 and i386.
We also excluded some packages from
security support.
Either because they are taking too much
time, like a security issue every 2 weeks
or that upstream is not cooperative
enough to be able to support it.
This list was basically made by the
current security team based on their
experience of doing security support.
If you look at the list there are some
important restrictions.
There's no xen, no kvm, no rails,
no browser. It sucks a bit.
But it's a way to get it started.
I think we can do better for wheezy.
Basically there is no virtualization
support on the host, only on the guest.
The security team helped to bootstrap
the LTS team but it is not the same team.
Obviously there are members of the
security team who also work on the LTS
team. They work in close collaboration.
We have regular contact and they watch our
mailing lists etc.
But the policies are different and the
infrastructure is separate,
which is a problem but I will talk about
that later.
We have a dedicated mailing list
debian-lts@lists.debian.org
and a dedicated IRC channel as well.
You are welcome to subscribe and to
join.
It's a new team which means new people
and new members.
Where do they come from?
The first idea was to get help from
people in various companies
who are already doing such in-house
support.
We had contact with EDF, and still have,
but they were one of the first
companies who were pushing for this
because they basically said
we are doing this already and we can
share with other companies.
The idea was to pool security support of
multiple companies.
We sent a press release asking
companies to join.
We had a few responses.
But I'll come back later to how it evolved
It's not really satisfying.
The other thing that we did is that we
offered companies the option to
fund the project to bring money and use
this to pay the work of
actual Debian contributors to do the
security updates that we need.
We have wiki pages listing all the ways
that companies can help with money.
In practice, most of the (wanting to be)
paid contributors joined together
under a single offer managed by
Freexian SARL which is my own company.
I'll quickly explain how this works.
Most companies don't want to bother
bringing human resources
They buy long term support contracts
from Freexian.
We have a rate. When you give €85 you
fund 1 hour of LTS work.
This is the current list of sponsors.
Top level gold sponsors sponsoring
more than 1 day of work per month.
On the other side we have Debian
contributors that are doing the work
and Freexian is paying them. There is a
small difference between the rate
to cover administration costs because I
have to handle the invoices
and some customers are using Paypal
which is taking a cut.
We ask contributors to follow some rules.
There is a requirement to publish a
monthly report on work done
on paid time. So they won't get paid until
they have published a report.
So everybody can know how the money
has been spent.
Currently we have 7 Debian contributors
and about 30 sponsors.
Some figures.
Who uploaded packages?
How has it evolved since June last year?
How is the funding evolving?
I just updated those figures a few
days ago.
I used this talk before at the mini
DebConf in Lyon in March,
but I updated it again.
The number of uploads is roughly over
one year since we started last year.
Over 300 uploads so it is not so much
but it is almost 1 per day.
So it is significant work.
I have given a state here of who paid
for the work and who did it on the left
The sponsors of Freexian are paying for
most of the uploads.
None is a separate category grouping all
Debian maintainers.
There are maintainers who are taking
care of their own packages in LTS.
Security team is members of the security
team who also work within the LTS team.
EDF is Électricité de France
Individuals are Debian developers that
have listed themselves as members of
the LTS team and did uploads for packages
of other maintainers not their own.
Credativ is a German company that you
probably know.
They have a booth here if you want a
job.
Toshiba, Univention, Catalyst etc
are other lower figures.
On the right are people. The top 5 people
are paid by Freexian.
Raphaël Geissert is working for EDF.
Thijs is a member of the security team.
Kurt is openssl maintainer so he maintains
his own packages.
He has a lot of work.
Mike Gabriel is also paid by Freexian.
Christoph Bieldl is mainly maintaining
the debian-security-support in Squeeze LTS
Nguyen Cong is employed by Toshiba.
Christoph Berg is employed by credativ
doing PostGreSQL maintainance.
How did it evolve over the year?
Again it is by affiliation.
The big blue part is paid contributors
You don't see it very well but the part
about maintainers is this one [points]
It tends to do better over the months
because here we started to
contact maintainers every time that we
have a new upload coming up
and ask them first 'do you want to handle
it yourself' so it slightly increased.
but the contribution of other companies
has not really increased over time.
Rather it has disappeared. It is
unfortunate but it looks like paid
contributors are more productive than
others.
In particular with EDF, they do the work,
but with some lag and we are faster
so they just reuse what we have done. I
want to talk to Raphaël to see
how we can do better towards this.
How did the sponsorship level evolve?
We have a steady increase, which is
rather nice. It is not a huge amount but
it is significant because we fund almost
80 hours per month.
It is close to our first goal. We wanted
that amount to be able to sustain
ourselves.
If you look at the sponsors, we have a
few big ones, possibly one very big
We can't give the name officially yet so
I won't.It will be a big jump in the graph
A few gold and many small sponsors.
I don't want to be dependent too much on
one big sponsor. I really prefer many
sponsors who are doing small donations
but donations which are sustainable
year after year because we are not here
for 1 year or 2.
We want to do it over the long term.
We have some figures about how many
hours have been funded since the start
Feel free to interrupt me if you have any
questions. I can take them at any time
That's it for evolution. Now, the future.
What do we expect for the future?
First keep doing what we have been up
to
Keep supporting the current set of packages.
But for wheezy long term support we
would really like to have more
supported packages. A browser would
be nice for desktop deployment.
Virtualization support is also important
for many companies so we should be
able to support something here.
Also we want to avoid some pitfalls that
we had with squeeze LTS.
As you know LTS users are currently
required to add a separate source
list entry with squeeze-lts repository. The
security.debian.org squeeze
repository is unused. It should be
possible for the LTS team to
continue using the same repository as
the security team once it no longer
use it. This will be the topic of a BoF
next week on Tue at 1800.
What's the problem with supporting the
current set of packages?
For example, we have MySQL right now in
Squeeze LTS in version 5.1
which is no longer supported by Oracle.
We don't even know if it's affected by
security issues because
Oracle doesn't give info on
non supported releases.
This is a problem, we should consider
switching to a newer version,
but newer versions involve library
transition, which is not really nice
in Long Term Support releases.
There is some work to do here if we want
to do something serious.
And we have other problems, other
packages which are similar.
From time to time, we backport, we take
newer upstream versions.
We did that for wireshark for example.
This is what I said before, the limited
scope sucks, we want to be able to
support more packages and we need more
support for this.
[Q] Speaking of wireshark, we switched to
Wheezy's version, so it's solved.
[Raphael] Yes, exactly, that's what I just
said.
It used to be a problem.
That's a part I did no update since March.
Additionally, the problem with a separate
repository is that
there is a propagation delay to the
mirrors
that we don't have with
security.debian.org.
I will speak a bit of how the team works.
Basically, the first step is triaging new
security issues.
We have a list of CVEs that comes in and
there are added to a text file
data/CVE/list.
Someone dispatches those on source
packages and then someone else reviews
status in each release.
Then the LTS team reviews the status in
Squeeze and members of security team
review status in Wheezy and Jessie.
And then, we decide what we must do with
the package.
Depending on the analysis, either we say
"We need to prepare an update", so we add
it to data/dla-needed.txt
and someone will have to take care of
preparing the update.
Or we say "The issue does not apply, does
not affect the current version"
or we ignore it because the package is
unsupported or because
the issue is really minor, not severe
enough.
Sometimes, it can be that the issue is
already fixed in Debian
due to some maintainer choices.
When we do this classification, we contact
the maintainer to keep him in the loop
and offer him the possibility to help us.
Here's what such a text file looks like.
The bold line is the line that we are
adding when we have decided
what we're doing with the packages.
This is all in the Subversion repository
on Alioth.
Then, someone has to prepare the security
update.
Basically, looking for a patch, it's often
useful to be able to look up at
RedHat or Ubuntu or upstream.
Best case is upstream because there are
nice upstream who are providing
patches also for older versions.
Usually there are already patches
available,
not always for the good version.
Sometimes we have to backport it.
Then we prepare an upload with this
+deb6uX suffix that we're using now
for security updates, stable updates.
This is a rather known territory for
package maintainers.
This is the hardest part sometimes,
testing the update, making sure the issue
is fixed.
When tools have test suites, it's nice,
but sometimes we have to set it up
ourselves.
Sometimes it is too hard, so we tend to,
out of safety measure,
to mail the mailing list and say
"Ok, I've done my best, but please double
check"
It doesn't happen often that we have
testers
but some lts users are willing to test it
before ???
It would be really nice if we had
more of those.
And if we don't get any negative feedback,
then we upload.
Then, you have to send
an announce.
We call that DLA for Debian LTS
Advisory.
We have a little script
./bin/gen-DLA
that generates the template mail
and you have to add
a little bit of description
and basically send it to
debian-lts-announce
with a GPG signature of someone in the
DD or DM keyring.
The process is rather open to
all maintainers
even the write rights to the secure
testing repository
where we have this data/DLA/list file
is writable by all Debian Developpers on
Alioth
so, really, the entrance barrier is
rather low for Debian Maintainers
and Debian Developpers.
There's a file which is automatically
updated when you generate this template
and which will update the tracker on the
web pages
because all this data is nicely browsable
on security-tracker.debian.org.
And it's all linked from the package
tracker.
That's it. If you have a few questions,
I'm here to answer you.
[inaudible question]
… library and operating server,
so that you have same API for clients,
or same ABI for clients and
a newer server that actually gets
security patches.
[RH] In my head, yes, but we did not look
further yet.
I really want to use the BOF that is
upcoming to discuss it
As I said, the amount of funding we have
allows us right now
to keep up with security update but
we do not have many spare cycles
for dealing with such things.
But it's slowly coming up,
as I said there's potentially a new
platinum sponsor.
We will have a few hours free to look
into this and I think
it's probably the best solution
but I don't know if it works in practice,
I have not tried it.
And since LTS, the end of life of Squeeze
happens in February,
it's not too far away so it's possibly
a nice solution
because upgrading to a newer upstream
version is harder.
[Q] I've heard ???
Freexian ???
you have enough contributors
but you could do more work
but you don't have the funding or
is it, like, you also need
more developpers?
[RH] A bit of both, actually,
the web page has always been very clear
on the rules
That means any Debian Developper who has
made security uploads on its own already
can apply and join the program and
be funded.
Fortunately, I did not have too many
because it would have been hard,
but it has been steadily growing over
the months.
Last year, there was only Thorsten,
Holger and me,
but in the meantime we got Ben Hutchings
who is helping us on the kernel
and a few more.
I'm always looking, trying to make sure
that we don't give too much money
to a single person
because it's Debian, there has been
??? in the past.
I've been involved, I know it.
I don't want anyone to have their life
depend on LTS work, really.
And I don't want the opposite way also,
LTS to depend on a single person
that can go away.
I try to limit it to maximum 20 ???
per month for each person
and right now, we're well below that
so it's good.
By the way, most Debian Developpers are
currently european,
the fact that everything is euro-based
is fine
but one developer is american and it's
no problem to pay in dollars
So the amount will vary with the rates
but it's not a problem.
And even more, I'm surprised to have no
contributors
in the countries where the rate of labor
is well bellow.
I mean, if we have south american
contributors,
or african or I don't know, I don't want
to stigmatize anyone,
but if there are people who could live
for much more,
if they would be payed 10 hours and
have made their month,
they can still join, it's not a problem,
it's not a high rate because we want only
europeans,
it's because we want to allow everybody
and if they can be payed for 10 hours
by Freexian and then spend
the rest of the month doing free work
on Debian, the better.
If you want to join the program,
get in touch,
there are details on the webpages
and the more people funded
the better.
[Q] How do you or other developers
motivate yourself
to do this kind of LTS work,
because obviously it's important
for companies to have
stable releases,
but it also, to me at least, doesn't seem
very exciting from a technology standpoint
[RH] Do you think it's exciting or
not exciting?
I'm not sure I understood.
Not exciting, ok.
I agree, it's not exciting.
[laughter]
[inaudible question]
[RH] A bit of both.
I want Debian to succeed and I know that
LTS support is important for Debian
to succeed
so I want to make sure the project is
working well and alive
but clearly, I'm doing it because…
Let's say, the security support work,
providing security update,
backporting code, it's for money and
because I need to live.
But organizing the LTS project is
something I do for free
because I want it for Debian
because Debian needs it.
[Q] I have a question.
Who announces the public relation,
the marketing stuff of
the Debian Long Term Support?
Do you do it yourself or do you have some
professional help?
[RH] I do not have any professional help.
Basically, the way we…
Well, we could do much better, I mean,
in number of sponsors.
There are almost no US-based sponsors,
there are lots of US companies
that could help.
It's true, I'm fighting between my time
spending free time on Debian,
doing LTS Debian.
I contact a few companies that people
suggest me to contact,
but I'm not doing much things
by myself.
[Q] So, you would benefit a lot from
some professional help
in the marketing department or
[RH] Yes
[Q] public relation
[RH] Yes, but I don't have money
to fund this or not much.
The 10€ difference is for
administrative cost.
Maybe a bit for this kind of stuff but
it doesn't look like a lot for this.
[Q] I just wanted to ask to the previous
questioner about
finding motivation for providing patches
for LTS packages and in my view
it's not very different from providing
security patches for stable.
Sometimes it's a bit harder because the
software is a bit older
but we overcame this situation
for wireshark for example
by updating wireshark because it was
quite impossible to backport older
security fixes.
I think it's part of the maintainance.
If you maintain a project, you should
maintain the security issues in stable
and if you care a bit more and if you have
a little more time,
you can care about the package in LTS
as well.
I did it for wireshark for free, because
it's easy for me.
[RH] I'm grateful, thank you.
I want to point out that it's a good
thing that
LTS and security are different,
because we have different policies like
we said,
importing a new upstream version is
something we can do
when it doesn't break too much.
The command line interface had not changed
so badly between both versions
so it was possible.
It's not possible for all projects, but we
are not so strict on new upstream versions
And if you look at what others have been
doing, RedHat,
they too import new upstream version
quite often, in fact,
even on libraries like nss.
[Q] Due to lack of browsers in LTS support
and Xen support
was it an issue with companies you are
working on, who are sponsoring Freexian?
[RH] I'm not sure I understood.
[Q] The lack of browsers support and
Xen support in the LTS version,
was that an issue with the companies
who you work with,
who are sponsoring your work?
[RH] Browsers, not so much.
Most sponsors are hosters and
stuff like that
but they were annoyed by the lack of
qemu or Xen support.
They did upgrade their host machines
but not their guest machines
for their customers.
Obviously, that would be
the first priority to fix.
Browser is nice, but it's also one of
those cases where we just need to
integrate newer upstream versions
??? doing it.
And I was hoping for Raphael Geissert
to do it because he does it for
Électricité de France.
I'll catch him.
Convince him.
I know him quite well, he's working in
Lyon near me, not far away.
[Q] Having sponsors for doing the LTS
work is a signal of
the need for LTS versions,
but do you have additional statistics on
the usage of LTS?
[RH] Not really.
Since LTS is not hosted in security
mirrors but on all mirrors,
we don't have access to all the
statistics.
But I know from mails, thank mails and
stuff like that
that it's really appreciated and used
quite a lot,
even from end users.
There are users who like the antiquated
GNOME stuff.
I'm fine with GNOME 3, by the way.
Yes, it's really widely used and widely
appreciated.
[Rhonda] Thanks for your attention.
[Applause]