-
[Talkmeister]: Welcome, our next talk will
be about the Debian Long Term support
-
team and the speaker is
Raphaël Hertzog.
-
[Raphaël Hertzog]: Hello.
-
Today I will speak a bit about Debian
long term support.
-
I guess most of you already know about
it.
-
Are there some people who have no
idea what this is about?
-
No, good.
-
I will make my talk in 3 parts.
-
First I will present the team, how it
works
-
I will give some facts about how it
evolved over the first years.
-
I took some time to collect statistics
and believe they are rather interesting
-
I will also speak about the future
-
but there will be a separate discussion
about this in a BoF later this week.
-
I will show you how to help because, like
any other team in Debian it is open
-
to anyone. We welcome help.
-
At the end I will answer your questions.
-
What is LTS about?
-
The idea is really simple.
-
We wanted to extend the support period
of all Debian releases.
-
Currently it is basically for 1 year after
the next stable release comes out.
-
We wanted to extend this to 5 years to
match, at least, Ubuntu's offering.
-
which is not our competitor, but for the
companies that are making choices
-
it is one of the important criteria.
So we wanted to do as well.
-
Since we publish new stable releases
every 2 years it is roughly 3 years.
-
A nice side benefit is that the user can
skip a full release.
-
We don't officially support dist-upgrade
over going from Debian 6 to 8
-
but you can do 2 dist-upgrades at
the same time, limiting the downtime
-
to once every 5 years.
-
By the way, in practice, in simple server
configurations, dist-upgrades tend to
-
work rather well even across 2 releases.
-
Keeping a distribution secure for 5 years
is a real challenge.
-
It is hard work that not everybody is
willing to do.
-
In Debian all the work is done by
volunteers who do the work they enjoy.
-
Generally we enjoy working on new
features on top of latest releases
-
and not really on backporting patches to
crud that was written 5 years ago.
-
The security team has limited resources
so we could not just ask the security
-
team to now do twice the work.
-
But they were still really interested in
the project and wanted to support the idea
-
and really helped to get it bootstrapped.
-
The security team has a slightly larger
scope.
-
They support all architectures, which
means you have lots of problems of
-
coordination when security updates do
not compile and stuff like that.
-
What did we do?
-
We restricted the scope by picking
the 2 most popular architectures
-
that most users care about.
-
We have had some demand for ARM
architectures but up to now we only
-
support amd64 and i386.
-
We also excluded some packages from
security support.
-
Either because they are taking too much
time, like a security issue every 2 weeks
-
or that upstream is not cooperative
enough to be able to support it.
-
This list was basically made by the
current security team based on their
-
experience of doing security support.
-
If you look at the list there are some
important restrictions.
-
There's no xen, no kvm, no rails,
no browser. It sucks a bit.
-
But it's a way to get it started.
-
I think we can do better for wheezy.
-
Basically there is no virtualization
support on the host, only on the guest.
-
The security team helped to bootstrap
the LTS team but it is not the same team.
-
Obviously there are members of the
security team who also work on the LTS
-
team. They work in close collaboration.
-
We have regular contact and they watch our
mailing lists etc.
-
But the policies are different and the
infrastructure is separate,
-
which is a problem but I will talk about
that later.
-
We have a dedicated mailing list
debian-lts@lists.debian.org
-
and a dedicated IRC channel as well.
-
You are welcome to subscribe and to
join.
-
It's a new team which means new people
and new members.
-
Where do they come from?
-
The first idea was to get help from
people in various companies
-
who are already doing such in-house
support.
-
We had contact with EDF, and still have,
but they were one of the first
-
companies who were pushing for this
because they basically said
-
we are doing this already and we can
share with other companies.
-
The idea was to pool security support of
multiple companies.
-
We sent a press release asking
companies to join.
-
We had a few responses.
-
But I'll come back later to how it evolved
It's not really satisfying.
-
The other thing that we did is that we
offered companies the option to
-
fund the project to bring money and use
this to pay the work of
-
actual Debian contributors to do the
security updates that we need.
-
We have wiki pages listing all the ways
that companies can help with money.
-
In practice, most of the (wanting to be)
paid contributors joined together
-
under a single offer managed by
Freexian SARL which is my own company.
-
I'll quickly explain how this works.
-
Most companies don't want to bother
bringing human resources
-
They buy long term support contracts
from Freexian.
-
We have a rate. When you give €85 you
fund 1 hour of LTS work.
-
This is the current list of sponsors.
-
Top level gold sponsors sponsoring
more than 1 day of work per month.
-
On the other side we have Debian
contributors that are doing the work
-
and Freexian is paying them. There is a
small difference between the rate
-
to cover administration costs because I
have to handle the invoices
-
and some customers are using Paypal
which is taking a cut.
-
We ask contributors to follow some rules.
-
There is a requirement to publish a
monthly report on work done
-
on paid time. So they won't get paid until
they have published a report.
-
So everybody can know how the money
has been spent.
-
Currently we have 7 Debian contributors
and about 30 sponsors.
-
Some figures.
-
Who uploaded packages?
How has it evolved since June last year?
-
How is the funding evolving?
-
I just updated those figures a few
days ago.
-
I used this talk before at the mini
DebConf in Lyon in March,
-
but I updated it again.
-
The number of uploads is roughly over
one year since we started last year.
-
Over 300 uploads so it is not so much
but it is almost 1 per day.
-
So it is significant work.
-
I have given a state here of who paid
for the work and who did it on the left
-
The sponsors of Freexian are paying for
most of the uploads.
-
None is a separate category grouping all
Debian maintainers.
-
There are maintainers who are taking
care of their own packages in LTS.
-
Security team is members of the security
team who also work within the LTS team.
-
EDF is Électricité de France
-
Individuals are Debian developers that
have listed themselves as members of
-
the LTS team and did uploads for packages
of other maintainers not their own.
-
Credativ is a German company that you
probably know.
-
They have a booth here if you want a
job.
-
Toshiba, Univention, Catalyst etc
are other lower figures.
-
On the right are people. The top 5 people
are paid by Freexian.
-
Raphaël Geissert is working for EDF.
-
Thijs is a member of the security team.
-
Kurt is openssl maintainer so he maintains
his own packages.
-
He has a lot of work.
-
Mike Gabriel is also paid by Freexian.
-
Christoph Bieldl is mainly maintaining
the debian-security-support in Squeeze LTS
-
Nguyen Cong is employed by Toshiba.
-
Christoph Berg is employed by credativ
doing PostGreSQL maintainance.
-
How did it evolve over the year?
-
Again it is by affiliation.
-
The big blue part is paid contributors
-
You don't see it very well but the part
about maintainers is this one [points]
-
It tends to do better over the months
because here we started to
-
contact maintainers every time that we
have a new upload coming up
-
and ask them first 'do you want to handle
it yourself' so it slightly increased.
-
but the contribution of other companies
has not really increased over time.
-
Rather it has disappeared. It is
unfortunate but it looks like paid
-
contributors are more productive than
others.
-
In particular with EDF, they do the work,
but with some lag and we are faster
-
so they just reuse what we have done. I
want to talk to Raphaël to see
-
how we can do better towards this.
-
How did the sponsorship level evolve?
-
We have a steady increase, which is
rather nice. It is not a huge amount but
-
it is significant because we fund almost
80 hours per month.
-
It is close to our first goal. We wanted
that amount to be able to sustain
-
ourselves.
-
If you look at the sponsors, we have a
few big ones, possibly one very big
-
We can't give the name officially yet so
I won't.It will be a big jump in the graph
-
A few gold and many small sponsors.
-
I don't want to be dependent too much on
one big sponsor. I really prefer many
-
sponsors who are doing small donations
but donations which are sustainable
-
year after year because we are not here
for 1 year or 2.
-
We want to do it over the long term.
-
We have some figures about how many
hours have been funded since the start
-
Feel free to interrupt me if you have any
questions. I can take them at any time
-
That's it for evolution. Now, the future.
-
What do we expect for the future?
-
First keep doing what we have been up
to
-
Keep supporting the current set of packages.
-
But for wheezy long term support we
would really like to have more
-
supported packages. A browser would
be nice for desktop deployment.
-
Virtualization support is also important
for many companies so we should be
-
able to support something here.
-
Also we want to avoid some pitfalls that
we had with squeeze LTS.
-
As you know LTS users are currently
required to add a separate source
-
list entry with squeeze-lts repository. The
security.debian.org squeeze
-
repository is unused. It should be
possible for the LTS team to
-
continue using the same repository as
the security team once it no longer
-
use it. This will be the topic of a BoF
next week on Tue at 1800.
-
What's the problem with supporting the
current set of packages?
-
For example, we have MySQL right now in
Squeeze LTS in version 5.1
-
which is no longer supported by Oracle.
-
We don't even know if it's affected by
security issues because
-
Oracle doesn't give info on
non supported releases.
-
This is a problem, we should consider
switching to a newer version,
-
but newer versions involve library
transition, which is not really nice
-
in Long Term Support releases.
-
There is some work to do here if we want
to do something serious.
-
And we have other problems, other
packages which are similar.
-
From time to time, we backport, we take
newer upstream versions.
-
We did that for wireshark for example.
-
This is what I said before, the limited
scope sucks, we want to be able to
-
support more packages and we need more
support for this.
-
[Q] Speaking of wireshark, we switched to
Wheezy's version, so it's solved.
-
[Raphael] Yes, exactly, that's what I just
said.
-
It used to be a problem.
-
That's a part I did no update since March.
-
Additionally, the problem with a separate
repository is that
-
there is a propagation delay to the
mirrors
-
that we don't have with
security.debian.org.
-
I will speak a bit of how the team works.
-
Basically, the first step is triaging new
security issues.
-
We have a list of CVEs that comes in and
there are added to a text file
-
data/CVE/list.
-
Someone dispatches those on source
packages and then someone else reviews
-
status in each release.
-
Then the LTS team reviews the status in
Squeeze and members of security team
-
review status in Wheezy and Jessie.
-
And then, we decide what we must do with
the package.
-
Depending on the analysis, either we say
-
"We need to prepare an update", so we add
it to data/dla-needed.txt
-
and someone will have to take care of
preparing the update.
-
Or we say "The issue does not apply, does
not affect the current version"
-
or we ignore it because the package is
unsupported or because
-
the issue is really minor, not severe
enough.
-
Sometimes, it can be that the issue is
already fixed in Debian
-
due to some maintainer choices.
-
When we do this classification, we contact
the maintainer to keep him in the loop
-
and offer him the possibility to help us.
-
Here's what such a text file looks like.
-
The bold line is the line that we are
adding when we have decided
-
what we're doing with the packages.
-
This is all in the Subversion repository
on Alioth.
-
Then, someone has to prepare the security
update.
-
Basically, looking for a patch, it's often
useful to be able to look up at
-
RedHat or Ubuntu or upstream.
-
Best case is upstream because there are
nice upstream who are providing
-
patches also for older versions.
-
Usually there are already patches
available,
-
not always for the good version.
-
Sometimes we have to backport it.
-
Then we prepare an upload with this
+deb6uX suffix that we're using now
-
for security updates, stable updates.
-
This is a rather known territory for
package maintainers.
-
This is the hardest part sometimes,
-
testing the update, making sure the issue
is fixed.
-
When tools have test suites, it's nice,
-
but sometimes we have to set it up
ourselves.
-
Sometimes it is too hard, so we tend to,
out of safety measure,
-
to mail the mailing list and say
-
"Ok, I've done my best, but please double
check"
-
It doesn't happen often that we have
testers
-
but some lts users are willing to test it
before ???
-
It would be really nice if we had
more of those.
-
And if we don't get any negative feedback,
-
then we upload.
-
Then, you have to send
an announce.
-
We call that DLA for Debian LTS
Advisory.
-
We have a little script
./bin/gen-DLA
-
that generates the template mail
and you have to add
-
a little bit of description
-
and basically send it to
debian-lts-announce
-
with a GPG signature of someone in the
DD or DM keyring.
-
The process is rather open to
all maintainers
-
even the write rights to the secure
testing repository
-
where we have this data/DLA/list file
-
is writable by all Debian Developpers on
Alioth
-
so, really, the entrance barrier is
rather low for Debian Maintainers
-
and Debian Developpers.
-
There's a file which is automatically
updated when you generate this template
-
and which will update the tracker on the
web pages
-
because all this data is nicely browsable
on security-tracker.debian.org.
-
And it's all linked from the package
tracker.
-
That's it. If you have a few questions,
I'm here to answer you.
-
[inaudible question]
-
… library and operating server,
-
so that you have same API for clients,
or same ABI for clients and
-
a newer server that actually gets
security patches.
-
[RH] In my head, yes, but we did not look
further yet.
-
I really want to use the BOF that is
upcoming to discuss it
-
As I said, the amount of funding we have
allows us right now
-
to keep up with security update but
we do not have many spare cycles
-
for dealing with such things.
-
But it's slowly coming up,
-
as I said there's potentially a new
platinum sponsor.
-
We will have a few hours free to look
into this and I think
-
it's probably the best solution
-
but I don't know if it works in practice,
I have not tried it.
-
And since LTS, the end of life of Squeeze
happens in February,
-
it's not too far away so it's possibly
a nice solution
-
because upgrading to a newer upstream
version is harder.
-
Not Synced
[Q] I've heard ???
Freexian ???
-
Not Synced
you have enough contributors
but you could do more work
-
Not Synced
but you don't have the funding or
is it, like, you also need
-
Not Synced
more developpers?
-
Not Synced
[RH] A bit of both, actually,
-
Not Synced
the web page has always been very clear
on the rules
-
Not Synced
That means any Debian Developper who has
made uploads on its own already
-
Not Synced
can apply and join the program and
be funded.
-
Not Synced
Fortunately, I did not have too many
-
Not Synced
because it would have been hard,
-
Not Synced
but it has been steadily growing over
the months.
-
Not Synced
Last year, there was only Thorsten,
Holger and me,
-
Not Synced
but in the meantime we got Ben Hutchings
who is helping us on the kernel
-
Not Synced
and a few more.
-
Not Synced
I'm always looking, trying to make sure
that we don't give too much money
-
Not Synced
to a single person
-
Not Synced
because it's Debian, there has been
??? in the past.
-
Not Synced
I've been involved, I know it.
-
Not Synced
I don't want anyone to have their life
depend on LTS work, really.
-
Not Synced
And I don't want the opposite way also,
LTS to depend on a single person
-
Not Synced
that can go away.
-
Not Synced
I try to limit it to maximum 20 ???
per month for each person
-
Not Synced
and right now, we're well below that
so it's good.
-
Not Synced
By the way, most Debian Developpers are
currently european,
-
Not Synced
the fact that everything is euro-based
is fine
-
Not Synced
but one developer is american and it's
no problem to pay in dollars
-
Not Synced
So the amount will vary with the rates
but it's not a problem.
-
Not Synced
And even more, I'm surprised to have no
contributors
-
Not Synced
in the countries where the rate of labor
is well bellow.
-
Not Synced
I mean, if we have south american
contributors,
-
Not Synced
or african or I don't know, I don't want
to stigmatize anyone,
-
Not Synced
but if there are people who could live
for much more,
-
Not Synced
if they would be payed 10 hours and
have made their month,
-
Not Synced
they can still join, it's not a problem,
-
Not Synced
it's not a high rate because we want only
europeans,
-
Not Synced
it's because we want to allow everybody
-
Not Synced
and if they can be payed for 10 hours
by Freexian and then spend
-
Not Synced
the rest of the month doing free work
on Debian, the better.
-
Not Synced
If you want to join the program,
get in touch,
-
Not Synced
there are details on the webpages
-
Not Synced
and the more people ???
the better.
-
Not Synced
[Q] How do you or other developers
motivate yourself
-
Not Synced
to do this kind of LTS work,
-
Not Synced
because obviously it's important
for companies to have
-
Not Synced
stable releases,
-
Not Synced
but it also, to me at least, doesn't seem
very exciting from a technology standpoint
-
Not Synced
[RH] Do you think it's exciting or
not exciting?
-
Not Synced
I'm not sure I understood.
-
Not Synced
Not exciting, ok.
-
Not Synced
I agree, it's not exciting.
-
Not Synced
[laughter]
-
Not Synced
[inaudible question]
-
Not Synced
[RH] A bit of both.
-
Not Synced
I want Debian to succeed and I know that
-
Not Synced
LTS support is important for Debian
to succeed
-
Not Synced
so I want to make sure the project is
working well and alive
-
Not Synced
but clearly, I'm doing it because…
-
Not Synced
Let's say, the security support work,
providing security update,
-
Not Synced
backporting code, it's for money and
because I need to live.
-
Not Synced
But organizing the LTS project is
something I do for free
-
Not Synced
because I want it for Debian
-
Not Synced
because Debian needs it.
-
Not Synced
[Q] I have a question.
-
Not Synced
Who announces the public relation,
the marketing stuff of
-
Not Synced
the Debian Long Term Support?
-
Not Synced
Do you do it yourself or do you have some
professional help?
-
Not Synced
[RH] I do not have any professional help.
-
Not Synced
Basically, the way we…
-
Not Synced
Well, we could do much better, I mean,
in number of sponsors.
-
Not Synced
There are almost no US-based sponsors,
-
Not Synced
there are lots of US companies
that could help.
-
Not Synced
It's true, I'm fighting between my time
-
Not Synced
spending free time on Debian,
doing LTS Debian.
-
Not Synced
I contact a few companies that people
suggest me to contact,
-
Not Synced
but I'm not doing much things
by myself.
-
Not Synced
[Q] So, you would benefit a lot from
some professional help
-
Not Synced
in the marketing department or
-
Not Synced
[RH] Yes
-
Not Synced
[Q] public relation
-
Not Synced
[RH] Yes, but I don't have money
to fund this or not much.
-
Not Synced
The 10€ difference is for
administrative cost.
-
Not Synced
Maybe a bit for this kind of stuff but
it doesn't look like a lot for this.
-
Not Synced
[Q] I just wanted to ask to the previous
questioner about
-
Not Synced
finding motivation for providing patches
for LTS packages and in my view
-
Not Synced
it's not very different from providing
security patches for stable.
-
Not Synced
Sometimes it's a bit harder because the
software is a bit older
-
Not Synced
but we overcame this situation
for wireshark for example
-
Not Synced
by updating wireshark because it was
quite impossible to backport older
-
Not Synced
security fixes.
-
Not Synced
I think it's part of the maintainers.
-
Not Synced
If you maintain a project, you should
maintain the security issues in stable
-
Not Synced
and if you care a bit more and if you have
a little more time,
-
Not Synced
you can care about the package in LTS
as well.
-
Not Synced
I did it for wireshark for free, because
it's easy for me.
-
Not Synced
[RH] I'm grateful, thank you.
-
Not Synced
I want to point out that it's a good
thing that
-
Not Synced
LTS and security are different,
-
Not Synced
because we have different policies like
we said,
-
Not Synced
importing a new upstream version is
something we can do
-
Not Synced
when it doesn't break too much.
-
Not Synced
The command line interface had not changed
so badly between both versions
-
Not Synced
so it was possible.
-
Not Synced
It's not possible for all projects, but we
are not so strict on new upstream versions
-
Not Synced
And if you look at what others have been
doing, RedHat,
-
Not Synced
they too import new upstream version
quite often, in fact,
-
Not Synced
even on libraries like dnss.
-
Not Synced
[Q] Due to lack of browsers in LTS support
and Xen support
-
Not Synced
was it an issue with companies you are
working on, who are sponsoring Freexian?
-
Not Synced
[RH] I'm not sure I understood.
-
Not Synced
[Q] The lack of browsers support and
Xen support in the LTS version,
-
Not Synced
was that an issue with the companies
who you work with,
-
Not Synced
who are sponsoring your work?
-
Not Synced
[RH] Browsers, not so much.
-
Not Synced
Most sponsors are hosters and
stuff like that
-
Not Synced
but they were annoyed by the lack of
qemu or Xen support.
-
Not Synced
They did upgrade their host machines
but not their guest machines
-
Not Synced
for their customers.
-
Not Synced
Obviously, that would be
the first priority to fix.
-
Not Synced
Browser is nice, but it's also one of
those cases where we just need to
-
Not Synced
integrate newer upstream versions
-
Not Synced
??? doing it.
-
Not Synced
And I was hoping for Raphael Geissert
to do it because he does it for
-
Not Synced
Électricité de France.
-
Not Synced
I'll catch him.
-
Not Synced
Convince him.
-
Not Synced
I know him quite well, he's working in
Lyon near me, not far away.
-
Not Synced
[Q] Having sponsors for doing the LTS
work is a signal of
-
Not Synced
the need for LTS versions,
-
Not Synced
but do you have additional statistics on
the usage of LTS?
-
Not Synced
[RH] Not really.
-
Not Synced
Since LTS is not hosted in security
mirrors but on all mirrors,
-
Not Synced
we don't have access to all the
statistics.
-
Not Synced
But I know from mails, thank mails and
stuff like that
-
Not Synced
that it's really appreciated and used
quite a lot,
-
Not Synced
even from end users.
-
Not Synced
There are users who like the antiquated
GNOME stuff.
-
Not Synced
I'm fine with GNOME 3, by the way.
-
Not Synced
Yes, it's really widely used and widely
appreciated.
-
Not Synced
[Rhonda] Thanks for your attention.
-
Not Synced
[Applause]