The_Debian_Long_Term_Support_Team_Past_Present_and_Future.webm

[Talkmeister]: Welcome, our next talk will
be about the Debian Long Term support

team and the speaker is
Raphaël Hertzog.

[Raphaël Hertzog]: Hello.

Today I will speak a bit about Debian
long term support.

I guess most of you already know about
it.

Are there some people who have no
idea what this is about?

No, good.

I will make my talk in 3 parts.

First I will present the team, how it
works

I will give some facts about how it
evolved over the first years.

I took some time to collect statistics
and believe they are rather interesting

I will also speak about the future

but there will be a separate discussion
about this in a BoF later this week.

I will show you how to help because, like
any other team in Debian it is open

to anyone. We welcome help.

At the end I will answer your questions.

What is LTS about?

The idea is really simple.

We wanted to extend the support period
of all Debian releases.

Currently it is basically for 1 year after
the next stable release comes out.

We wanted to extend this to 5 years to
match, at least, Ubuntu's offering.

which is not our competitor, but for the
companies that are making choices

it is one of the important criteria.
So we wanted to do as well.

Since we publish new stable releases
every 2 years it is roughly 3 years.

A nice side benefit is that the user can
skip a full release.

We don't officially support dist-upgrade
over going from Debian 6 to 8

but you can do 2 dist-upgrades at
the same time, limiting the downtime

to once every 5 years.

By the way, in practice, in simple server
configurations, dist-upgrades tend to

work rather well even across 2 releases.

Keeping a distribution secure for 5 years
is a real challenge.

It is hard work that not everybody is
willing to do.

In Debian all the work is done by
volunteers who do the work they enjoy.

Generally we enjoy working on new
features on top of latest releases

and not really on backporting patches to
crud that was written 5 years ago.

The security team has limited resources
so we could not just ask the security

team to now do twice the work.

But they were still really interested in
the project and wanted to support the idea

and really helped to get it bootstrapped.

The security team has a slightly larger
scope.

They support all architectures, which
means you have lots of problems of

coordination when security updates do
not compile and stuff like that.

What did we do?

We restricted the scope by picking
the 2 most popular architectures

that most users care about.

We have had some demand for ARM
architectures but up to now we only

support amd64 and i386.

We also excluded some packages from
security support.

Either because they are taking too much
time, like a security issue every 2 weeks

or that upstream is not cooperative
enough to be able to support it.

This list was basically made by the
current security team based on their

experience of doing security support.

If you look at the list there are some
important restrictions.

There's no xen, no kvm, no rails,
no browser. It sucks a bit.

But it's a way to get it started.

I think we can do better for wheezy.

Basically there is no virtualization
support on the host, only on the guest.

The security team helped to bootstrap
the LTS team but it is not the same team.

Obviously there are members of the
security team who also work on the LTS

team. They work in close collaboration.

We have regular contact and they watch our
mailing lists etc.

But the policies are different and the
infrastructure is separate,

which is a problem but I will talk about
that later.

We have a dedicated mailing list
[e-mail]

and a dedicated IRC channel as well.

You are welcome to subscribe and to
join.

It's a new team which means new people
and new members.

Where do they come from?

The first idea was to get help from
people in various companies

who are already doing such in-house
support.

We had contact with EDF, and still have,
but they were one of the first

companies who were pushing for this
because they basically said

we are doing this already and we can
share with other companies.

The idea was to pool security support of
multiple companies.

We sent a press release asking
companies to join.

We had a few responses.

But I'll come back later to how it evolved
It's not really satisfying.

The other thing that we did is that we
offered companies the option to

fund the project to bring money and use
this to pay the work of

actual Debian contributors to do the
security updates that we need.

We have wiki pages listing all the ways
that companies can help with money.

In practice, most of the (wanting to be)
paid contributors joined together

under a single offer managed by
Freexian SARL which is my own company.

I'll quickly explain how this works.

Most companies don't want to bother
bringing human resources

They buy long term support contracts
from Freexian.

We have a rate. When you give €85 you
fund 1 hour of LTS work.

This is the current list of sponsors.

Top level gold sponsors sponsoring
more than 1 day of work per month.

On the other side we have Debian
contributors that are doing the work

and Freexian is paying them. There is a
small difference between the rate

to cover administration costs because I
have to handle the invoices

and some customers are using Paypal
which is taking a cut.

We ask contributors to follow some rules.

There is a requirement to publish a
monthly report on work done

on paid time. So they won't get paid until
they have published a report.

So everybody can know how the money
has been spent.

Currently we have 7 Debian contributors
and about 30 sponsors.

Some figures.

Who uploaded packages?
How has it evolved since June last year?

How is the funding evolving?

I just updated those figures a few
days ago.

I used this talk before at the mini
DebConf in Lyon in March,

but I updated it again.

The number of uploads is roughly over
one year since we started last year.

Over 300 uploads so it is not so much
but it is almost 1 per day.

So it is significant work.

I have given a state here of who paid
for the work and who did it on the left

The sponsors of Freexian are paying for
most of the uploads.

None is a separate category grouping all
Debian maintainers.

There are maintainers who are taking
care of their own packages in LTS.

Security team is members of the security
team who also work within the LTS team.

EDF is Électricité de France

Individuals are Debian developers that
have listed themselves as members of

the LTS team and did uploads for packages
of other maintainers not their own.

Credativ is a German company that you
probably know.

They have a booth here if you want a
job.

Toshiba, Univention, Catalyst etc
are other lower figures.

On the right are people. The top 5 people
are paid by Freexian.

Raphaël Geissert is working for EDF.

Thijs is a member of the security team.

Kurt is openssl maintainer so he maintains
his own packages.

He has a lot of work.

Mike Gabriel is also paid by Freexian.

Christoph Bieldl is mainly maintaining
the debian-security-support in Squeeze LTS

Nguyen Cong is employed by Toshiba.

Christoph Berg is employed by credativ
doing PostGreSQL maintainance.

How did it evolve over the year?

Again it is by affiliation.

The big blue part is paid contributors

You don't see it very well but the part
about maintainers is this one [points]

It tends to do better over the months
because here we started to

contact maintainers every time that we
have a new upload coming up

and ask them first 'do you want to handle
it yourself' so it slightly increased.

but the contribution of other companies
has not really increased over time.

Rather it has disappeared. It is
unfortunate but it looks like paid

contributors are more productive than
others.

In particular with EDF, they do the work,
but with some lag and we are faster

so they just reuse what we have done. I
want to talk to Raphaël to see

how we can do better towards this.

How did the sponsorship level evolve?

We have a steady increase, which is
rather nice. It is not a huge amount but

it is significant because we fund almost
80 hours per month.

It is close to our first goal. We wanted
that amount to be able to sustain

ourselves.

If you look at the sponsors, we have a
few big ones, possibly one very big

We can't give the name officially yet so
I won't.It will be a big jump in the graph

A few gold and many small sponsors.

I don't want to be dependent too much on
one big sponsor. I really prefer many

sponsors who are doing small donations
but donations which are sustainable

year after year because we are not here
for 1 year or 2.

We want to do it over the long term.

We have some figures about how many
hours have been funded since the start

Feel free to interrupt me if you have any
questions. I can take them at any time

That's it for evolution. Now, the future.

What do we expect for the future?

First keep doing what we have been up
to

Keep supporting the current set of packages.

But for wheezy long term support we
would really like to have more

supported packages. A browser would
be nice for desktop deployment.

Virtualization support is also important
for many companies so we should be

able to support something here.

Also we want to avoid some pitfalls that
we had with squeeze LTS.

As you know LTS users are currently
required to add a separate source

list entry with squeeze-lts repository. The
security.debian.org squeeze

repository is unused. It should be
possible for the LTS team to

continue using the same repository as
the security team once it no longer

use it. This will be the topic of a BoF
next week on Tue at 1800.

What's the problem with supporting the
current set of packages?

For example, we have MySQL right now in
Squeeze LTS in version 5.1

which is no longer supported by Oracle.

We don't even know if it's affected by
security issues because

Oracle doesn't give info on
non supported releases.

This is a problem, we should consider
switching to a newer version,

but newer versions involve library
transition, which is not really nice

in Long Term Support releases.

There is some work to do here if we want
to do something serious.

And we have other problems, other
packages which are similar.

From time to time, we backport, we take
newer upstream versions.

We did that for wireshark for example.

This is what I said before, the limited
scope sucks, we want to be able to

support more packages and we need more
support for this.

[Q] Speaking of wireshark, we switched to
Wheezy's version, so it's solved.

[Raphael] Yes, exactly, that's what I just
said.

It used to be a problem.

That's a part I did no update since March.

Additionally, the problem with a separate
repository is that

there is a propagation delay to the
mirrors

that we don't have with
security.debian.org.

I will speak a bit of how the team works.

Basically, the first step is triaging new
security issues.

We have a list of CVEs that comes in and
there are added to a text file

data/CVE/list.

Someone dispatches those on source
packages and then someone else reviews

status in each release.

Then the LTS team reviews the status in
Squeeze and members of security team

review status in Wheezy and Jessie.

And then, we decide what we must do with
the package.

Depending on the analysis, either we say

"We need to prepare an update", so we add
it to data/dla-needed.txt

and someone will have to take care of
preparing the update.

Or we say "The issue does not apply, does
not affect the current version"

or we ignore it because the package is
unsupported or because

the issue is really minor, not severe
enough.

Sometimes, it can be that the issue is
already fixed in Debian

due to some maintainer choices.

When we do this classification, we contact
the maintainer to keep him in the loop

and offer him the possibility to help us.

Here's what such a text file looks like.

The bold line is the line that we are
adding when we have decided

what we're doing with the packages.

This is all in the Subversion repository
on Alioth.

Then, someone has to prepare the security
update.

Basically, looking for a patch, it's often
useful to be able to look up at

RedHat or Ubuntu or upstream.

Best case is upstream because there are
nice upstream who are providing

patches also for older versions.

Usually there are already patches
available,

not always for the good version.

Sometimes we have to backport it.

Then we prepare an upload with this
+deb6uX suffix that we're using now

for security updates, stable updates.

This is a rather known territory for
package maintainers.

This is the hardest part sometimes,

testing the update, making sure the issue
is fixed.

When tools have test suites, it's nice,

but sometimes we have to set it up
ourselves.

Sometimes it is too hard, so we tend to,
out of safety measure,

to mail the mailing list and say

"Ok, I've done my best, but please double
check"

It doesn't happen often that we have
testers

but some lts users are willing to test it
before ???

It would be really nice if we had
more of those.

And if we don't get any negative feedback,

then we upload.

Then, you have to send
an announce.

We call that DLA for Debian LTS
Advisory.

We have a little script
./bin/gen-DLA

that generates the template mail
and you have to add

a little bit of description

and basically send it to
debian-lts-announce

with a GPG signature of someone in the
DD or DM keyring.

The process is rather open to
all maintainers

even the write rights to the secure
testing repository

where we have this data/DLA/list file

is writable by all Debian Developpers on
Alioth

so, really, the entrance barrier is
rather low for Debian Maintainers

and Debian Developpers.

There's a file which is automatically
updated when you generate this template

and which will update the tracker on the
web pages

because all this data is nicely browsable
on security-tracker.debian.org.

And it's all linked from the package
tracker.

That's it. If you have a few questions,
I'm here to answer you.

[inaudible question]

… library and operating server,

so that you have same API for clients,
or same ABI for clients and

a newer server that actually gets
security patches.

[RH] In my head, yes, but we did not look
further yet.

I really want to use the BOF that is
upcoming to discuss it

As I said, the amount of funding we have
allows us right now

to keep up with security update but
we do not have many spare cycles

for dealing with such things.

But it's slowly coming up,

as I said there's potentially a new
platinum sponsor.

We will have a few hours free to look
into this and I think

it's probably the best solution

but I don't know if it works in practice,
I have not tried it.

And since LTS, the end of life of Squeeze
happens in February,

it's not too far away so it's possibly
a nice solution

because upgrading to a newer upstream
version is harder.

[Q] I've heard ???
Freexian ???

you have enough contributors
but you could do more work

but you don't have the funding or
is it, like, you also need

more developpers?

[RH] A bit of both, actually,

the web page has always been very clear
on the rules

That means any Debian Developper who has
made security uploads on its own already

can apply and join the program and
be funded.

Fortunately, I did not have too many

because it would have been hard,

but it has been steadily growing over
the months.

Last year, there was only Thorsten,
Holger and me,

but in the meantime we got Ben Hutchings
who is helping us on the kernel

and a few more.

I'm always looking, trying to make sure
that we don't give too much money

to a single person

because it's Debian, there has been
??? in the past.

I've been involved, I know it.

I don't want anyone to have their life
depend on LTS work, really.

And I don't want the opposite way also,
LTS to depend on a single person

that can go away.

I try to limit it to maximum 20 ???
per month for each person

and right now, we're well below that
so it's good.

By the way, most Debian Developpers are
currently european,

the fact that everything is euro-based
is fine

but one developer is american and it's
no problem to pay in dollars

So the amount will vary with the rates
but it's not a problem.

And even more, I'm surprised to have no
contributors

in the countries where the rate of labor
is well bellow.

I mean, if we have south american
contributors,

or african or I don't know, I don't want
to stigmatize anyone,

but if there are people who could live
for much more,

if they would be payed 10 hours and
have made their month,

they can still join, it's not a problem,

it's not a high rate because we want only
europeans,

it's because we want to allow everybody

and if they can be payed for 10 hours
by Freexian and then spend

the rest of the month doing free work
on Debian, the better.

If you want to join the program,
get in touch,

there are details on the webpages

and the more people funded
the better.

[Q] How do you or other developers
motivate yourself

to do this kind of LTS work,

because obviously it's important
for companies to have

stable releases,

but it also, to me at least, doesn't seem
very exciting from a technology standpoint

[RH] Do you think it's exciting or
not exciting?

I'm not sure I understood.

Not exciting, ok.

I agree, it's not exciting.

[laughter]

[inaudible question]

[RH] A bit of both.

I want Debian to succeed and I know that

LTS support is important for Debian
to succeed

so I want to make sure the project is
working well and alive

but clearly, I'm doing it because…

Let's say, the security support work,
providing security update,

backporting code, it's for money and
because I need to live.

But organizing the LTS project is
something I do for free

because I want it for Debian

because Debian needs it.

[Q] I have a question.

Who announces the public relation,
the marketing stuff of

the Debian Long Term Support?

Do you do it yourself or do you have some
professional help?

[RH] I do not have any professional help.

Basically, the way we…

Well, we could do much better, I mean,
in number of sponsors.

There are almost no US-based sponsors,

there are lots of US companies
that could help.

It's true, I'm fighting between my time

spending free time on Debian,
doing LTS Debian.

I contact a few companies that people
suggest me to contact,

but I'm not doing much things
by myself.

[Q] So, you would benefit a lot from
some professional help

in the marketing department or

[RH] Yes

[Q] public relation

[RH] Yes, but I don't have money
to fund this or not much.

The 10€ difference is for
administrative cost.

Maybe a bit for this kind of stuff but
it doesn't look like a lot for this.

[Q] I just wanted to ask to the previous
questioner about

finding motivation for providing patches
for LTS packages and in my view

it's not very different from providing
security patches for stable.

Sometimes it's a bit harder because the
software is a bit older

but we overcame this situation
for wireshark for example

by updating wireshark because it was
quite impossible to backport older

security fixes.

I think it's part of the maintainance.

If you maintain a project, you should
maintain the security issues in stable

and if you care a bit more and if you have
a little more time,

you can care about the package in LTS
as well.

I did it for wireshark for free, because
it's easy for me.

[RH] I'm grateful, thank you.

I want to point out that it's a good
thing that

LTS and security are different,

because we have different policies like
we said,

importing a new upstream version is
something we can do

when it doesn't break too much.

The command line interface had not changed
so badly between both versions

so it was possible.

It's not possible for all projects, but we
are not so strict on new upstream versions

And if you look at what others have been
doing, RedHat,

they too import new upstream version
quite often, in fact,

even on libraries like nss.

[Q] Due to lack of browsers in LTS support
and Xen support

was it an issue with companies you are
working on, who are sponsoring Freexian?

[RH] I'm not sure I understood.

[Q] The lack of browsers support and
Xen support in the LTS version,

was that an issue with the companies
who you work with,

who are sponsoring your work?

[RH] Browsers, not so much.

Most sponsors are hosters and
stuff like that

but they were annoyed by the lack of
qemu or Xen support.

They did upgrade their host machines
but not their guest machines

for their customers.

Obviously, that would be
the first priority to fix.

Browser is nice, but it's also one of
those cases where we just need to

integrate newer upstream versions

??? doing it.

And I was hoping for Raphael Geissert
to do it because he does it for

Électricité de France.

I'll catch him.

Convince him.

I know him quite well, he's working in
Lyon near me, not far away.

[Q] Having sponsors for doing the LTS
work is a signal of

the need for LTS versions,

but do you have additional statistics on
the usage of LTS?

[RH] Not really.

Since LTS is not hosted in security
mirrors but on all mirrors,

we don't have access to all the
statistics.

But I know from mails, thank mails and
stuff like that

that it's really appreciated and used
quite a lot,

even from end users.

There are users who like the antiquated
GNOME stuff.

I'm fine with GNOME 3, by the way.

Yes, it's really widely used and widely
appreciated.

[Rhonda] Thanks for your attention.

[Applause]