[Talkmeister]: Welcome, our next talk will
be about the Debian Long Term support

team and the speaker is
Raphaël Hertzog.

[Raphaël Hertzog]: Hello.

Today I will speak a bit about Debian
long term support.

I guess most of you already know about
it.

Are there some people who have no
idea what this is about?

No, good.

I will make my talk in 3 parts.

First I will present the team, how it
works

I will give some facts about how it
evolved over the first years.

I took some time to collect statistics
and believe they are rather interesting

I will also speak about the future

but there will be a separate discussion
about this in a BoF later this week.

I will show you how to help because, like
any other team in Debian it is open

to anyone. We welcome help.

At the end I will answer your questions.

What is LTS about?

The idea is really simple.

We wanted to extend the support period
of all Debian releases.

Currently it is basically for 1 year after
the next stable release comes out.

We wanted to extend this to 5 years to
match, at least, Ubuntu's offering.

which is not our competitor, but for the
companies that are making choices

it is one of the important criteria.
So we wanted to do as well.

Since we publish new stable releases
every 2 years it is roughly 3 years.

A nice side benefit is that the user can
skip a full release.

We don't officially support dist-upgrade
over going from Debian 6 to 8

but you can do 2 dist-upgrades at
the same time, limiting the downtime

to once every 5 years.

By the way, in practice, in simple server
configurations, dist-upgrades tend to

work rather well even across 2 releases.

Keeping a distribution secure for 5 years
is a real challenge.

It is hard work that not everybody is
willing to do.

In Debian all the work is done by
volunteers who do the work they enjoy.

Generally we enjoy working on new
features on top of latest releases

and not really on backporting patches to
crud that was written 5 years ago.

The security team has limited resources
so we could not just ask the security

team to now do twice the work.

But they were still really interested in
the project and wanted to support the idea

and really helped to get it bootstrapped.

The security team has a slightly larger
scope.

They support all architectures, which
means you have lots of problems of

coordination when security updates do
not compile and stuff like that.

What did we do?

We restricted the scope by picking
the 2 most popular architectures

that most users care about.

We have had some demand for ARM
architectures but up to now we only

support amd64 and i386.

We also excluded some packages from
security support.

Either because they are taking too much
time, like a security issue every 2 weeks

or that upstream is not cooperative
enough to be able to support it.

This list was basically made by the
current security team based on their

experience of doing security support.

If you look at the list there are some
important restrictions.

There's no xen, no kvm, no rails,
no browser. It sucks a bit.

But it's a way to get it started.

I think we can do better for wheezy.

Basically there is no virtualization
support on the host, only on the guest.

The security team helped to bootstrap
the LTS team but it is not the same team.

Obviously there are members of the
security team who also work on the LTS

team. They work in close collaboration.

We have regular contact and they watch our
mailing lists etc.

But the policies are different and the
infrastructure is separate,

which is a problem but I will talk about
that later.

We have a dedicated mailing list

and a dedicated IRC channel as well.

You are welcome to subscribe and to
join.

It's a new team which means new people
and new members.

Where do they come from?

The first idea was to get help from
people in various companies

who are already doing such in-house
support.

We had contact with EDF, and still have,
but they were one of the first

companies who were pushing for this
because they basically said

we are doing this already and we can
share with other companies.

The idea was to pool security support of
multiple companies.

We sent a press release asking
companies to join.

We had a few responses.

But I'll come back later to how it evolved
It's not really satisfying.

The other thing that we did is that we
offered companies the option to

fund the project to bring money and use
this to pay the work of

actual Debian contributors to do the
security updates that we need.

We have wiki pages listing all the ways
that companies can help with money.

In practice, most of the (wanting to be)
paid contributors joined together

under a single offer managed by
Freexian SARL which is my own company.

I'll quickly explain how this works.

Most companies don't want to bother
bringing human resources ??? (08:25)

They buy long term support contracts
from Freexian.

We have a rate. When you give €85 you
fund 1 hour of LTS work.

This is the current list of sponsors.

Top level gold sponsors sponsoring
more than 1 day of work per month.

On the other side we have Debian 
contributors that are doing the work

and Freexian is paying them. There is a
small difference between the rate

to cover administration costs because I
have to handle the invoices

and some customers are using Paypal
which is taking a cut.

We ask contributors to follow some rules.

There is a requirement to publish a
monthly report on work done

on paid time. So they won't get paid until
they have published a report.

So everybody can know how the money
has been spent.

Currently we have 7 Debian contributors
and about 30 sponsors.

Some figures.

Who uploaded packages?
How has it evolved since June last year?

How is the funding evolving?

I just updated those figures a few
days ago.

I used this talk before at the mini
DebConf in Lyon in March,

but I updated it again.

The number of uploads is roughly over
one year since we started last year.

Over 300 uploads so it is not so much
but it is almost 1 per day.

So it is significant work.

I have given a state here of who paid
for the work and who did it on the left

The sponsors of Freexian are paying for
most of the uploads. ???

None is a separate category grouping all
Debian maintainers.

There are maintainers who are taking
care of their own packages in LTS.

Security team is members of the security
team who also work within the LTS team.

EDF is Électricité de France

Individuals are Debian developers that
have listed themselves as members of

the LTS team and did uploads for packages
of other maintainers not their own.

Credativ is a German company that you
probably know.

They have a booth here if you want a
job.

Toshiba, Univention, Catalyst etc
are other lower ???

On the right are people. The top 5 people
are paid by Freexian.

Raphaël Geissert is working for EDF.

Thijs is a member of the security team.

Kurt is openssl maintainer ???

Mike Gabriel is also paid by Freexian.

Christoph Bieldl is mainly maintaining
the debian-security-support in Squeeze LTS

Nguyen Cong is employed by Toshiba.

Christoph Berg is employed by creditv
doing postgresql maintainence.

How did it evolve over the year?

Again it is by affiliation.

The big blue part is paid contributors

You don't see it very well but the part
about maintainers is this one [points]

It tends to do better over the months
because here we started to

contact maintainers every time that we
have a new upload coming up

and ask them first 'do you want to handle
it yourself' so it slightly increased.

but the contribution of other companies
has not really increased over time.

Rather it has disappeared. It is
unfortunate but it looks like paid

contributors are more productive than
others.

In particular with EDF, they do the work,
but with some lag and we are faster

so they just reuse what we have done. I
want to talk to Raphaël to see

how we can do better towards this.

How did the sponsorship level evolve?

We have a steady increase, which is
rather nice. It is not a huge amount but

it is significant because we fund almost
80 hours per month.

It is close to our first goal. We wanted
that amount to be able to sustain

ourselves.

If you look at the sponsors, we have a
few big ones, possibly one very big

We can't give the name officially yet so
I won't.It will be a big jump in the graph

A few gold and many small sponsors.

I don't want to be dependent too much on
one big sponsor. I really prefer many

sponsors who are doing small donations
but donations which are sustainable

year after year because we are not here
for 1 year or 2.

We want to do it over the long term.

We have some figures about how many
hours have been funded since the start

Feel free to interrupt me if you have any
questions. I can take them at any time

That's it for evolution. Now, the future.

What do we expect for the future?

First keep doing what we have been up
to

Keep supporting the current set of packages.

But for wheezy long term support we
would really like to have more

supported packages. A browser would
be nice for desktop deployment.

Virtualization support is also important
for many companies so we should be

able to support something here.

Also we want to avoid some pitfalls that
we had with squeeze LTS.

As you know LTS users are currently
required to add a separate source

list entry with squeeze-lts repository. The
security.debian.org squeeze

repository is unused. It should be
possible for the LTS team to

continue using the same repository as
the security team once it no longer

use it. This will be the topic of a BoF
next week on Tue at 1800.

What's the problem with supporting the
current set of packages?

For example, we have MySQL right now in
Squeeze LTS in version 5.1

which is no longer supported by Oracle.

We don't even know if it's affected by
security issues because

Oracle doesn't give info on
??? release.

This is a problem, we should consider
switching to a newer version,

but newer versions involve library
transition, which is not really nice

in Long Term Support releases.

There is some work to do here if we want
to do something serious.

And we have other problems, other
packages which are similar.

From time to time, we backport, we take
newer upstream versions.

We did that for wireshark for example.

This is what I said before, the limited
scope sucks, we want to be able to

support more packages and we need more
support for this.

[Q] Speaking of wireshark, we switched to
Wheezy's version, so it's solved.

[Raphael] Yes, exactly, that's what I just
said.

It used to be a problem.

That's a part I did no update since March.

Additionally, the problem with a separate
repository is that

there is a propagation delay to the
mirrors

that we don't have with
security.debian.org.

I will speak a bit of how the team works.

Basically, the first step is triaging new
security issues.

We have a list of CVEs that comes in and
there are added to a text file

data/CVE/list.

Some dispatches those on source packages
and then someone else reviews

status in each release.

Then the LTS team reviews the status in
Squeeze and members of security team

review status in Wheezy and Jessie.

And then, we decide what we must do with
the package.

Depending on the analysis, either we say

"We need to prepare an update", so we add
it to data/dla-needed.txt

and someone will have to take care of
preparing the update.

Or we say "The issue does not apply, does
not affect the current version"

or we ignore it because the package is
unsupported or because

the issue is really minor, not severe
enough.

Sometimes, it can be that the issue is
already fixed in Debian

due to some maintainer choices.

When we do this classification, we contact
the maintainer to keep him in the loop

and offer him the possibility to help us.

Here's what such a text file looks like.

The bold line is the line that we are
adding when we have decided

what we're doing with the packages.

This is all in the Subversion repository
on Alioth.

Then, someone has to prepare the security
update.

Basically, looking for a patch, it's often
useful to be able to look up at

RedHat or Ubuntu or upstream.

Best case is upstream because there are
nice upstream who are providing

patches also for older versions.

Usually there are already patches
available,

not always for the good version.

Sometimes we have to backport it.

Then we prepare an upload with this
+deb6uX suffix that we're using now

for security updates, stable updates.

This is a rather known territory for
package maintainers.

This is the hardest part sometimes,

testing the update, making sure the issue
is fixed.

When tools have test suites, it's nice,

but sometimes we have to set it up
ourselves.

Sometimes it is too hard, so we tend to,
out of safety measure,

to mail the mailing list and say

"Ok, I've done my best, but please double
check"

It doesn't happen often that we have
testers

but some lts users are willing to test it
before ???

It would be really nice if we had
more of those.

And if we don't get any negative feedback,

then we upload.