Preferred_Debian_Packaging.webm

Thank you very much.

Thanks everybody for coming,…

If you are packaging software and you want
me to work on with you,

this is how you can do that.

It is a very self-centered talk:

I just want to explain some of the things
that I like about,

some practice that I prefer
about Debian packaging,

and I don't pretend this is any sort of…

official, permanent or final thing.

I just wanted to share some ideas
that I have about

the way that I work with packages,
in the hope that maybe…

For two hopes:

One is that I hope that I can show you
something that you have not heard of,

or maybe you were doing differently,

or maybe you think it is
the right think to do

and it is just nice to see
somebody else doing it.

My second hope is that you can tell me
what I am doing wrong,

and you can help me learn and improve
on my own packaging techniques.

If you see something that
I am proposing up here,

and you think there is a problem with it,
I would like to hear about it too.

I just want to see more of the culture
within Debian,

of people who are doing packaging,
explaining what they are doing,

and so I thought I would
just step up and explain:

"Here is some of the practice that I do",

In the hope that other people will do the
same and explain what they are doing,

and maybe they can learn from
me and I can learn from them.

Without much further ado I am
just going to dive into it.

If you have questions, I am perfectly
happy to be interrupted,

we have some folks with
walking mics in the crowd:

you can just raise your hand.

If you have got a question or an
interruption or whatever,

that is fine.

I doubt I will go the whole 15 minutes,
I think there are 20 minutes,

I doubt I will go the whole time,
so there will be also

time for questions at
the end if you prefer.

But I do not mind being interrupted.

So, this is all on this web page here,

you could probably skip this talk and go
read the web page,

but then you would not have the nice
in-person interactions,

and it is easier to tell me that I am
wrong in person,

so I would like to have that happen.

I put this up on the Debian wiki,

because I want anyone
to be able to find it.

If you think you have got some good ideas,
you should put it on the Debian Wiki too:

other people can take advantage of the
ideas that you have got.

First baseline is:
I really like revision control.

And I know that it makes me
a certain flavor on nerd,

but when we are working with things that
are as complicated as software packages,

hmmm…

I think a lot of people don't get
that in Debian you are not

just working on one
software package:

you are actually probably, if you
are doing a responsibly work,

on at least two software packages,
and maybe 5.

So you have got the version
that is unstable,

and you have got the version that you
try to maintain for stable as well.

And we are committing to
doing maintenance work.

A lot of our work in the project is
janitorial in nature:

we want to clean up the mess and
we want us to stay out of the way

and make sure things work,
functionally,

for people who are relying on the
operating system to not get in their way.

So revision control I think is really
helpful because it means you can

keep track of what changes you have done
on different branches of the project

while you are maintaining both of them.

Basically, I'd require working with
the revision system I am comfortable with,

I prefer Git, I am not going to have a
religious war about it.

If upstream uses Git, I am
even happier, and I try to make

my packaging depend on
upstream's revision control.

I like to use 'git-buildpackage', and I
like to use it with debhelper.

If you have not tried out
'git-buildpackage',

we are going to have a
'git-buildpackage' skill share session

later on today, and I welcome you
to come and share your tricks with it,

or learn some tricks from other people.

It is a particular way that you can keep
your Debian packaging in a Git repository,

and it helps you to keep track of all of
the changes that have happened within

your packaging and within upstream to
make sure you are not accidentally

making other changes.

So it is very easy to go back and
review what you have done.

I find that really useful.

I definitely also like to keep
upstream's source code

in the same revision control system.

I like to keep the tarballs in the
revision control system

because it means that
if someone is interested,

they can uses a tool called
'debcheckout'.

You can use 'debcheckout' with
a name of a package:

you just say "I am really
interested in package 'foo',

let me see the source code for that":
'debcheckout foo'

You get the source code, and
you get the source code…

from a revision control system
that you can now track

and you can just propose changes on.

You can also extract the tarball from that
revision control system.

'debcheckout' actually works even if you
do not have upstream stuff in there,

but I like to keep it all in one
revision control system,

it is just easier to find everything
when you want.

Some of these things that
I prefer have to do with

what the upstream software
developer has done,

so I am less inclined to try the package,
an upstream software project,

if they just throw tarballs
here over the wall

to an FTP side every now and then.

It makes it more difficult for me to
know what they are doing,

and why they are doing it.

So i like it, I have already said,
when upstream uses Git,

I also like it when upstream
signs their releases,

and says "hey, this is specific release",

because that is a signal
that I can use,

or somebody else that
understands the project.

As said, "we think that this is
something that other people can use",

or "this is a particular version that
we would like other people to test".

There are a lot of other situations where
maybe it is not so important.

And having that be cryptographically
signed is really useful.

I care about cryptographic signature on
software because I want to know that

what I am running is related to the code
that somebody else out should be run.

And if you don't verify your
software cryptographically,

anyone who can intercept
the network connection

between you and that software, can
modify the software before it gets to you.

And the cryptographic signature just says:

"look, this is a version
that I am OK with.

I am putting it out there
and it comes from me".

So I can have a trace back
to that point.

Just let me talk about briefly about
how you do cryptographic verification

of upstream
One is you might know upstream:

you might know them personally, you
know their key already, that is fine.

That is not the usual case:
we work on the Internet.

In the situation where your upstream
is signing their tarballs

and you have not met them,
you do not have to sign their key,

you do not have to say
"I announce this is their key".

But it is probably the same one
that is signing every release,

so you should keep track of that.

Debian has a nice way to
keep track of that:

you can tell Debian how to find the new
version of the upstream tarball.

This is in the Debian 'watch' file.

If you type 'man uscan', you can learn
more about Debian 'watch',

and Debian 'watch' now has
a feature that lets you say

"that is not only this way
you find the tarball,

but upstream publishes signatures
and the signatures look like this".

You know, they got a '.sig' at the end.

So there is a particular arcane way to
specify that, but if you specify that,

then 'uscan' can find
not only the upstream tarball,

it can find the upstream signature.

And, if you drop
upstream's signing key

- which of course I did not
put on the wiki page,

someone should
edit that and fix it -

you can put the upstream signing
key in 'debian/upstream/signing-key.asc'.

And then if you do that, when you say
'uscan', you can tell…

Maybe some people here do not
know how to use 'uscan':

'uscan' is a very simple tool,

you run it from a software package
that has a 'debian' directory,

or even one level up if you keep all of
your software packages in one folder.

You can go one level up
and say 'uscan',

and it will look in all of the folders
that are children of it,

and look for new versions

by trying to find a new upstream
version in 'debian/watch'.

And if you have configured
'debian/watch' properly,

it can find the new upstream signatures,

and if you have got the
'upstream/signing-key.asc',

then it will actually verify
the signatures for you

as part of fetching the
new upstream tarball.

So you can get all of those things just
by setting a pre-packaging that way.

There is a hand up down there, could we
get the mic down to the hand ?

Thanks.

Or to the person who has that hand,
it is not just a hand.

[public laugh]

[attendee] Publish a tarball,
and a hash, '.sha1',

and sign that hash,
'.sha1.asc'.

Can 'uscan' cope with this and
check the signature on the hash

and that the hash belongs
to that tarball ?

[Daniel] I do not believe that 'uscan'
can do that currently.

So anybody out there who wants to
make things better for the world

should go hack on 'uscan': that is a
pretty straightforward thing

that we should fix because
I agree that is common pattern.

[attendee] I have no answer to this
question by I have another question:

how do you convince upstreams
who do not release tarballs

or who do not set tags in Git ?

[Daniel] Who do not make tags in Git ?

[someone] Yes, if there is no tags
you can not check out a tarball.

Is there any good way to
convince upstream to do this ?

[Daniel] Git has this nice feature, which
is that you can create a tag,

which is associated with
a particular revision,

and you would like to have a tag

everywhere that a tarball
has been released from.

I am tempted to pull up a Git
view and show people some tags.

The question that you ask is a
social one tho, not just a technical one,

and I actually find that my upstreams
are pretty responsive.

Usually I frame my request as

"hey, it like you made this tarball
from this particular commit 'id'.

If you could tag you releases,
it would be really helpful to me,

and here is the command
that I would use to tag the release".

And I say "git tag…"

and of course I can never
remember so first I look it up,

but it is either 'tag name' 'commit id'
or 'commit id' 'tag name'.

But I would look it up and
I would write the email so that

all they have to do is they read it,

understand my argument,

and execute one command.

I mean, it doesn't get them in the habit
but it start them towards it

And if you say 'tag -s',

then your tag will be
cryptographically signed,

which I think is a really
good thing to do too.

So, cryptographic verification
of upstream.

As I said, I want to keep upstream's code
in the revision control system.

I also like to keep…

In my ideal case upstream is using Git:
I am using Git for packaging.

I actually like to keep upsteam's Git
history fully in my repository,

so that I do not just have the tarballs,
but I actually have all of their commits.

And that turns out to be really useful
for two specific cases:

In one case, there is a common scenario
where upstream will fix a bug,

but they have not made a release yet.

And that bug is really,
really obviously problematic

for the folks who are using Debian,

so I want to fix it.

All I can do, because I have
their full revision history,

I can use Git to "cherry pick"
the upstream commit.

And then I "cherry pick"
that upstream commit

and I can have it applied separately,

and release a Debian version
that has the fix,

even before upstream has made
a release with the fix.

So one nice thing about
having upstream revision

is that I can pull fixes from upstream
before they decided to release it.

The other advantage is the other
way around.

Often when I am doing packaging,
I discover a problem,

and maybe I can fix the problem.

And in fact maybe I am already shipping
a Debian package that fixes the problem.

If my Debian fixes can be
directly applied to upstream,

then I can use whatever their preferred
upstream patch submission guidelines are,

whether it is a Github pull request,
or a patch to a mailing list,

or a "hey can you pull this from my Git
repository over here", e-mail…

The fact that I am using the same
Git history that they are using

makes it much easier for me
to push my changes back to them.

So, it sort of smooths the interaction
if you can consolidate

and use the same revision control
system as their.

Towards that aim,

I use a system now
called 'patch queue',

which is part of 'git-buildpackage'.

So 'git buildpackage' is 'gbp',

'patch queue' is 'pq',

so to deal with 'patch queue'
you say 'gbp pq'

and then you have some commands.

And what that does, is it takes…

How many of you are Debian packagers ?

How many of you package
software for Debian ?

[most attendees raise their hand]

A very large percentage, but not everyone.

I hope some folks are considering starting
packaging if you have not done it yet.

Of those of you who package software,

how many of you package software
with modifications,

how many of you ship a
modified version of upstream sources ?

[most attendees raise their hand]

Beyond the 'debian' directory,
just Debian patches ?

So the common way to do that,

for the Debian 3.0 quilt
packaging skill,

is that in your 'debian' directory you
have a 'patches' sub-directory

that has a set of individual patches
that apply certain changes,

and they are applied in order based on
the file called 'debian/patches/series'.

So maintaining that is kind of a drag
when upstream makes big changes:

then all of sudden you have got this set
of patches and they do not quite apply…

It is a drag even you do not have it
in the 'debian/patches/' directory.

But what Debian 'patch queue' does is
it maps that directory of patches

into a little branch on your
Git revision history.

So when you get a new upstream version,
you can say 'patch queue rebase',

and it treats it just as Git:
it takes the 'patch queue'…

You have already imported the new version,
and it re-applies your patches,

and sometimes that means
some minor adjustments.

Git is really good at figuring out what
the right minor adjustments are to make,

and so all of the sudden
the patch queue is re-based,

you refresh it in your revision
control system…

and there you go.

So I like to use
git-buildpackage patch queue,

tagging, as already brought up,
thank you for that,

I like to to tag
everything that I release,

I like to push that
as soon as I can,

so that other people
who are following my work

can know where my releases
come from.

The reason that I like other people
following my work is

they can fix my bugs easier.

I make mistakes,
everybody makes mistakes,

and it is really important to me that
if someone catches one of my mistakes,

I can accept their feedback,
their criticism, their improvements,

as easily as possible.

I want a low barrier to entry for people
to help me fix my problems:

it is selfishness.

So I try to patch it and publish these
things for people can find it.

I'm going to rattle through some of these pretty
fast because were are almost out of time.

I like to put my repo in some place
where other people get to the them,

at the moment I like to put them in
'collab-maint',

it has some problems but it is better
than not publishing your stuff,

and it is nice because it is sort of
a public use.

I like to standardize how of
my branches are named,

so if I am working on something
that has got a stable version,

that is for Jessie, I will
name the branch 'jessie',

because I will probably making changes,
like I said, editing multiple of software

I try to push as frequently as I have made
something that looks sensible.

I do not feel obliged to push
my commits to a public repository

when I am still experimenting,

I actually really like to
experiment,

and I also like to keep track of my
experiments while I am doing them.

So I try to push when there is
a sensible set of changes,

and I am trying to get myself

to a point where I can understand
what I have done, even if it is wrong.

If I can get myself to a conceptual
point where it is done,

I will push my changes so other people
can see what I am working on,

and then work from there.

That is OK to push something
that is wrong,

as long as you push something that
people can understand.

When you make a 'git commit'
(if you are working with Git),

one of the things that helps me to think
for commit messages…

People often think that commit messages
should say "what change you made".

I think that the 'git patch' shows what
change what change you have made,

and I thin your commit messages should
say "why you made the change".

That is what people really want to read.

If you need to explain technically
why the thing that you did

maps to the conceptual thing
that you wanted to do,

that is fine: do that in
your commit message too.

But it is really important to say
why you made the change.

It is not just like
"initialize variable to 'no'":

OK, we can see that from the patch,

but what you are really saying is
"there was a crash if someone did 'x',

and we are avoiding that crash by
setting this to 'no'.

So I like to send patches via email,

so I try to configure Git email,

which make it really easy to just
push patches back upstream.

If I am starting taking over a project
that somebody else has past on,

and they did not use Git,
I will try to restore all of the imports.

I would be happy to talk with people
about how to do that,

if you have questions come find me.

I like to keep my files
nice and simple:

there is a tool 'wrap-and-sort',

that just canonicalizes your files
to make them look

in a simple and sensible way.

And it is nice because it
means that everything is…

It does things like alphabetize
your list of build depends,

and brake them out one per line.

The nice thing about that,
since you are using revision control,

when you make a change
to your build depends,

the changes become
very easy to see:

"oh, they added one new package here,
there is a single '+'".

When ????
so you can see that kind of thing.

I like to use ??? deb five ??? to format
Debian copyright to be machine readable,

it is nice for people who are
doing scans of the archive

and try reason about
what the patterns are,

and licensing of free software.

And if I am doing something really crazy,
that is going to make a big change,

I like to use a feature branch in
revision control.

So we have got one minute left,
I want to open it up for other questions.

????

[attendee] You said you are using
'wrap-and-sort' which is nice,

I had learned that ???? editors
- 'cme' - do the same job,

and somehow does a better job:
it also ??? standard version

if it does not fit, or it makes VCS
fields properly has it should be.

'cme fix dpkg-control' fixes your
control file.

[Daniel] 'cme' ? And it is in
what package ?

[attendee] The package 'cme', in
unstable ????. In Jessie it is ????

[Daniel] You are developing in unstable,
that is OK.

'cme'
OK, thank you.

Other questions or suggestions,
or complains ?

[attendee] If you change the original
source code, and do some commits,

how do you convert that into a series
of ??? patches ?

[Daniel] I use 'patch q' for that as well,
so what I do is I say

"I want to move over to my 'patch q'
view of the tree",

and then I make may changes, I make
my commits,

and then I say
'gbp pq export',

so that 'patch q export',

and it takes the 'patch q'
that I am on

and dumps it back into
the Debian patches directory.

If you have not use 'gbp pq', I
recommend looking into it.

It takes a little while to get used to,
and I still screwed it up sometimes,

but it makes easy to fix your
mistakes too.

[organizer] Last question ?

[attendee] Do you think it is possible

to make this 'patch q' branch
"pullable" by upstream ?

[Daniel] I do not actually think it is
possible to make it directly

"pullable" by upstream: I think upstream
can cherry pick patches from it,

but I do not see how to
make it "pullable".

If someone else does, I would be
happy to learn.

[organizer] This was "before last",
so last.

[attendee] Do you have a recording of
you using the tools that you mentioned,

a video recording would be great,
just to show your workflow ?

[Daniel] I do not really know how
to do that:

if somebody wants to help me do that
I would be happy to do it.

I am going to give one last plug,

I know we are out of time here,
sorry.

This tool is called 'gitk'.

This is an example…

- sorry we should leave -

but this the way that I visualize
my revision control system.

We could do a whole other session
about 'gitk'.

If you do not try to visualize your
revision control system,

you are missing out:

I recommend try to find a way
to visualize stuff,

find one that works for you.

Thanks for coming.

[organizer] Thank you.