salsa.debian.org state of affairs

Talk given by Alexander Wirt at Minidebconf Hamburg 18
https://meetings-archive.debian.net/pub/debian-meetings/2018/miniconf-hamburg/2018-05-19/salsa.debian.org_state.webm

Ok, welcome back to the second session
of the day.

It's going to be Alexander Wirt talking
about salsa.debian.org.

[Applause]

Thank you, good morning.

I usually don't give talks in english,
so please be nice to me.

However, I'm here.

I want to talk today about our journey
for Alioth

which is still running, but not for long
anymore,

to our new service, salsa.

I want to get a little bit into the history
of old things

and what we have already achieved,
what we still need to achieve

and what are our plans for the future.

Let's start with the basic things,
who am I.

I am the guy who rejects the mails
on lists.debian.org,

I am a listmaster.

I am the guy that rejects your backports.

I am the backports ftp master.

And I am the guy that will destroy
alioth.debian.org.

For the last ten years

[Applause]

I was an admin by accident of
alioth.debian.org.

This is another story I will tell you
in a few minutes.

Beside from that, I work as an OpenSource
consultant at credativ,

which is a small company in Germany
which is specialized in OpenSource,

we only do OpenSource consulting
in Germany.

We do what today is called DevOps,
we do every kind of consulting.

If you do something with OpenSource,
we are probably the ones you can talk with.

I am a father of two wonderful girls,

they're not here unfortunately,

but otherwise I wouldn't be able
to work.

And in my little bit spare time, I do
role playing games and Tabletop games.

In theory there should be a picture now.

There's a picture missing,
I don't know why,

which should tell "We need you".

A little bit of advertisement, if you
want to do OpenSource work in Germany,

paid,

and you need a job, please talk to me.

We are always looking for good people,
especially in C development,

kernel development, but also of course
consulting.

So please talk to me.

Some steps in history.

Some years ago, ???
2008, 2009,

I told the alioth channel

"Hey, if you need help, I can help with
system administration,

not the GForge stuff which is running
above,

but if you need help, tell me."

[Audience] Big mistake

Yeah.

One or two years went by,
and step by step

all alioth admins left.

We were alone in the channel.

And around that time, I detected

"Hey, I have sudo permissions
and I'm admin"

Somebody made me an admin.

So, I had to decide that I will be
the person that is the future alioth admin

and I stepped in.

So it was the beginning of our alioth
journey.

Then, in DebConf15, we had a long
'Birds of a Feather'

where we talked about several security
problems in collab-maint,

some of you are maybe not aware of it,

but since we use git at filesystem level
on alioth,

we are introducing a number of interesting
security problems

like if someone writes a hook, that hook
gets executed every time someone pushes.

So you have basically shell access.

And of course you execute it as
your own uid.

So, if some DM (Debian Maintainer) or even
not DM, nearly the whole world

has write access to collab-maint,

drops some hooks in,

it can make you execute code on Alioth
at your uid, which is a problem.

We did some things to solve that problem,
but the main problem remained.

So, along that time, we decided that we
would need a successor for git.debian.org.

At that point, we are talking about gitolite

which we evaluated at that time.

However, as ???

Two years went into the land and
nothing real happened,

we just played with it.

Then, May 2017, a thread comes up,
"Moving away from fusionforge".

What nobody was really aware of, is that
alioth is on a Wheezy machine

and Wheezy is ??? out of security
support end of the month.

So time was running up.

The thread was long as usual on
debian-devel and

we decided to do a few steps, like
evaluating things

and in June 2017, I did a survey about
our new alioth services.

It was clear at that point that I wouldn't
be able to maintain all the things

alioth had in the future

so we decided to just bring over
the important things.

What is important? For everyone,
everything else is important

so I decided to do a survey which was
pretty successful

with a few hundreds submissions.

Then, in…

Then we evaluated… "we" as probably "me",

evaluated a few solutions, named pagure,
which is the git solution Fedora is using,

which is a Python thing based on gitolite,

gitlab, which is the biggest Github
competitor

gogs/gitea, which is some golang-based
small git service.

pagure turned out to be not stable enough
for our needs

and we would have to do to much coding
inside pagure to use it in our infrastructure

because pagure is very strongly ???
with the Fedora infrastructure,

specially its user authentication and
user management stuff.

Gitlab had an other problem called
"opencore" and

"contributor license agreement"
which means

I and others were not very happy with
contributing code to Gitlab

which is something that will always
happen if you maintain such a service.

And gogs and gitea is nice but it's small

It will not be able to manage 10,000s
of repositories.

Next step happened in August 2017 when
we had a sprint here in Hamburg

at the hackerlab CCC on the other side
of the building,

where we talked about it.

After long discussions, we decided to go
with Gitlab

because Gitlab, at that point, was
the best solution that was already ready.

We didn't have to adapt too much, we don't
need to patch it

which turned out it isn't true, but it's
an other problem

It had features like continuous integration
ready,

it had features like code review ready,
wiki pretty good working

and ??? very scalable
in all directions

Every component is scalable which is
good for us.

This is a TODO point, I wanted to add
an image about the restaurant

where we decided on the name "salsa".

Somebody of you may ask yourself where
the name is coming from.

There's a small mexican restaurant
a few hundred meters from here

where you can get great burritos and
they have a painting at the back

with the term "salsa" written

and we were deciding on a name which
just not describes the type of service on it

so we wanted…

Yes, it's also a sauce. So salsa had sauce.

I wanted to call it Klaus, but we decided
against it so somebody came up

in the restaurant with the name "salsa"
and so it's called salsa.

In the meanwhile, we talked a lot with
the Gitlab people

which were very kind and helped us
with our problems.

We also talked with them about the CLA
problem and after some discussions,

the lawyer of SPI was also involved,

we made them to remove the CLA
and replace it with something better.

Contributing patches to Gitlab is now
much easier and better

which is something we are very proud of

[Applause]

And between November and the 25th of
December, we implemented salsa two times

First time on ???.debian.net where we had
root but

after more discussions we decided having
this maintained at a (debian).org box

would be better, which made us
??? ansible stuff

and develop a ??? to be able to install
gitlab as a non-privileged user

but we did that.

In Christmas, he was able to release
salsa into public beta.

Things went well, which allowed, at the
end of January, salsa to leave the beta

Since then it's official, our official
git successor.

What will happen in the future?

Oh no, this is already past.

On May, we disable user and project
creation on alioth.

Still in May, we disabled the not so much
used version control systems,

bazaar, mercurial and darcs

On Thursday (May 17th 2018), I disabled
projects web sites.

And this is future, at the end the month,

all other remaining version control systems
on alioth will get disabled.

So if you have anything running on alioth,
still running on alioth,

cron jobs are also disabled so
you don't have cron jobs enabled anymore

Be it whatever you think of, remove it.

1st of June, alioth will be off, you won't
be able to get any data anymore

from alioth.

You can get the ??? via DSA to get
subsequent backups, that's up to you

but I don't recommend it and they won't
like it.

Yeah

In June, alioth will come to an end.

It served us well for 10, 15 years, but
its time is over.

Some numbers.
Where are we now?

Yesterday (May 18th 2018), we had
23,700 repositories on gitlab,

3200 users, 400 groups, which sums up
around 90GB on disk, which is nice.

For a service running for more or less
6 months, it's a pretty nice number.

What are our future plans.

??? Docker registry, by now
you can use external registries

which is working

You can the gitlab registry for
Docker images

but it will be nicer to have our own
registry.

That is pretty high on my todo list, after
alioth is gone.

We want more runners, so you are able to
sponsor runners, if you have machines or

some money you want to spend on runners,
please tell us.

What are runners? Runners are the things
that are used by Gitlab CI to build code

or test code, or do things.

You can use it to build your packages,
you can use it to autopkgtest you packages

you can use it to build websites or
whatever you like.

It's pretty useful and I think using CI more
will be a big step forward for Debian.

We should really get more into it.

There are already some projects like
the reproducible builds, the debci guys

that are working on such stuff

and now we have the infrastructure that
every DD, every developer or package maintainer

can use it.

There's also an other feature called

"devops" which is based on kubernetes
which allows you to even

deploy and test things properly.

So if you have package which implements
a web service, you can even run

??? kubernetes part which runs
a web server,

you can test it, you can even record it,
do QA test and so on

all based on this devops feature which
would also be a nice thing.

By now, we don't have a kubernetes instance
we can use for it,

so if you have a spare kubernetes instance
you want to offer Debian,

please talk to us.

And integration with sso.debian.org,
which is another side project of mine

and some of ??? students, sitting there.

We want to build a successor for
the command sso.debian.org

which has a problem that it doesn't have
a user backend,

the user backend is alioth,
you see the problem

But it just the case for our guest users.

The official Debian Developers come from
the ldap which will still work,

but we have a problem with guest users,
so we currently don't have a way to

source for managing those guest users,
especially give additional groups like

"Hey, the user's a DM"

I would love to give all DMs access to
the Debian group, write access,

but I can't currently because I'm not able
to ???

which is something we want to solve with
the new sso.debian.org feature.

sso.debian.org should also develop a new
authentication protocol like OAuth2,

which we will use for salsa but new
services can also rely on,

??? a way from this certificate stuff
which is somewhat nice

but it's not that good integrated in most
browsers anymore

and it doesn't work that well.

We hope to have, we already have
a prototype, and we hope to have it live

until the end of the summer.

What we left behind.

We don't have shells anymore.

So you won't be able to run any cron jobs
or other stuff on salsa and

please don't ask, we won't give anyone
a shell on salsa.debian.org or godard

which is the host hosting it.

We hape APIs, several of them,
I will show them

Please use them, we won't run any
cron jobs or custom stuff on gitlab,

it was a nightmare on alioth to maintain
and to administrate

and I will never, never want to get
into this again.

What we also don't have are custom domains
which is a feature gitlab has, but

DSA decided against it, so you will have
to live with

projectname.pages.debian.net until
someone decides for that feature.

We also left behind old version…
not so much anymore version control systems

like darcs, bazaar, subversion which isn't
a problem,

but we also don't have cvs anymore,
which may be a surprise for someone

but Debian is still a heavy user of cvs,
especially for our web site

and translations.

But maybe they will now migrate faster
away from cvs.

They are working on it, I know,
they're working on it for 10 years

but things are getting faster and
they're making progress

in migrating away from cvs.

Yeah, ???, that's right,
we also left mercurial,

or whatever people have in their
home directory.

Yeah we also had rcs on alioth, there
were rcs repos, yes.

What we got instead.

We got a bunch of new features
we didn't have before.

So, this is such… maybe a start of new
ways of working in Debian,

we got a bunch of collaboration features.

In the past, collaboration often meant
finding the right mailing list,

sending a patch and hoping.

Now we can use merge requests, which
allow people to easily fork and

modify packages or repositories, and after
they are done, they can just hit a button

or whatever and create a nice merge
request which is already heavily used

by some projects like apt or dak or my own
redirector.

That allows ???, the admins
of those repositories/projects

to review code easily, they can add
comments, they can discuss with

??? people out of the mailing list.

If people update a bunch and they
commited, those merge requests

get updated which is a workflow we are
also using very heavily in our company

which is pretty nice in my eyes.

This also allows contribution to packages
from outside people

It lowers the barrier for people to
collaborate with Debian,

which is in my eyes a good feature,

something I always liked on Github and
I'm happy we are having it too now.

Gitlab has a nice feature of good, well
designed web frontend,

some things could be better, but it's
always the case,

but in most cases Gitlab is still
blazingly fast

except if you've hit some of the bugs
in the API

but that's an other problem.

And you can work with it.

If you don't like the web frontend,
use the API,

nearly everything the web frontend
supports is exposed via the API

And there are also a bunch of
command line clients

which can integrate into git to allow
things like merge requests,

allow you to process merge requests
from the command line

if you don't like web frontends.

You can also open merge requests
by e-mail if you still like it

you can just hit the right buttons,
you'll get a mail address

that you can use.

And if you send a patch to that mail address
you will create a merge request,

some of the not so known research.

Issues.

You can track todo items or bugs.

Please, this is not intended for Debian
packages,

so please don't replace the BTS

but using it as an issue tracker or todo
lists is great.

We are using it all the time.

We're also having some upstream projects
on salsa, like sane or ???

which is ???

So, they're using issues, that's fine too.

Issues are disabled by default for
a project,

but every project has ??? to just
enable it and to use it.

You have boards where you can organize
your work,

you can add sprints, you can add
milestones and other things,

all the basic stuff you need to have
an issue tracker is included.

And we also enabled reply by mail so
you don't have to use the web frontend,

you can just use your mail client
to reply ??? into gitlab.

You can also close issues by merge requests.

So, similar to our BTS, Gitlab has
this "closes" feature.

It's all the same. So "Close", "Closes"…
and so on, it's all the same

and we close here your issues.

You can even close issues in other
projects,

so if you have projects related together
and you fix something in another project

you can even close it with that syntax.

You can also create issues by mail,
which is basically the same

as for merge requests,

you have that "email new issue" button
where you get a custom mail address you can use

and then you can use that mail address
for the future

to submit bugs if you don't want to use
the issue tracker.

What we also got are webhooks.

Custom hooks are not anymore possible
because you don't have access to

the repositories directly

but what you can use are webhooks.

Webhooks are common standard in the
web world,

you can use them to react to events
in your repository,

events may be things like someone created
an issue, someone created a pull request,

someone pushed something, someone took
something, things like that.

And you can use those events to create
IRC notifications,

we have two IRC bots available for you
to use, which is KGB

and my own irker instance.

You can automatically close or tag
bugs

If you look into our documentation,
wiki.debian.org,

you find a small paragraph about it
where you can just,

as we did before, if you close a bug
and you enable the tag pending,

tag pending webhook, your bug will
be tagged automatically as pending

like before if you used the ???
hooks on alioth.

And you can also trigger external CI QA
systems, like Jenkins or SonarQube

or whatever you like to test you code.

In the future, we will also use it
for collab, for the collaboration stuff

from tincho, where we will just forward
every push happened on the whole salsa system

so you don't have to configure that
manually, it will happen automatically

So if you contribute something to Debian,
it will come up on collab.debian.net

If you want to provide webhooks but you
don't want to run your own web server,

you can come to us, which means you have
to code Ruby.

We have our own webhook server implementation
for salsa.debian.org,

which is currently also running on salsa,
but that must be the case in the future.

So, if you want to run a webhook, provide us
a patch for our webhook implementation

which is pluggable, so write a plugin which
listens to your webhooks,

provide a patch, a merge request and we'll
happily add it to our webhook implementation

so it can be used for everybody.

Documentation is in the wiki.

Currently provided hooks are, as already
mentioned, tagpending

which allows you to tag bug as pending if
you mention them in you changelog

and some project directly working with
commits are using the close webhook

which allows you to directly close
a bug with a commit

which is used by some web servers and
other stuff directly used in Debian.

One of the most powerful features we got
is Gitlab CI.

Gitlab CI is a system that allows
a continuous integration,

continuous development on salsa

and that allows you to build, test and
eventually deploy software and packages

from within Gitlab.

You can nearly do whatever you want
in this CI stuff,

you can compile ???, run linter,
run autopkgtest,

whatever you can imagine you can do.

We have two runners provided.

One of it is running as an ???
on Google cloud,

the other one is hardware sponsored
by a sponsor

and for every CI run, we launch a docker
container in it,

You can even provide an image you want
to use as this one

and then you can do whatever you want
with it.

But please don't do bitmining or
something like that,

be kind to them, we all have to use them
and we have only two of them,

so please, if you want to do something
bigger, talk to us

like the KDE people already did.

How to use it?

Using Gitlab CI is surprisingly easy.

There is this gitlab-ci.yml file which is
usually in the root of your repository

but you can add configuration to your
repository, for example to

add it to your /debian repository
which works better for

??? packages

or whatever you have, if you don't want to
clutter the upstream directories

of your gitlab-ci file.

[Question about potential conflicts with
gitlab-ci files from upstream]

We already have a bug opened on the
Gitlab issue tracker

that allows us to change the default name
of the gitlab-ci file because

currently, if you import an external
repository, which has a gitlab-ci file

which can happen,

if we ??? run on our infrastructure
and for example,

it's ansible or some other project
???

for every upstream commit, salsa will
??? run our runners

and build the pipeline.

After you edit your file, that's it.

From then on, you can watch every commit
happening on your pipeline.

This is a simple gitlab-ci file, gitlab-ci
files are yaml-based,

documentation is in the Gitlab repository,
the documentation repository and

as you can see, it's pretty easy, you have
a pre-step, which allows you to do things

like installing dependencies which is
what's happening here.

Since Gitlab CI is running a detached
head, if you want to ???

use git buildpackage, we will have to
checkout master

for it to properly work,

then you can do a git pull, git buildpackage
and, after that,

you have build your package.

That's basically all.

You can also use artifacts.

Artifacts allow you to… keep a build
artifact for downloading, so

if you want someone to use a package,
you can just add an artifact stanza here

and that allows you to later download
your deb files.

Now that doesn't allow you to create
a repository ???

but it's an other problem.

If it's too much, you can also use the
??? yesterday

This is a prepared docker container which
is prepared for git buildpackage and

all you have to do is to execute this.

After that, you have Gitlab CI.

[Applause]

I don't know who provided it, but it popped
up in the wiki yesterday

or something like that.

We also have Gitlab pages.

Gitlab pages are like Github pages and
allow you to host web sites,

static web sites from within Gitlab.

Internally they also use Gitlab CI, so
you provide a Gitlab CI job

that just deploys your website, so
it's nothing to do

and here's our build artifact feature.

All we do here is just add those public
files in the public directory to our pages

and we only do this on the master branch

and basically that's it.

The magic is happening here, it's the
"pages" step and

if you correctly configure pages in your
repository configuration,

you have a Gitlab page after that.

You can also do more fancy things like
a Hugo web site,

just depend on a docker image which has
hugo installed

and then execute a script which builds
Hugo, add some artifacts

and after that you have a Hugo website.

You can also use it for blogs, you can
use it in personal repositories,

then for example for your own web site
or blogs.

Of course, it's not intended to serve
big web sites

but providing blogs for planet.debian.org
is perfectly fine, for example

or web site of ??? project or
whatever is Debian-related.

This brings me to an other topic
not mentioned in my slides

some people asked us what is fine to host
on salsa.

As long as it's open source, as long as
it's intended to be Debian-related

or open source related or can be included
in Debian,

it's perfectly fine to host it on salsa.

So we invite every upstream which is
looking for a home,

like the SANE guys, to host them
on salsa.

What we got with the latest major version
is a web editor

which is pretty new,

probably buggy, but it works.

So, if you don't want to clone
a repository

or you have just to have simple changes,
you can add your file in the web editor,

you get a web editor with syntax
highlighting,

you get even a markdown preview if you
just do documentation,

so that's great for every one just doing
documentation

that doesn't want to ??? with git

or the code inside, you can even
preview it,

then you can write a commit message,

and that's it.

What we also have is two-factor
authentication, which is a security feature

that allows you to add a second factor
to your Gitlab login.

I can only recommend to use it, that's
well integrated and adds a lot of security.

It works with Yubikeys or any U2F-compatible
key

and also with software solutions that
implement TOTP,

time-based one time passwords.

So every TOTP-compatible generator also
works, for example the Google authenticator

but there are also others which are
open source, that all works.

Adding it is easy.

???

It's easy.

What you can't see now is me getting
my smartphone out,

scanning the bar code, generating a PIN
code,

and in 2 seconds I will enter the PIN code

What you see here are recovery codes,

I will mention them in a few minutes.

You can use them to recover your account
if you lost your one-time password generator

So, if I log in now,

I have to use my smartphone to generate
an authentication code, add it,

and now I'm in, that's it.

So it's pretty easy.

Some people say "Oh, what if I lost
my token, that is such much work."

No, it's easy.

If you want to recover your token,
I do that all the time,

you can just use SSH and do
"ssh [e-mail]"

and the command "2fa_recovery_codes"
which will generate you

a number of new recovery codes.