WEBVTT 99:59:59.999 --> 99:59:59.999 Ok, welcome back to the second session of the day. 99:59:59.999 --> 99:59:59.999 It's going to be Alexander Wirt talking about salsa.debian.org. 99:59:59.999 --> 99:59:59.999 [Applause] 99:59:59.999 --> 99:59:59.999 Thank you, good morning. 99:59:59.999 --> 99:59:59.999 I usually don't give talks in english, so please be nice to me. 99:59:59.999 --> 99:59:59.999 However, I'm here. 99:59:59.999 --> 99:59:59.999 I want to talk today about our journey for Alioth 99:59:59.999 --> 99:59:59.999 which is still running, but not for long anymore, 99:59:59.999 --> 99:59:59.999 to our new service, salsa. 99:59:59.999 --> 99:59:59.999 I want to get a little bit into the history of old things 99:59:59.999 --> 99:59:59.999 and what we have already achieved, what we still need to achieve 99:59:59.999 --> 99:59:59.999 and what are our plans for the future. 99:59:59.999 --> 99:59:59.999 Let's start with the basic things, who am I. 99:59:59.999 --> 99:59:59.999 I am the guy who rejects the mails on lists.debian.org, 99:59:59.999 --> 99:59:59.999 I am a listmaster. 99:59:59.999 --> 99:59:59.999 I am the guy that rejects your backports. 99:59:59.999 --> 99:59:59.999 I am the backports ftp master. 99:59:59.999 --> 99:59:59.999 And I am the guy that will destroy alioth.debian.org. 99:59:59.999 --> 99:59:59.999 For the last ten years 99:59:59.999 --> 99:59:59.999 [Applause] 99:59:59.999 --> 99:59:59.999 I was an admin by accident of alioth.debian.org. 99:59:59.999 --> 99:59:59.999 This is another story I will tell you in a few minutes. 99:59:59.999 --> 99:59:59.999 Beside from that, I work as an OpenSource consultant at credativ, 99:59:59.999 --> 99:59:59.999 which is a small company in Germany which is specialized in OpenSource, 99:59:59.999 --> 99:59:59.999 we only do OpenSource consulting in Germany. 99:59:59.999 --> 99:59:59.999 We do what today is called DevOps, we do every kind of consulting. 99:59:59.999 --> 99:59:59.999 If you do something with OpenSource, we are probably the ones you can talk with. 99:59:59.999 --> 99:59:59.999 I am a father of two wonderful girls, 99:59:59.999 --> 99:59:59.999 they're not here unfortunately, 99:59:59.999 --> 99:59:59.999 but otherwise I wouldn't be able to work. 99:59:59.999 --> 99:59:59.999 And in my little bit spare time, I do role playing games and Tabletop games. 99:59:59.999 --> 99:59:59.999 In theory there should be a picture now. 99:59:59.999 --> 99:59:59.999 There's a picture missing, I don't know why, 99:59:59.999 --> 99:59:59.999 which should tell "We need you". 99:59:59.999 --> 99:59:59.999 A little bit of advertisement, if you want to do OpenSource work in Germany, 99:59:59.999 --> 99:59:59.999 paid, 99:59:59.999 --> 99:59:59.999 and you need a job, please talk to me. 99:59:59.999 --> 99:59:59.999 We are always looking for good people, especially in C development, 99:59:59.999 --> 99:59:59.999 kernel development, but also of course consulting. 99:59:59.999 --> 99:59:59.999 So please talk to me. 99:59:59.999 --> 99:59:59.999 Some steps in history. 99:59:59.999 --> 99:59:59.999 Some years ago, ??? 2008, 2009, 99:59:59.999 --> 99:59:59.999 I told the alioth channel 99:59:59.999 --> 99:59:59.999 "Hey, if you need help, I can help with system administration, 99:59:59.999 --> 99:59:59.999 not the GForge stuff which is running above, 99:59:59.999 --> 99:59:59.999 but if you need help, tell me." 99:59:59.999 --> 99:59:59.999 [Audience] Big mistake 99:59:59.999 --> 99:59:59.999 Yeah. 99:59:59.999 --> 99:59:59.999 One or two years went by, and step by step 99:59:59.999 --> 99:59:59.999 all alioth admins left. 99:59:59.999 --> 99:59:59.999 We were alone in the channel. 99:59:59.999 --> 99:59:59.999 And around that time, I detected 99:59:59.999 --> 99:59:59.999 "Hey, I have sudo permissions and I'm admin" 99:59:59.999 --> 99:59:59.999 Somebody made me an admin. 99:59:59.999 --> 99:59:59.999 So, I had to decide that I will be the person that is the future alioth admin 99:59:59.999 --> 99:59:59.999 and I stepped in. 99:59:59.999 --> 99:59:59.999 So it was the beginning of our alioth journey. 99:59:59.999 --> 99:59:59.999 Then, in DebConf15, we had a long 'Birds of a Feather' 99:59:59.999 --> 99:59:59.999 where we talked about several security problems in collab-maint, 99:59:59.999 --> 99:59:59.999 some of you are maybe not aware of it, 99:59:59.999 --> 99:59:59.999 but since we use git at filesystem level on alioth, 99:59:59.999 --> 99:59:59.999 we are introducing a number of interesting security problems 99:59:59.999 --> 99:59:59.999 like if someone writes a hook, that hook gets executed every time someone pushes. 99:59:59.999 --> 99:59:59.999 So you have basically shell access. 99:59:59.999 --> 99:59:59.999 And of course you execute it as your own uid. 99:59:59.999 --> 99:59:59.999 So, if some DM (Debian Maintainer) or even not DM, nearly the whole world 99:59:59.999 --> 99:59:59.999 has write access to collab-maint, 99:59:59.999 --> 99:59:59.999 drops some hooks in, 99:59:59.999 --> 99:59:59.999 it can make you execute code on Alioth at your uid, which is a problem. 99:59:59.999 --> 99:59:59.999 We did some things to solve that problem, but the main problem remained. 99:59:59.999 --> 99:59:59.999 So, along that time, we decided that we would need a successor for git.debian.org. 99:59:59.999 --> 99:59:59.999 At that point, we are talking about gitolite 99:59:59.999 --> 99:59:59.999 which we evaluated at that time. 99:59:59.999 --> 99:59:59.999 However, as ??? 99:59:59.999 --> 99:59:59.999 Two years went into the land and nothing real happened, 99:59:59.999 --> 99:59:59.999 we just played with it. 99:59:59.999 --> 99:59:59.999 Then, May 2017, a thread comes up, "Moving away from fusionforge". 99:59:59.999 --> 99:59:59.999 What nobody was really aware of, is that alioth is on a Wheezy machine 99:59:59.999 --> 99:59:59.999 and Wheezy is ??? out of security support end of the month. 99:59:59.999 --> 99:59:59.999 So time was running up. 99:59:59.999 --> 99:59:59.999 The thread was long as usual on debian-devel and 99:59:59.999 --> 99:59:59.999 we decided to do a few steps, like evaluating things 99:59:59.999 --> 99:59:59.999 and in June 2017, I did a survey about our new alioth services. 99:59:59.999 --> 99:59:59.999 It was clear at that point that I wouldn't be able to maintain all the things 99:59:59.999 --> 99:59:59.999 alioth had in the future 99:59:59.999 --> 99:59:59.999 so we decided to just bring over the important things. 99:59:59.999 --> 99:59:59.999 What is important? For everyone, everything else is important 99:59:59.999 --> 99:59:59.999 so I decided to do a survey which was pretty successful 99:59:59.999 --> 99:59:59.999 with a few hundreds submissions. 99:59:59.999 --> 99:59:59.999 Then, in… 99:59:59.999 --> 99:59:59.999 Then we evaluated… "we" as probably "me", 99:59:59.999 --> 99:59:59.999 evaluated a few solutions, named pagure, which is the git solution Fedora is using, 99:59:59.999 --> 99:59:59.999 which is a Python thing based on gitolite, 99:59:59.999 --> 99:59:59.999 gitlab, which is the biggest Github competitor 99:59:59.999 --> 99:59:59.999 gogs/gitea, which is some golang-based small git service. 99:59:59.999 --> 99:59:59.999 pagure turned out to be not stable enough for our needs 99:59:59.999 --> 99:59:59.999 and we would have to do to much coding inside pagure to use it in our infrastructure 99:59:59.999 --> 99:59:59.999 because pagure is very strongly ??? with the Fedora infrastructure, 99:59:59.999 --> 99:59:59.999 specially its user authentication and user management stuff. 99:59:59.999 --> 99:59:59.999 Gitlab had an other problem called "opencore" and 99:59:59.999 --> 99:59:59.999 "contributor license agreement" which means 99:59:59.999 --> 99:59:59.999 I and others were not very happy with contributing code to Gitlab 99:59:59.999 --> 99:59:59.999 which is something that will always happen if you maintain such a service. 99:59:59.999 --> 99:59:59.999 And gogs and gitea is nice but it's small 99:59:59.999 --> 99:59:59.999 It will not be able to manage 10,000s of repositories. 99:59:59.999 --> 99:59:59.999 Next step happened in August 2017 when we had a sprint here in Hamburg 99:59:59.999 --> 99:59:59.999 at the hackerlab CCC on the other side of the building, 99:59:59.999 --> 99:59:59.999 where we talked about it. 99:59:59.999 --> 99:59:59.999 After long discussions, we decided to go with Gitlab 99:59:59.999 --> 99:59:59.999 because Gitlab, at that point, was the best solution that was already ready. 99:59:59.999 --> 99:59:59.999 We didn't have to adapt too much, we don't need to patch it 99:59:59.999 --> 99:59:59.999 which turned out it isn't true, but it's an other problem 99:59:59.999 --> 99:59:59.999 It had features like continuous integration ready, 99:59:59.999 --> 99:59:59.999 it had features like code review ready, wiki pretty good working 99:59:59.999 --> 99:59:59.999 and ??? very scalable in all directions 99:59:59.999 --> 99:59:59.999 Every component is scalable which is good for us. 99:59:59.999 --> 99:59:59.999 This is a TODO point, I wanted to add an image about the restaurant 99:59:59.999 --> 99:59:59.999 where we decided on the name "salsa". 99:59:59.999 --> 99:59:59.999 Somebody of you may ask yourself where the name is coming from. 99:59:59.999 --> 99:59:59.999 There's a small mexican restaurant a few hundred meters from here 99:59:59.999 --> 99:59:59.999 where you can get great burritos and they have a painting at the back 99:59:59.999 --> 99:59:59.999 with the term "salsa" written 99:59:59.999 --> 99:59:59.999 and we were deciding on a name which just not describes the type of service on it 99:59:59.999 --> 99:59:59.999 so we wanted… 99:59:59.999 --> 99:59:59.999 Yes, it's also a sauce. So salsa had sauce. 99:59:59.999 --> 99:59:59.999 I wanted to call it Klaus, but we decided against it so somebody came up 99:59:59.999 --> 99:59:59.999 in the restaurant with the name "salsa" and so it's called salsa. 99:59:59.999 --> 99:59:59.999 In the meanwhile, we talked a lot with the Gitlab people 99:59:59.999 --> 99:59:59.999 which were very kind and helped us with our problems. 99:59:59.999 --> 99:59:59.999 We also talked with them about the CLA problem and after some discussions, 99:59:59.999 --> 99:59:59.999 the lawyer of SPI was also involved, 99:59:59.999 --> 99:59:59.999 we made them to remove the CLA and replace it with something better. 99:59:59.999 --> 99:59:59.999 Contributing patches to Gitlab is now much easier and better 99:59:59.999 --> 99:59:59.999 which is something we are very proud of 99:59:59.999 --> 99:59:59.999 [Applause] 99:59:59.999 --> 99:59:59.999 And between November and the 25th of December, we implemented salsa two times 99:59:59.999 --> 99:59:59.999 First time on ???.debian.net where we had root but 99:59:59.999 --> 99:59:59.999 after more discussions we decided having this maintained at a (debian).org box 99:59:59.999 --> 99:59:59.999 would be better, which made us ??? ansible stuff 99:59:59.999 --> 99:59:59.999 and develop a ??? to be able to install gitlab as a non-privileged user 99:59:59.999 --> 99:59:59.999 but we did that. 99:59:59.999 --> 99:59:59.999 In Christmas, he was able to release salsa into public beta. 99:59:59.999 --> 99:59:59.999 Things went well, which allowed, at the end of January, salsa to leave the beta 99:59:59.999 --> 99:59:59.999 Since then it's official, our official git successor. 99:59:59.999 --> 99:59:59.999 What will happen in the future? 99:59:59.999 --> 99:59:59.999 Oh no, this is already past. 99:59:59.999 --> 99:59:59.999 On May, we disable user and project creation on alioth. 99:59:59.999 --> 99:59:59.999 Still in May, we disabled the not so much used version control systems, 99:59:59.999 --> 99:59:59.999 bazaar, mercurial and darcs 99:59:59.999 --> 99:59:59.999 On Thursday (May 17th 2018), I disabled projects web sites. 99:59:59.999 --> 99:59:59.999 And this is future, at the end the month, 99:59:59.999 --> 99:59:59.999 all other remaining version control systems on alioth will get disabled. 99:59:59.999 --> 99:59:59.999 So if you have anything running on alioth, still running on alioth, 99:59:59.999 --> 99:59:59.999 cron jobs are also disabled so you don't have cron jobs enabled anymore 99:59:59.999 --> 99:59:59.999 Be it whatever you think of, remove it. 99:59:59.999 --> 99:59:59.999 1st of June, alioth will be off, you won't be able to get any data anymore 99:59:59.999 --> 99:59:59.999 from alioth. 99:59:59.999 --> 99:59:59.999 You can get the ??? via DSA to get subsequent backups, that's up to you 99:59:59.999 --> 99:59:59.999 but I don't recommend it and they won't like it. 99:59:59.999 --> 99:59:59.999 Yeah 99:59:59.999 --> 99:59:59.999 In June, alioth will come to an end. 99:59:59.999 --> 99:59:59.999 It served us well for 10, 15 years, but its time is over. 99:59:59.999 --> 99:59:59.999 Some numbers. Where are we now? 99:59:59.999 --> 99:59:59.999 Yesterday (May 18th 2018), we had 23,700 repositories on gitlab, 99:59:59.999 --> 99:59:59.999 3200 users, 400 groups, which sums up around 90GB on disk, which is nice. 99:59:59.999 --> 99:59:59.999 For a service running for more or less 6 months, it's a pretty nice number. 99:59:59.999 --> 99:59:59.999 What are our future plans. 99:59:59.999 --> 99:59:59.999 ??? Docker registry, by now you can use external registries 99:59:59.999 --> 99:59:59.999 which is working 99:59:59.999 --> 99:59:59.999 You can the gitlab registry for Docker images 99:59:59.999 --> 99:59:59.999 but it will be nicer to have our own registry. 99:59:59.999 --> 99:59:59.999 That is pretty high on my todo list, after alioth is gone. 99:59:59.999 --> 99:59:59.999 We want more runners, so you are able to sponsor runners, if you have machines or 99:59:59.999 --> 99:59:59.999 some money you want to spend on runners, please tell us. 99:59:59.999 --> 99:59:59.999 What are runners? Runners are the things that are used by Gitlab CI to build code 99:59:59.999 --> 99:59:59.999 or test code, or do things. 99:59:59.999 --> 99:59:59.999 You can use it to build your packages, you can use it to autopkgtest you packages 99:59:59.999 --> 99:59:59.999 you can use it to build websites or whatever you like. 99:59:59.999 --> 99:59:59.999 It's pretty useful and I think using CI more will be a big step forward for Debian. 99:59:59.999 --> 99:59:59.999 We should really get more into it. 99:59:59.999 --> 99:59:59.999 There are already some projects like the reproducible builds, the debci guys 99:59:59.999 --> 99:59:59.999 that are working on such stuff 99:59:59.999 --> 99:59:59.999 and now we have the infrastructure that every DD, every developer or package maintainer 99:59:59.999 --> 99:59:59.999 can use it. 99:59:59.999 --> 99:59:59.999 There's also an other feature called 99:59:59.999 --> 99:59:59.999 "devops" which is based on kubernetes which allows you to even 99:59:59.999 --> 99:59:59.999 deploy and test things properly. 99:59:59.999 --> 99:59:59.999 So if you have package which implements a web service, you can even run 99:59:59.999 --> 99:59:59.999 ??? kubernetes part which runs a web server, 99:59:59.999 --> 99:59:59.999 you can test it, you can even record it, do QA test and so on 99:59:59.999 --> 99:59:59.999 all based on this devops feature which would also be a nice thing. 99:59:59.999 --> 99:59:59.999 By now, we don't have a kubernetes instance we can use for it, 99:59:59.999 --> 99:59:59.999 so if you have a spare kubernetes instance you want to offer Debian, 99:59:59.999 --> 99:59:59.999 please talk to us.