1 99:59:59,999 --> 99:59:59,999 I am Nicolas Dandrimont. 2 99:59:59,999 --> 99:59:59,999 I am going to talk to you about a year of fedmsg in Debian. 3 99:59:59,999 --> 99:59:59,999 We had a problem before with infrastructure in distributions. 4 99:59:59,999 --> 99:59:59,999 All services are bit like people. 5 99:59:59,999 --> 99:59:59,999 There are dozen of services maintained by many people 6 99:59:59,999 --> 99:59:59,999 and each of those services has its own way of communicating with the rest of the world 7 99:59:59,999 --> 99:59:59,999 Meaning that if you want to spin up a new service that needs to talk to other services in the distribution 8 99:59:59,999 --> 99:59:59,999 which is basically any service you want to include 9 99:59:59,999 --> 99:59:59,999 you will need to implement a bunch of communication systems 10 99:59:59,999 --> 99:59:59,999 For instance, in the Debian infrastructure 11 99:59:59,999 --> 99:59:59,999 we have our archive software, which is dak, 12 99:59:59,999 --> 99:59:59,999 that mostly uses emails and databases to communicate. 13 99:59:59,999 --> 99:59:59,999 The metadat is available in a RFC822 format with no real API. 14 99:59:59,999 --> 99:59:59,999 The database is not public either. 15 99:59:59,999 --> 99:59:59,999 The build queue management software, which is called wanna-build, 16 99:59:59,999 --> 99:59:59,999 polls a database every so often to know what needs to get built. 17 99:59:59,999 --> 99:59:59,999 There is no API outside of its database 18 99:59:59,999 --> 99:59:59,999 that isn't public either 19 99:59:59,999 --> 99:59:59,999 Our bug tracking system, which is called debbugs, 20 99:59:59,999 --> 99:59:59,999 works via email, stores its data in flat files, for now, 21 99:59:59,999 --> 99:59:59,999 and exposes a read-only SOAP API. 22 99:59:59,999 --> 99:59:59,999 Our source control managament pushes in the distribution-provided repositories on alioth 23 99:59:59,999 --> 99:59:59,999 can trigger an IRC bot or some emails 24 99:59:59,999 --> 99:59:59,999 but there is no real central notification mechanism. 25 99:59:59,999 --> 99:59:59,999 We have some kludges that are available to overcome those issues. 26 99:59:59,999 --> 99:59:59,999 We have the Ultimate Debian Database 27 99:59:59,999 --> 99:59:59,999 which contains a snapshot of a lot of the databases that are underlying the Debian infrastructure 28 99:59:59,999 --> 99:59:59,999 This means that every so often, 29 99:59:59,999 --> 99:59:59,999 there is a cron that runs and imports data from a service here, a service there. 30 99:59:59,999 --> 99:59:59,999 There is no realtime data. 31 99:59:59,999 --> 99:59:59,999 It's useful for distro-wide Q&A stuff because you don't need to have realtime data 32 99:59:59,999 --> 99:59:59,999 But when you want some notification for trying to build a new package or something 33 99:59:59,999 --> 99:59:59,999 That doesn't work very well 34 99:59:59,999 --> 99:59:59,999 and the consistency between the different data sources is not guaranteed. 35 99:59:59,999 --> 99:59:59,999 We have another central notification system which the package tracking system 36 99:59:59,999 --> 99:59:59,999 which also is cron-triggered or email-triggered 37 99:59:59,999 --> 99:59:59,999 You can update the data from the BTS using ?? 38 99:59:59,999 --> 99:59:59,999 You can subscribe to email updates on a given package 39 99:59:59,999 --> 99:59:59,999 But the messages are not uniform, 40 99:59:59,999 --> 99:59:59,999 they can be machine parsed. 41 99:59:59,999 --> 99:59:59,999 There are a few headers but they are not sufficient to know what the message is about. 42 99:59:59,999 --> 99:59:59,999 And it's still not realtime. 43 99:59:59,999 --> 99:59:59,999 The Fedora people invented something that could improve stuff which is called fedmsg. 44 99:59:59,999 --> 99:59:59,999 It was actually introduced in 2009. 45 99:59:59,999 --> 99:59:59,999 It's an unified message bus that can reduce the coupling between the different services in a distribution. 46 99:59:59,999 --> 99:59:59,999 The idea is that services can subscribe to one or several message topics, register callbacks and react to events 47 99:59:59,999 --> 99:59:59,999 that are triggered by all the services in the distribution. 48 99:59:59,999 --> 99:59:59,999 There is a bunch of stuff that is already implemented in fedmsg. 49 99:59:59,999 --> 99:59:59,999 You get a stream of data with all the activity in your infrastructure which allows you to do statistics for instance 50 99:59:59,999 --> 99:59:59,999 You decouple interdepent services because you can swap something for another 51 99:59:59,999 --> 99:59:59,999 Or just listen to the messages and start doing stuff directly without having to fiddle a database or something. 52 99:59:59,999 --> 99:59:59,999 You can get a pluggable unified notification system that can gather all the events in the project and send them by email, by IRC, 53 99:59:59,999 --> 99:59:59,999 on your mobile phone, on your desktop, everywhere you want. 54 99:59:59,999 --> 99:59:59,999 Fedora people use fedmsg to implement a badge system 55 99:59:59,999 --> 99:59:59,999 which is some kind of gamification of the development process of the distribution. 56 99:59:59,999 --> 99:59:59,999 They implemented a live web dashboard. 57 99:59:59,999 --> 99:59:59,999 They implemented IRC feed. 58 99:59:59,999 --> 99:59:59,999 And then they also got some bot bans on social networks because they were flooding. 59 99:59:59,999 --> 99:59:59,999 How does it work? 60 99:59:59,999 --> 99:59:59,999 Well, the first idea was to use AMQP as implemented by qpid. 61 99:59:59,999 --> 99:59:59,999 Basically, you take all your services and you have them send their messages in a central broker. 62 99:59:59,999 --> 99:59:59,999 Then you have several listeners that can send messages to clients. 63 99:59:59,999 --> 99:59:59,999 There were a few issues with this. 64 99:59:59,999 --> 99:59:59,999 Basically, you have a single point of failure at the central broker. 65 99:59:59,999 --> 99:59:59,999 And the brokers weren't really reliable. 66 99:59:59,999 --> 99:59:59,999 When they tested it under load, the brokers were tipping over. 67 99:59:59,999 --> 99:59:59,999 The actual implementation of fedmsg uses 0mq. 68 99:59:59,999 --> 99:59:59,999 Basically what you get is not a single broker. 69 99:59:59,999 --> 99:59:59,999 You get a mesh of interconnected services. 70 99:59:59,999 --> 99:59:59,999 Basically, you can connect only to the services that you want to listen to. 71 99:59:59,999 --> 99:59:59,999 The big drawback of this is that each and every service has to open up a port on the public Internet 72 99:59:59,999 --> 99:59:59,999 for people to be able to connect to it. 73 99:59:59,999 --> 99:59:59,999 There are some solutions for that which I will talk about. 74 99:59:59,999 --> 99:59:59,999 But the main advantage is that you have no central broker 75 99:59:59,999 --> 99:59:59,999 and they got like a hundred-fold speedup over the previous implementation. 76 99:59:59,999 --> 99:59:59,999 You also have an issue with service discovery. 77 99:59:59,999 --> 99:59:59,999 You can write a broker which gives you back your single point of failure. 78 99:59:59,999 --> 99:59:59,999 You can use DNS which means that can say "Hey I added a new service, let's use this SRV record to get to it" 79 99:59:59,999 --> 99:59:59,999 Or you can distribute a text file. 80 99:59:59,999 --> 99:59:59,999 Last year, during the Google Summer of Code, I mentored Simon Choppin 81 99:59:59,999 --> 99:59:59,999 ...who implemented the DNS solution for integration in fedmsg in Debian. 82 99:59:59,999 --> 99:59:59,999 The Fedora people as they control their whole infrastructure just distribute a text file 83 99:59:59,999 --> 99:59:59,999 ...with the list of servers that are sending fedmsg messages. 84 99:59:59,999 --> 99:59:59,999 How do you use it? 85 99:59:59,999 --> 99:59:59,999 This is the Fedora topology. 86 99:59:59,999 --> 99:59:59,999 I didn't have much time to do the Debian one. 87 99:59:59,999 --> 99:59:59,999 It's really simpler. I'll talk about it later. 88 99:59:59,999 --> 99:59:59,999 Basically, the messages are split in topics where you have a hierarchy of topics. 89 99:59:59,999 --> 99:59:59,999 It's really easy to filter out the things that you want to listen to. 90 99:59:59,999 --> 99:59:59,999 For instance, you can filter all the messages that concern package upload by using the dak service. 91 99:59:59,999 --> 99:59:59,999 Or everything that involves a given package or something else. 92 99:59:59,999 --> 99:59:59,999 Publishing messages is really trivial. 93 99:59:59,999 --> 99:59:59,999 From Python, you only have to import the module, 94 99:59:59,999 --> 99:59:59,999 do fedmsg.publish with a dict of the data that you want to send. 95 99:59:59,999 --> 99:59:59,999 And that's it, your message is published. 96 99:59:59,999 --> 99:59:59,999 From the shell, it's really easy too. 97 99:59:59,999 --> 99:59:59,999 You just have a command called fedmsg-logger that you can pipe some input to. 98 99:59:59,999 --> 99:59:59,999 And it goes on the bus, so it's really simple. 99 99:59:59,999 --> 99:59:59,999 Receiving messages is trivial too. 100 99:59:59,999 --> 99:59:59,999 In Python, you load the configuration 101 99:59:59,999 --> 99:59:59,999 ...and you just have an iterator 102 99:59:59,999 --> 99:59:59,999 [audio stops] 103 99:59:59,999 --> 99:59:59,999 was a replay mechanism with just a sequence number 104 99:59:59,999 --> 99:59:59,999 which will have your client query the event sender for new messages that you would have missed 105 99:59:59,999 --> 99:59:59,999 ...in case of a network failure or anything. 106 99:59:59,999 --> 99:59:59,999 That's how basically the system works. 107 99:59:59,999 --> 99:59:59,999 Now, what about fedmsg in Debian? 108 99:59:59,999 --> 99:59:59,999 During the last Google Summer of code, a lot happened thanks to Simon Chopin's involvement. 109 99:59:59,999 --> 99:59:59,999 He did most of the packaging of fedmsg and its dependencies 110 99:59:59,999 --> 99:59:59,999 ...which means that you can just apt-get install fedmsg and get it running. 111 99:59:59,999 --> 99:59:59,999 It's available in sid, jessie and wheezy-backports. 112 99:59:59,999 --> 99:59:59,999 He adapted the code of fedmsg to make it distribution agnostic. 113 99:59:59,999 --> 99:59:59,999 He had a lot of support from upstream developers in Fedora to make that happen. 114 99:59:59,999 --> 99:59:59,999 They are really excited to have their stuff being used by Debian or by other organizations, 115 99:59:59,999 --> 99:59:59,999 ...that fedmsg was the right solution for event notification. 116 99:59:59,999 --> 99:59:59,999 And finally, we bootstrapped the Debian bus by using mailing-list subscriptions 117 99:59:59,999 --> 99:59:59,999 ...to get bug notifications and package upload notifications 118 99:59:59,999 --> 99:59:59,999 ...and on mentors.debian.net which is a service I can control, so it's easy to add new stuff to it. 119 99:59:59,999 --> 99:59:59,999 What then? 120 99:59:59,999 --> 99:59:59,999 After the Google Summer of Code, there was some packaging adaptations to make it easier to run services based on fedmsg, 121 99:59:59,999 --> 99:59:59,999 ...proper backports and maintainance of the bus 122 99:59:59,999 --> 99:59:59,999 ...which mostly means keeping the software up-to-date 123 99:59:59,999 --> 99:59:59,999 ...because the upstream is really active and responsive to bug reports. 124 99:59:59,999 --> 99:59:59,999 It's really nice to work with them. 125 99:59:59,999 --> 99:59:59,999 Since July 14th 2013 which is the day we started sending messages on the bus, 126 99:59:59,999 --> 99:59:59,999 ...we had around 200k messages split accross 155k bug mails and 45k uploads 127 99:59:59,999 --> 99:59:59,999 ...which proves that Debian is a really active project, I guess. 128 99:59:59,999 --> 99:59:59,999 [laughs] 129 99:59:59,999 --> 99:59:59,999 The latest developments with fedmsg is the packaging of Datanommer 130 99:59:59,999 --> 99:59:59,999 ...which is a database component that can store messages that has been sent to the bus. 131 99:59:59,999 --> 99:59:59,999 It allows Fedora to do queries on their messages 132 99:59:59,999 --> 99:59:59,999 ...and give people the achievements that they did like "yeah, you had a hundred build failures" 133 99:59:59,999 --> 99:59:59,999 ...or stuff like that. [laughs] 134 99:59:59,999 --> 99:59:59,999 One big issue with fedmsg that I said earlier is that Debian services are widely distributed. 135 99:59:59,999 --> 99:59:59,999 Some of the times, firewall restrictions are out of Debian control, 136 99:59:59,999 --> 99:59:59,999 ...which is also the case of with the Fedora infrastructure 137 99:59:59,999 --> 99:59:59,999 ...because some of their servers are hosted within Redhat 138 99:59:59,999 --> 99:59:59,999 ...and Redhat networking sometimes don't want to open firewall ports. 139 99:59:59,999 --> 99:59:59,999 So we need a way for services to push their messages instead of having clients pull the messages. 140 99:59:59,999 --> 99:59:59,999 There is a component in fedmsg which have been created by the Fedora people which is called fedmsg-relay 141 99:59:59,999 --> 99:59:59,999 ...which basically is just a tube where you push your message using a 0mq socket 142 99:59:59,999 --> 99:59:59,999 ...and it then pushes it to the subscribers on the other side. 143 99:59:59,999 --> 99:59:59,999 It just allows to bypass firwalls. 144 99:59:59,999 --> 99:59:59,999 The issue is that it uses a non-standard port and a non-standard protocol. 145 99:59:59,999 --> 99:59:59,999 It's just 0mq so it basically put your data on the wire and that's it. 146 99:59:59,999 --> 99:59:59,999 So, I am pondering a way for services to push their messages using more classic web services. 147 99:59:59,999 --> 99:59:59,999 You will take your JSON dictionary and push it by POST through HTTPS. 148 99:59:59,999 --> 99:59:59,999 And then after that send the message to the bus 149 99:59:59,999 --> 99:59:59,999 ...which I think will make it easier to integrate with other Debian services. 150 99:59:59,999 --> 99:59:59,999 This was a really short talk. 151 99:59:59,999 --> 99:59:59,999 I hope there is some discussions afterwards. 152 99:59:59,999 --> 99:59:59,999 In conclusion, I am really glad it works. 153 99:59:59,999 --> 99:59:59,999 For the moment, it's really apart from the Debian infrastructure. 154 99:59:59,999 --> 99:59:59,999 So the big challenge will be to try to integrate fedmsg to Debian infrastructure 155 99:59:59,999 --> 99:59:59,999 ...and use it for real. 156 99:59:59,999 --> 99:59:59,999 If you want to contact me, I am olasd, 157 99:59:59,999 --> 99:59:59,999 ...I am here for the whole conference. 158 99:59:59,999 --> 99:59:59,999 If you want to talk to me about it, if you want to help me, 159 99:59:59,999 --> 99:59:59,999 ...I am a little bit alone on this project, so I'll be glad if someone would join. 160 99:59:59,999 --> 99:59:59,999 I'll be glad to hold an hacking session later this week. 161 99:59:59,999 --> 99:59:59,999 Thanks for your attention! 162 99:59:59,999 --> 99:59:59,999 [applause] 163 99:59:59,999 --> 99:59:59,999 Was it this clear? 164 99:59:59,999 --> 99:59:59,999 You talked about the ??? use to publish SRV record. 165 99:59:59,999 --> 99:59:59,999 I missed some of the details of what that means. 166 99:59:59,999 --> 99:59:59,999 What is in a SRV record and how do I do discovery on it? 167 99:59:59,999 --> 99:59:59,999 The idea is that to actually receive messages, you need the host and the port of the sender. 168 99:59:59,999 --> 99:59:59,999 If you have several WSGI workers, you have several ports that you need to listen to. 169 99:59:59,999 --> 99:59:59,999 What we do with the SRV record is basically under the domain name of the service, 170 99:59:59,999 --> 99:59:59,999 ...for example ftp-master.debian.org, we would have fedmsg.tcp.ftp-master.debian.org 171 99:59:59,999 --> 99:59:59,999 ...which will point to the four or five workers that you would use to get the messages. 172 99:59:59,999 --> 99:59:59,999 So if I don't know that ftp-master.debian.org is something that I want to subscribe to as a mechanism for getting the details, 173 99:59:59,999 --> 99:59:59,999 ...is there something which tells me that ftp-master.debian.org is a an host to begin with? 174 99:59:59,999 --> 99:59:59,999 No, not yet. 175 99:59:59,999 --> 99:59:59,999 Only part of the problem is solved. 176 99:59:59,999 --> 99:59:59,999 Currently there is no list of every single services that publish messages. 177 99:59:59,999 --> 99:59:59,999 What they do in Fedora and what we do in Debian too, for public consumption, 178 99:59:59,999 --> 99:59:59,999 ...there is a component called the gateway which will connect to all the message sources 179 99:59:59,999 --> 99:59:59,999 ...and rewrite the messages to send them to clients. 180 99:59:59,999 --> 99:59:59,999 You don't get the replay mechanism because it works only for a single source 181 99:59:59,999 --> 99:59:59,999 ...but you solve your discovery problem but you get back the single point of failure. 182 99:59:59,999 --> 99:59:59,999