9:59:59.000,9:59:59.000 I am Nicolas Dandrimont. 9:59:59.000,9:59:59.000 I am going to talk to you about a year of fedmsg in Debian. 9:59:59.000,9:59:59.000 We had a problem before with infrastructure in distributions. 9:59:59.000,9:59:59.000 All services are bit like people. 9:59:59.000,9:59:59.000 There are dozen of services maintained by many people 9:59:59.000,9:59:59.000 and each of those services has its own way of communicating with the rest of the world 9:59:59.000,9:59:59.000 Meaning that if you want to spin up a new service[br]that needs to talk to other services in the distribution 9:59:59.000,9:59:59.000 which is basically any service you want to include 9:59:59.000,9:59:59.000 you will need to implement a bunch of communication systems 9:59:59.000,9:59:59.000 For instance, in the Debian infrastructure 9:59:59.000,9:59:59.000 we have our archive software, which is dak, 9:59:59.000,9:59:59.000 that mostly uses emails and databases to communicate. 9:59:59.000,9:59:59.000 The metadat is available in a RFC822 format with no real API. 9:59:59.000,9:59:59.000 The database is not public either. 9:59:59.000,9:59:59.000 The build queue management software, which is called wanna-build, 9:59:59.000,9:59:59.000 polls a database every so often to know what needs to get built. 9:59:59.000,9:59:59.000 There is no API outside of its database 9:59:59.000,9:59:59.000 that isn't public either 9:59:59.000,9:59:59.000 Our bug tracking system, which is called debbugs, 9:59:59.000,9:59:59.000 works via email, stores its data in flat files, for now, 9:59:59.000,9:59:59.000 and exposes a read-only SOAP API. 9:59:59.000,9:59:59.000 Our source control managament pushes in the distribution-provided repositories on alioth 9:59:59.000,9:59:59.000 can trigger an IRC bot or some emails 9:59:59.000,9:59:59.000 but there is no real central notification mechanism. 9:59:59.000,9:59:59.000 We have some kludges that are available to overcome those issues. 9:59:59.000,9:59:59.000 We have the Ultimate Debian Database 9:59:59.000,9:59:59.000 which contains a snapshot of a lot of the databases that are underlying the Debian infrastructure 9:59:59.000,9:59:59.000 This means that every so often, 9:59:59.000,9:59:59.000 there is a cron that runs and imports data from a service here, a service there. 9:59:59.000,9:59:59.000 There is no realtime data. 9:59:59.000,9:59:59.000 It's useful for distro-wide Q&A stuff because you don't need to have realtime data 9:59:59.000,9:59:59.000 But when you want some notification for trying to build a new package or something 9:59:59.000,9:59:59.000 That doesn't work very well 9:59:59.000,9:59:59.000 and the consistency between the different data sources is not guaranteed. 9:59:59.000,9:59:59.000 We have another central notification system which the package tracking system 9:59:59.000,9:59:59.000 which also is cron-triggered or email-triggered 9:59:59.000,9:59:59.000 You can update the data from the BTS using ?? 9:59:59.000,9:59:59.000 You can subscribe to email updates on a given package 9:59:59.000,9:59:59.000 But the messages are not uniform, 9:59:59.000,9:59:59.000 they can be machine parsed. 9:59:59.000,9:59:59.000 There are a few headers but they are not sufficient to know what the message is about. 9:59:59.000,9:59:59.000 And it's still not realtime. 9:59:59.000,9:59:59.000 The Fedora people invented something that could improve stuff which is called fedmsg. 9:59:59.000,9:59:59.000 It was actually introduced in 2009. 9:59:59.000,9:59:59.000 It's an unified message bus that can reduce the coupling between the different services in a distribution. 9:59:59.000,9:59:59.000 The idea is that services can subscribe to one or several message topics, register callbacks and react to events 9:59:59.000,9:59:59.000 that are triggered by all the services in the distribution. 9:59:59.000,9:59:59.000 There is a bunch of stuff that is already implemented in fedmsg. 9:59:59.000,9:59:59.000 You get a stream of data with all the activity in your infrastructure which allows you to do statistics for instance 9:59:59.000,9:59:59.000 You decouple interdepent services because you can swap something for another 9:59:59.000,9:59:59.000 Or just listen to the messages and start doing stuff directly without having to fiddle a database or something. 9:59:59.000,9:59:59.000 You can get a pluggable unified notification system that can gather all the events in the project and send them by email, by IRC, 9:59:59.000,9:59:59.000 on your mobile phone, on your desktop, everywhere you want. 9:59:59.000,9:59:59.000 Fedora people use fedmsg to implement a badge system 9:59:59.000,9:59:59.000 which is some kind of gamification of the development process of the distribution. 9:59:59.000,9:59:59.000 They implemented a live web dashboard. 9:59:59.000,9:59:59.000 They implemented IRC feed. 9:59:59.000,9:59:59.000 And then they also got some bot bans on social networks because they were flooding. 9:59:59.000,9:59:59.000 How does it work? 9:59:59.000,9:59:59.000 Well, the first idea was to use AMQP as implemented by qpid. 9:59:59.000,9:59:59.000 Basically, you take all your services and you have them send their messages in a central broker. 9:59:59.000,9:59:59.000 Then you have several listeners that can send messages to clients. 9:59:59.000,9:59:59.000 There were a few issues with this. 9:59:59.000,9:59:59.000 Basically, you have a single point of failure at the central broker. 9:59:59.000,9:59:59.000 And the brokers weren't really reliable. 9:59:59.000,9:59:59.000 When they tested it under load, the brokers were tipping over. 9:59:59.000,9:59:59.000 The actual implementation of fedmsg uses 0mq. 9:59:59.000,9:59:59.000 Basically what you get is not a single broker. 9:59:59.000,9:59:59.000 You get a mesh of interconnected services. 9:59:59.000,9:59:59.000 Basically, you can connect only to the services that you want to listen to. 9:59:59.000,9:59:59.000 The big drawback of this is that each and every service has to open up a port on the public Internet 9:59:59.000,9:59:59.000 for people to be able to connect to it. 9:59:59.000,9:59:59.000 There are some solutions for that which I will talk about. 9:59:59.000,9:59:59.000 But the main advantage is that you have no central broker 9:59:59.000,9:59:59.000 and they got like a hundred-fold speedup over the previous implementation. 9:59:59.000,9:59:59.000 You also have an issue with service discovery. 9:59:59.000,9:59:59.000 You can write a broker which gives you back your single point of failure. 9:59:59.000,9:59:59.000 You can use DNS which means that can say "Hey I added a new service, let's use this SRV record to get to it" 9:59:59.000,9:59:59.000 Or you can distribute a text file. 9:59:59.000,9:59:59.000 Last year, during the Google Summer of Code, I mentored Simon Choppin 9:59:59.000,9:59:59.000 ...who implemented the DNS solution for integration in fedmsg in Debian. 9:59:59.000,9:59:59.000 The Fedora people as they control their whole infrastructure just distribute a text file 9:59:59.000,9:59:59.000 ...with the list of servers that are sending fedmsg messages. 9:59:59.000,9:59:59.000 How do you use it? 9:59:59.000,9:59:59.000 This is the Fedora topology. 9:59:59.000,9:59:59.000 I didn't have much time to do the Debian one. 9:59:59.000,9:59:59.000 It's really simpler. I'll talk about it later. 9:59:59.000,9:59:59.000 Basically, the messages are split in topics where you have a hierarchy of topics. 9:59:59.000,9:59:59.000 It's really easy to filter out the things that you want to listen to. 9:59:59.000,9:59:59.000 For instance, you can filter all the messages that concern package upload by using the dak service. 9:59:59.000,9:59:59.000 Or everything that involves a given package or something else. 9:59:59.000,9:59:59.000 Publishing messages is really trivial. 9:59:59.000,9:59:59.000 From Python, you only have to import the module, 9:59:59.000,9:59:59.000 do fedmsg.publish with a dict of the data that you want to send. 9:59:59.000,9:59:59.000 And that's it, your message is published. 9:59:59.000,9:59:59.000 From the shell, it's really easy too. 9:59:59.000,9:59:59.000 You just have a command called fedmsg-logger that you can pipe some input to. 9:59:59.000,9:59:59.000 And it goes on the bus, so it's really simple. 9:59:59.000,9:59:59.000 Receiving messages is trivial too. 9:59:59.000,9:59:59.000 In Python, you load the configuration 9:59:59.000,9:59:59.000 ...and you just have an iterator 9:59:59.000,9:59:59.000 [audio stops] 9:59:59.000,9:59:59.000 was a replay mechanism with just a sequence number 9:59:59.000,9:59:59.000 which will have your client query the event sender for new messages that you would have missed 9:59:59.000,9:59:59.000 ...in case of a network failure or anything. 9:59:59.000,9:59:59.000 That's how basically the system works. 9:59:59.000,9:59:59.000 Now, what about fedmsg in Debian? 9:59:59.000,9:59:59.000 During the last Google Summer of code, a lot happened thanks to Simon Chopin's involvement. 9:59:59.000,9:59:59.000 He did most of the packaging of fedmsg and its dependencies 9:59:59.000,9:59:59.000 ...which means that you can just apt-get install fedmsg and get it running. 9:59:59.000,9:59:59.000 It's available in sid, jessie and wheezy-backports. 9:59:59.000,9:59:59.000 He adapted the code of fedmsg to make it distribution agnostic. 9:59:59.000,9:59:59.000 He had a lot of support from upstream developers in Fedora to make that happen. 9:59:59.000,9:59:59.000 They are really excited to have their stuff being used by Debian or by other organizations, 9:59:59.000,9:59:59.000 ...that fedmsg was the right solution for event notification. 9:59:59.000,9:59:59.000 And finally, we bootstrapped the Debian bus by using mailing-list subscriptions 9:59:59.000,9:59:59.000 ...to get bug notifications and package upload notifications 9:59:59.000,9:59:59.000 ...and on mentors.debian.net which is a service I can control, so it's easy to add new stuff to it. 9:59:59.000,9:59:59.000 What then? 9:59:59.000,9:59:59.000 After the Google Summer of Code, there was some packaging adaptations to make it easier to run services based on fedmsg, 9:59:59.000,9:59:59.000 ...proper backports and maintainance of the bus 9:59:59.000,9:59:59.000 ...which mostly means keeping the software up-to-date 9:59:59.000,9:59:59.000 ...because the upstream is really active and responsive to bug reports. 9:59:59.000,9:59:59.000 It's really nice to work with them. 9:59:59.000,9:59:59.000 Since July 14th 2013 which is the day we started sending messages on the bus, 9:59:59.000,9:59:59.000 ...we had around 200k messages split accross 155k bug mails and 45k uploads 9:59:59.000,9:59:59.000 ...which proves that Debian is a really active project, I guess. 9:59:59.000,9:59:59.000 [laughs] 9:59:59.000,9:59:59.000 The latest developments with fedmsg is the packaging of Datanommer 9:59:59.000,9:59:59.000 ...which is a database component that can store messages that has been sent to the bus. 9:59:59.000,9:59:59.000 It allows Fedora to do queries on their messages 9:59:59.000,9:59:59.000 ...and give people the achievements that they did like "yeah, you had a hundred build failures" 9:59:59.000,9:59:59.000 ...or stuff like that.[br][laughs] 9:59:59.000,9:59:59.000 One big issue with fedmsg that I said earlier is that Debian services are widely distributed. 9:59:59.000,9:59:59.000 Some of the times, firewall restrictions are out of Debian control, 9:59:59.000,9:59:59.000 ...which is also the case of with the Fedora infrastructure 9:59:59.000,9:59:59.000 ...because some of their servers are hosted within Redhat 9:59:59.000,9:59:59.000 ...and Redhat networking sometimes don't want to open firewall ports. 9:59:59.000,9:59:59.000 So we need a way for services to push their messages instead of having clients pull the messages. 9:59:59.000,9:59:59.000 There is a component in fedmsg which have been created by the Fedora people which is called fedmsg-relay 9:59:59.000,9:59:59.000 ...which basically is just a tube where you push your message using a 0mq socket 9:59:59.000,9:59:59.000 ...and it then pushes it to the subscribers on the other side. 9:59:59.000,9:59:59.000 It just allows to bypass firwalls. 9:59:59.000,9:59:59.000 The issue is that it uses a non-standard port and a non-standard protocol. 9:59:59.000,9:59:59.000 It's just 0mq so it basically put your data on the wire and that's it. 9:59:59.000,9:59:59.000 So, I am pondering a way for services to push their messages using more classic web services. 9:59:59.000,9:59:59.000 You will take your JSON dictionary and push it by POST through HTTPS. 9:59:59.000,9:59:59.000 And then after that send the message to the bus 9:59:59.000,9:59:59.000 ...which I think will make it easier to integrate with other Debian services. 9:59:59.000,9:59:59.000 This was a really short talk. 9:59:59.000,9:59:59.000 I hope there is some discussions afterwards. 9:59:59.000,9:59:59.000 In conclusion, I am really glad it works. 9:59:59.000,9:59:59.000 For the moment, it's really apart from the Debian infrastructure. 9:59:59.000,9:59:59.000 So the big challenge will be to try to integrate fedmsg to Debian infrastructure 9:59:59.000,9:59:59.000 ...and use it for real. 9:59:59.000,9:59:59.000 If you want to contact me, I am olasd, 9:59:59.000,9:59:59.000 ...I am here for the whole conference. 9:59:59.000,9:59:59.000 If you want to talk to me about it, if you want to help me, 9:59:59.000,9:59:59.000 ...I am a little bit alone on this project, so I'll be glad if someone would join. 9:59:59.000,9:59:59.000 I'll be glad to hold an hacking session later this week. 9:59:59.000,9:59:59.000 Thanks for your attention! 9:59:59.000,9:59:59.000 [applause] 9:59:59.000,9:59:59.000 Was it this clear? 9:59:59.000,9:59:59.000 You talked about the ??? use to publish SRV record. 9:59:59.000,9:59:59.000 I missed some of the details of what that means. 9:59:59.000,9:59:59.000 What is in a SRV record and how do I do discovery on it? 9:59:59.000,9:59:59.000 The idea is that to actually receive messages, you need the host and the port of the sender. 9:59:59.000,9:59:59.000 If you have several WSGI workers, you have several ports that you need to listen to. 9:59:59.000,9:59:59.000 What we do with the SRV record is basically under the domain name of the service, 9:59:59.000,9:59:59.000 ...for example ftp-master.debian.org, we would have fedmsg.tcp.ftp-master.debian.org 9:59:59.000,9:59:59.000 ...which will point to the four or five workers that you would use to get the messages. 9:59:59.000,9:59:59.000 So if I don't know that ftp-master.debian.org is something that I want to subscribe to as a mechanism for getting the details, 9:59:59.000,9:59:59.000 ...is there something which tells me that ftp-master.debian.org is a an host to begin with? 9:59:59.000,9:59:59.000 No, not yet. 9:59:59.000,9:59:59.000 Only part of the problem is solved. 9:59:59.000,9:59:59.000 Currently there is no list of every single services that publish messages. 9:59:59.000,9:59:59.000 What they do in Fedora and what we do in Debian too, for public consumption, 9:59:59.000,9:59:59.000 ...there is a component called the gateway which will connect to all the message sources 9:59:59.000,9:59:59.000 ...and rewrite the messages to send them to clients. 9:59:59.000,9:59:59.000 You don't get the replay mechanism because it works only for a single source 9:59:59.000,9:59:59.000 ...but you solve your discovery problem but you get back the single point of failure. 9:59:59.000,9:59:59.000