WEBVTT

00:00:02.440 --> 00:00:05.281
I am Nicolas Dandrimont.

00:00:06.080 --> 00:00:08.801
I am going to talk to you about a year of fedmsg in Debian.

00:00:09.151 --> 00:00:12.120
We had a problem before with infrastructure in distributions.

00:00:13.841 --> 00:00:15.760
All services are bit like people.

99:59:59.999 --> 99:59:59.999
There are dozen of services maintained by many people

99:59:59.999 --> 99:59:59.999
and each of those services has its own way of communicating with the rest of the world

99:59:59.999 --> 99:59:59.999
Meaning that if you want to spin up a new service
that needs to talk to other services in the distribution

99:59:59.999 --> 99:59:59.999
which is basically any service you want to include

99:59:59.999 --> 99:59:59.999
you will need to implement a bunch of communication systems

99:59:59.999 --> 99:59:59.999
For instance, in the Debian infrastructure

99:59:59.999 --> 99:59:59.999
we have our archive software, which is dak,

99:59:59.999 --> 99:59:59.999
that mostly uses emails and databases to communicate.

99:59:59.999 --> 99:59:59.999
The metadat is available in a RFC822 format with no real API.

99:59:59.999 --> 99:59:59.999
The database is not public either.

99:59:59.999 --> 99:59:59.999
The build queue management software, which is called wanna-build,

99:59:59.999 --> 99:59:59.999
polls a database every so often to know what needs to get built.

99:59:59.999 --> 99:59:59.999
There is no API outside of its database

99:59:59.999 --> 99:59:59.999
that isn't public either

99:59:59.999 --> 99:59:59.999
Our bug tracking system, which is called debbugs,

99:59:59.999 --> 99:59:59.999
works via email, stores its data in flat files, for now,

99:59:59.999 --> 99:59:59.999
and exposes a read-only SOAP API.

99:59:59.999 --> 99:59:59.999
Our source control managament pushes in the distribution-provided repositories on alioth

99:59:59.999 --> 99:59:59.999
can trigger an IRC bot or some emails

99:59:59.999 --> 99:59:59.999
but there is no real central notification mechanism.

99:59:59.999 --> 99:59:59.999
We have some kludges that are available to overcome those issues.

99:59:59.999 --> 99:59:59.999
We have the Ultimate Debian Database

99:59:59.999 --> 99:59:59.999
which contains a snapshot of a lot of the databases that are underlying the Debian infrastructure

99:59:59.999 --> 99:59:59.999
This means that every so often,

99:59:59.999 --> 99:59:59.999
there is a cron that runs and imports data from a service here, a service there.

99:59:59.999 --> 99:59:59.999
There is no realtime data.

99:59:59.999 --> 99:59:59.999
It's useful for distro-wide Q&A stuff because you don't need to have realtime data

99:59:59.999 --> 99:59:59.999
But when you want some notification for trying to build a new package or something

99:59:59.999 --> 99:59:59.999
That doesn't work very well

99:59:59.999 --> 99:59:59.999
and the consistency between the different data sources is not guaranteed.

99:59:59.999 --> 99:59:59.999
We have another central notification system which the package tracking system

99:59:59.999 --> 99:59:59.999
which also is cron-triggered or email-triggered

99:59:59.999 --> 99:59:59.999
You can update the data from the BTS using ??

99:59:59.999 --> 99:59:59.999
You can subscribe to email updates on a given package

99:59:59.999 --> 99:59:59.999
But the messages are not uniform,

99:59:59.999 --> 99:59:59.999
they can be machine parsed.

99:59:59.999 --> 99:59:59.999
There are a few headers but they are not sufficient to know what the message is about.

99:59:59.999 --> 99:59:59.999
And it's still not realtime.

99:59:59.999 --> 99:59:59.999
The Fedora people invented something that could improve stuff which is called fedmsg.

99:59:59.999 --> 99:59:59.999
It was actually introduced in 2009.

99:59:59.999 --> 99:59:59.999
It's an unified message bus that can reduce the coupling between the different services in a distribution.

99:59:59.999 --> 99:59:59.999
The idea is that services can subscribe to one or several message topics, register callbacks and react to events

99:59:59.999 --> 99:59:59.999
that are triggered by all the services in the distribution.

99:59:59.999 --> 99:59:59.999
There is a bunch of stuff that is already implemented in fedmsg.

99:59:59.999 --> 99:59:59.999
You get a stream of data with all the activity in your infrastructure which allows you to do statistics for instance

99:59:59.999 --> 99:59:59.999
You decouple interdepent services because you can swap something for another

99:59:59.999 --> 99:59:59.999
Or just listen to the messages and start doing stuff directly without having to fiddle a database or something.

99:59:59.999 --> 99:59:59.999
You can get a pluggable unified notification system that can gather all the events in the project and send them by email, by IRC,

99:59:59.999 --> 99:59:59.999
on your mobile phone, on your desktop, everywhere you want.

99:59:59.999 --> 99:59:59.999
Fedora people use fedmsg to implement a badge system

99:59:59.999 --> 99:59:59.999
which is some kind of gamification of the development process of the distribution.

99:59:59.999 --> 99:59:59.999
They implemented a live web dashboard.

99:59:59.999 --> 99:59:59.999
They implemented IRC feed.

99:59:59.999 --> 99:59:59.999
And then they also got some bot bans on social networks because they were flooding.

99:59:59.999 --> 99:59:59.999
How does it work?

99:59:59.999 --> 99:59:59.999
Well, the first idea was to use AMQP as implemented by qpid.

99:59:59.999 --> 99:59:59.999
Basically, you take all your services and you have them send their messages in a central broker.

99:59:59.999 --> 99:59:59.999
Then you have several listeners that can send messages to clients.

99:59:59.999 --> 99:59:59.999
There were a few issues with this.

99:59:59.999 --> 99:59:59.999
Basically, you have a single point of failure at the central broker.

99:59:59.999 --> 99:59:59.999
And the brokers weren't really reliable.

99:59:59.999 --> 99:59:59.999
When they tested it under load, the brokers were tipping over.

99:59:59.999 --> 99:59:59.999
The actual implementation of fedmsg uses 0mq.

99:59:59.999 --> 99:59:59.999
Basically what you get is not a single broker.

99:59:59.999 --> 99:59:59.999
You get a mesh of interconnected services.

99:59:59.999 --> 99:59:59.999
Basically, you can connect only to the services that you want to listen to.

99:59:59.999 --> 99:59:59.999
The big drawback of this is that each and every service has to open up a port on the public Internet

99:59:59.999 --> 99:59:59.999
for people to be able to connect to it.

99:59:59.999 --> 99:59:59.999
There are some solutions for that which I will talk about.

99:59:59.999 --> 99:59:59.999
But the main advantage is that you have no central broker

99:59:59.999 --> 99:59:59.999
and they got like a hundred-fold speedup over the previous implementation.

99:59:59.999 --> 99:59:59.999
You also have an issue with service discovery.

99:59:59.999 --> 99:59:59.999
You can write a broker which gives you back your single point of failure.

99:59:59.999 --> 99:59:59.999
You can use DNS which means that can say "Hey I added a new service, let's use this SRV record to get to it"

99:59:59.999 --> 99:59:59.999
Or you can distribute a text file.

99:59:59.999 --> 99:59:59.999
Last year, during the Google Summer of Code, I mentored Simon Choppin

99:59:59.999 --> 99:59:59.999
...who implemented the DNS solution for integration in fedmsg in Debian.

99:59:59.999 --> 99:59:59.999
The Fedora people as they control their whole infrastructure just distribute a text file

99:59:59.999 --> 99:59:59.999
...with the list of servers that are sending fedmsg messages.

99:59:59.999 --> 99:59:59.999
How do you use it?

99:59:59.999 --> 99:59:59.999
This is the Fedora topology.

99:59:59.999 --> 99:59:59.999
I didn't have much time to do the Debian one.

99:59:59.999 --> 99:59:59.999
It's really simpler. I'll talk about it later.

99:59:59.999 --> 99:59:59.999
Basically, the messages are split in topics where you have a hierarchy of topics.

99:59:59.999 --> 99:59:59.999
It's really easy to filter out the things that you want to listen to.

99:59:59.999 --> 99:59:59.999
For instance, you can filter all the messages that concern package upload by using the dak service.

99:59:59.999 --> 99:59:59.999
Or everything that involves a given package or something else.

99:59:59.999 --> 99:59:59.999
Publishing messages is really trivial.

99:59:59.999 --> 99:59:59.999
From Python, you only have to import the module,

99:59:59.999 --> 99:59:59.999
do fedmsg.publish with a dict of the data that you want to send.

99:59:59.999 --> 99:59:59.999
And that's it, your message is published.

99:59:59.999 --> 99:59:59.999
From the shell, it's really easy too.

99:59:59.999 --> 99:59:59.999
You just have a command called fedmsg-logger that you can pipe some input to.

99:59:59.999 --> 99:59:59.999
And it goes on the bus, so it's really simple.

99:59:59.999 --> 99:59:59.999
Receiving messages is trivial too.

99:59:59.999 --> 99:59:59.999
In Python, you load the configuration

99:59:59.999 --> 99:59:59.999
...and you just have an iterator

99:59:59.999 --> 99:59:59.999
[audio stops]

99:59:59.999 --> 99:59:59.999
was a replay mechanism with just a sequence number

99:59:59.999 --> 99:59:59.999
which will have your client query the event sender for new messages that you would have missed

99:59:59.999 --> 99:59:59.999
...in case of a network failure or anything.

99:59:59.999 --> 99:59:59.999
That's how basically the system works.

99:59:59.999 --> 99:59:59.999
Now, what about fedmsg in Debian?

99:59:59.999 --> 99:59:59.999
During the last Google Summer of code, a lot happened thanks to Simon Chopin's involvement.

99:59:59.999 --> 99:59:59.999
He did most of the packaging of fedmsg and its dependencies

99:59:59.999 --> 99:59:59.999
...which means that you can just apt-get install fedmsg and get it running.

99:59:59.999 --> 99:59:59.999
It's available in sid, jessie and wheezy-backports.

99:59:59.999 --> 99:59:59.999
He adapted the code of fedmsg to make it distribution agnostic.

99:59:59.999 --> 99:59:59.999
He had a lot of support from upstream developers in Fedora to make that happen.

99:59:59.999 --> 99:59:59.999
They are really excited to have their stuff being used by Debian or by other organizations,

99:59:59.999 --> 99:59:59.999
...that fedmsg was the right solution for event notification.

99:59:59.999 --> 99:59:59.999
And finally, we bootstrapped the Debian bus by using mailing-list subscriptions

99:59:59.999 --> 99:59:59.999
...to get bug notifications and package upload notifications

99:59:59.999 --> 99:59:59.999
...and on mentors.debian.net which is a service I can control, so it's easy to add new stuff to it.

99:59:59.999 --> 99:59:59.999
What then?

99:59:59.999 --> 99:59:59.999
After the Google Summer of Code, there was some packaging adaptations to make it easier to run services based on fedmsg,

99:59:59.999 --> 99:59:59.999
...proper backports and maintainance of the bus

99:59:59.999 --> 99:59:59.999
...which mostly means keeping the software up-to-date

99:59:59.999 --> 99:59:59.999
...because the upstream is really active and responsive to bug reports.

99:59:59.999 --> 99:59:59.999
It's really nice to work with them.

99:59:59.999 --> 99:59:59.999
Since July 14th 2013 which is the day we started sending messages on the bus,

99:59:59.999 --> 99:59:59.999
...we had around 200k messages split accross 155k bug mails and 45k uploads

99:59:59.999 --> 99:59:59.999
...which proves that Debian is a really active project, I guess.

99:59:59.999 --> 99:59:59.999
[laughs]

99:59:59.999 --> 99:59:59.999
The latest developments with fedmsg is the packaging of Datanommer

99:59:59.999 --> 99:59:59.999
...which is a database component that can store messages that has been sent to the bus.

99:59:59.999 --> 99:59:59.999
It allows Fedora to do queries on their messages

99:59:59.999 --> 99:59:59.999
...and give people the achievements that they did like "yeah, you had a hundred build failures"

99:59:59.999 --> 99:59:59.999
...or stuff like that.
[laughs]

99:59:59.999 --> 99:59:59.999
One big issue with fedmsg that I said earlier is that Debian services are widely distributed.

99:59:59.999 --> 99:59:59.999
Some of the times, firewall restrictions are out of Debian control,

99:59:59.999 --> 99:59:59.999
...which is also the case of with the Fedora infrastructure

99:59:59.999 --> 99:59:59.999
...because some of their servers are hosted within Redhat

99:59:59.999 --> 99:59:59.999
...and Redhat networking sometimes don't want to open firewall ports.

99:59:59.999 --> 99:59:59.999
So we need a way for services to push their messages instead of having clients pull the messages.

99:59:59.999 --> 99:59:59.999
There is a component in fedmsg which have been created by the Fedora people which is called fedmsg-relay

99:59:59.999 --> 99:59:59.999
...which basically is just a tube where you push your message using a 0mq socket

99:59:59.999 --> 99:59:59.999
...and it then pushes it to the subscribers on the other side.

99:59:59.999 --> 99:59:59.999
It just allows to bypass firwalls.

99:59:59.999 --> 99:59:59.999
The issue is that it uses a non-standard port and a non-standard protocol.

99:59:59.999 --> 99:59:59.999
It's just 0mq so it basically put your data on the wire and that's it.

99:59:59.999 --> 99:59:59.999
So, I am pondering a way for services to push their messages using more classic web services.

99:59:59.999 --> 99:59:59.999
You will take your JSON dictionary and push it by POST through HTTPS.

99:59:59.999 --> 99:59:59.999
And then after that send the message to the bus

99:59:59.999 --> 99:59:59.999
...which I think will make it easier to integrate with other Debian services.

99:59:59.999 --> 99:59:59.999
This was a really short talk.

99:59:59.999 --> 99:59:59.999
I hope there is some discussions afterwards.

99:59:59.999 --> 99:59:59.999
In conclusion, I am really glad it works.

99:59:59.999 --> 99:59:59.999
For the moment, it's really apart from the Debian infrastructure.

99:59:59.999 --> 99:59:59.999
So the big challenge will be to try to integrate fedmsg to Debian infrastructure

99:59:59.999 --> 99:59:59.999
...and use it for real.

99:59:59.999 --> 99:59:59.999
If you want to contact me, I am olasd,

99:59:59.999 --> 99:59:59.999
...I am here for the whole conference.

99:59:59.999 --> 99:59:59.999
If you want to talk to me about it, if you want to help me,

99:59:59.999 --> 99:59:59.999
...I am a little bit alone on this project, so I'll be glad if someone would join.

99:59:59.999 --> 99:59:59.999
I'll be glad to hold an hacking session later this week.

99:59:59.999 --> 99:59:59.999
Thanks for your attention!

99:59:59.999 --> 99:59:59.999
[applause]

99:59:59.999 --> 99:59:59.999
Was it this clear?

99:59:59.999 --> 99:59:59.999
You talked about the ??? use to publish SRV record.

99:59:59.999 --> 99:59:59.999
I missed some of the details of what that means.

99:59:59.999 --> 99:59:59.999
What is in a SRV record and how do I do discovery on it?

99:59:59.999 --> 99:59:59.999
The idea is that to actually receive messages, you need the host and the port of the sender.

99:59:59.999 --> 99:59:59.999
If you have several WSGI workers, you have several ports that you need to listen to.

99:59:59.999 --> 99:59:59.999
What we do with the SRV record is basically under the domain name of the service,

99:59:59.999 --> 99:59:59.999
...for example ftp-master.debian.org, we would have <u>fedmsg.</u>tcp.ftp-master.debian.org

99:59:59.999 --> 99:59:59.999
...which will point to the four or five workers that you would use to get the messages.

99:59:59.999 --> 99:59:59.999
So if I don't know that ftp-master.debian.org is something that I want to subscribe to as a mechanism for getting the details,

99:59:59.999 --> 99:59:59.999
...is there something which tells me that ftp-master.debian.org is a an host to begin with?

99:59:59.999 --> 99:59:59.999
No, not yet.

99:59:59.999 --> 99:59:59.999
Only part of the problem is solved.

99:59:59.999 --> 99:59:59.999
Currently there is no list of every single services that publish messages.

99:59:59.999 --> 99:59:59.999
What they do in Fedora and what we do in Debian too, for public consumption,

99:59:59.999 --> 99:59:59.999
...there is a component called the gateway which will connect to all the message sources

99:59:59.999 --> 99:59:59.999
...and rewrite the messages to send them to clients.

99:59:59.999 --> 99:59:59.999
You don't get the replay mechanism because it works only for a single source

99:59:59.999 --> 99:59:59.999
...but you solve your discovery problem but you get back the single point of failure.

99:59:59.999 --> 99:59:59.999