9:59:59.000,9:59:59.000
I am Nicolas Dandrimont.

9:59:59.000,9:59:59.000
I am going to talk to you about a year of fedmsg in Debian.

9:59:59.000,9:59:59.000
We had a problem before with infrastructure in distributions.

9:59:59.000,9:59:59.000
Services are bit like people.

9:59:59.000,9:59:59.000
There are dozen of services maintained by many people

9:59:59.000,9:59:59.000
and each of those services has its own way of communicating with the rest of the world

9:59:59.000,9:59:59.000
Meaning that if you want to spin up a new service

9:59:59.000,9:59:59.000
that needs to talk to other services in the distribution

9:59:59.000,9:59:59.000
which is basically any service you want to include

9:59:59.000,9:59:59.000
you will need to implement a bunch of communication systems

9:59:59.000,9:59:59.000
For instance, in the Debian infrastructure

9:59:59.000,9:59:59.000
we have our archive software, which is dak,

9:59:59.000,9:59:59.000
that mostly uses emails and databases to communicate.

9:59:59.000,9:59:59.000
The metadat is available in a RFC822 format with no real API.

9:59:59.000,9:59:59.000
The database is not public either.

9:59:59.000,9:59:59.000
The build queue management software, which is wanna-build,

9:59:59.000,9:59:59.000
polls a database every so often to know what needs to get built.

9:59:59.000,9:59:59.000
There is no API outside of its database

9:59:59.000,9:59:59.000
that isn't public either

9:59:59.000,9:59:59.000
Out bug tracking system, which is called debbugs,

9:59:59.000,9:59:59.000
works via email, stores its data in flat files, for now,

9:59:59.000,9:59:59.000
and exposes a read-only SOAP API.

9:59:59.000,9:59:59.000
Our source control managament pushes in the distro-provided repos on alioth

9:59:59.000,9:59:59.000
can trigger an IRC bot or some emails

9:59:59.000,9:59:59.000
but there is no real central notification mechanism.

9:59:59.000,9:59:59.000
We have some kludges that are available to overcome those issues.

9:59:59.000,9:59:59.000
We have the Ultimate Debian Database

9:59:59.000,9:59:59.000
which contains a snapshot of a lot of the databases that are underlying the Debian infrastructure

9:59:59.000,9:59:59.000
This means that every so often,

9:59:59.000,9:59:59.000
there is a cron that runs and imports data from a service here, a service there.

9:59:59.000,9:59:59.000
There is no realtime data.

9:59:59.000,9:59:59.000
It's useful for distro-wide Q&A stuff because you don't need to have realtime data

9:59:59.000,9:59:59.000
But when you want some notification for trying to build a new package or something

9:59:59.000,9:59:59.000
That doesn't work very well

9:59:59.000,9:59:59.000
and the consistency between the data sources is not guaranteed.

9:59:59.000,9:59:59.000
We have another central notification system which the package tracking system

9:59:59.000,9:59:59.000
which also is cron-triggered or email-triggered

9:59:59.000,9:59:59.000
You can update the data from the BTS using ??

9:59:59.000,9:59:59.000
But you can subscribe to email updates on a given package

9:59:59.000,9:59:59.000
But the messages are not uniform,

9:59:59.000,9:59:59.000
they can be machine parsed.

9:59:59.000,9:59:59.000
There are a few headers but they are not sufficient to know what the messages are about.

9:59:59.000,9:59:59.000
And it's still not realtime.

9:59:59.000,9:59:59.000
The Fedora people invented something that could improve stuff which is called fedmsg.

9:59:59.000,9:59:59.000
It was actually introduced in 2009.

9:59:59.000,9:59:59.000
It's an unified message bus that can reduce the coupling between different services of a distribution.

9:59:59.000,9:59:59.000
That services can subscribe to one or several message topics, register callbacks and react to events

9:59:59.000,9:59:59.000
that are triggered by all the services in the distribution.

9:59:59.000,9:59:59.000
There is a bunch of stuff that are already implemented in fedmsg.

9:59:59.000,9:59:59.000
You get a stream of data with all the activity in your infrastructure which allows you to do statistics for instance

9:59:59.000,9:59:59.000
You decouple interdepent services because you can swap one thing with another

9:59:59.000,9:59:59.000
Or just listen to the messages and start doing stuff directly without having to ?? a database or something.

9:59:59.000,9:59:59.000
You can get a pluggable unified notification system that can gather all the events in the project and send them by email, by IRC

9:59:59.000,9:59:59.000
on your mobile phone, on your desktop, everywhere you want.

9:59:59.000,9:59:59.000
Fedora people use fedmsg to implement a badge system

9:59:59.000,9:59:59.000
which is some kind of gamification of the development process of the distribution

9:59:59.000,9:59:59.000
They implemented a live web dashboard

9:59:59.000,9:59:59.000
They implemented IRC feed.

9:59:59.000,9:59:59.000
And then they als go some bot bans on social networks because they were flooding

9:59:59.000,9:59:59.000
How does it work?

9:59:59.000,9:59:59.000
Well, the first idea was to use AMQP as implemented by qpid

9:59:59.000,9:59:59.000
Basically, you take all your services and you have them send their messages in a central broker

9:59:59.000,9:59:59.000
and then you have several listeners that can send messages to clients.

9:59:59.000,9:59:59.000
There were a few issues with this.

9:59:59.000,9:59:59.000
Basically, you have a single point of failure at the central broker

9:59:59.000,9:59:59.000
And the brokers weren't really reliable.

9:59:59.000,9:59:59.000
When they tested it under load, the brokers were tipping over ??

9:59:59.000,9:59:59.000
The actual implementation of fedmsg uses 0mq.

9:59:59.000,9:59:59.000
Basically what you get is not a single broker.

9:59:59.000,9:59:59.000
You get a mesh of interconnected services.

9:59:59.000,9:59:59.000
Basically, you can connect only to the service that you want to listen to.

9:59:59.000,9:59:59.000
The big drawback of this is that each and every service has to open up a port on the public Internet

9:59:59.000,9:59:59.000
for people to be able to connect to it.

9:59:59.000,9:59:59.000
There are some solutions for that which I will talk about.

9:59:59.000,9:59:59.000
But the main advantages is that you have no central broker

9:59:59.000,9:59:59.000
And they got like a hundred-fold speedup over the previous implementation.

9:59:59.000,9:59:59.000
You also have an issue with service discovery

9:59:59.000,9:59:59.000
You can write a broker which gives you back your single point of failure

9:59:59.000,9:59:59.000
You can use DNS which means that can say "Hey I added a new service, let's use this SRV record to get to it"

9:59:59.000,9:59:59.000
Or you can distribute a text file.

9:59:59.000,9:59:59.000
Last year, during the Google Summer of Code, I mentored Simon Choping

9:59:59.000,9:59:59.000
who implemented the DNS solution for integration in fedmsg in Debian.

9:59:59.000,9:59:59.000
The Fedora people as they control their whole infrastructure just distribute a text file

9:59:59.000,9:59:59.000
with the list of servers that are sending fedmsg messages

9:59:59.000,9:59:59.000
How do you use it?

9:59:59.000,9:59:59.000
This is the Fedora topology.

9:59:59.000,9:59:59.000
I didn't have much time to do the Debian one.

9:59:59.000,9:59:59.000
It's really simpler. I'll talk about it later.

9:59:59.000,9:59:59.000
Basically, the messages are split in topics where you have a hierarchy of topics.

9:59:59.000,9:59:59.000
It's really easy to filter out the things that you want to listen to.

9:59:59.000,9:59:59.000
For instance, you can filter all the messages that concern package upload by using the dak service.

9:59:59.000,9:59:59.000
Or everything that involves a given package or something else.

9:59:59.000,9:59:59.000
Publishing messages is really trivial.

9:59:59.000,9:59:59.000
From Python, you only have to import the module,

9:59:59.000,9:59:59.000
do fedmsg.publish with a dict of the data that you want to send

9:59:59.000,9:59:59.000
And that's it, your message is published.

9:59:59.000,9:59:59.000
From the shell, it's really easy too.

9:59:59.000,9:59:59.000
You just have a command called fedmsg-logger that you can pipe some input to

9:59:59.000,9:59:59.000
And it goes on the bus, so it's really simple.

9:59:59.000,9:59:59.000
Receiving messages is trivial too.

9:59:59.000,9:59:59.000
In Python, you load the configuration

9:59:59.000,9:59:59.000
and you just have an iterator

9:59:59.000,9:59:59.000
(video problems, resume at 10:10)

9:59:59.000,9:59:59.000
was a replay mechanism with just a sequence number

9:59:59.000,9:59:59.000
which will have your client query the event senders for new messages that you would have missed

9:59:59.000,9:59:59.000
in case of a network failure ??

9:59:59.000,9:59:59.000
That's how basically the system works.

9:59:59.000,9:59:59.000
Now, what about fedmsg in Debian

9:59:59.000,9:59:59.000
During the last Google Summer of code, a lot happened thanks to Simon Chopin's involvement

9:59:59.000,9:59:59.000
He did most of the packaging of fedmsg and its dependencies

9:59:59.000,9:59:59.000
It means that you can just apt-get install fedmsg and get it running

9:59:59.000,9:59:59.000
It's available in sid, jessie and wheezy-backports

9:59:59.000,9:59:59.000
He adapted the code of fedmsg to make it distribution agnostic

9:59:59.000,9:59:59.000
So he had a lot of support from upstream developers in Fedora to make that happen

9:59:59.000,9:59:59.000
They are really excited to have their stuff being used by Debian or by other organizations

9:59:59.000,9:59:59.000
?? fedmsg was the right solution for event notification

9:59:59.000,9:59:59.000
And finally, we bootstrapped the Debian bus by using mailing-list subscriptions

9:59:59.000,9:59:59.000
to get bug notifications and package upload notifications

9:59:59.000,9:59:59.000
and on mentors.debian.net which is a service I can control, so it's easy to add new stuff to it.

9:59:59.000,9:59:59.000
What then?

9:59:59.000,9:59:59.000
After the Google Summer of Code, there was some packaging adaptations to make it easier to run services based on fedmsg,

9:59:59.000,9:59:59.000
proper backports and maintainance of the bus

9:59:59.000,9:59:59.000
Which mostly means keeping the software up-to-date

9:59:59.000,9:59:59.000
because the upstream is really active and responsive to bug reports

9:59:59.000,9:59:59.000
It's really nice to work with them

9:59:59.000,9:59:59.000
Since July 14th 2013 which is the day we started sending messages on the bus

9:59:59.000,9:59:59.000
we had around 200k messages split accross 155k bug mails and 45k uploads

9:59:59.000,9:59:59.000
which proves that Debian is a really active project, I guess

9:59:59.000,9:59:59.000
[laughs]

9:59:59.000,9:59:59.000
The latest developments with fedmsg is the packaging of Datanommer

9:59:59.000,9:59:59.000
Which is a database component that can store messages that has been sent to the bus

9:59:59.000,9:59:59.000
It allows Fedora to do queries on their messages

9:59:59.000,9:59:59.000
and give people the achievements that they did like "yeah, you got a hundred build failures"

9:59:59.000,9:59:59.000
or stuff like that [laughs]

9:59:59.000,9:59:59.000
One big issue with fedmsg that I said earlier is that Debian services are widely distributed

9:59:59.000,9:59:59.000
Some of the times, firewall restrictions are out of Debian control

9:59:59.000,9:59:59.000
which is also the case of with the Fedora infrastructure

9:59:59.000,9:59:59.000
because some of their servers are hosted within Redhat

9:59:59.000,9:59:59.000
and Redhat networking sometimes don't want to open firewall ports

9:59:59.000,9:59:59.000
So we need a way for services to push their messages instead of having clients pull the messages

9:59:59.000,9:59:59.000
There is a component in fedmsg which have been created by the Fedora people which is called fedmsg-relay

9:59:59.000,9:59:59.000
Which basically is just a tube where you push your message using a 0mq socket

9:59:59.000,9:59:59.000
and it then pushes it to the subscribers on the other side

9:59:59.000,9:59:59.000
It just allows to bypass firwalls

9:59:59.000,9:59:59.000
The issue is that it uses a non-standard port and a non-standard protocol

9:59:59.000,9:59:59.000
It's just 0mq so it basically put your data on the wire and that's it.

9:59:59.000,9:59:59.000
So, I am pondering a way for services to push their messages using more classic web services

9:59:59.000,9:59:59.000
You will take your JSON dictionary and push it by POST through HTTPS

9:59:59.000,9:59:59.000
And then after that send the message to the bus

9:59:59.000,9:59:59.000
Which I think will make it easier to integrate with other Debian services

9:59:59.000,9:59:59.000
This was a really short talk

9:59:59.000,9:59:59.000
I hope there is some discussions afterwards

9:59:59.000,9:59:59.000
In conclusion, ??

9:59:59.000,9:59:59.000
I am really glad ??

9:59:59.000,9:59:59.000
For the moment, it's really apart from the Debian infrastructure

9:59:59.000,9:59:59.000
So the big challenge will be to try to integrate fedmsg to Debian infrastructure

9:59:59.000,9:59:59.000
Use it for real

9:59:59.000,9:59:59.000
If you want to contact me, I am olasd

9:59:59.000,9:59:59.000
I am here for the whole conference

9:59:59.000,9:59:59.000
If you want to talk to me about it, if you want to help me,

9:59:59.000,9:59:59.000
I am a little bit alone on this project, so I'll be glad if someone would join

9:59:59.000,9:59:59.000
I'll be glad to hold an hacking session later this week

9:59:59.000,9:59:59.000
Thanks for your attention

9:59:59.000,9:59:59.000
[applause]

9:59:59.000,9:59:59.000
Was it this clear?