1
99:59:59,999 --> 99:59:59,999
I am Nicolas Dandrimont.

2
99:59:59,999 --> 99:59:59,999
I am going to talk to you about a year of fedmsg in Debian.

3
99:59:59,999 --> 99:59:59,999
We had a problem before with infrastructure in distributions.

4
99:59:59,999 --> 99:59:59,999
Services are bit like people.

5
99:59:59,999 --> 99:59:59,999
There are dozen of services maintained by many people

6
99:59:59,999 --> 99:59:59,999
and each of those services has its own way of communicating with the rest of the world

7
99:59:59,999 --> 99:59:59,999
Meaning that if you want to spin up a new service

8
99:59:59,999 --> 99:59:59,999
that needs to talk to other services in the distribution

9
99:59:59,999 --> 99:59:59,999
which is basically any service you want to include

10
99:59:59,999 --> 99:59:59,999
you will need to implement a bunch of communication systems

11
99:59:59,999 --> 99:59:59,999
For instance, in the Debian infrastructure

12
99:59:59,999 --> 99:59:59,999
we have our archive software, which is dak,

13
99:59:59,999 --> 99:59:59,999
that mostly uses emails and databases to communicate.

14
99:59:59,999 --> 99:59:59,999
The metadat is available in a RFC822 format with no real API.

15
99:59:59,999 --> 99:59:59,999
The database is not public either.

16
99:59:59,999 --> 99:59:59,999
The build queue management software, which is wanna-build,

17
99:59:59,999 --> 99:59:59,999
polls a database every so often to know what needs to get built.

18
99:59:59,999 --> 99:59:59,999
There is no API outside of its database

19
99:59:59,999 --> 99:59:59,999
that isn't public either

20
99:59:59,999 --> 99:59:59,999
Out bug tracking system, which is called debbugs,

21
99:59:59,999 --> 99:59:59,999
works via email, stores its data in flat files, for now,

22
99:59:59,999 --> 99:59:59,999
and exposes a read-only SOAP API.

23
99:59:59,999 --> 99:59:59,999
Our source control managament pushes in the distro-provided repos on alioth

24
99:59:59,999 --> 99:59:59,999
can trigger an IRC bot or some emails

25
99:59:59,999 --> 99:59:59,999
but there is no real central notification mechanism.

26
99:59:59,999 --> 99:59:59,999
We have some kludges that are available to overcome those issues.

27
99:59:59,999 --> 99:59:59,999
We have the Ultimate Debian Database

28
99:59:59,999 --> 99:59:59,999
which contains a snapshot of a lot of the databases that are underlying the Debian infrastructure

29
99:59:59,999 --> 99:59:59,999
This means that every so often,

30
99:59:59,999 --> 99:59:59,999
there is a cron that runs and imports data from a service here, a service there.

31
99:59:59,999 --> 99:59:59,999
There is no realtime data.

32
99:59:59,999 --> 99:59:59,999
It's useful for distro-wide Q&A stuff because you don't need to have realtime data

33
99:59:59,999 --> 99:59:59,999
But when you want some notification for trying to build a new package or something

34
99:59:59,999 --> 99:59:59,999
That doesn't work very well

35
99:59:59,999 --> 99:59:59,999
and the consistency between the data sources is not guaranteed.

36
99:59:59,999 --> 99:59:59,999
We have another central notification system which the package tracking system

37
99:59:59,999 --> 99:59:59,999
which also is cron-triggered or email-triggered

38
99:59:59,999 --> 99:59:59,999
You can update the data from the BTS using ??

39
99:59:59,999 --> 99:59:59,999
But you can subscribe to email updates on a given package

40
99:59:59,999 --> 99:59:59,999
But the messages are not uniform,

41
99:59:59,999 --> 99:59:59,999
they can be machine parsed.

42
99:59:59,999 --> 99:59:59,999
There are a few headers but they are not sufficient to know what the messages are about.

43
99:59:59,999 --> 99:59:59,999
And it's still not realtime.

44
99:59:59,999 --> 99:59:59,999
The Fedora people invented something that could improve stuff which is called fedmsg.

45
99:59:59,999 --> 99:59:59,999
It was actually introduced in 2009.

46
99:59:59,999 --> 99:59:59,999
It's an unified message bus that can reduce the coupling between different services of a distribution.

47
99:59:59,999 --> 99:59:59,999
That services can subscribe to one or several message topics, register callbacks and react to events

48
99:59:59,999 --> 99:59:59,999
that are triggered by all the services in the distribution.

49
99:59:59,999 --> 99:59:59,999
There is a bunch of stuff that are already implemented in fedmsg.

50
99:59:59,999 --> 99:59:59,999
You get a stream of data with all the activity in your infrastructure which allows you to do statistics for instance

51
99:59:59,999 --> 99:59:59,999
You decouple interdepent services because you can swap one thing with another

52
99:59:59,999 --> 99:59:59,999
Or just listen to the messages and start doing stuff directly without having to ?? a database or something.

53
99:59:59,999 --> 99:59:59,999
You can get a pluggable unified notification system that can gather all the events in the project and send them by email, by IRC

54
99:59:59,999 --> 99:59:59,999
on your mobile phone, on your desktop, everywhere you want.

55
99:59:59,999 --> 99:59:59,999
Fedora people use fedmsg to implement a badge system

56
99:59:59,999 --> 99:59:59,999
which is some kind of gamification of the development process of the distribution

57
99:59:59,999 --> 99:59:59,999
They implemented a live web dashboard

58
99:59:59,999 --> 99:59:59,999
They implemented IRC feed.

59
99:59:59,999 --> 99:59:59,999
And then they als go some bot bans on social networks because they were flooding

60
99:59:59,999 --> 99:59:59,999
How does it work?

61
99:59:59,999 --> 99:59:59,999
Well, the first idea was to use AMQP as implemented by qpid

62
99:59:59,999 --> 99:59:59,999
Basically, you take all your services and you have them send their messages in a central broker

63
99:59:59,999 --> 99:59:59,999
and then you have several listeners that can send messages to clients.

64
99:59:59,999 --> 99:59:59,999
There were a few issues with this.

65
99:59:59,999 --> 99:59:59,999
Basically, you have a single point of failure at the central broker

66
99:59:59,999 --> 99:59:59,999
And the brokers weren't really reliable.

67
99:59:59,999 --> 99:59:59,999
When they tested it under load, the brokers were tipping over ??

68
99:59:59,999 --> 99:59:59,999
The actual implementation of fedmsg uses 0mq.

69
99:59:59,999 --> 99:59:59,999
Basically what you get is not a single broker.

70
99:59:59,999 --> 99:59:59,999
You get a mesh of interconnected services.

71
99:59:59,999 --> 99:59:59,999
Basically, you can connect only to the service that you want to listen to.

72
99:59:59,999 --> 99:59:59,999
The big drawback of this is that each and every service has to open up a port on the public Internet

73
99:59:59,999 --> 99:59:59,999
for people to be able to connect to it.

74
99:59:59,999 --> 99:59:59,999
There are some solutions for that which I will talk about.

75
99:59:59,999 --> 99:59:59,999
But the main advantages is that you have no central broker

76
99:59:59,999 --> 99:59:59,999
And they got like a hundred-fold speedup over the previous implementation.

77
99:59:59,999 --> 99:59:59,999
You also have an issue with service discovery

78
99:59:59,999 --> 99:59:59,999
You can write a broker which gives you back your single point of failure

79
99:59:59,999 --> 99:59:59,999
You can use DNS which means that can say "Hey I added a new service, let's use this SRV record to get to it"

80
99:59:59,999 --> 99:59:59,999
Or you can distribute a text file.

81
99:59:59,999 --> 99:59:59,999
Last year, during the Google Summer of Code, I mentored Simon Choping

82
99:59:59,999 --> 99:59:59,999
who implemented the DNS solution for integration in fedmsg in Debian.

83
99:59:59,999 --> 99:59:59,999
The Fedora people as they control their whole infrastructure just distribute a text file

84
99:59:59,999 --> 99:59:59,999
with the list of servers that are sending fedmsg messages

85
99:59:59,999 --> 99:59:59,999
How do you use it?

86
99:59:59,999 --> 99:59:59,999
This is the Fedora topology.

87
99:59:59,999 --> 99:59:59,999
I didn't have much time to do the Debian one.

88
99:59:59,999 --> 99:59:59,999
It's really simpler. I'll talk about it later.

89
99:59:59,999 --> 99:59:59,999
Basically, the messages are split in topics where you have a hierarchy of topics.

90
99:59:59,999 --> 99:59:59,999
It's really easy to filter out the things that you want to listen to.

91
99:59:59,999 --> 99:59:59,999
For instance, you can filter all the messages that concern package upload by using the dak service.

92
99:59:59,999 --> 99:59:59,999
Or everything that involves a given package or something else.

93
99:59:59,999 --> 99:59:59,999
Publishing messages is really trivial.

94
99:59:59,999 --> 99:59:59,999
From Python, you only have to import the module,

95
99:59:59,999 --> 99:59:59,999
do fedmsg.publish with a dict of the data that you want to send

96
99:59:59,999 --> 99:59:59,999
And that's it, your message is published.

97
99:59:59,999 --> 99:59:59,999
From the shell, it's really easy too.

98
99:59:59,999 --> 99:59:59,999
You just have a command called fedmsg-logger that you can pipe some input to

99
99:59:59,999 --> 99:59:59,999
And it goes on the bus, so it's really simple.

100
99:59:59,999 --> 99:59:59,999
Receiving messages is trivial too.

101
99:59:59,999 --> 99:59:59,999
In Python, you load the configuration

102
99:59:59,999 --> 99:59:59,999
and you just have an iterator

103
99:59:59,999 --> 99:59:59,999
(video problems, resume at 10:10)

104
99:59:59,999 --> 99:59:59,999
was a replay mechanism with just a sequence number

105
99:59:59,999 --> 99:59:59,999
which will have your client query the event senders for new messages that you would have missed

106
99:59:59,999 --> 99:59:59,999
in case of a network failure ??

107
99:59:59,999 --> 99:59:59,999
That's how basically the system works.

108
99:59:59,999 --> 99:59:59,999
Now, what about fedmsg in Debian

109
99:59:59,999 --> 99:59:59,999
During the last Google Summer of code, a lot happened thanks to Simon Chopin's involvement

110
99:59:59,999 --> 99:59:59,999
He did most of the packaging of fedmsg and its dependencies

111
99:59:59,999 --> 99:59:59,999
It means that you can just apt-get install fedmsg and get it running

112
99:59:59,999 --> 99:59:59,999
It's available in sid, jessie and wheezy-backports

113
99:59:59,999 --> 99:59:59,999
He adapted the code of fedmsg to make it distribution agnostic

114
99:59:59,999 --> 99:59:59,999
So he had a lot of support from upstream developers in Fedora to make that happen

115
99:59:59,999 --> 99:59:59,999
They are really excited to have their stuff being used by Debian or by other organizations

116
99:59:59,999 --> 99:59:59,999
?? fedmsg was the right solution for event notification

117
99:59:59,999 --> 99:59:59,999
And finally, we bootstrapped the Debian bus by using mailing-list subscriptions

118
99:59:59,999 --> 99:59:59,999
to get bug notifications and package upload notifications

119
99:59:59,999 --> 99:59:59,999
and on mentors.debian.net which is a service I can control, so it's easy to add new stuff to it.

120
99:59:59,999 --> 99:59:59,999
What then?

121
99:59:59,999 --> 99:59:59,999
After the Google Summer of Code, there was some packaging adaptations to make it easier to run services based on fedmsg,

122
99:59:59,999 --> 99:59:59,999
proper backports and maintainance of the bus

123
99:59:59,999 --> 99:59:59,999
Which mostly means keeping the software up-to-date

124
99:59:59,999 --> 99:59:59,999
because the upstream is really active and responsive to bug reports

125
99:59:59,999 --> 99:59:59,999
It's really nice to work with them

126
99:59:59,999 --> 99:59:59,999
Since July 14th 2013 which is the day we started sending messages on the bus

127
99:59:59,999 --> 99:59:59,999
we had around 200k messages split accross 155k bug mails and 45k uploads

128
99:59:59,999 --> 99:59:59,999
which proves that Debian is a really active project, I guess

129
99:59:59,999 --> 99:59:59,999
[laughs]

130
99:59:59,999 --> 99:59:59,999
The latest developments with fedmsg is the packaging of Datanommer

131
99:59:59,999 --> 99:59:59,999
Which is a database component that can store messages that has been sent to the bus

132
99:59:59,999 --> 99:59:59,999
It allows Fedora to do queries on their messages

133
99:59:59,999 --> 99:59:59,999
and give people the achievements that they did like "yeah, you got a hundred build failures"

134
99:59:59,999 --> 99:59:59,999
or stuff like that [laughs]

135
99:59:59,999 --> 99:59:59,999
One big issue with fedmsg that I said earlier is that Debian services are widely distributed

136
99:59:59,999 --> 99:59:59,999
Some of the times, firewall restrictions are out of Debian control

137
99:59:59,999 --> 99:59:59,999
which is also the case of with the Fedora infrastructure

138
99:59:59,999 --> 99:59:59,999
because some of their servers are hosted within Redhat

139
99:59:59,999 --> 99:59:59,999
and Redhat networking sometimes don't want to open firewall ports

140
99:59:59,999 --> 99:59:59,999
So we need a way for services to push their messages instead of having clients pull the messages

141
99:59:59,999 --> 99:59:59,999
There is a component in fedmsg which have been created by the Fedora people which is called fedmsg-relay

142
99:59:59,999 --> 99:59:59,999
Which basically is just a tube where you push your message using a 0mq socket

143
99:59:59,999 --> 99:59:59,999
and it then pushes it to the subscribers on the other side

144
99:59:59,999 --> 99:59:59,999
It just allows to bypass firwalls

145
99:59:59,999 --> 99:59:59,999
The issue is that it uses a non-standard port and a non-standard protocol

146
99:59:59,999 --> 99:59:59,999
It's just 0mq so it basically put your data on the wire and that's it.

147
99:59:59,999 --> 99:59:59,999
So, I am pondering a way for services to push their messages using more classic web services

148
99:59:59,999 --> 99:59:59,999
You will take your JSON dictionary and push it by POST through HTTPS

149
99:59:59,999 --> 99:59:59,999
And then after that send the message to the bus

150
99:59:59,999 --> 99:59:59,999
Which I think will make it easier to integrate with other Debian services

151
99:59:59,999 --> 99:59:59,999
This was a really short talk

152
99:59:59,999 --> 99:59:59,999
I hope there is some discussions afterwards

153
99:59:59,999 --> 99:59:59,999
In conclusion, ??

154
99:59:59,999 --> 99:59:59,999
I am really glad ??

155
99:59:59,999 --> 99:59:59,999
For the moment, it's really apart from the Debian infrastructure

156
99:59:59,999 --> 99:59:59,999
So the big challenge will be to try to integrate fedmsg to Debian infrastructure

157
99:59:59,999 --> 99:59:59,999
Use it for real

158
99:59:59,999 --> 99:59:59,999
If you want to contact me, I am olasd

159
99:59:59,999 --> 99:59:59,999
I am here for the whole conference

160
99:59:59,999 --> 99:59:59,999
If you want to talk to me about it, if you want to help me,

161
99:59:59,999 --> 99:59:59,999
I am a little bit alone on this project, so I'll be glad if someone would join

162
99:59:59,999 --> 99:59:59,999
I'll be glad to hold an hacking session later this week

163
99:59:59,999 --> 99:59:59,999
Thanks for your attention

164
99:59:59,999 --> 99:59:59,999
[applause]

165
99:59:59,999 --> 99:59:59,999
Was it this clear?