[Script Info]
Title: 
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,0:00:02.44,0:00:05.28,Default,,0000,0000,0000,,I am Nicolas Dandrimont.
Dialogue: 0,0:00:06.08,0:00:08.80,Default,,0000,0000,0000,,I am going to talk to you about a year of fedmsg in Debian.
Dialogue: 0,0:00:09.15,0:00:12.12,Default,,0000,0000,0000,,We had a problem before with infrastructure in distributions.
Dialogue: 0,0:00:13.84,0:00:15.76,Default,,0000,0000,0000,,All services are bit like people.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There are dozen of services maintained by many people
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and each of those services has its own way of communicating with the rest of the world
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Meaning that if you want to spin up a new service\Nthat needs to talk to other services in the distribution
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,which is basically any service you want to include
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you will need to implement a bunch of communication systems
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,For instance, in the Debian infrastructure
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,we have our archive software, which is dak,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,that mostly uses emails and databases to communicate.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The metadat is available in a RFC822 format with no real API.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The database is not public either.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The build queue management software, which is called wanna-build,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,polls a database every so often to know what needs to get built.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There is no API outside of its database
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,that isn't public either
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Our bug tracking system, which is called debbugs,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,works via email, stores its data in flat files, for now,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and exposes a read-only SOAP API.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Our source control managament pushes in the distribution-provided repositories on alioth
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,can trigger an IRC bot or some emails
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,but there is no real central notification mechanism.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We have some kludges that are available to overcome those issues.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We have the Ultimate Debian Database
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,which contains a snapshot of a lot of the databases that are underlying the Debian infrastructure
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This means that every so often,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,there is a cron that runs and imports data from a service here, a service there.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There is no realtime data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's useful for distro-wide Q&A stuff because you don't need to have realtime data
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But when you want some notification for trying to build a new package or something
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That doesn't work very well
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and the consistency between the different data sources is not guaranteed.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We have another central notification system which the package tracking system
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,which also is cron-triggered or email-triggered
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can update the data from the BTS using ??
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can subscribe to email updates on a given package
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But the messages are not uniform,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,they can be machine parsed.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There are a few headers but they are not sufficient to know what the message is about.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And it's still not realtime.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The Fedora people invented something that could improve stuff which is called fedmsg.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It was actually introduced in 2009.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's an unified message bus that can reduce the coupling between the different services in a distribution.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The idea is that services can subscribe to one or several message topics, register callbacks and react to events
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,that are triggered by all the services in the distribution.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There is a bunch of stuff that is already implemented in fedmsg.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You get a stream of data with all the activity in your infrastructure which allows you to do statistics for instance
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You decouple interdepent services because you can swap something for another
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Or just listen to the messages and start doing stuff directly without having to fiddle a database or something.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can get a pluggable unified notification system that can gather all the events in the project and send them by email, by IRC,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,on your mobile phone, on your desktop, everywhere you want.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Fedora people use fedmsg to implement a badge system
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,which is some kind of gamification of the development process of the distribution.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,They implemented a live web dashboard.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,They implemented IRC feed.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And then they also got some bot bans on social networks because they were flooding.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,How does it work?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Well, the first idea was to use AMQP as implemented by qpid.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Basically, you take all your services and you have them send their messages in a central broker.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Then you have several listeners that can send messages to clients.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There were a few issues with this.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Basically, you have a single point of failure at the central broker.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And the brokers weren't really reliable.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,When they tested it under load, the brokers were tipping over.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The actual implementation of fedmsg uses 0mq.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Basically what you get is not a single broker.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You get a mesh of interconnected services.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Basically, you can connect only to the services that you want to listen to.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The big drawback of this is that each and every service has to open up a port on the public Internet
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,for people to be able to connect to it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There are some solutions for that which I will talk about.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But the main advantage is that you have no central broker
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and they got like a hundred-fold speedup over the previous implementation.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You also have an issue with service discovery.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can write a broker which gives you back your single point of failure.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can use DNS which means that can say "Hey I added a new service, let's use this SRV record to get to it"
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Or you can distribute a text file.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Last year, during the Google Summer of Code, I mentored Simon Choppin
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...who implemented the DNS solution for integration in fedmsg in Debian.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The Fedora people as they control their whole infrastructure just distribute a text file
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...with the list of servers that are sending fedmsg messages.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,How do you use it?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is the Fedora topology.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I didn't have much time to do the Debian one.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's really simpler. I'll talk about it later.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Basically, the messages are split in topics where you have a hierarchy of topics.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's really easy to filter out the things that you want to listen to.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,For instance, you can filter all the messages that concern package upload by using the dak service.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Or everything that involves a given package or something else.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Publishing messages is really trivial.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,From Python, you only have to import the module,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,do fedmsg.publish with a dict of the data that you want to send.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And that's it, your message is published.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,From the shell, it's really easy too.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You just have a command called fedmsg-logger that you can pipe some input to.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And it goes on the bus, so it's really simple.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Receiving messages is trivial too.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,In Python, you load the configuration
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...and you just have an iterator
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,[audio stops]
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,was a replay mechanism with just a sequence number
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,which will have your client query the event sender for new messages that you would have missed
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...in case of a network failure or anything.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's how basically the system works.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, what about fedmsg in Debian?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,During the last Google Summer of code, a lot happened thanks to Simon Chopin's involvement.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,He did most of the packaging of fedmsg and its dependencies
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...which means that you can just apt-get install fedmsg and get it running.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's available in sid, jessie and wheezy-backports.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,He adapted the code of fedmsg to make it distribution agnostic.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,He had a lot of support from upstream developers in Fedora to make that happen.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,They are really excited to have their stuff being used by Debian or by other organizations,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...that fedmsg was the right solution for event notification.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And finally, we bootstrapped the Debian bus by using mailing-list subscriptions
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...to get bug notifications and package upload notifications
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...and on mentors.debian.net which is a service I can control, so it's easy to add new stuff to it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What then?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,After the Google Summer of Code, there was some packaging adaptations to make it easier to run services based on fedmsg,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...proper backports and maintainance of the bus
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...which mostly means keeping the software up-to-date
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...because the upstream is really active and responsive to bug reports.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's really nice to work with them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Since July 14th 2013 which is the day we started sending messages on the bus,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...we had around 200k messages split accross 155k bug mails and 45k uploads
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...which proves that Debian is a really active project, I guess.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,[laughs]
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The latest developments with fedmsg is the packaging of Datanommer
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...which is a database component that can store messages that has been sent to the bus.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It allows Fedora to do queries on their messages
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...and give people the achievements that they did like "yeah, you had a hundred build failures"
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...or stuff like that.\N[laughs]
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,One big issue with fedmsg that I said earlier is that Debian services are widely distributed.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Some of the times, firewall restrictions are out of Debian control,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...which is also the case of with the Fedora infrastructure
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...because some of their servers are hosted within Redhat
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...and Redhat networking sometimes don't want to open firewall ports.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So we need a way for services to push their messages instead of having clients pull the messages.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There is a component in fedmsg which have been created by the Fedora people which is called fedmsg-relay
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...which basically is just a tube where you push your message using a 0mq socket
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...and it then pushes it to the subscribers on the other side.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It just allows to bypass firwalls.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The issue is that it uses a non-standard port and a non-standard protocol.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's just 0mq so it basically put your data on the wire and that's it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, I am pondering a way for services to push their messages using more classic web services.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You will take your JSON dictionary and push it by POST through HTTPS.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And then after that send the message to the bus
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...which I think will make it easier to integrate with other Debian services.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This was a really short talk.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I hope there is some discussions afterwards.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,In conclusion, I am really glad it works.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,For the moment, it's really apart from the Debian infrastructure.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So the big challenge will be to try to integrate fedmsg to Debian infrastructure
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...and use it for real.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you want to contact me, I am olasd,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...I am here for the whole conference.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you want to talk to me about it, if you want to help me,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...I am a little bit alone on this project, so I'll be glad if someone would join.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'll be glad to hold an hacking session later this week.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Thanks for your attention!
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,[applause]
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Was it this clear?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You talked about the ??? use to publish SRV record.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I missed some of the details of what that means.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What is in a SRV record and how do I do discovery on it?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The idea is that to actually receive messages, you need the host and the port of the sender.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you have several WSGI workers, you have several ports that you need to listen to.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What we do with the SRV record is basically under the domain name of the service,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...for example ftp-master.debian.org, we would have {\u1}fedmsg.{\u0}tcp.ftp-master.debian.org
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...which will point to the four or five workers that you would use to get the messages.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So if I don't know that ftp-master.debian.org is something that I want to subscribe to as a mechanism for getting the details,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...is there something which tells me that ftp-master.debian.org is a an host to begin with?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,No, not yet.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Only part of the problem is solved.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Currently there is no list of every single services that publish messages.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What they do in Fedora and what we do in Debian too, for public consumption,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...there is a component called the gateway which will connect to all the message sources
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...and rewrite the messages to send them to clients.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You don't get the replay mechanism because it works only for a single source
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,...but you solve your discovery problem but you get back the single point of failure.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,