Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Fedora Infrastructure (http://www.linux-archive.org/fedora-infrastructure/)
-   -   announcing repository updates over Fedora Message Bus (http://www.linux-archive.org/fedora-infrastructure/702285-announcing-repository-updates-over-fedora-message-bus.html)

Kamil Paral 09-10-2012 12:56 PM

announcing repository updates over Fedora Message Bus
 
I wonder, is it possible for Fedora Infrastructure team to implement announcements of new repository pushes over Fedora Message Bus[1]?

By new repository pushes I mean updates to 'fedora', 'updates' and 'updates-testing' repositories for all Fedora releases as well as updates to 'branched' and 'rawhide' trees.

The benefits of this solution:
1. Mirrors that download from dl.fp.o could start syncing very soon after an update is pushed (after a random timeout to decrease the load).
2. Those mirrors wouldn't have to periodically query the server for updates.
3. The duration when those mirrors are outdated would be shorter.

I can just guess about other people, but in our office I believe the message bus would help a lot. Currently we query the master server every 4 hours for updates. That presents a lot of bandwidth/IO load on both sides. Using the message bug there would be no useless queries. Also it would help us stay up-to-date more often that we currently are (if 'branched' is updated an hour after our last query, there are 3 hours wasted until we run rsync again).

This solution could be extended in the future to all Fedora public mirrors, so that they could announce using our message bus when they have finished syncing and users that sync from that particular mirror would know when to start downloading from it. (E.g. if I use mirrors1.kernel.org to sync my private mirror, I would wait for their announcement on our message bus).

What do you think?

[1] join #fedora-fedmsg and stay there for a while, if you don't know what message bus is
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

09-10-2012 01:21 PM

announcing repository updates over Fedora Message Bus
 
Love the idea. Please describe how to hook up applications to receive such notifications. I'd love to have MirrorManager's update-master-directory-list cronjob listen for such instead.

--
Matt Domsch
Technology Strategist
Dell | Office of the CTO
________________________________________
From: infrastructure-bounces@lists.fedoraproject.org [infrastructure-bounces@lists.fedoraproject.org] On Behalf Of Kamil Paral [kparal@redhat.com]
Sent: Monday, September 10, 2012 7:56 AM
To: infrastructure@lists.fedoraproject.org
Subject: announcing repository updates over Fedora Message Bus

I wonder, is it possible for Fedora Infrastructure team to implement announcements of new repository pushes over Fedora Message Bus[1]?

By new repository pushes I mean updates to 'fedora', 'updates' and 'updates-testing' repositories for all Fedora releases as well as updates to 'branched' and 'rawhide' trees.

The benefits of this solution:
1. Mirrors that download from dl.fp.o could start syncing very soon after an update is pushed (after a random timeout to decrease the load).
2. Those mirrors wouldn't have to periodically query the server for updates.
3. The duration when those mirrors are outdated would be shorter.

I can just guess about other people, but in our office I believe the message bus would help a lot. Currently we query the master server every 4 hours for updates. That presents a lot of bandwidth/IO load on both sides. Using the message bug there would be no useless queries. Also it would help us stay up-to-date more often that we currently are (if 'branched' is updated an hour after our last query, there are 3 hours wasted until we run rsync again).

This solution could be extended in the future to all Fedora public mirrors, so that they could announce using our message bus when they have finished syncing and users that sync from that particular mirror would know when to start downloading from it. (E.g. if I use mirrors1.kernel.org to sync my private mirror, I would wait for their announcement on our message bus).

What do you think?

[1] join #fedora-fedmsg and stay there for a while, if you don't know what message bus is
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Kamil Paral 09-10-2012 02:14 PM

announcing repository updates over Fedora Message Bus
 
> Love the idea. Please describe how to hook up applications to
> receive such notifications. I'd love to have MirrorManager's
> update-master-directory-list cronjob listen for such instead.

I don't have any hands-on experience with Fedora Message Bus. But we can ask the Messaging SIG guys [1]. On their mindmap they even have MirrorManager mentioned as one of their use cases.

My naive idea is that there would be a simple script (Python or whatever the qpid library has bindings for) that would connect to the bus and wait for the user-defined event, then trigger a user-defined command. We could distribute the script as part of the MirrorManager toolkit. I don't have any MirrorManager experience either, so maybe this could be easily integrated into the existing set of scripts.

This is basically why I propose this, to learn more about the topic and see whether there is someone willing to go ahead and try to implement this.


[1] http://fedoraproject.org/wiki/Messaging_SIG
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Brennan Ashton 09-10-2012 02:26 PM

announcing repository updates over Fedora Message Bus
 
On Sep 10, 2012, at 7:15, Kamil Paral <kparal@redhat.com> wrote:

>> Love the idea. Please describe how to hook up applications to
>> receive such notifications. I'd love to have MirrorManager's
>> update-master-directory-list cronjob listen for such instead.
>
> I don't have any hands-on experience with Fedora Message Bus. But we can ask the Messaging SIG guys [1]. On their mindmap they even have MirrorManager mentioned as one of their use cases.
>
> My naive idea is that there would be a simple script (Python or whatever the qpid library has bindings for) that would connect to the bus and wait for the user-defined event, then trigger a user-defined command. We could distribute the script as part of the MirrorManager toolkit. I don't have any MirrorManager experience either, so maybe this could be easily integrated into the existing set of scripts.
>
> This is basically why I propose this, to learn more about the topic and see whether there is someone willing to go ahead and try to implement this.
>
>
> [1] http://fedoraproject.org/wiki/Messaging_SIG
> _______________________________________________
> infrastructure mailing list
> infrastructure@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/infrastructure

I wrote some scripts to do similar functionality a few years ago when
we were talking AMQP. Users could subscribe to packages and get gnome
notifications when bohdi or koji events took place. I also actually
made that mindmap at the Toronto fudcon a few years back. Maybe I can
look into this again in the next week when I get back from vacation.

Brennan Ashton
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Ralph Bean 09-10-2012 03:30 PM

announcing repository updates over Fedora Message Bus
 
On Mon, Sep 10, 2012 at 08:56:26AM -0400, Kamil Paral wrote:
> I wonder, is it possible for Fedora Infrastructure team to implement announcements of new repository pushes over Fedora Message Bus[1]?
>
> By new repository pushes I mean updates to 'fedora', 'updates' and 'updates-testing' repositories for all Fedora releases as well as updates to 'branched' and 'rawhide' trees.

Some of this should already be happening. There is a
"org.fedoraproject.prod.bodhi.mashtask.sync.do ne" message that should
be getting emitted, but we don't have a way to check the history and
verify yet. https://bugzilla.redhat.com/show_bug.cgi?id=853252 should
take care of that.

"branched" and "rawhide" are another story. We'd need to add
"fedmsg-logger" statements to some of the scripts in
http://git.fedorahosted.org/cgit/releng/tree/scripts, but I'm not sure
where yet.

> The benefits of this solution:
> 1. Mirrors that download from dl.fp.o could start syncing very soon after an update is pushed (after a random timeout to decrease the load).
> 2. Those mirrors wouldn't have to periodically query the server for updates.
> 3. The duration when those mirrors are outdated would be shorter.
>
> I can just guess about other people, but in our office I believe the message bus would help a lot. Currently we query the master server every 4 hours for updates. That presents a lot of bandwidth/IO load on both sides. Using the message bug there would be no useless queries. Also it would help us stay up-to-date more often that we currently are (if 'branched' is updated an hour after our last query, there are 3 hours wasted until we run rsync again).
>
> This solution could be extended in the future to all Fedora public mirrors, so that they could announce using our message bus when they have finished syncing and users that sync from that particular mirror would know when to start downloading from it. (E.g. if I use mirrors1.kernel.org to sync my private mirror, I would wait for their announcement on our message bus).
>
> What do you think?

This sounds great for the future. I've been working on docs this week
(not quite done yet). I'll have docs out soon with examples on how to
do something like this.
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Kamil Paral 09-10-2012 04:07 PM

announcing repository updates over Fedora Message Bus
 
> > What do you think?
>
> This sounds great for the future. I've been working on docs this
> week
> (not quite done yet). I'll have docs out soon with examples on how
> to
> do something like this.

Exciting! Please don't forget to share on planet.fp.o once you have something. This is great stuff.
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Ralph Bean 09-11-2012 06:44 PM

announcing repository updates over Fedora Message Bus
 
On Mon, Sep 10, 2012 at 12:07:12PM -0400, Kamil Paral wrote:
> Exciting! Please don't forget to share on planet.fp.o once you have something. This is great stuff.

Updated docs:

- http://fedmsg.rtfd.org/
- http://fedmsg.readthedocs.org/en/latest/consuming.html

I'm going to hold off sharing on the planet until we can get a port
open for external message consumption of the production bus. Right
now, only staging is available (with practically no activity).

After the freeze, hopefully we can get that in place. An announcement
pointing folks to the docs will be more timely then. :)
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Ralph Bean 09-12-2012 03:38 PM

announcing repository updates over Fedora Message Bus
 
On Mon, Sep 10, 2012 at 11:30:15AM -0400, Ralph Bean wrote:
> "branched" and "rawhide" are another story. We'd need to add
> "fedmsg-logger" statements to some of the scripts in
> http://git.fedorahosted.org/cgit/releng/tree/scripts, but I'm not sure
> where yet.

Just an update:

I talked with dgilmore about this today and he'd like to wait until a
"composedb" app is written so we can have all the fedmsg statements in
a nice, centralized place (not strewn out across all the releng
scripts).
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Luke Macken 09-12-2012 04:14 PM

announcing repository updates over Fedora Message Bus
 
On Wed, Sep 12, 2012 at 11:38:50AM -0400, Ralph Bean wrote:
> On Mon, Sep 10, 2012 at 11:30:15AM -0400, Ralph Bean wrote:
> > "branched" and "rawhide" are another story. We'd need to add
> > "fedmsg-logger" statements to some of the scripts in
> > http://git.fedorahosted.org/cgit/releng/tree/scripts, but I'm not sure
> > where yet.
>
> Just an update:
>
> I talked with dgilmore about this today and he'd like to wait until a
> "composedb" app is written so we can have all the fedmsg statements in
> a nice, centralized place (not strewn out across all the releng
> scripts).

How far along is the composedb? Considering the patch to enable fedmsg
for rawhide/branched mashes is a few trivial 1-liners[0], it seems like
a quick win to just apply/deploy them. The transition to the composedb
will be transparent to the message consumers, as the topics won't
change.

Ralph currently doesn't have privs to commit/test them, but I'd be happy
to help.

luke

[0]: http://threebean.org/patches/0001-Add-fedmsg-logger-statements-to-compose-scripts.patch
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure


All times are GMT. The time now is 09:27 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.