FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Development

 
 
LinkBack Thread Tools
 
Old 03-05-2010, 12:08 PM
Seth Vidal
 
Default Speedup the availability of updates (was: Push scripts, mash) pushes in Bodhi (urgent call for feedback))

On Fri, 5 Mar 2010, Till Maas wrote:

> Hi,
>
> I have some ideas to speedup the availability of updates. Are there any
> reasons except that the tools to do this do not exist yet, to switch to
> this? I created a wiki page for this:
> https://fedoraproject.org/wiki/User:Till/update_availability_speedup_ideas
>
> The basic idea is to create new repos $repo-recent, that only includes
> metadata for the new rpms, but references to the same rpms on the
> filesystem that $repo uses.

the problem is you have to depsolve both sets of pkgs separately keeping
in mind stable vs unstable. And the depsolving impacts the multilib
selection (and vice versa).

-sv

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 12:19 PM
Till Maas
 
Default Speedup the availability of updates (was: Push scripts, mash) pushes in Bodhi (urgent call for feedback))

On Fri, Mar 05, 2010 at 08:08:09AM -0500, Seth Vidal wrote:
>
>
> On Fri, 5 Mar 2010, Till Maas wrote:
>
> > Hi,
> >
> > I have some ideas to speedup the availability of updates. Are there any
> > reasons except that the tools to do this do not exist yet, to switch to
> > this? I created a wiki page for this:
> > https://fedoraproject.org/wiki/User:Till/update_availability_speedup_ideas
> >
> > The basic idea is to create new repos $repo-recent, that only includes
> > metadata for the new rpms, but references to the same rpms on the
> > filesystem that $repo uses.
>
> the problem is you have to depsolve both sets of pkgs separately keeping
> in mind stable vs unstable. And the depsolving impacts the multilib
> selection (and vice versa).

I do not understand the problem, can you maybe give an example?
Does the current creation of the updates repo need information from the
Everything repo?

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 12:26 PM
Seth Vidal
 
Default Speedup the availability of updates (was: Push scripts, mash) pushes in Bodhi (urgent call for feedback))

On Fri, 5 Mar 2010, Till Maas wrote:

>>
>> the problem is you have to depsolve both sets of pkgs separately keeping
>> in mind stable vs unstable. And the depsolving impacts the multilib
>> selection (and vice versa).
>
> I do not understand the problem, can you maybe give an example?
> Does the current creation of the updates repo need information from the
> Everything repo?

If you have 20 pkgs in updates-testing and 5 of them are depend on each
other.

If only 3 of those 5 make it through updates-testing into updates, then
you have to figure out if the other 3 actually need the versions of the
other 2 or if they can work with what's already available in GA or
updates.

-sv

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 01:25 PM
Kevin Kofler
 
Default Speedup the availability of updates (was: Push scripts, mash) pushes in Bodhi (urgent call for feedback))

Seth Vidal wrote:
> If only 3 of those 5 make it through updates-testing into updates, then
> you have to figure out if the other 3 actually need the versions of the
> other 2 or if they can work with what's already available in GA or
> updates.

How's that relevant to his proposal? Or more precisely: Why would this be
any different under his proposal than now? It's the maintainer's job to push
stuff which needs to go out together in the same push (ideally as a grouped
update).

I think you're misunderstanding his proposal. It only impacts the internal
organization of the repositories and metadata, to allow for faster
mirroring. The workflow visible to packagers would stay exactly the same,
the repositories presented to the user would only get augmented by an
updates-recent (which is expected to be always enabled in most cases, often
updated, but only a very small metadata download, but which of course could
also be disabled, which would just lead to not getting updates as often, but
still always a consistent set, at least as consistent as updates are now).

Unlike your proposal, the goal here is NOT to let packages sit longer in
testing (his proposal doesn't touch this at all!), but to make mirroring
more efficient, which could also make more than 1 push/day possible, while
potentially leading to even smaller metadata downloads than now (because the
full metadata could be regenerated less often than now, though of course
there's a tradeoff there as the less we regenerate the full metadata, the
more the incremental one grows).

Kevin Kofler

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 04:23 PM
Bill Nottingham
 
Default Speedup the availability of updates (was: Push scripts, mash) pushes in Bodhi (urgent call for feedback))

Till Maas (opensource@till.name) said:
> I have some ideas to speedup the availability of updates. Are there any
> reasons except that the tools to do this do not exist yet, to switch to
> this? I created a wiki page for this:
> https://fedoraproject.org/wiki/User:Till/update_availability_speedup_ideas

It seems to be missing something - it says 'all rpms that are not included
in the prior metadata will be deleted', but there's nothing in that proposal
as written that will cause rpms to fall out of the metadata.

With respect to hardlinks, the updates repos that are created *are*
hardlinks (to the koji packages); the only copying that's involved is
pushing them from the koji storage to the mirror storage.

Bill
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 04:36 PM
Till Maas
 
Default Speedup the availability of updates (was: Push scripts, mash) pushes in Bodhi (urgent call for feedback))

On Fri, Mar 05, 2010 at 12:23:17PM -0500, Bill Nottingham wrote:
> Till Maas (opensource@till.name) said:
> > I have some ideas to speedup the availability of updates. Are there any
> > reasons except that the tools to do this do not exist yet, to switch to
> > this? I created a wiki page for this:
> > https://fedoraproject.org/wiki/User:Till/update_availability_speedup_ideas
>
> It seems to be missing something - it says 'all rpms that are not included
> in the prior metadata will be deleted', but there's nothing in that proposal
> as written that will cause rpms to fall out of the metadata.

It was probably to unclear. This metadata is generated the same way as
it happens currently, e.g. only for the latest build of a package. I
added this to the wiki.

> With respect to hardlinks, the updates repos that are created *are*
> hardlinks (to the koji packages); the only copying that's involved is
> pushing them from the koji storage to the mirror storage.

The hardlinking is targetted at the mirror storage. Afaik currently
packages vanish first from updates-testing and reappear later in
updates, so there is no hardlinking possible and the packages need to be
copied again from the master mirror to the other mirrors. There was a
mail about this recently on this list.

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 04:49 PM
Bill Nottingham
 
Default Speedup the availability of updates (was: Push scripts, mash) pushes in Bodhi (urgent call for feedback))

Till Maas (opensource@till.name) said:
> > It seems to be missing something - it says 'all rpms that are not included
> > in the prior metadata will be deleted', but there's nothing in that proposal
> > as written that will cause rpms to fall out of the metadata.
>
> It was probably to unclear. This metadata is generated the same way as
> it happens currently, e.g. only for the latest build of a package. I
> added this to the wiki.

Oh. That would require at least some code outside of createrepo, as that's
not an option for createrepo.

Bill
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 05:29 PM
Mike McGrath
 
Default Speedup the availability of updates (was: Push scripts, mash) pushes in Bodhi (urgent call for feedback))

On Fri, 5 Mar 2010, Bill Nottingham wrote:

> Till Maas (opensource@till.name) said:
> > I have some ideas to speedup the availability of updates. Are there any
> > reasons except that the tools to do this do not exist yet, to switch to
> > this? I created a wiki page for this:
> > https://fedoraproject.org/wiki/User:Till/update_availability_speedup_ideas
>
> It seems to be missing something - it says 'all rpms that are not included
> in the prior metadata will be deleted', but there's nothing in that proposal
> as written that will cause rpms to fall out of the metadata.
>
> With respect to hardlinks, the updates repos that are created *are*
> hardlinks (to the koji packages); the only copying that's involved is
> pushing them from the koji storage to the mirror storage.
>

Side note about this, it's a 10T ext3 filesystem exportetd via nfs. Does
anyone happen to know if hardlinking is particularly expensive in this
setup? We're doing profiling and things and haven't actually tested this
yet but I thought I'd ask.

-Mike
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 09:07 PM
Till Maas
 
Default Speedup the availability of updates (was: Push scripts, mash) pushes in Bodhi (urgent call for feedback))

On Fri, Mar 05, 2010 at 12:49:09PM -0500, Bill Nottingham wrote:
> Till Maas (opensource@till.name) said:
> > > It seems to be missing something - it says 'all rpms that are not included
> > > in the prior metadata will be deleted', but there's nothing in that proposal
> > > as written that will cause rpms to fall out of the metadata.
> >
> > It was probably to unclear. This metadata is generated the same way as
> > it happens currently, e.g. only for the latest build of a package. I
> > added this to the wiki.
>
> Oh. That would require at least some code outside of createrepo, as that's
> not an option for createrepo.

Just to be sure: You mean code to detect which packages have been
removed in the metadata? According to "repodiff --help", it might do the
job:
| repodiff: take 2 or more repositories and return a list of added,
| removed and changed packages.

So only the old metadata needs to be stored and then it could be
compared to the new one to get a list of packages that can be removed.
To get all packages that have been removed from updates-testing but do
not go into updates, because the update was unpushed, the list of
removed packages for the updates-testing repo could be compared with the
list of added packages for the updates repo.

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-08-2010, 11:34 PM
Jesse Keating
 
Default Speedup the availability of updates (was: Push scripts, mash) pushes in Bodhi (urgent call for feedback))

On Fri, 2010-03-05 at 13:49 +0100, Till Maas wrote:
> Hi,
>
> I have some ideas to speedup the availability of updates. Are there any
> reasons except that the tools to do this do not exist yet, to switch to
> this? I created a wiki page for this:
> https://fedoraproject.org/wiki/User:Till/update_availability_speedup_ideas
>
> The basic idea is to create new repos $repo-recent, that only includes
> metadata for the new rpms, but references to the same rpms on the
> filesystem that $repo uses.
>

Giving feedback here but you can throw it in the wiki if you want.

I do like the idea of having the rpms in a single directory. I had
toyed with the thought a while ago, just never got around to fleshing
out a proposal. When I was thinking of the mechanics of it however I
was thinking of using hardlinks into the updates-testing and whatever
other directory. This helps keep the packages with the repodata and
prevents mirroring issues if a mirror hosts updates in a different
location from development. It would also help us more easily detect
when a package is no longer in use in any of the repos.

I'm not sure I like some of the other suggestions, repodata for -recent
or adding the rpms to the mirrors as soon as the bodhi ticket is filed.
I think those would be a bit more complicated to accomplish.

Here is a walk through of what I think we could easily accomplish:
Bodhi pushes would put rpms in an updates-packages/ type directory,
hardlink specific things to updates/ and updates-testing/.
Rsync updates-packages, updates, updates-testing without --delete to
the master mirror.
Rsync updates and updates-testing with --delete to the master mirror.
Nightly branched compose would rsync with --delete and --link-dest to
updates-packages
Tail end of nightly branched compose would examine updates-packages
and look for items with no links and no pending bodhi update, or items
only linked to development/13/ and prune them.

I think this would ensure that the package stays on the filesystem at
all times as it moves from one repo into another, only deleting it once
it has gone into stable, or has been removed all together from bodhi.

This is pretty close what you have, but I think it's a bit more doable
from the infrastructure side. Not something I would want to turn on now
though, not without a lot of testing first, there are already too many
moving parts right now trying to settle into a groove.

--
Jesse Keating
Fedora -- Freedom˛ is a feature!
identi.ca: http://identi.ca/jkeating
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 

Thread Tools




All times are GMT. The time now is 06:31 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org