FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Build System

 
 
LinkBack Thread Tools
 
Old 02-10-2010, 08:24 PM
Mike McLean
 
Default createrepo run after mock

On 02/10/2010 04:08 PM, Seth Vidal wrote:
> 1. have repodata immediately available for all koji scratch builds for
> easier testing
> 2. possibly speed up assembling repos from multiple koji builds

Now this is something I'm interested in but haven't had time to poke at.
Certainly anything that speeds up repo generation is a win for koji.

I would also be interested in this from another angle -- if the main
createrepo task in koji were to merely merge a ton of 1-build repos the
we could probably eliminate the need for having /mnt/koji mounted on the
builders that handle these tasks.

> 3. it's a pretty light-weight hit to go ahead and generate this data
> 4. it means if you know the taskid then you can find another good index to
> slurp down all the files related to it.
>
> those are ones I can think of off the top of my head.
>
> I don't have a strong preference toward this being in mock - when we were
> talking about it on irc yesterday it seemed the easiest place to put it.

I'm curious at what level folks would like the repo created. By that I
mean do we:
a) make one repo for the entire build (all arches)
b) make one repo for each arch within each build

The latter is mostly (apart from src and noarch subpackages) what the
mock approach would get. However when this issue has come up in the past
I definitely remember folks asking for the former.

Granted if we have an eye towards speeding up the generation of koji's
main repos I suppose we should also add
c) make one repo for each /canonical/ arch within each build
...which is a little different.
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-10-2010, 08:27 PM
Dennis Gregorovic
 
Default createrepo run after mock

On Wed, 2010-02-10 at 16:09 -0500, Seth Vidal wrote:
>
> On Wed, 10 Feb 2010, Mike McLean wrote:
>
> > On 02/10/2010 03:41 PM, Roland McGrath wrote:
> >> I'd certainly like it to be de rigeur for all the canonical forms of
> >> producing a passel of rpms. A directory full of fresh built rpms that is
> >> not directly usable as a repo is just silly!
> >
> > Not every set of rpms makes sense as a repo.
> >
> > If you really feel this way, perhaps you should request that yum support
> > a dir of rpms as a repo.
>
> repoquery --repofrompath and the yum tmprepo plugin.
>
> -sv

I agree with Seth that there would be significant utility in having
repodata readily available for brew builds. I suggest that for regular
builds the repodata live at /data/repodata within the package dir. For
scratch builds, the repodata directory could live right inside the
task_123456 dir.

I think Mike is right that if we do add repodata creation, having it on
the hub would make the most sense.

Cheers
-- Dennis

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-10-2010, 08:39 PM
Roland McGrath
 
Default createrepo run after mock

> I guess I've never understood how this is helpful. In order to use it,
> you'd first need to configure the repo, which means creating/editing a
> config file. And you go to that effort so that yum can resolve a handful
> of inter-subpackage deps for you?

A large handful (build a gcc sometime), that you have to pick and download.
And yum should have --repofrompath like repoquery does (I thought it did,
actually). i.e.:

yum upgrade --repofrompath=test,http://koji.fedoraproject.org/scratch/roland/task_1965799

> I've just never had any trouble running yum localinstall against the
> resulting rpms.

Nor have I, after either spending some time operating my mouse to download
the right ones, and trying three different times to have it tell me I
missed one I forgot was actually a dependency of the one I needed to test,
or else spending the same 10 minutes again every time (I know, I should
just write it down) to figure out how to make wget or curl download the
whole directory full without filling my world with files called
'index.html?M=D.2' or following too many links and trying to download the
entire koji site.


Thanks,
Roland
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-10-2010, 08:48 PM
Dennis Gregorovic
 
Default createrepo run after mock

On Wed, 2010-02-10 at 16:24 -0500, Mike McLean wrote:
> On 02/10/2010 04:08 PM, Seth Vidal wrote:
> > 1. have repodata immediately available for all koji scratch builds for
> > easier testing
> > 2. possibly speed up assembling repos from multiple koji builds
>
> Now this is something I'm interested in but haven't had time to poke at.
> Certainly anything that speeds up repo generation is a win for koji.
>
> I would also be interested in this from another angle -- if the main
> createrepo task in koji were to merely merge a ton of 1-build repos the
> we could probably eliminate the need for having /mnt/koji mounted on the
> builders that handle these tasks.

Interesting indeed. Yes, I think this could work.
>
> > 3. it's a pretty light-weight hit to go ahead and generate this data
> > 4. it means if you know the taskid then you can find another good index to
> > slurp down all the files related to it.
> >
> > those are ones I can think of off the top of my head.
> >
> > I don't have a strong preference toward this being in mock - when we were
> > talking about it on irc yesterday it seemed the easiest place to put it.
>
> I'm curious at what level folks would like the repo created. By that I
> mean do we:
> a) make one repo for the entire build (all arches)
> b) make one repo for each arch within each build
>
> The latter is mostly (apart from src and noarch subpackages) what the
> mock approach would get. However when this issue has come up in the past
> I definitely remember folks asking for the former.
>
> Granted if we have an eye towards speeding up the generation of koji's
> main repos I suppose we should also add
> c) make one repo for each /canonical/ arch within each build
> ...which is a little different.

My preference would be (a). After that it be (c) then (b).

Cheers
-- Dennis

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-10-2010, 08:48 PM
Roland McGrath
 
Default createrepo run after mock

> I'm curious at what level folks would like the repo created. By that I
> mean do we:
> a) make one repo for the entire build (all arches)
> b) make one repo for each arch within each build
>
> The latter is mostly (apart from src and noarch subpackages) what the
> mock approach would get. However when this issue has come up in the past
> I definitely remember folks asking for the former.

What I want is, "I can easily install it from a scratch build."
The single repo makes this easy, but that is the only reason to care.
So if it were per-arch repos in subdirectories, that would be fine
if it's simple to have a '/$arch' on the end of a URL and it dtrt.

IOW, the innards really do not matter to the end user (me).
If some different innards have benefits for feeding into the
usual koji/newrepo/compose/whatnot stuff, rock on.

What I want is:

yum install --kojirepo=scratch/roland/task_1234567 foo
yum upgrade --kojirepo=packages/kernel/2.6.37/92.9.f22

and Just Make It Happen. Plugins, scripts, one repo, arch repos,
do what you want. Just make it easy to do the stuff that every
package maintainer winds up wanting to do all the time.


Thanks,
Roland
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-10-2010, 09:24 PM
James Antill
 
Default createrepo run after mock

On Wed, 2010-02-10 at 13:39 -0800, Roland McGrath wrote:
> > I guess I've never understood how this is helpful. In order to use it,
> > you'd first need to configure the repo, which means creating/editing a
> > config file. And you go to that effort so that yum can resolve a handful
> > of inter-subpackage deps for you?
>
> A large handful (build a gcc sometime), that you have to pick and download.
> And yum should have --repofrompath like repoquery does (I thought it did,
> actually). i.e.:
>
> yum upgrade --repofrompath=test,http://koji.fedoraproject.org/scratch/roland/task_1965799
>

Currently you have to do:

yum upgrade
--tmprepo=http://koji.fedoraproject.org/scratch/roland/task_1965799/repodata/repomd.xml

...I'm looking at updating the tmprepo plugin so you could do something
like:

yum upgrade --tmprepo=koji:roland,1965799

> > I've just never had any trouble running yum localinstall against the
> > resulting rpms.
>
> Nor have I, after either spending some time operating my mouse to download
> the right ones, and trying three different times to have it tell me I
> missed one I forgot was actually a dependency of the one I needed to test,
> or else spending the same 10 minutes again every time (I know, I should
> just write it down) to figure out how to make wget or curl download the
> whole directory full without filling my world with files called
> 'index.html?M=D.2' or following too many links and trying to download the
> entire koji site.

Note that in rawhide/3.2.26 yum you can just paste the http URLs to yum
install/localinstall. Of course having real repos. will still be much
nicer, IMNSHO.

--
James Antill - james@fedoraproject.org
http://yum.baseurl.org/wiki/releases
http://yum.baseurl.org/wiki/whatsnew/3.2.26
http://yum.baseurl.org/wiki/YumMultipleMachineCaching
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-10-2010, 09:35 PM
Clark Williams
 
Default createrepo run after mock

On Wed, 10 Feb 2010 12:41:34 -0800 (PST)
Roland McGrath <roland@redhat.com> wrote:

> > On 02/09/2010 08:56 PM, Seth Vidal wrote:
> > > This is a patch to run createrepo on the contents of the resultdir after
> > > a successful mock run. In theory this lets us suck those contents over
> > > for koji to have available.
> >
> > If the target is koji, then mock is the wrong place for this code.
> >
> > ...unless folks that use plain mock need this?
>
> I'd certainly like it to be de rigeur for all the canonical forms of
> producing a passel of rpms. A directory full of fresh built rpms that is
> not directly usable as a repo is just silly!
> --
> buildsys mailing list
> buildsys@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/buildsys

I just added Seth's patch to mock and spun up version 1.0.4. Should be
pushed to testing in the morning

Clark
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-11-2010, 07:06 AM
Mike Bonnet
 
Default createrepo run after mock

On 02/10/2010 11:39 PM, Roland McGrath wrote:
>> I guess I've never understood how this is helpful. In order to use it,
>> you'd first need to configure the repo, which means creating/editing a
>> config file. And you go to that effort so that yum can resolve a handful
>> of inter-subpackage deps for you?
>
> A large handful (build a gcc sometime), that you have to pick and download.
> And yum should have --repofrompath like repoquery does (I thought it did,
> actually). i.e.:
>
> yum upgrade --repofrompath=test,http://koji.fedoraproject.org/scratch/roland/task_1965799
>
>> I've just never had any trouble running yum localinstall against the
>> resulting rpms.
>
> Nor have I, after either spending some time operating my mouse to download
> the right ones, and trying three different times to have it tell me I
> missed one I forgot was actually a dependency of the one I needed to test,
> or else spending the same 10 minutes again every time (I know, I should
> just write it down) to figure out how to make wget or curl download the
> whole directory full without filling my world with files called
> 'index.html?M=D.2' or following too many links and trying to download the
> entire koji site.

Have you tried "koji download-build" (for official builds) or
http://people.redhat.com/mikeb/scripts/download-scratch.py (for
--scratch builds)?
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-11-2010, 07:12 AM
Mike Bonnet
 
Default createrepo run after mock

On 02/10/2010 11:27 PM, Dennis Gregorovic wrote:
> On Wed, 2010-02-10 at 16:09 -0500, Seth Vidal wrote:
>>
>> On Wed, 10 Feb 2010, Mike McLean wrote:
>>
>>> On 02/10/2010 03:41 PM, Roland McGrath wrote:
>>>> I'd certainly like it to be de rigeur for all the canonical forms of
>>>> producing a passel of rpms. A directory full of fresh built rpms that is
>>>> not directly usable as a repo is just silly!
>>>
>>> Not every set of rpms makes sense as a repo.
>>>
>>> If you really feel this way, perhaps you should request that yum support
>>> a dir of rpms as a repo.
>>
>> repoquery --repofrompath and the yum tmprepo plugin.
>>
>> -sv
>
> I agree with Seth that there would be significant utility in having
> repodata readily available for brew builds. I suggest that for regular
> builds the repodata live at /data/repodata within the package dir. For
> scratch builds, the repodata directory could live right inside the
> task_123456 dir.
>
> I think Mike is right that if we do add repodata creation, having it on
> the hub would make the most sense.

We have a Koji hub plugin system now, which makes implementing this kind
of post-build operation pretty simple. I think the last time we talked
about something like this, we were concerned about the load on the hub
from constantly running createrepos. However, I guess running a
createrepo on a handful of packages won't cause a huge increase in load,
and if it's really useful it's worth it.
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-11-2010, 04:50 PM
Dennis Gregorovic
 
Default createrepo run after mock

On Thu, 2010-02-11 at 10:12 +0200, Mike Bonnet wrote:
> On 02/10/2010 11:27 PM, Dennis Gregorovic wrote:
> > On Wed, 2010-02-10 at 16:09 -0500, Seth Vidal wrote:
> >>
> >> On Wed, 10 Feb 2010, Mike McLean wrote:
> >>
> >>> On 02/10/2010 03:41 PM, Roland McGrath wrote:
> >>>> I'd certainly like it to be de rigeur for all the canonical forms of
> >>>> producing a passel of rpms. A directory full of fresh built rpms that is
> >>>> not directly usable as a repo is just silly!
> >>>
> >>> Not every set of rpms makes sense as a repo.
> >>>
> >>> If you really feel this way, perhaps you should request that yum support
> >>> a dir of rpms as a repo.
> >>
> >> repoquery --repofrompath and the yum tmprepo plugin.
> >>
> >> -sv
> >
> > I agree with Seth that there would be significant utility in having
> > repodata readily available for brew builds. I suggest that for regular
> > builds the repodata live at /data/repodata within the package dir. For
> > scratch builds, the repodata directory could live right inside the
> > task_123456 dir.
> >
> > I think Mike is right that if we do add repodata creation, having it on
> > the hub would make the most sense.
>
> We have a Koji hub plugin system now, which makes implementing this kind
> of post-build operation pretty simple. I think the last time we talked
> about something like this, we were concerned about the load on the hub
> from constantly running createrepos. However, I guess running a
> createrepo on a handful of packages won't cause a huge increase in load,
> and if it's really useful it's worth it.

Using a plugin would be handy if we want to store the repodata separate
from the packages. Otherwise, we would need to add a mechanism to
upload the repodata back to the hub (assuming that the box running
createrepo doesn't have rw access to /mnt/koji). However, adding some
generic routines for uploading content to the /data and /scratch
directories could be useful down the line.

-- Dennis

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 

Thread Tools




All times are GMT. The time now is 06:42 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org