FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Build System

 
 
LinkBack Thread Tools
 
Old 02-26-2010, 06:15 PM
Doug Reiland
 
Default koji chain-build usage

>
> You can emulate chain-build with a series of build and wait-repo
> commands if you like.
>

I want to be able to improve build throughput.

Is the following valid approach?

koji build --skip-tag mytag srpm1 srpm2 ....
after done
koji tag-pkg mytag pkg1 pkg2 ....
koji regen-repo ...
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-26-2010, 06:27 PM
Mike Bonnet
 
Default koji chain-build usage

On 02/26/2010 02:15 PM, Doug Reiland wrote:
>>
>> You can emulate chain-build with a series of build and wait-repo
>> commands if you like.
>>
>
> I want to be able to improve build throughput.
>
> Is the following valid approach?
>
> koji build --skip-tag mytag srpm1 srpm2 ....
> after done
> koji tag-pkg mytag pkg1 pkg2 ....
> koji regen-repo ...

I think you're assuming that a repo-regen will block other builds from
taking place. That's not the case. A build will use the most recently
created valid repo for the tag. Other builds can go on while a repo is
being regenerated. When the repo regen is complete, it will be marked
as the new latest repo, and any builds submitted after that will use the
new repo instead of the old. Using --skip-tag doesn't improve
throughput at all over using chain-build or wait-repo in between
(dependent) builds.
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-26-2010, 07:14 PM
Doug Reiland
 
Default koji chain-build usage

ok, I just seem to be waiting for createrepo alot.
Making very slow progress in package builds.

On Fri, Feb 26, 2010 at 2:27 PM, Mike Bonnet <mikeb@redhat.com> wrote:
> On 02/26/2010 02:15 PM, Doug Reiland wrote:
>>>
>>> You can emulate chain-build with a series of build and wait-repo
>>> commands if you like.
>>>
>>
>> I want to be able to improve build throughput.
>>
>> Is the following valid approach?
>>
>> koji build --skip-tag mytag srpm1 srpm2 ....
>> after done
>> koji tag-pkg mytag pkg1 pkg2 ....
>> koji regen-repo ...
>
> I think you're assuming that a repo-regen will block other builds from
> taking place. *That's not the case. *A build will use the most recently
> created valid repo for the tag. *Other builds can go on while a repo is
> being regenerated. *When the repo regen is complete, it will be marked
> as the new latest repo, and any builds submitted after that will use the
> new repo instead of the old. *Using --skip-tag doesn't improve
> throughput at all over using chain-build or wait-repo in between
> (dependent) builds.
> --
> buildsys mailing list
> buildsys@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-26-2010, 07:28 PM
Mike Bonnet
 
Default koji chain-build usage

On 02/26/2010 03:14 PM, Doug Reiland wrote:
> ok, I just seem to be waiting for createrepo alot.
> Making very slow progress in package builds.

Is that because you only have one builder, and its busy processing
newRepo tasks instead of build tasks? The simple answer is to add more
builders.

The only time you should be waiting for repo regens is if your package
depends on a previously-built package. In that case chain-build or
scripting up something using wait-repo is the best option.

> On Fri, Feb 26, 2010 at 2:27 PM, Mike Bonnet <mikeb@redhat.com> wrote:
>> On 02/26/2010 02:15 PM, Doug Reiland wrote:
>>>>
>>>> You can emulate chain-build with a series of build and wait-repo
>>>> commands if you like.
>>>>
>>>
>>> I want to be able to improve build throughput.
>>>
>>> Is the following valid approach?
>>>
>>> koji build --skip-tag mytag srpm1 srpm2 ....
>>> after done
>>> koji tag-pkg mytag pkg1 pkg2 ....
>>> koji regen-repo ...
>>
>> I think you're assuming that a repo-regen will block other builds from
>> taking place. That's not the case. A build will use the most recently
>> created valid repo for the tag. Other builds can go on while a repo is
>> being regenerated. When the repo regen is complete, it will be marked
>> as the new latest repo, and any builds submitted after that will use the
>> new repo instead of the old. Using --skip-tag doesn't improve
>> throughput at all over using chain-build or wait-repo in between
>> (dependent) builds.
>> --
>> buildsys mailing list
>> buildsys@lists.fedoraproject.org
>> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>>
> --
> buildsys mailing list
> buildsys@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/buildsys

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-26-2010, 08:06 PM
Doug Reiland
 
Default koji chain-build usage

All I have is srpms.
Is at least one createrepo builder required?

If I remove my builder from createrepo channel until a bunch of builds
are done, ...

I don't under how wait-repo would work here. Or how you would script this.
Does/would it assume package dependency knowledge and just plan the
builds in a order that reduces dependencies?

I know, I am clueless.


On Fri, Feb 26, 2010 at 3:28 PM, Mike Bonnet <mikeb@redhat.com> wrote:
> On 02/26/2010 03:14 PM, Doug Reiland wrote:
>> ok, I just seem to be waiting for createrepo alot.
>> Making very slow progress in package builds.
>
> Is that because you only have one builder, and its busy processing
> newRepo tasks instead of build tasks? *The simple answer is to add more
> builders.
>
> The only time you should be waiting for repo regens is if your package
> depends on a previously-built package. *In that case chain-build or
> scripting up something using wait-repo is the best option.
>
>> On Fri, Feb 26, 2010 at 2:27 PM, Mike Bonnet <mikeb@redhat.com> wrote:
>>> On 02/26/2010 02:15 PM, Doug Reiland wrote:
>>>>>
>>>>> You can emulate chain-build with a series of build and wait-repo
>>>>> commands if you like.
>>>>>
>>>>
>>>> I want to be able to improve build throughput.
>>>>
>>>> Is the following valid approach?
>>>>
>>>> koji build --skip-tag mytag srpm1 srpm2 ....
>>>> after done
>>>> koji tag-pkg mytag pkg1 pkg2 ....
>>>> koji regen-repo ...
>>>
>>> I think you're assuming that a repo-regen will block other builds from
>>> taking place. *That's not the case. *A build will use the most recently
>>> created valid repo for the tag. *Other builds can go on while a repo is
>>> being regenerated. *When the repo regen is complete, it will be marked
>>> as the new latest repo, and any builds submitted after that will use the
>>> new repo instead of the old. *Using --skip-tag doesn't improve
>>> throughput at all over using chain-build or wait-repo in between
>>> (dependent) builds.
>>> --
>>> buildsys mailing list
>>> buildsys@lists.fedoraproject.org
>>> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>>>
>> --
>> buildsys mailing list
>> buildsys@lists.fedoraproject.org
>> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>
> --
> buildsys mailing list
> buildsys@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-26-2010, 09:14 PM
Mike Bonnet
 
Default koji chain-build usage

On 02/26/2010 04:06 PM, Doug Reiland wrote:
> All I have is srpms.
> Is at least one createrepo builder required?
>
> If I remove my builder from createrepo channel until a bunch of builds
> are done, ...

Then no createrepo tasks would be processed until all your builds are
done. Is all of this because of a lack of hardware? The easiest thing
would be to add more builders to your koji setup.

> I don't under how wait-repo would work here. Or how you would script this.
> Does/would it assume package dependency knowledge and just plan the
> builds in a order that reduces dependencies?

Yes, if package bar depends on package foo then you could do something like:

koji build my-target foo-1.0-1.src.rpm &&
koji wait-repo --target my-target --build foo-1.0-1 &&
koji build my-target bar-1.0-1.src.rpm

But you need to know the dependency order. If your packages have no
interdependencies (or those dependencies are satisfied by packages
already built or in the external repo) then you can just throw them all
at Koji at once.

> I know, I am clueless.
>
>
> On Fri, Feb 26, 2010 at 3:28 PM, Mike Bonnet <mikeb@redhat.com> wrote:
>> On 02/26/2010 03:14 PM, Doug Reiland wrote:
>>> ok, I just seem to be waiting for createrepo alot.
>>> Making very slow progress in package builds.
>>
>> Is that because you only have one builder, and its busy processing
>> newRepo tasks instead of build tasks? The simple answer is to add more
>> builders.
>>
>> The only time you should be waiting for repo regens is if your package
>> depends on a previously-built package. In that case chain-build or
>> scripting up something using wait-repo is the best option.
>>
>>> On Fri, Feb 26, 2010 at 2:27 PM, Mike Bonnet <mikeb@redhat.com> wrote:
>>>> On 02/26/2010 02:15 PM, Doug Reiland wrote:
>>>>>>
>>>>>> You can emulate chain-build with a series of build and wait-repo
>>>>>> commands if you like.
>>>>>>
>>>>>
>>>>> I want to be able to improve build throughput.
>>>>>
>>>>> Is the following valid approach?
>>>>>
>>>>> koji build --skip-tag mytag srpm1 srpm2 ....
>>>>> after done
>>>>> koji tag-pkg mytag pkg1 pkg2 ....
>>>>> koji regen-repo ...
>>>>
>>>> I think you're assuming that a repo-regen will block other builds from
>>>> taking place. That's not the case. A build will use the most recently
>>>> created valid repo for the tag. Other builds can go on while a repo is
>>>> being regenerated. When the repo regen is complete, it will be marked
>>>> as the new latest repo, and any builds submitted after that will use the
>>>> new repo instead of the old. Using --skip-tag doesn't improve
>>>> throughput at all over using chain-build or wait-repo in between
>>>> (dependent) builds.
>>>> --
>>>> buildsys mailing list
>>>> buildsys@lists.fedoraproject.org
>>>> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>>>>
>>> --
>>> buildsys mailing list
>>> buildsys@lists.fedoraproject.org
>>> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>>
>> --
>> buildsys mailing list
>> buildsys@lists.fedoraproject.org
>> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>>
> --
> buildsys mailing list
> buildsys@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/buildsys

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-26-2010, 09:56 PM
Doug Reiland
 
Default koji chain-build usage

just, lack of hardware.
I am trying to do proof of concept before go too far.

Maybe I cut loose on the server bootstrapping howto too fast.
I have a local mirror so all source rpms and rpms.
Source imports worked fine, but lots of binary rpms couldn't find a
source rpm and it looked like I made a mess.
If a srpm builds more than one binary rpm, is the import binary going to work.





On Fri, Feb 26, 2010 at 5:14 PM, Mike Bonnet <mikeb@redhat.com> wrote:
> On 02/26/2010 04:06 PM, Doug Reiland wrote:
>> All I have is srpms.
>> Is at least one createrepo builder required?
>>
>> If I remove my builder from createrepo channel until a bunch of builds
>> are done, ...
>
> Then no createrepo tasks would be processed until all your builds are
> done. *Is all of this because of a lack of hardware? *The easiest thing
> would be to add more builders to your koji setup.
>
>> I don't under how wait-repo would work here. Or how you would script this.
>> Does/would it assume package dependency knowledge and just plan the
>> builds in a order that reduces dependencies?
>
> Yes, if package bar depends on package foo then you could do something like:
>
> koji build my-target foo-1.0-1.src.rpm &&
> *koji wait-repo --target my-target --build foo-1.0-1 &&
> *koji build my-target bar-1.0-1.src.rpm
>
> But you need to know the dependency order. *If your packages have no
> interdependencies (or those dependencies are satisfied by packages
> already built or in the external repo) then you can just throw them all
> at Koji at once.
>
>> I know, I am clueless.
>>
>>
>> On Fri, Feb 26, 2010 at 3:28 PM, Mike Bonnet <mikeb@redhat.com> wrote:
>>> On 02/26/2010 03:14 PM, Doug Reiland wrote:
>>>> ok, I just seem to be waiting for createrepo alot.
>>>> Making very slow progress in package builds.
>>>
>>> Is that because you only have one builder, and its busy processing
>>> newRepo tasks instead of build tasks? *The simple answer is to add more
>>> builders.
>>>
>>> The only time you should be waiting for repo regens is if your package
>>> depends on a previously-built package. *In that case chain-build or
>>> scripting up something using wait-repo is the best option.
>>>
>>>> On Fri, Feb 26, 2010 at 2:27 PM, Mike Bonnet <mikeb@redhat.com> wrote:
>>>>> On 02/26/2010 02:15 PM, Doug Reiland wrote:
>>>>>>>
>>>>>>> You can emulate chain-build with a series of build and wait-repo
>>>>>>> commands if you like.
>>>>>>>
>>>>>>
>>>>>> I want to be able to improve build throughput.
>>>>>>
>>>>>> Is the following valid approach?
>>>>>>
>>>>>> koji build --skip-tag mytag srpm1 srpm2 ....
>>>>>> after done
>>>>>> koji tag-pkg mytag pkg1 pkg2 ....
>>>>>> koji regen-repo ...
>>>>>
>>>>> I think you're assuming that a repo-regen will block other builds from
>>>>> taking place. *That's not the case. *A build will use the most recently
>>>>> created valid repo for the tag. *Other builds can go on while a repo is
>>>>> being regenerated. *When the repo regen is complete, it will be marked
>>>>> as the new latest repo, and any builds submitted after that will use the
>>>>> new repo instead of the old. *Using --skip-tag doesn't improve
>>>>> throughput at all over using chain-build or wait-repo in between
>>>>> (dependent) builds.
>>>>> --
>>>>> buildsys mailing list
>>>>> buildsys@lists.fedoraproject.org
>>>>> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>>>>>
>>>> --
>>>> buildsys mailing list
>>>> buildsys@lists.fedoraproject.org
>>>> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>>>
>>> --
>>> buildsys mailing list
>>> buildsys@lists.fedoraproject.org
>>> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>>>
>> --
>> buildsys mailing list
>> buildsys@lists.fedoraproject.org
>> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>
> --
> buildsys mailing list
> buildsys@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 03-04-2010, 07:50 PM
Doug Reiland
 
Default koji chain-build usage

I am trying to bring another builder on line.
Would I copy the certs. from the hub over to the other system? I
generated 2 builders at that time, like the examples.
On system2, called fedora2: would I use the same koji_ca_cert.crt file

Doug

On Fri, Feb 26, 2010 at 3:28 PM, Mike Bonnet <mikeb@redhat.com> wrote:
> On 02/26/2010 03:14 PM, Doug Reiland wrote:
>> ok, I just seem to be waiting for createrepo alot.
>> Making very slow progress in package builds.
>
> Is that because you only have one builder, and its busy processing
> newRepo tasks instead of build tasks? *The simple answer is to add more
> builders.
>
> The only time you should be waiting for repo regens is if your package
> depends on a previously-built package. *In that case chain-build or
> scripting up something using wait-repo is the best option.
>
>> On Fri, Feb 26, 2010 at 2:27 PM, Mike Bonnet <mikeb@redhat.com> wrote:
>>> On 02/26/2010 02:15 PM, Doug Reiland wrote:
>>>>>
>>>>> You can emulate chain-build with a series of build and wait-repo
>>>>> commands if you like.
>>>>>
>>>>
>>>> I want to be able to improve build throughput.
>>>>
>>>> Is the following valid approach?
>>>>
>>>> koji build --skip-tag mytag srpm1 srpm2 ....
>>>> after done
>>>> koji tag-pkg mytag pkg1 pkg2 ....
>>>> koji regen-repo ...
>>>
>>> I think you're assuming that a repo-regen will block other builds from
>>> taking place. *That's not the case. *A build will use the most recently
>>> created valid repo for the tag. *Other builds can go on while a repo is
>>> being regenerated. *When the repo regen is complete, it will be marked
>>> as the new latest repo, and any builds submitted after that will use the
>>> new repo instead of the old. *Using --skip-tag doesn't improve
>>> throughput at all over using chain-build or wait-repo in between
>>> (dependent) builds.
>>> --
>>> buildsys mailing list
>>> buildsys@lists.fedoraproject.org
>>> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>>>
>> --
>> buildsys mailing list
>> buildsys@lists.fedoraproject.org
>> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>
> --
> buildsys mailing list
> buildsys@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/buildsys
>
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 

Thread Tools




All times are GMT. The time now is 07:39 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org