FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Build System

 
 
LinkBack Thread Tools
 
Old 02-26-2010, 10:43 AM
"Doug Reiland"
 
Default buildcapacity - # of builders

Does it make any sense of run more one builder on a
system or do you just control via host.capacity in the database?
*
On every package build completion does it trigger a
createrepo event?
Are my package builds serialized between
createrepos?
*
Thanks
*
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-26-2010, 10:46 AM
Jitesh Shah
 
Default buildcapacity - # of builders

On Fri, 2010-02-26 at 03:43 -0800, Doug Reiland wrote:

> On every package build completion does it trigger a createrepo event?
Every tag event triggers a regen-repo

> Are my package builds serialized between createrepos?
Nope. See "koji chain-build" to do that.

Jitesh

>
> Thanks
>

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-26-2010, 11:02 AM
Jeroen van Meeuwen
 
Default buildcapacity - # of builders

On 02/26/2010 12:43 PM, Doug Reiland wrote:
> Does it make any sense of run more one builder on a system or do you
> just control via host.capacity in the database?
>

You don't want more then one builder per node. You may want multiple
nodes on one hardware box through virtualization, but each node would
still only have one koji-builder.

You control the amount of labour the host can/may/will do through
host.capacity in the psql database, as well as the maxjobs setting in
/etc/kojid/kojid.conf (maximum 5 parallel jobs by default).

> On every package build completion does it trigger a createrepo event?

Not every package build completion triggers a newRepo task to be
executed. Only build tags in build targets need a repository to be
available within Koji itself. Should the build tag inherit the
destination tag of a successful build task, then of course a build task
completion does trigger the regeneration of the repository metadata, but
for the build tag (using/including the inherited packages and builds
from the destination tag).

Here's another scenario:

dist-f12 and dist-f12-updates are inherited by a dist-f12-build tag. The
dist-f12-updates build target uses destination tag
dist-f12-updates-candidate. So far, a successful build will not trigger
a newRepo task to be executed.

Some other system (or someone, manually) tags successful builds tagged
with tag 'dist-f12-updates-candidate' to 'dist-f12-updates', preferably
not one-by-one but per-the-bunch of course, which will cause a change to
occur in the inherited tags for the build tag - triggering a newRepo
task to be executed.

> Are my package builds serialized between createrepos?
>

I'm not sure what you mean by serialized, I'm sorry.

Kind regards,

Jeroen van Meeuwen
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-26-2010, 11:23 AM
"Doug Reiland"
 
Default buildcapacity - # of builders

Thanks for both of you for these details.
By "serialized", I mean non-parallel package builds because of the
waitrepos.

Can host.capacity be changed on the fly (pop into psql while stuff is
running)? If not, how well does the system handle a kojid stop with a bunch
of outstanding work?
/etc/init.d/kojid stop
change capacity
/etc/init.d/kojid start

and outstanding tasks are handled cleanly.

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-26-2010, 11:46 AM
Jeroen van Meeuwen
 
Default buildcapacity - # of builders

On 02/26/2010 12:46 PM, Jitesh Shah wrote:
> On Fri, 2010-02-26 at 03:43 -0800, Doug Reiland wrote:
>
>> On every package build completion does it trigger a createrepo event?
> Every tag event triggers a regen-repo
>

That depends on whether the destination tag is part of a build tag
(inherited tags included); ergo whether there is a static-repo for the
destination tag.

Most downstream koji deployments have this though, so you're probably
right 9 out of 10 times ;-)

-- Jeroen
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-26-2010, 11:49 AM
Jeroen van Meeuwen
 
Default buildcapacity - # of builders

On 02/26/2010 01:23 PM, Doug Reiland wrote:
> Thanks for both of you for these details.
> By "serialized", I mean non-parallel package builds because of the
> waitrepos.
>
> Can host.capacity be changed on the fly (pop into psql while stuff is
> running)? If not, how well does the system handle a kojid stop with a
> bunch of outstanding work?
> /etc/init.d/kojid stop
> change capacity
> /etc/init.d/kojid start
>
> and outstanding tasks are handled cleanly.

waitRepo is a special task; In my experience, if there is no version of
the repository yet, a task against a build root will automatically have
a waitRepo task (before even adding the buildArch tasks to the task queue).

You can add a waitRepo task to a build task on a per-build basis, either
through the command-line (koji wait-repo iirc), and possibly as an
option to the "koji build <options> <tag> (...)" command as well (have
no experience with such).

-- Jeroen
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-26-2010, 12:03 PM
Jeroen van Meeuwen
 
Default buildcapacity - # of builders

On 02/26/2010 01:02 PM, Jeroen van Meeuwen wrote:
> On 02/26/2010 12:43 PM, Doug Reiland wrote:
>> Does it make any sense of run more one builder on a system or do you
>> just control via host.capacity in the database?
>>
>
> You don't want more then one builder per node. You may want multiple
> nodes on one hardware box through virtualization, but each node would
> still only have one koji-builder.
>
> You control the amount of labour the host can/may/will do through
> host.capacity in the psql database, as well as the maxjobs setting in
> /etc/kojid/kojid.conf (maximum 5 parallel jobs by default).
>

Also note that new tasks won't be taken up by the koji-builder if the
load average of the node is 8.00 or greater (I don't recall having
modified that setting ever before).

-- Jeroen
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-26-2010, 01:58 PM
Mike Bonnet
 
Default buildcapacity - # of builders

On 02/26/2010 07:23 AM, Doug Reiland wrote:
> Thanks for both of you for these details.
> By "serialized", I mean non-parallel package builds because of the
> waitrepos.
>
> Can host.capacity be changed on the fly (pop into psql while stuff is
> running)?

Yes, host.capacity can be changed on the fly and kojid will handle it
just fine.

> If not, how well does the system handle a kojid stop with a bunch
> of outstanding work?
> /etc/init.d/kojid stop
> change capacity
> /etc/init.d/kojid start
>
> and outstanding tasks are handled cleanly.

If kojid shuts down cleanly it will free all the tasks it is currently
working on, to make them available for another builder to process. On
startup is also frees any tasks it owns before starting to process new
tasks. In general a kojid restart may result in running tasks being
started over, but nothing worse than that.
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-26-2010, 02:02 PM
Mike Bonnet
 
Default buildcapacity - # of builders

On 02/26/2010 08:03 AM, Jeroen van Meeuwen wrote:
> On 02/26/2010 01:02 PM, Jeroen van Meeuwen wrote:
>> On 02/26/2010 12:43 PM, Doug Reiland wrote:
>>> Does it make any sense of run more one builder on a system or do you
>>> just control via host.capacity in the database?
>>>
>>
>> You don't want more then one builder per node. You may want multiple
>> nodes on one hardware box through virtualization, but each node would
>> still only have one koji-builder.
>>
>> You control the amount of labour the host can/may/will do through
>> host.capacity in the psql database, as well as the maxjobs setting in
>> /etc/kojid/kojid.conf (maximum 5 parallel jobs by default).
>>
>
> Also note that new tasks won't be taken up by the koji-builder if the
> load average of the node is 8.00 or greater (I don't recall having
> modified that setting ever before).

The heuristic is actually load > 4 * (# of processors). It's sort of a
last-resort test so if some rogue process is running amok on a machine,
kojid there doesn't try to take tasks.
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 02-26-2010, 02:03 PM
Dennis Gilmore
 
Default buildcapacity - # of builders

On Friday 26 February 2010 07:03:33 am Jeroen van Meeuwen wrote:
> On 02/26/2010 01:02 PM, Jeroen van Meeuwen wrote:
> > On 02/26/2010 12:43 PM, Doug Reiland wrote:
> >> Does it make any sense of run more one builder on a system or do you
> >> just control via host.capacity in the database?
> >
> > You don't want more then one builder per node. You may want multiple
> > nodes on one hardware box through virtualization, but each node would
> > still only have one koji-builder.
> >
> > You control the amount of labour the host can/may/will do through
> > host.capacity in the psql database, as well as the maxjobs setting in
> > /etc/kojid/kojid.conf (maximum 5 parallel jobs by default).
>
> Also note that new tasks won't be taken up by the koji-builder if the
> load average of the node is 8.00 or greater (I don't recall having
> modified that setting ever before).

that is not true. the sparc builders have a capacity of 16 and regularly hit
max capacity. they also regullary have a load of 30-40

this is the code that limits task takeup by system load its 4x the number of
cpus so a quad core system can go to a load of 16 before it wont take on new
tasks

loadavgs = os.getloadavg()
# this likely treats HT processors the same as real ones
# but that's fine, it's a conservative test
maxload = 4.0 * os.sysconf('SC_NPROCESSORS_ONLN')
if loadavgs[0] > maxload:
self.status = "Load average %.2f > %.2f" % (loadavgs[0], maxload)
self.logger.info(self.status)
return False
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 

Thread Tools




All times are GMT. The time now is 09:48 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org