koji builders in failure conditions
 

Hi,
If a build job has been sent to a builder and the builder crashes (let's
say the hw dies) - what happens to that job? Does koji send it to another builder to work on?


-sv


--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

koji builders in failure conditions
 

On Wed, 11 Jul 2012, Seth Vidal wrote:


Hi,
If a build job has been sent to a builder and the builder crashes (let's say
the hw dies) - what happens to that job? Does koji send it to another
builder to work on?




I got some answers on irc - but I wanted to expand on this a bit more:

I would like to stop having to disable/enable builders in koji manually in
order to rebuild/reboot the systems.


I'd like the kojihub to check for the last time any builder checked in.

If it has not checked in in more than N minutes then it frees any task
sent to that builder and doesn't send any new tasks to it until it does
check in.


Seems like we have all the requisite information to make the hub a bit
smarter about the presence or death of a builder. Is there a significant
component of this I'm missing?



-sv

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

koji builders in failure conditions
 

On 07/11/2012 02:10 PM, Seth Vidal wrote:
>
>
>
> On Wed, 11 Jul 2012, Seth Vidal wrote:
>
>> Hi,
>> If a build job has been sent to a builder and the builder crashes
>> (let's say the hw dies) - what happens to that job? Does koji send it
>> to another builder to work on?
>>
>
> I got some answers on irc - but I wanted to expand on this a bit more:
>
> I would like to stop having to disable/enable builders in koji manually
> in order to rebuild/reboot the systems.
>
> I'd like the kojihub to check for the last time any builder checked in.
>
> If it has not checked in in more than N minutes then it frees any task
> sent to that builder and doesn't send any new tasks to it until it does
> check in.
>
> Seems like we have all the requisite information to make the hub a bit
> smarter about the presence or death of a builder. Is there a significant
> component of this I'm missing?
>
>
> -sv
>
> --
> buildsys mailing list
> buildsys@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/buildsys

I can't really do more than arm-chair design these days but I'd like to
see builders be a less of a manual effort too. It'd be cool if it could
handle one-time builders too, such as EC2 instances spun up to handle
load spikes.

- Jay
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

koji builders in failure conditions
 

On 07/11/2012 11:10 AM, Seth Vidal wrote:

On Wed, 11 Jul 2012, Seth Vidal wrote:


Hi,
If a build job has been sent to a builder and the builder crashes
(let's say the hw dies) - what happens to that job? Does koji send it
to another builder to work on?



I got some answers on irc - but I wanted to expand on this a bit more:

I would like to stop having to disable/enable builders in koji manually
in order to rebuild/reboot the systems.

I'd like the kojihub to check for the last time any builder checked in.

If it has not checked in in more than N minutes then it frees any task
sent to that builder and doesn't send any new tasks to it until it does
check in.

Seems like we have all the requisite information to make the hub a bit
smarter about the presence or death of a builder. Is there a significant
component of this I'm missing?


All task claiming/completing/freeing is handled by the builders
themselves, the hub is completely passive. So if a builder is not
checking in, it cannot claim any new tasks.


If a builder dies while running tasks, those tasks will remain claimed
by that builder until it starts checking in again (at which point, if
the daemon was restarted, they will be freed). There is no daemon that
checks for tasks owned by non-responsive builders and frees them. That
logic could be added to a daemon like kojira.

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

koji builders in failure conditions
 

On Wed, 11 Jul 2012, Jay Greguske wrote:


On 07/11/2012 02:10 PM, Seth Vidal wrote:




On Wed, 11 Jul 2012, Seth Vidal wrote:


Hi,
If a build job has been sent to a builder and the builder crashes
(let's say the hw dies) - what happens to that job? Does koji send it
to another builder to work on?



I got some answers on irc - but I wanted to expand on this a bit more:

I would like to stop having to disable/enable builders in koji manually
in order to rebuild/reboot the systems.

I'd like the kojihub to check for the last time any builder checked in.

If it has not checked in in more than N minutes then it frees any task
sent to that builder and doesn't send any new tasks to it until it does
check in.

Seems like we have all the requisite information to make the hub a bit
smarter about the presence or death of a builder. Is there a significant
component of this I'm missing?


-sv

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys


I can't really do more than arm-chair design these days but I'd like to
see builders be a less of a manual effort too. It'd be cool if it could
handle one-time builders too, such as EC2 instances spun up to handle
load spikes.


That's pretty much what I'd like to do.Though not on EC2 - but in a CLOUD.
I'd like to be able to double-duty some buildsystems between koji and

something else. So it would be good if I didn't have to mess around
with koji admin privileges when a box goes up/down.

I spoke with dgilmore on irc a bit and he said - we'd need some sort of
monitor process that watched for the builder checkins. If a builder didn't
check in normally then look for open tasks, free them then disable the
builder until it checks back in.



-sv

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

koji builders in failure conditions
 

On Wed, 11 Jul 2012, Mike Bonnet wrote:


On 07/11/2012 11:10 AM, Seth Vidal wrote:

On Wed, 11 Jul 2012, Seth Vidal wrote:


Hi,
If a build job has been sent to a builder and the builder crashes
(let's say the hw dies) - what happens to that job? Does koji send it
to another builder to work on?



I got some answers on irc - but I wanted to expand on this a bit more:

I would like to stop having to disable/enable builders in koji manually
in order to rebuild/reboot the systems.

I'd like the kojihub to check for the last time any builder checked in.

If it has not checked in in more than N minutes then it frees any task
sent to that builder and doesn't send any new tasks to it until it does
check in.

Seems like we have all the requisite information to make the hub a bit
smarter about the presence or death of a builder. Is there a significant
component of this I'm missing?


All task claiming/completing/freeing is handled by the builders themselves,
the hub is completely passive. So if a builder is not checking in, it cannot
claim any new tasks.


If a builder dies while running tasks, those tasks will remain claimed by
that builder until it starts checking in again (at which point, if the daemon
was restarted, they will be freed). There is no daemon that checks for tasks
owned by non-responsive builders and frees them. That logic could be added
to a daemon like kojira.


The use case I have the builder will come back up - later - but it will
probably be a brand new install with an old builder key/cert. So it won't
know about any of the old jobs.


I guess I don't want to have to worry about a builder checking in.

Kojira runs on the kojihub itself?

-sv

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

koji builders in failure conditions
 

On 07/11/2012 11:32 AM, Seth Vidal wrote:




On Wed, 11 Jul 2012, Mike Bonnet wrote:


On 07/11/2012 11:10 AM, Seth Vidal wrote:

On Wed, 11 Jul 2012, Seth Vidal wrote:


Hi,
If a build job has been sent to a builder and the builder crashes
(let's say the hw dies) - what happens to that job? Does koji send it
to another builder to work on?



I got some answers on irc - but I wanted to expand on this a bit more:

I would like to stop having to disable/enable builders in koji manually
in order to rebuild/reboot the systems.

I'd like the kojihub to check for the last time any builder checked in.

If it has not checked in in more than N minutes then it frees any task
sent to that builder and doesn't send any new tasks to it until it does
check in.

Seems like we have all the requisite information to make the hub a bit
smarter about the presence or death of a builder. Is there a significant
component of this I'm missing?


All task claiming/completing/freeing is handled by the builders
themselves, the hub is completely passive. So if a builder is not
checking in, it cannot claim any new tasks.

If a builder dies while running tasks, those tasks will remain claimed
by that builder until it starts checking in again (at which point, if
the daemon was restarted, they will be freed). There is no daemon
that checks for tasks owned by non-responsive builders and frees
them. That logic could be added to a daemon like kojira.


The use case I have the builder will come back up - later - but it will
probably be a brand new install with an old builder key/cert. So it
won't know about any of the old jobs.


Right, that's essentially the same as restarting a builder. The builder
process doesn't know about any old jobs, so it frees any jobs that are
assigned to it on the hub.



I guess I don't want to have to worry about a builder checking in.


I'm not sure why you would.


Kojira runs on the kojihub itself?


It can run anywhere. It's the daemon that monitors tags for changes and
triggers repo regens when necessary. It seems like a reasonable place
for this kind of recurring cleanup task. Or just run a cron job.

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

koji builders in failure conditions
 

Right, that's essentially the same as restarting a builder. The builder
process doesn't know about any old jobs, so it frees any jobs that are
assigned to it on the hub.



I guess I don't want to have to worry about a builder checking in.


I'm not sure why you would.



I have to if I ever want those jobs to be freed and there completed by
some other builder.




Kojira runs on the kojihub itself?


It can run anywhere. It's the daemon that monitors tags for changes and
triggers repo regens when necessary. It seems like a reasonable place for
this kind of recurring cleanup task. Or just run a cron job.


I'll take a look at the kojira code and see if I can figure out something
there.


-sv

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys