Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Fedora Infrastructure (http://www.linux-archive.org/fedora-infrastructure/)
-   -   Jenkins in the Fedora infrastructure (http://www.linux-archive.org/fedora-infrastructure/709895-jenkins-fedora-infrastructure.html)

Pierre-Yves Chibon 10-05-2012 05:00 PM

Jenkins in the Fedora infrastructure
 
Hi,

This week Seth, Toshio and I have been thinking about and playing with
Jenkins.

The current jenkins we used is administrted by Luke at:
http://jenkins.turbogears.org/
and runs on hardware which is not within the Fedora infrastructure.
This machine is:
Processor: Dual Xeon @ 2.50GHz (on a dual quad-core Xen dom0)
Memory: 1G allocated; 12G on dom0
OS: Red Hat Enterprise Linux Server 5.8
Python: python-2.4, 2.5, 2.6 and 2.7

This week had two co-occurring events:
- fedora-review did not build on this instance of jenkins due to missing
dependencies on the system
- Toshio started to port Kitchen to python3 and had no place to run his
unit-tests in an automated way.

So we thought about using our new cloud system for setting up jenkins
build nodes.

We now have two build nodes within our cloud, one running Fedora 17 and
one running EL6 (down right now as it is being rebuilt).
[http://jenkins.turbogears.org/computer/]

Where do we stand from this:
- We can create nodes on our cloud
- Seth created an Ansible routine to configure the nodes directly after
their creation [http://fpaste.org/jRX1/raw/]

So adding new nodes to a Jenkins instance becomes really easy and rather
fast.


If we want to run our own jenkins master:

This is the system I can think of:
* Configure the Jenkins master in a machine within the Fedora
infrastructure
* This master is not allowed to do build
* The master can send emails (current jenkins can not due to mail server
restrictions)
* All the builds ran in nodes in the cloud
* Nodes are reinstalled every 6 month, when there is a new version of
Fedora or when needed (via Ansible)
* Nodes can be thrown away at any time

Maintenance wise:
* Upstream provides a rpm and a repo
* the rpm is pretty much a .jar file and an init script doing java -jar
everything else is extracted the first time the app is deployed and goes
into /var/lib/jenkins
* we should be able to use mod_proxy or iptable to redirect the port
8080 (default) to 80.
* Master would have backup, but we should also be able to have an
Ansible routine to re-install it (up to jenkins' configuration)


Thoughs/Questions/Suggestions/Comments?

Regards,
Pierre
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Robyn Bergeron 10-07-2012 01:59 PM

Jenkins in the Fedora infrastructure
 
On 10/05/2012 10:00 AM, Pierre-Yves Chibon wrote:

Hi,

This week Seth, Toshio and I have been thinking about and playing with
Jenkins.

Awesome.

The current jenkins we used is administrted by Luke at:
http://jenkins.turbogears.org/
and runs on hardware which is not within the Fedora infrastructure.
This machine is:
Processor: Dual Xeon @ 2.50GHz (on a dual quad-core Xen dom0)
Memory: 1G allocated; 12G on dom0
OS: Red Hat Enterprise Linux Server 5.8
Python: python-2.4, 2.5, 2.6 and 2.7

This week had two co-occurring events:
- fedora-review did not build on this instance of jenkins due to missing
dependencies on the system
- Toshio started to port Kitchen to python3 and had no place to run his
unit-tests in an automated way.

So we thought about using our new cloud system for setting up jenkins
build nodes.

Awesome.


We now have two build nodes within our cloud, one running Fedora 17 and
one running EL6 (down right now as it is being rebuilt).
[http://jenkins.turbogears.org/computer/]

Where do we stand from this:
- We can create nodes on our cloud
- Seth created an Ansible routine to configure the nodes directly after
their creation [http://fpaste.org/jRX1/raw/]

So adding new nodes to a Jenkins instance becomes really easy and rather
fast.


If we want to run our own jenkins master:

This is the system I can think of:
* Configure the Jenkins master in a machine within the Fedora
infrastructure
* This master is not allowed to do build
* The master can send emails (current jenkins can not due to mail server
restrictions)
* All the builds ran in nodes in the cloud
* Nodes are reinstalled every 6 month, when there is a new version of
Fedora or when needed (via Ansible)
* Nodes can be thrown away at any time

Maintenance wise:
* Upstream provides a rpm and a repo
* the rpm is pretty much a .jar file and an init script doing java -jar
everything else is extracted the first time the app is deployed and goes
into /var/lib/jenkins
* we should be able to use mod_proxy or iptable to redirect the port
8080 (default) to 80.
* Master would have backup, but we should also be able to have an
Ansible routine to re-install it (up to jenkins' configuration)


Thoughs/Questions/Suggestions/Comments?

Did I mention this is awesome? I think I did.

I will disclaim that I have am now a good chunk of the way into the
Continuous Delivery book and have been thinking about how some things
might dovetail in the context of how we do things 'round these parts, so
that is likely some of the inspiration for some of the questions I have....


What are we "building"? Packages? Packages + test against multiple
versions? Build package + unit-test/run against nightly? Package + test
against multiple+"nightly"/latest, if passes then automagically
incorporate into new "nightly"? Or am I TOTALLY off base and you're
just thinking about how we build/test/redeploy our infrastructure apps?


Yes, I realize that a bunch of those are like, totally jumping the gun
and starting small is good, but curious if that is sort of the roadmap.
Or if there is a roadmap or if this is just "checking things out to see
what they might do for us."


To that end - I think it would be super cool for others observing to
know... what is the problem(s) we're trying to solve, or what is the
gain we're hoping to see? And yes - "we heard this was kind of cool so
we're just checking it out to see what it even does" is perfectly
reasonable (and, ahem, awesome). But we have all this new stuff going
on - "we haz a cloud," autoqa is continuing to evolve, new archs like
arm, etc... - and some of it could potentially solve things. So I'm
wondering... is there is a basic "things we could solve/improve" list?
Maybe this would be a cool topic for FUDCon if nothing else? :)


-robyn



Regards,
Pierre
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure



_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Garrett Holmstrom 10-07-2012 10:58 PM

Jenkins in the Fedora infrastructure
 
On 2012-10-05 10:00, Pierre-Yves Chibon wrote:

This week Seth, Toshio and I have been thinking about and playing with
Jenkins.

The current jenkins we used is administrted by Luke at:
http://jenkins.turbogears.org/
and runs on hardware which is not within the Fedora infrastructure.
This machine is:
Processor: Dual Xeon @ 2.50GHz (on a dual quad-core Xen dom0)
Memory: 1G allocated; 12G on dom0
OS: Red Hat Enterprise Linux Server 5.8
Python: python-2.4, 2.5, 2.6 and 2.7

This week had two co-occurring events:
- fedora-review did not build on this instance of jenkins due to missing
dependencies on the system
- Toshio started to port Kitchen to python3 and had no place to run his
unit-tests in an automated way.

So we thought about using our new cloud system for setting up jenkins
build nodes.

We now have two build nodes within our cloud, one running Fedora 17 and
one running EL6 (down right now as it is being rebuilt).
[http://jenkins.turbogears.org/computer/]

Where do we stand from this:
- We can create nodes on our cloud
- Seth created an Ansible routine to configure the nodes directly after
their creation [http://fpaste.org/jRX1/raw/]

So adding new nodes to a Jenkins instance becomes really easy and rather
fast.


If we want to run our own jenkins master:

This is the system I can think of:
* Configure the Jenkins master in a machine within the Fedora
infrastructure
* This master is not allowed to do build
* The master can send emails (current jenkins can not due to mail server
restrictions)
* All the builds ran in nodes in the cloud
* Nodes are reinstalled every 6 month, when there is a new version of
Fedora or when needed (via Ansible)
* Nodes can be thrown away at any time


Jenkins has an EC2 plugin that works with Eucalyptus (or at least
version 1.14 does; I haven't tested anything newer at $dayjob). You add
entries to the master's config that point it toward the right images in
the cloud, label those config entries, and add matching labels to your
Jenkins jobs. Then when it goes to run a job it automatically spins up
new instances with the right labels if there are no open slots. It will
also kill off instances that have been idle for a while.


Combine that with the Copy to Slave plugin and you've got yourself a
nice little build pipeline.

_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

David Nalley 10-07-2012 11:15 PM

Jenkins in the Fedora infrastructure
 
On Sun, Oct 7, 2012 at 6:58 PM, Garrett Holmstrom
<gholms@fedoraproject.org> wrote:
> On 2012-10-05 10:00, Pierre-Yves Chibon wrote:
>>
>> This week Seth, Toshio and I have been thinking about and playing with
>> Jenkins.
>>
>> The current jenkins we used is administrted by Luke at:
>> http://jenkins.turbogears.org/
>> and runs on hardware which is not within the Fedora infrastructure.
>> This machine is:
>> Processor: Dual Xeon @ 2.50GHz (on a dual quad-core Xen dom0)
>> Memory: 1G allocated; 12G on dom0
>> OS: Red Hat Enterprise Linux Server 5.8
>> Python: python-2.4, 2.5, 2.6 and 2.7
>>
>> This week had two co-occurring events:
>> - fedora-review did not build on this instance of jenkins due to missing
>> dependencies on the system
>> - Toshio started to port Kitchen to python3 and had no place to run his
>> unit-tests in an automated way.
>>
>> So we thought about using our new cloud system for setting up jenkins
>> build nodes.
>>
>> We now have two build nodes within our cloud, one running Fedora 17 and
>> one running EL6 (down right now as it is being rebuilt).
>> [http://jenkins.turbogears.org/computer/]
>>
>> Where do we stand from this:
>> - We can create nodes on our cloud
>> - Seth created an Ansible routine to configure the nodes directly after
>> their creation [http://fpaste.org/jRX1/raw/]
>>
>> So adding new nodes to a Jenkins instance becomes really easy and rather
>> fast.
>>
>>
>> If we want to run our own jenkins master:
>>
>> This is the system I can think of:
>> * Configure the Jenkins master in a machine within the Fedora
>> infrastructure
>> * This master is not allowed to do build
>> * The master can send emails (current jenkins can not due to mail server
>> restrictions)
>> * All the builds ran in nodes in the cloud
>> * Nodes are reinstalled every 6 month, when there is a new version of
>> Fedora or when needed (via Ansible)
>> * Nodes can be thrown away at any time
>
>
> Jenkins has an EC2 plugin that works with Eucalyptus (or at least version
> 1.14 does; I haven't tested anything newer at $dayjob). You add entries to
> the master's config that point it toward the right images in the cloud,
> label those config entries, and add matching labels to your Jenkins jobs.
> Then when it goes to run a job it automatically spins up new instances with
> the right labels if there are no open slots. It will also kill off
> instances that have been idle for a while.
>
> Combine that with the Copy to Slave plugin and you've got yourself a nice
> little build pipeline.

There's also a jclouds plugin that works with well nigh every cloud
platform known to man, and plenty of things that aren't really clouds.
The jclouds folks are almost maniacal in their efforts to keep things
working with the latest and greatest versions.

--David
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Seth Vidal 10-08-2012 03:09 PM

Jenkins in the Fedora infrastructure
 
On Sun, 7 Oct 2012, Garrett Holmstrom wrote:



Jenkins has an EC2 plugin that works with Eucalyptus (or at least version
1.14 does; I haven't tested anything newer at $dayjob). You add entries to
the master's config that point it toward the right images in the cloud, label
those config entries, and add matching labels to your Jenkins jobs. Then
when it goes to run a job it automatically spins up new instances with the
right labels if there are no open slots. It will also kill off instances
that have been idle for a while.




I think I'd rather keep us cloud-neutral for a while. That's where
the ansible playbooks come in. All ansible expects is an instance started with
an ssh key on it that it can use. That's pretty neutral ime.


-sv

_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Pierre-Yves Chibon 10-08-2012 03:38 PM

Jenkins in the Fedora infrastructure
 
On Sun, 2012-10-07 at 06:59 -0700, Robyn Bergeron wrote:
>
> What are we "building"? Packages? Packages + test against multiple
> versions? Build package + unit-test/run against nightly? Package +
> test against multiple+"nightly"/latest, if passes then automagically
> incorporate into new "nightly"? Or am I TOTALLY off base and you're
> just thinking about how we build/test/redeploy our infrastructure
> apps?

In this case, we use Jenkins for running unit-tests of project for which
we are upstream (and which a) have unit-tests b) are interested in
having a Continuous Integration approach).
At the moment projects like python-kitchen, Bodhi, FedoraReview are in
Jenkins.

I remember thoughts of doing mass-rebuild using the clouds at some
point, but I don't remember from who.

> Yes, I realize that a bunch of those are like, totally jumping the gun
> and starting small is good, but curious if that is sort of the
> roadmap.
> Or if there is a roadmap or if this is just "checking things out to
> see what they might do for us."

Regarding Jenkins, I do not think there is a clear roadmap. We had a
need for an environment in which we could test python3 and the current
Jenkins instance was going to be hard to adapt for this and we thought
the cloud infra might be a nice solution to this.

However, I am convinced all developers here agree that we could improve
on unit-tests coverage for the project we maintain.
I guess that this will be something to work on when we port our current
(web-)app to newer framework.

> To that end - I think it would be super cool for others observing to
> know... what is the problem(s) we're trying to solve, or what is the
> gain we're hoping to see? And yes - "we heard this was kind of cool so
> we're just checking it out to see what it even does" is perfectly
> reasonable (and, ahem, awesome). But we have all this new stuff going
> on - "we haz a cloud," autoqa is continuing to evolve, new archs like
> arm, etc... - and some of it could potentially solve things. So I'm
> wondering... is there is a basic "things we could solve/improve" list?

I am sure the QA guys could benefit from the fact that we have our own
cloud now.

> Maybe this would be a cool topic for FUDCon if nothing else? :)

I don't doubt it would be :)

Pierre
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

"Adam M. Dutko" 10-08-2012 09:27 PM

Jenkins in the Fedora infrastructure
 
It's awesome Fedora Infra is starting to use Jenkins. At work I use it to keep our systems scripts code base in sync across our infrastructure after pulling fresh bits from our git repositories. We also use it to build RPMs and maintain key stores. To Robyn's point, I look forward to reading how Jenkins might complement the existing Fedora build infrastructure.


On Oct 8, 2012 11:38 AM, "Pierre-Yves Chibon" <pingou@pingoured.fr> wrote:
On Sun, 2012-10-07 at 06:59 -0700, Robyn Bergeron wrote:

>

> What are we "building"? Packages? Packages + test against multiple

> versions? Build package + unit-test/run against nightly? *Package +

> test against multiple+"nightly"/latest, if passes then automagically

> incorporate into new "nightly"? *Or am I TOTALLY off base and you're

> just thinking about how we build/test/redeploy our infrastructure

> apps?



In this case, we use Jenkins for running unit-tests of project for which

we are upstream (and which a) have unit-tests b) are interested in

having a Continuous Integration approach).

At the moment projects like python-kitchen, Bodhi, FedoraReview are in

Jenkins.



I remember thoughts of doing mass-rebuild using the clouds at some

point, but I don't remember from who.



> Yes, I realize that a bunch of those are like, totally jumping the gun

> and starting small is good, but curious if that is sort of the

> roadmap.

> Or if there is a roadmap or if this is just "checking things out to

> see what they might do for us."



Regarding Jenkins, I do not think there is a clear roadmap. We had a

need for an environment in which we could test python3 and the current

Jenkins instance was going to be hard to adapt for this and we thought

the cloud infra might be a nice solution to this.



However, I am convinced all developers here agree that we could improve

on unit-tests coverage for the project we maintain.

I guess that this will be something to work on when we port our current

(web-)app to newer framework.



> To that end - I think it would be super cool for others observing to

> know... what is the problem(s) we're trying to solve, or what is the

> gain we're hoping to see? And yes - "we heard this was kind of cool so

> we're just checking it out to see what it even does" is perfectly

> reasonable (and, ahem, awesome). *But we have all this new stuff going

> on - "we haz a cloud," autoqa is continuing to evolve, new archs like

> arm, etc... - and some of it could potentially solve things. So I'm

> wondering... is there is a basic "things we could solve/improve" list?



I am sure the QA guys could benefit from the fact that we have our own

cloud now.



> *Maybe this would be a cool topic for FUDCon if nothing else? :)



I don't doubt it would be :)



Pierre

_______________________________________________

infrastructure mailing list

infrastructure@lists.fedoraproject.org

https://admin.fedoraproject.org/mailman/listinfo/infrastructure
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure


All times are GMT. The time now is 12:25 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.