FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS Development

 
 
LinkBack Thread Tools
 
Old 12-29-2011, 03:22 PM
夜神 岩男
 
Default custom iso spin

On 12/30/2011 01:02 AM, Larry Brigman wrote:
> That sounds good. I have built ISO's with upstream packages that
> weren't available
> even from EPEL.

This is part of what we're doing in my situation. There are a lot of
custom ERP/office automation, CAD, etc things that no distros include
just yet, *and* the customer uses the systems for showroom stuff so
rebranding (at least surface stuff that's readily visible) is an issue
as well. There is also the matter of our Fluendo codecs and media things
that can't go into a community distro because they have to get paid for
somewhere along the way, and the applications that are being ported to
PostgreSQL 9.1 from evil-empire type software...

Some of this goes beyond what ks can do alone, and having clean isos
that the users can carry for demos and things go a long way to reducing
confusion and manpower needs (sales != devepment types...).

> I take it the tools are in anaconda-runtime or their dependency set?
>
>
> > Is there some documentation on building a custom iso spin?
> > I would also like to be able to drive the tools.
>
> What does that mean 'drive the tools' ?
>
>
> I have used buildinstall from anaconda-runtime package but trying
> my same recipe that I used from Centos 5 doesn't work. So I wonder,
> if I was giving the correct input to the tool or if the project has a
> set of
> scripts that provide that input in some manner that would better
> document the tools?

This command works so long as the ../Packages/ directory actually has
all the packages necessary for the system to work correctly. This boots
straight to the installer, no livecd stuff, and gives you the chance to
install whatever is in Packages.

A command very similar to this is what gets fed automatically to
/usr/lib/anaconda-runtime/buildinstall by other tools.

/usr/lib/anaconda-runtime/buildinstall --version 6 --brand $COMPANY
--product $PRODUCTNAME --release "Tester" --output /home/iwao/remaster/
/home/iwao/remaster/Packages/

One quirk is you still have to do this as root. Looking for a way around
that. Using revisor you have to disable selinux as well (eh?). So there
are some weird things that I feel were never worked out properly because
the drive to "virtualize everything!" got in the way of writing for
proper system segregation at the user level. So this root/disable
SELinux thing is likely a development oversight that was never
straightened out, but may not be all that hard to fix.

I'm curious what the CentOS build infrastructure looks like. Fedora's is
quite robust/complex but works pretty well considering the things they
put it through from time to time.
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 12-29-2011, 06:22 PM
Johnny Hughes
 
Default custom iso spin

On 12/29/2011 10:22 AM, 夜神 岩男 wrote:
> On 12/30/2011 01:02 AM, Larry Brigman wrote:
>> That sounds good. I have built ISO's with upstream packages that
>> weren't available
>> even from EPEL.
>
> This is part of what we're doing in my situation. There are a lot of
> custom ERP/office automation, CAD, etc things that no distros include
> just yet, *and* the customer uses the systems for showroom stuff so
> rebranding (at least surface stuff that's readily visible) is an issue
> as well. There is also the matter of our Fluendo codecs and media things
> that can't go into a community distro because they have to get paid for
> somewhere along the way, and the applications that are being ported to
> PostgreSQL 9.1 from evil-empire type software...
>
> Some of this goes beyond what ks can do alone, and having clean isos
> that the users can carry for demos and things go a long way to reducing
> confusion and manpower needs (sales != devepment types...).
>
>> I take it the tools are in anaconda-runtime or their dependency set?
>>
>>
>> > Is there some documentation on building a custom iso spin?
>> > I would also like to be able to drive the tools.
>>
>> What does that mean 'drive the tools' ?
>>
>>
>> I have used buildinstall from anaconda-runtime package but trying
>> my same recipe that I used from Centos 5 doesn't work. So I wonder,
>> if I was giving the correct input to the tool or if the project has a
>> set of
>> scripts that provide that input in some manner that would better
>> document the tools?
>
> This command works so long as the ../Packages/ directory actually has
> all the packages necessary for the system to work correctly. This boots
> straight to the installer, no livecd stuff, and gives you the chance to
> install whatever is in Packages.
>
> A command very similar to this is what gets fed automatically to
> /usr/lib/anaconda-runtime/buildinstall by other tools.
>
> /usr/lib/anaconda-runtime/buildinstall --version 6 --brand $COMPANY
> --product $PRODUCTNAME --release "Tester" --output /home/iwao/remaster/
> /home/iwao/remaster/Packages/
>
> One quirk is you still have to do this as root. Looking for a way around
> that. Using revisor you have to disable selinux as well (eh?). So there
> are some weird things that I feel were never worked out properly because
> the drive to "virtualize everything!" got in the way of writing for
> proper system segregation at the user level. So this root/disable
> SELinux thing is likely a development oversight that was never
> straightened out, but may not be all that hard to fix.
>
> I'm curious what the CentOS build infrastructure looks like. Fedora's is
> quite robust/complex but works pretty well considering the things they
> put it through from time to time.

Well, when KB says we will add stuff, obviously he means we will add
items that are GPL or other fully recognized OSS licenses that all us to
obtain sources and properly redistribute the packages.

If it is things like the upstream supplementary disk items where one
would need some kind of special agreement to distribute, then we can't
do that.


_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 12-30-2011, 03:45 AM
Karanbir Singh
 
Default custom iso spin

On 12/29/2011 04:22 PM, 夜神 岩男 wrote:
> This is part of what we're doing in my situation. There are a lot of
> custom ERP/office automation, CAD, etc things that no distros include
> just yet, *and* the customer uses the systems for showroom stuff so
> rebranding (at least surface stuff that's readily visible) is an issue
> as well. There is also the matter of our Fluendo codecs and media things
> that can't go into a community distro because they have to get paid for
> somewhere along the way, and the applications that are being ported to
> PostgreSQL 9.1 from evil-empire type software...
>

... remember that you dont need to rebuild the installer to change the
package
manifests ... or to add custom repos on media ... you only need to
rebuild the
installer if you actually change the code inside the installer ( even
then, an updates.img is sufficient )

my talk on application hosting on CentOS from Fosdem 2008 went into some
details about this, there should be a copy of the talk online.

> I'm curious what the CentOS build infrastructure looks like. Fedora's is
> quite robust/complex but works pretty well considering the things they
> put it through from time to time.

happy to answer specific questions... our build services are fairly
robust as well ( remember over 100k builds were done leading to 6.0 );
the plague setup in place for c4/c5 has been fairly resilient as well.

- KB
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 12-30-2011, 09:53 AM
夜神 岩男
 
Default custom iso spin

On 12/30/2011 04:22 AM, Johnny Hughes wrote:
> Well, when KB says we will add stuff, obviously he means we will add
> items that are GPL or other fully recognized OSS licenses that all us to
> obtain sources and properly redistribute the packages.
>
> If it is things like the upstream supplementary disk items where one
> would need some kind of special agreement to distribute, then we can't
> do that.

Certainly! I what I was describing was the build situation where I need
a simple way to build installation ISOs. That's just one situation, and
the specific packages being assembled is quite a different problem from
the issue of finding a simple ISO creation system that can be automated
with profiles.

Maybe have different profiles inherit from a master ks or something. Not
sure. Anyway...

Cobbler actually looks a bit promising, but I've yet to explore it more
fully. It loooks like it can be simplified a lot, too -- there are some
pretty weird error message conditions in the code that don't play nicely
with SELinux and would probably confuse folks. I'm also unsure if it
supports building an installer from scratch by just pointing it at a
list of packages (which would need to include anaconda, of course). If
it could do it from nothing but a list of src.rpms that would be great.
I'll probably discover whether it can tonight, and if not what needs to
be added. A koji+cobbler that is easy to setup and doesn't require root
to run (other than initial setup) or turning selinux off would be ideal.

Some managers have problem with things like the installer displaying
branding information that isn't *their* local brand. Some customers also
get tickled by seeing their own logos and names flash by on the screen.
Technically pointless? Yes. I sort of hate all that. *but* it does sell.
Actually, if the Linux community understood why pixels and chicken
lipstick does sell so well we would be in a very different situation
today than we are now.
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 12-30-2011, 09:55 AM
夜神 岩男
 
Default custom iso spin

On 12/30/2011 01:45 PM, Karanbir Singh wrote:
> my talk on application hosting on CentOS from Fosdem 2008 went into some
> details about this, there should be a copy of the talk online.

I appreciate the reference, I'll give it a look.

> happy to answer specific questions... our build services are fairly
> robust as well ( remember over 100k builds were done leading to 6.0 );
> the plague setup in place for c4/c5 has been fairly resilient as well.

Specific question. You mentioned plague. Are you using Koji now for
build management or something else?
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 12-30-2011, 12:28 PM
Johnny Hughes
 
Default custom iso spin

On 12/30/2011 04:55 AM, 夜神 岩男 wrote:
> On 12/30/2011 01:45 PM, Karanbir Singh wrote:
>> my talk on application hosting on CentOS from Fosdem 2008 went into some
>> details about this, there should be a copy of the talk online.
>
> I appreciate the reference, I'll give it a look.
>
>> happy to answer specific questions... our build services are fairly
>> robust as well ( remember over 100k builds were done leading to 6.0 );
>> the plague setup in place for c4/c5 has been fairly resilient as well.
>
> Specific question. You mentioned plague. Are you using Koji now for
> build management or something else?

Something else ... no koji at all here.




_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 12-30-2011, 01:55 PM
Lamar Owen
 
Default custom iso spin

On Thursday, December 29, 2011 10:46:22 AM Karanbir Singh wrote:
> We could, should there be a need, even have rpms that are not a part of
> the distro ( eg. somethings from an external third party repo ). There
> would be some stuff that needs resolved and make sure all licenses etc
> are in place, and some level of assurances are in place for upgrade path
> etc. But it can be done, fairly easily.
...
> > Is there some documentation on building a custom iso spin?
> > I would also like to be able to drive the tools.
>
> What does that mean 'drive the tools' ?

Karabir, and all,

I'm interested in this as well, for some other reasons.

I'd like to make an install USB with all the CentOS packages, sorted into repositories, and with updates integrated. Also, I'd like to put some select third party repos, and be able to use them at install time (anaconda is capable of using any repo at install time, of course). With 16 and 32GB USB bootable USB sticks somewhat affordable now, I can see great utility in being able to carry a key around to do installs to non-network attached PC's (of which we have a few) as well as being able to yum update off of that media (as documented in a form in the CentOS-Media.repo file).

And, like the OP, I'd like to drive the tools, meaning build it myself so that I can include packages that CentOS can't distribute (in particular, some site-developed packages that for various reasons we can't or won't distribute). The SL project documents a good deal of this process in their 'sites' documentation, incidentally.
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 12-30-2011, 03:10 PM
夜神 岩男
 
Default custom iso spin

On 12/30/2011 10:28 PM, Johnny Hughes wrote:
> On 12/30/2011 04:55 AM, 夜神 岩男 wrote:
>> On 12/30/2011 01:45 PM, Karanbir Singh wrote:
>>> my talk on application hosting on CentOS from Fosdem 2008 went into some
>>> details about this, there should be a copy of the talk online.
>>
>> I appreciate the reference, I'll give it a look.
>>
>>> happy to answer specific questions... our build services are fairly
>>> robust as well ( remember over 100k builds were done leading to 6.0 );
>>> the plague setup in place for c4/c5 has been fairly resilient as well.
>>
>> Specific question. You mentioned plague. Are you using Koji now for
>> build management or something else?
>
> Something else ... no koji at all here.

...which would be?

Or was that supposed to be mysterious and intriguing?
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 12-30-2011, 03:25 PM
Johnny Hughes
 
Default custom iso spin

On 12/30/2011 10:10 AM, 夜神 岩男 wrote:
> On 12/30/2011 10:28 PM, Johnny Hughes wrote:
>> On 12/30/2011 04:55 AM, 夜神 岩男 wrote:
>>> On 12/30/2011 01:45 PM, Karanbir Singh wrote:
>>>> my talk on application hosting on CentOS from Fosdem 2008 went into some
>>>> details about this, there should be a copy of the talk online.
>>>
>>> I appreciate the reference, I'll give it a look.
>>>
>>>> happy to answer specific questions... our build services are fairly
>>>> robust as well ( remember over 100k builds were done leading to 6.0 );
>>>> the plague setup in place for c4/c5 has been fairly resilient as well.
>>>
>>> Specific question. You mentioned plague. Are you using Koji now for
>>> build management or something else?
>>
>> Something else ... no koji at all here.
>
> ...which would be?
>
> Or was that supposed to be mysterious and intriguing?

No, It was supposed to answer the question that you asked. We are not
using Koji ... we are using something else.

We have a custom build system. It uses a beanstalkd workqueue to submit
packages to mock and can scale up or down as required and submit
packages to several build machines. If we need to build 2500 packages,
we can add extra machines to the system.

But mock on an x86_64 machine is what is actually building packages.
The buildsystem around it could just as easily be the command line
passing the proper variables to mock. In fact, it is just individual
jobs being passed into mock and some other way to "queue" up the packages.

_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 12-30-2011, 07:47 PM
Karanbir Singh
 
Default custom iso spin

On 12/30/2011 04:25 PM, Johnny Hughes wrote:
> We have a custom build system. It uses a beanstalkd workqueue to submit
> packages to mock and can scale up or down as required and submit

Just to add a few more bits : the queue system is meant to extend in
both directions ( import and tracking of sources on one end - and
testing + release management at the other end ). At the moment there is
a bunch of band-aid and Johnny and me typing up stuff to bridge those
bits, but I hope to have the entire setup in place end to end by end of
March 2012. Apart from the cli, some of the other interesting interfaces
that are semi functional at this point include an irc-bot, a rest api
and a publicly available http callback server that allows arbitrary
subscription.

There is, by design, no web interface. The plan is to focus on the
mechanics and make the entire system work, with enough hooks that allows
anyone / everyone to do the client side work. Eg. being able to write a
yum plugin that would harvest update metadata from the buildsystem and
be able to report on 'upcoming updates' ( a PoC for this is functional
now.. )

The entire system is called 'reimzul'

As soon as I have got the basic functional stuff in place, and some docs
around it I'll have it up in a git repo where people can clone and join
the fun. And yes, this is very custom done for the CentOS ecosystem and
address's the problems and hurdles that we run into with the localised
process. Koji is a great system, just like OBS is - but they solve
different problems and do so in a way that makes both of them really bad
solutions for the distro-rebuild and tracking process's.

In other interesting artifacts ( and mostly from free win's ) include
being able to test external third party repo's for pre-updates-release
breakage's.... Another feature is the constant compose ( the
next-release-tree is constantly composed on every update build, so were
pretty much ready to go for the next release every night.. a feature
that came in handy for the 6.2 release.)

again, just to be clear - this system works for c6 now, and will for c7
etc ( unless we come up with something else ). We are very much
committed to using plague for c4/c5 for the time being. The pain
associated with migrating and then proving the system for c5 would be
more than is justifiable at this point.

Bit of a brain dump, but there you have it.

--
Karanbir Singh
+44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh
ICQ: 2522219 | Yahoo IM: z00dax | Gtalk: z00dax
GnuPG Key : http://www.karan.org/publickey.asc
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 

Thread Tools




All times are GMT. The time now is 03:48 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org