FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Embedded

 
 
LinkBack Thread Tools
 
Old 11-30-2009, 05:24 PM
Shinkan
 
Default Cross Dev Tricks + Hardened questions

Hi there,

I'm quite sorry to annoy you, but I'm quite new to
Gentoo, and the Embedded Handbook (which appears to be the best
information source for what I want to do) does not make it clear for me.
I hope you'll be able to help me a little.


I have a base host system, which is a x86 hardened-profiled Gentoo 10.0.
I would like to use it to build target systems with the most control Gentoo can offer.
I first looked at Catalyst but Cross-dev things seemed more accurate for what I want to do.




Let's say I want to make a "example-v1" target system.
I want my host to build on a "example-v1-build" directory everything needed to build and emerge binaries for my target system.
I
want my host to build with "example-v1-build" files and toolchain a
"example-v1-target" directory which would contain emerged system.

"example-v1-target" would contain a very minimal system, with no gcc,
emerge or dev things. It just have to be able to run C/C++ binaries for
a given arch.

I would like to use crossdev because I want to specify which version of glibc/gcc/etc I will use for one given target.

Furthermore, I would like my toolchain to build "hardened" binaries, as those we get by using an hardened stage3.

I don't get the process from the Embedded Handbook.
Do I have to set some CHOST, CTARGET, ROOT or PORTAGE_CONFDIR env vars or crossdev will handle that some way ?



How can I tell to crossdev : "build the cross toolchain to my example-v1-build directory" ?
How
can I then tell to emerge "emerge and build from my example-v1-build
directory to example-v1-target" ? Do I have to chroot in
"example-v1-build" ? In this case wouldn't I lost all emerge
capabilities ?


Many thanks in advance for your help.
Sorry I can't figure it all by myself despite the good docs.
--
Pierre.
"Sometimes when I'm talking, my words can't keep up with my thoughts. I wonder why we think faster than we speak. Probably so we can think twice." - Bill Watterson
 
Old 11-30-2009, 08:12 PM
Peter Stuge
 
Default Cross Dev Tricks + Hardened questions

Shinkan wrote:
> I have a base host system, which is a x86 hardened-profiled Gentoo 10.0.
> I would like to use it to build target systems with the most
> control Gentoo can offer.
> I first looked at Catalyst but Cross-dev things seemed more accurate
> for what I want to do.

USE CATALYST!


> I would like to use crossdev because I want to specify which
> version of glibc/gcc/etc I will use for one given target.

To do this you will create stage2, stage3 and stage4 spec files
(maybe also stage1, I'm not sure on that) for catalyst. It can
produce exactly what you want.


> Furthermore, I would like my toolchain to build "hardened"
> binaries, as those we get by using an hardened stage3.

Fine, I believe this would be controlled by USE flags for stage2 and
up.


> Sorry I can't figure it all by myself despite the good docs.

Please just start playing with the catalyst examples. I hope you will
quickly learn what they produce.

Everything that catalyst produces goes into /var/tmp/catalyst/


//Peter
 
Old 12-01-2009, 07:44 AM
Shinkan
 
Default Cross Dev Tricks + Hardened questions

2009/11/30 Peter Stuge <peter@stuge.se>





USE CATALYST!



To do this you will create stage2, stage3 and stage4 spec files

(maybe also stage1, I'm not sure on that) for catalyst. It can

produce exactly what you want.



Fine, I believe this would be controlled by USE flags for stage2 and

up.



Please just start playing with the catalyst examples. I hope you will

quickly learn what they produce.

I already looked a lot into Catalyst but some points doesn't fit to my plans :
- I have to use a profile if I want to specify things (for instance ports version) for base stage. I really don't want to build a profile because they're hard to maintain in a wide use scheme, and because that's overkill.


- Catatalyst seems to build by substraction (I mean on livecd, it builds from a stage3, then unmerges and removes things). I want to build with additive steps (from nothing).
- I want to be able to just emerge one new port or update one on a target, and with Catalyst I cant. I must rebuild all (yeah, cache is there but...), and I really need to build just one port on some cases.



That's why I thought about a "build" directory built from my host with crossdev. Then I use this build env to build my target with gcc/libc I want. If I have to build just one port, I can use my build env for this target again.


Since I choose what to build from nothing, I don't have to use profile to define what I put in my target or build. I don't break system by removing things.
My build standard "make.conf" serves as usual, I have nothing more than a crossdev, a make.conf filling, and some emerge to a given root.



That's what I want, but I don't know how to achieve this.
If Catalyst can offer me this control, I would be glad to use it.


--
Pierre.
"Sometimes when I'm talking, my words can't keep up with my thoughts. I wonder why we think faster than we speak. Probably so we can think twice." - Bill Watterson
 
Old 12-01-2009, 04:48 PM
Ned Ludd
 
Default Cross Dev Tricks + Hardened questions

On Tue, 2009-12-01 at 09:44 +0100, Shinkan wrote:
>
> 2009/11/30 Peter Stuge <peter@stuge.se>
>
>
> USE CATALYST!
>
>
> To do this you will create stage2, stage3 and stage4 spec
> files
> (maybe also stage1, I'm not sure on that) for catalyst. It can
> produce exactly what you want.
>
>
> Fine, I believe this would be controlled by USE flags for
> stage2 and
> up.
>
>
> Please just start playing with the catalyst examples. I hope
> you will
> quickly learn what they produce.
>
> I already looked a lot into Catalyst but some points doesn't fit to my
> plans :
> - I have to use a profile if I want to specify things (for instance
> ports version) for base stage. I really don't want to build a profile
> because they're hard to maintain in a wide use scheme, and because
> that's overkill.
> - Catatalyst seems to build by substraction (I mean on livecd, it
> builds from a stage3, then unmerges and removes things). I want to
> build with additive steps (from nothing).
> - I want to be able to just emerge one new port or update one on a
> target, and with Catalyst I cant. I must rebuild all (yeah, cache is
> there but...), and I really need to build just one port on some cases.
>
> That's why I thought about a "build" directory built from my host with
> crossdev. Then I use this build env to build my target with gcc/libc I
> want. If I have to build just one port, I can use my build env for
> this target again.
> Since I choose what to build from nothing, I don't have to use profile
> to define what I put in my target or build. I don't break system by
> removing things.
> My build standard "make.conf" serves as usual, I have nothing more
> than a crossdev, a make.conf filling, and some emerge to a given root.
>
> That's what I want, but I don't know how to achieve this.
> If Catalyst can offer me this control, I would be glad to use it.


catalyst can do none of the above and it's not cross-compile aware at
all. Building from nothing is the right way to handle what you are
trying to accomplish.


Good luck.


--
Ned Ludd <solar@gentoo.org>
Gentoo Linux
 
Old 12-01-2009, 06:03 PM
Ed W
 
Default Cross Dev Tricks + Hardened questions

Ned Ludd wrote:

catalyst can do none of the above and it's not cross-compile aware at
all. Building from nothing is the right way to handle what you are
trying to accomplish.





Oh, fair enough

There is also a build tool in Funtoo... Might be worth seeing if that's
any help?


Good luck

Ed
 
Old 12-01-2009, 06:03 PM
Peter Stuge
 
Default Cross Dev Tricks + Hardened questions

Shinkan wrote:
> I already looked a lot into Catalyst but some points doesn't fit to
> my plans :
> - I have to use a profile if I want to specify things (for instance
> ports version) for base stage.

Hm, what is ports version? Do you mean the portage snapshot? That can
be set in the spec files though?


> I really don't want to build a profile because they're hard to
> maintain in a wide use scheme, and because that's overkill.

Would it not work to put a profile in /etc/catalyst/mytarget/profile
and give a relative path like ../../../etc/catalyst/mytarget/profile
in the catalyst spec file?


> - Catatalyst seems to build by substraction (I mean on livecd, it
> builds from a stage3, then unmerges and removes things). I want
> to build with additive steps (from nothing).

catalyst starts somewhere too, and it uses spec files (written by
you) to create all stages. You would create your own stage3, which
would suit your needs. And a stage4. And probably use the stage4 as
source for the livecd.


> - I want to be able to just emerge one new port or update one on a
> target, and with Catalyst I cant.

Sure you can. I do this a lot with my systems.


> I must rebuild all (yeah, cache is there but...),

But what? I find that the most time is spent in the rsync, done first
for a catalyst stage. Binpkgs install very quickly.


> and I really need to build just one port on some cases.

Right. I get tbz2s which I install on my target.


> That's why I thought about a "build" directory built from my host
> with crossdev. Then I use this build env to build my target with
> gcc/libc I want. If I have to build just one port, I can use my
> build env for this target again.

It also works, but I think there are nice benefits with catalyst.


> Since I choose what to build from nothing, I don't have to use
> profile to define what I put in my target or build. I don't break
> system by removing things.

"system" is not a great term. Yes, your stage4 will not have a build
environment, so the next catalyst run will take a little longer.
Maybe that's a big problem in your scenario, but if you can live with
that, I don't see a problem.


> My build standard "make.conf" serves as usual, I have nothing more
> than a crossdev, a make.conf filling, and some emerge to a given
> root.
>
> That's what I want, but I don't know how to achieve this.

profile, stage1, stage2, stage3 and stage4. I agree that it is a
detour, but it does solve the problem.


> If Catalyst can offer me this control, I would be glad to use it.

Ned Ludd wrote:
> catalyst can do none of the above

catalyst can produce the same result, but it needs different input of
course.


> and it's not cross-compile aware at all.

In this case it will work, since build is amd64 and target x86.


> Building from nothing is the right way to handle what you
> are trying to accomplish.

The (big, I think) benefit with catalyst is that it gathers all
recipes and files and configuration in a single tool.

Of course, emerge can be scripted, but that seems like what catalyst
is already doing, right?


Hey - another, more novel, idea would be to create some new catalyst
targets, that combines both these methods and builds from nothing,
but with direction.

I insist on catalyst because it takes care of a good deal of things
that have to be done manually otherwise - documentation, filesystem
overlay, create binpkgs, create tarball, create iso..


//Peter
 
Old 12-01-2009, 07:51 PM
Peter Stuge
 
Default Cross Dev Tricks + Hardened questions

Ed W wrote:
> There is also a build tool in Funtoo... Might be worth seeing if
> that's any help?

Nod, metro. I've looked at it a little but so far not found big
benefits over catalyst for me.


//Peter
 
Old 12-01-2009, 07:52 PM
Shinkan
 
Default Cross Dev Tricks + Hardened questions

2009/12/1 Peter Stuge <peter@stuge.se>


Hm, what is ports version? Do you mean the portage snapshot? That can

be set in the spec files though?

I wanted to say for instance "gcc-3.4.3" and not "current gcc".
Because I want some targets to be built against a specific version of glibc.


*




> I really don't want to build a profile because they're hard to

> maintain in a wide use scheme, and because that's overkill.



Would it not work to put a profile in /etc/catalyst/mytarget/profile

and give a relative path like ../../../etc/catalyst/mytarget/profile

in the catalyst spec file?

I thought about that. But :
1) That's a dirty hack. What if gentoo decides to put profiles in /usr/portage/boringstuffs/profiles/ ?
They can because of official transparent "eselect profile" way of selecting a profile.


2) Each of my target can be different. WCS, we have one profile per target, per architecture, per type of target... per version.
I want my target build env to be self-containing, so I don't want 10K profiles moving around my host tree.


Plus, I lost profiles parenting if I put things in /etc/catalyst/mytarget/
*


> - I want to be able to just emerge one new port or update one on a

> * target, and with Catalyst I cant.



Sure you can. I do this a lot with my systems.





> * I must rebuild all (yeah, cache is there but...),



But what? I find that the most time is spent in the rsync, done first

for a catalyst stage. Binpkgs install very quickly.

I would have to audit catalyst code to tell cache won't break some of my needs.
Because I will do nasty things on the portage tree. I don't want catalyst to think it can "pick a bin in a cache" (that's sound funny isn't it ?), but in fact ebuild with same version has changed because of some patches I applied.


Cache is really a killing feat of catalyst, but I can't afford it for my things. I fear the cache.
*


Right. I get tbz2s which I install on my target.

That's fine. But I can do this with simple emerges too.
*

"system" is not a great term. Yes, your stage4 will not have a build

environment, so the next catalyst run will take a little longer.

Maybe that's a big problem in your scenario, but if you can live with

that, I don't see a problem.

I don't want my "stage4" to have a build env. I want my stage4 to be very minimal.
I would expect catalyst to do this in fact :
1) Take a official stage3.


2) Build a build env with this stage3. Build env would have specified arch/gcc/libc/binutils version, and portage with a specified tree.
3) From this build env, catalyst would build a target with JUST the packages I specified, so the target would be totally empty at begining.


Then I would like the kernel comp on build env, and image cped to my target.
Then I would like catalyst to emerge kernel dependant packages to my target.
Then I would be done with a "stage4", which is my target.


4) Then my target could be put on a live cd with some more mechanic catalyst already offers.
For all step there would be a spec file, and possibility to specify each dest dirs.
*


Ned Ludd wrote:

> catalyst can do none of the above

Hey that was a little rude I think !
*


In this case it will work, since build is amd64 and target x86.

I confirm in my case it would work. I build on amd64, I target x86 and amd64, but with different glibc bases.






Hey - another, more novel, idea would be to create some new catalyst

targets, that combines both these methods and builds from nothing,

but with direction.



I insist on catalyst because it takes care of a good deal of things

that have to be done manually otherwise - documentation, filesystem

overlay, create binpkgs, create tarball, create iso..

I really liked catalyst when I read it spec examples.
After dealing a little with basic scenarios and totally outdated examples, I started to find that I had needs it can't answer.


I was really sad, I promise.
What I miss the most is the "I will just have to write 2 spec files per target, YAY !!!" dream, and the "I take care of the Live CD boring stuffs".
*Catalyst should provide a "solar glasses stuffed cowboy" mode, where it could act as an evolved spec-files base cross compiler/emerger, and where we could build from nothing.



--
Pierre.
"Sometimes when I'm talking, my words can't keep up with my thoughts. I wonder why we think faster than we speak. Probably so we can think twice." - Bill Watterson
 
Old 12-01-2009, 09:24 PM
Shinkan
 
Default Cross Dev Tricks + Hardened questions

2009/12/1 Peter Stuge <peter@stuge.se>


Nod, metro. I've looked at it a little but so far not found big

benefits over catalyst for me.

Same to me. It seems weird.


--
Pierre.
"Sometimes when I'm talking, my words can't keep up with my thoughts. I wonder why we think faster than we speak. Probably so we can think twice." - Bill Watterson
 
Old 12-01-2009, 10:57 PM
Peter Stuge
 
Default Cross Dev Tricks + Hardened questions

Shinkan wrote:
> > Hm, what is ports version? Do you mean the portage snapshot? That
> > can be set in the spec files though?
>
> I wanted to say for instance "gcc-3.4.3" and not "current gcc".
> Because I want some targets to be built against a specific version
> of glibc.

No problem. Just specify the versions you want in your stage2. Of
course you need those ebuilds somewhere, but that can easily be in an
overlay.


> > > I really don't want to build a profile because they're hard to
> > > maintain in a wide use scheme, and because that's overkill.
> >
> > Would it not work to put a profile in /etc/catalyst/mytarget/profile
> > and give a relative path like ../../../etc/catalyst/mytarget/profile
> > in the catalyst spec file?
>
> I thought about that. But :
> 1) That's a dirty hack. What if gentoo decides to put profiles in
> /usr/portage/boringstuffs/profiles/ ?
> They can because of official transparent "eselect profile" way of
> selecting a profile.

I agree it's a bit of a hack. But it will work very well until you
upgrade catalyst, at which point you should probably be prepared to
review your spec files anyway.


> 2) Each of my target can be different. WCS, we have one profile per
> target, per architecture, per type of target... per version.

I don't quite understand this, since you haven't given very many
details about what you want to do.


> I want my target build env to be self-containing, so I don't want
> 10K profiles moving around my host tree.
> Plus, I lost profiles parenting if I put things in
> /etc/catalyst/mytarget/

Can profiles be in an overlay by the way? In that case you don't have
to mess with symlinks at all, and could still use inheritance.


> > > I must rebuild all (yeah, cache is there but...),
> >
> > But what? I find that the most time is spent in the rsync, done
> > first for a catalyst stage. Binpkgs install very quickly.
>
> I would have to audit catalyst code to tell cache won't break some
> of my needs.

catalyst does not implement any caching of it's own. The caching is
all done by emerge, and can be used fine also without catalyst.


> Because I will do nasty things on the portage tree.

Then you must expect everything to break.


> I don't want catalyst to think it can "pick a bin in a cache"
> (that's sound funny isn't it ?), but in fact ebuild with same
> version has changed because of some patches I applied.

This is not proper use of portage and ebuilds. Please read a bit
about how version numbers are handled in ebuilds. If you make local
additions to an upstream version you shall append -r1 to the ebuild
version. Next local change to same upstream version shall be -r2, and
so on.


> Cache is really a killing feat of catalyst, but I can't afford it
> for my things. I fear the cache.

I think you should learn how it works instead, so that you can take
full advantage of it.

If you create proper ebuilds and follow the rules for ebuild
versioning, you will find that the cache works wonderfully well.

(Again, the "cache" is simply emerge --usepkg --buildpkg.)


> > "system" is not a great term. Yes, your stage4 will not have a
> > build environment, so the next catalyst run will take a little
> > longer. Maybe that's a big problem in your scenario, but if you
> > can live with that, I don't see a problem.
>
> I don't want my "stage4" to have a build env. I want my stage4 to
> be very minimal.

Yes, I know this. Please read again what I wrote.

Your stage4 will not have a build environment.


> I would expect catalyst to do this in fact :
> 1) Take a official stage3.
> 2) Build a build env with this stage3. Build env would have
> specified arch/gcc/libc/binutils version, and portage with a
> specified tree.
> 3) From this build env, catalyst would build a target with JUST the
> packages I specified, so the target would be totally empty at
> begining.
> Then I would like the kernel comp on build env, and image cped to my target.
> Then I would like catalyst to emerge kernel dependant packages to my target.
> Then I would be done with a "stage4", which is my target.
> 4) Then my target could be put on a live cd with some more mechanic
> catalyst already offers.
> For all step there would be a spec file, and possibility to specify
> each dest dirs.

No.

You would start by creating a stage1 or stage2 spec file. stage1
might not be neccessary, stage2 may be enough.

A stage2 spec file starts from a stage1 tarball.
A stage2 spec file makes catalyst build toolchain and libc.

A stage3 spec file starts from the stage2 tarball produced above.
A stage3 spec file normally installs some basic packages. Your stage3
may in fact be a no-op. But stuff like iproute2 could be here.

A stage4 spec file starts from the stage3 tarball produced above.
A stage4 spec file normally installs anything beyond the basic
system, and cleans out all unwanted things. The stage4 also builds a
kernel.

Since all your stages would be minimal, there is not a lot of
cleaning, so re-running the stage4 will also not take very long time.

In the profile and in most if not all stage spec files you will
specify the packages you want for that stage.


> I confirm in my case it would work. I build on amd64, I target x86
> and amd64, but with different glibc bases.

That's ok. I admit that this would need different sets of spec files
for catalyst, but you'd also need different scripts if doing it
manually.


> > I insist on catalyst because it takes care of a good deal of things
> > that have to be done manually otherwise - documentation, filesystem
> > overlay, create binpkgs, create tarball, create iso..
>
> I really liked catalyst when I read it spec examples.
> After dealing a little with basic scenarios and totally outdated
> examples, I started to find that I had needs it can't answer.

This is why I wrote earlier that I think you should make some
experiments and see for yourself what it can do. catalyst is not
really a very advanced tool - it just collects all the small tasks in
a single utility, which makes it very handy.


> What I miss the most is the "I will just have to write 2 spec files
> per target, YAY !!!" dream, and the "I take care of the Live CD
> boring stuffs".

You'll have to do a bit more than that, but not too much. 5 or so
spec files, including the one for the CD. Create the profile, the
configuration files that you want of course. In any case you also
have to create an overlay for local ebuilds. I find layman very easy
to work with. There's an XML file for my overlay at
http://stuge.se/overlay.xml


> Catalyst should provide a "solar glasses stuffed cowboy" mode,
> where it could act as an evolved spec-files base cross
> compiler/emerger, and where we could build from nothing.

I guess this is kind of what I proposed. Create a new target type or
types, other than stage1..4 and livecd1,2, which do not start from a
source stage, but start from an empty directory. But someone has to
do it, and I think releng are pretty busy doing their own job.
(catalyst is their tool.)


//Peter
 

Thread Tools




All times are GMT. The time now is 05:13 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org