Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Gentoo Alt (http://www.linux-archive.org/gentoo-alt/)
-   -   Gentoo prefix with host system toolchain? (http://www.linux-archive.org/gentoo-alt/347791-gentoo-prefix-host-system-toolchain.html)

Dag Sverre Seljebotn 03-27-2010 07:35 AM

Gentoo prefix with host system toolchain?
 
Just stumbled over Gentoo prefix and it seems very exciting (see my
usecase below). My question:


Is it possible to use the host system's bash/gcc/... when having a
Gentoo prefix on Mac or Linux? I.e. a system for setting up software
which doesn't have *too* much redundancy with the host system. One would
then set PATH, LD_LIBRARY_PATH, MANPATH etc. to use it as an overlay
over the host system.


It wouldn't need to support all the Gentoo packages, just a select subset.

All the bootstrapping guides assume setting up a seperate toolchain for
Gentoo prefix. I know very little about Gentoo, but am eager to read up
on it. But just a tiny link to the right docs can save me massive
amounts of time now in the beginning.


Background:

The Python scientific/engineering open source community has a (widely
recognized) problem of distributing our software and integrating that
distribution mechanism into our development workflow.


Gentoo prefix seems like a perfect fit:
- Lots of scientific users don't have root access (run software on
clusters)
- Need to distribute software fast and iteratively to coworkers on the
other side of the world with a totally different setup

- Linux, Mac, Solaris, Windows...

There's lot of half-baked distribution solution available (sagemath.org,
femhub.org, enthought.com, pythonxy.com), but Gentoo seems like a better
choice for my project for several reasons.


Dag Sverre Seljebotn

Michael Haubenwallner 03-29-2010 03:50 PM

Gentoo prefix with host system toolchain?
 
Hi Dag,

Dag Sverre Seljebotn wrote:
> Just stumbled over Gentoo prefix and it seems very exciting (see my
> usecase below). My question:

You're welcome!

> Is it possible to use the host system's bash/gcc/... when having a
> Gentoo prefix on Mac or Linux? I.e. a system for setting up software
> which doesn't have *too* much redundancy with the host system. One would
> then set PATH, LD_LIBRARY_PATH, MANPATH etc. to use it as an overlay
> over the host system.

While it should be "possible" to some degree to do this, we can not recommend
and of course not really support this - although you might get hints for specific
problems here and there.
First hint: Do _not_ use LD_LIBRARY_PATH (on a regular basis) for various reasons.

> It wouldn't need to support all the Gentoo packages, just a select subset.
>
> All the bootstrapping guides assume setting up a seperate toolchain for
> Gentoo prefix. I know very little about Gentoo, but am eager to read up
> on it. But just a tiny link to the right docs can save me massive
> amounts of time now in the beginning.

You really should put a toolchain (binutils, gcc, libtool, other?) into
your Prefix, they provide the magic necessary to avoid LD_LIBRARY_PATH.

For other packages: You can try to package.provide[1] them and see what
breaks. Eventually you might have to create symlinks inside your Prefix
pointing to host utilitis, or you might have to modify ebuilds...

Second hint:
It does help to be a more experienced (at least) user (Gentoo Linux
and/or Gentoo Prefix) when trying to reuse host packages.

<snip>
> - Need to distribute software fast and iteratively to coworkers on the
> other side of the world with a totally different setup
> - Linux, Mac, Solaris, Windows...

Third hint:
Having that many different systems and users feels like becoming a maintenance
nightmare when trying to use as much as possible packages from the host system.

[1] http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=3&chap=5#doc_chap3
where /etc/portage/profile/package.provided is below your Prefix.

HTH,
/haubi/
--
Michael Haubenwallner
Gentoo on a different level

Jeremy Olexa 03-29-2010 04:03 PM

Gentoo prefix with host system toolchain?
 
On Mon, 29 Mar 2010 17:50:20 +0200, Michael Haubenwallner
<haubi@gentoo.org> wrote:
> Third hint:
> Having that many different systems and users feels like becoming a
> maintenance nightmare when trying to use as much as possible
> packages from the host system.

Hello,
I would even go one step farther and say it is VERY difficult to provide
such a setup with host tools. For example, can one guarantee that every
host that someone uses has bash-3.2 or greater? Half of my systems don't.
So, I would argue that you should have more tools in the target prefix
rather than less from a supportability standpoint (where support is coming
from you ;) ). It goes without saying that the Gentoo Prefix team is never
able to support this due to all the variables.

With that being said, please first build up a Gentoo Prefix environment
like it was intended and then try to replicate it with host tools. You will
need the experience of knowing how it is suppose to work first and
foremost.

Good luck,
Jeremy

Peter Abrahamsen 03-29-2010 06:45 PM

Gentoo prefix with host system toolchain?
 
I have to agree with everyone else. Especially if you're trying to
provide a foundation for a number of people, building on top of a
variety of moving targets, i.e. peoples' OSs, will be a nightmare.

drobbins says he's going to rewrite Metro in the next few months. Once
he does, I might be interested in working on some scripts for setting
up prefix instances.

Peter

On Mon, Mar 29, 2010 at 9:03 AM, Jeremy Olexa <darkside@gentoo.org> wrote:
>
> On Mon, 29 Mar 2010 17:50:20 +0200, Michael Haubenwallner
> <haubi@gentoo.org> wrote:
>> Third hint:
>> Having that many different systems and users feels like becoming a
>> maintenance nightmare when trying to use as much as possible
>> packages from the host system.
>
> Hello,
> I would even go one step farther and say it is VERY difficult to provide
> such a setup with host tools. For example, can one guarantee that every
> host that someone uses has bash-3.2 or greater? Half of my systems don't.
> So, I would argue that you should have more tools in the target prefix
> rather than less from a supportability standpoint (where support is coming
> from you ;) ). It goes without saying that the Gentoo Prefix team is never
> able to support this due to all the variables.
>
> With that being said, please first build up a Gentoo Prefix environment
> like it was intended and then try to replicate it with host tools. You will
> need the experience of knowing how it is suppose to work first and
> foremost.
>
> Good luck,
> Jeremy
>
>

Dag Sverre Seljebotn 03-29-2010 07:22 PM

Gentoo prefix with host system toolchain?
 
Michael Haubenwallner wrote:

Hi Dag,

Dag Sverre Seljebotn wrote:


Just stumbled over Gentoo prefix and it seems very exciting (see my
usecase below). My question:



You're welcome!



Is it possible to use the host system's bash/gcc/... when having a
Gentoo prefix on Mac or Linux? I.e. a system for setting up software
which doesn't have *too* much redundancy with the host system. One would
then set PATH, LD_LIBRARY_PATH, MANPATH etc. to use it as an overlay
over the host system.



While it should be "possible" to some degree to do this, we can not recommend
and of course not really support this - although you might get hints for specific
problems here and there.


Thanks for the input, it's very helpful.


First hint: Do _not_ use LD_LIBRARY_PATH (on a regular basis) for various reasons.

I keep seeing this statement repeated on the web, but somehow always
feel the reasons given don't apply in my usecase (I might of course be
wrong..). And using LD_LIBRARY_PATH is incredibly convenient for a
number of reasons: It allows moving directories around, moving programs
to our cluster for execution without recompiling, ship a binary tarball
to others with the same system.


I realize people have strong opinions about this. Please, I don't want
to start any flamewars. (In particular I don't feel strongly for arguing
about one vs. the other, just saying that I'm currently a happy
LD_LIBRARY_PATH user.)



<snip>


- Need to distribute software fast and iteratively to coworkers on the
other side of the world with a totally different setup
- Linux, Mac, Solaris, Windows...



Third hint:
Having that many different systems and users feels like becoming a maintenance
nightmare when trying to use as much as possible packages from the host system.

Something that might help is the simplicity of the packages I want to
deploy, but I do see your point. Others have already taken this approach
though, such as Sage (http://sagemath.org) and it works pretty well.
(Well, there's no Windows.)


It is a weakness shared with the *current* approach used by pretty much
everybody, which is to provide my package, list the 15 dependencies, and
have people sort it out by themselves -- which is a major waste of time
(to make an understatement, not all scientists are experts in building
software). To the point that people rewrite complicated stuff themselves
just to avoid the dependencies...


(I live in a world where people just don't have root access to their
boxes. Using package managers to install software only works if the
software is stable and one can ask sysadmins to do it and wait some
days; which drastically lengthens the turnaround time.)


Another alternative is writing packages for Sagemath --
http://sagemath.org. I already know that the Gentoo community don't
particularly care for the Sagemath approach though :-) :


http://bugs.gentoo.org/show_bug.cgi?id=240622

Thanks for the input, I'll just have to see where this takes me...

Dag Sverre

Fabian Groffen 03-29-2010 07:28 PM

Gentoo prefix with host system toolchain?
 
On 29-03-2010 21:22:12 +0200, Dag Sverre Seljebotn wrote:
> > First hint: Do _not_ use LD_LIBRARY_PATH (on a regular basis) for various reasons.
> >
> I keep seeing this statement repeated on the web, but somehow always
> feel the reasons given don't apply in my usecase (I might of course be
> wrong..). And using LD_LIBRARY_PATH is incredibly convenient for a
> number of reasons: It allows moving directories around, moving programs
> to our cluster for execution without recompiling, ship a binary tarball
> to others with the same system.
>
> I realize people have strong opinions about this. Please, I don't want
> to start any flamewars. (In particular I don't feel strongly for arguing
> about one vs. the other, just saying that I'm currently a happy
> LD_LIBRARY_PATH user.)

Gentoo Prefix doesn't work with LD_LIBRARY_PATH, so if your environment
is designed to use it, then we simply cannot offer a solution to you.
> It is a weakness shared with the *current* approach used by pretty much
> everybody, which is to provide my package, list the 15 dependencies, and
> have people sort it out by themselves -- which is a major waste of time
> (to make an understatement, not all scientists are experts in building
> software). To the point that people rewrite complicated stuff themselves
> just to avoid the dependencies...

You could consider building binary packages and offering them to your
scientists. Gentoo Prefix can install binary packages in different
locations than they are built for, although -- to be honest -- it's a
sufficiently scaring technology to try and avoid it if possible (by
having the same install location everywhere).


--
Fabian Groffen
Gentoo on a different level

Dag Sverre Seljebotn 03-29-2010 07:37 PM

Gentoo prefix with host system toolchain?
 
Fabian Groffen wrote:

On 29-03-2010 21:22:12 +0200, Dag Sverre Seljebotn wrote:


First hint: Do _not_ use LD_LIBRARY_PATH (on a regular basis) for various reasons.


I keep seeing this statement repeated on the web, but somehow always
feel the reasons given don't apply in my usecase (I might of course be
wrong..). And using LD_LIBRARY_PATH is incredibly convenient for a
number of reasons: It allows moving directories around, moving programs
to our cluster for execution without recompiling, ship a binary tarball
to others with the same system.


I realize people have strong opinions about this. Please, I don't want
to start any flamewars. (In particular I don't feel strongly for arguing
about one vs. the other, just saying that I'm currently a happy
LD_LIBRARY_PATH user.)



Gentoo Prefix doesn't work with LD_LIBRARY_PATH, so if your environment
is designed to use it, then we simply cannot offer a solution to you.

It's not a total dealbreaker, but it does mean quite a few extra builds,
i.e. wasted time before getting results (as the home dir on the cluster
I use don't match the home dir on my development machine).


I'm not looking for a ready solution BTW, just a good starting point. Of
course, it might just be too much work.


Dag Sverre

Michael Haubenwallner 03-29-2010 08:34 PM

Gentoo prefix with host system toolchain?
 
Dag Sverre Seljebotn wrote:
> There's lot of half-baked distribution solution available (sagemath.org,
> femhub.org, enthought.com, pythonxy.com), but Gentoo seems like a better
> choice for my project for several reasons.

After a short look on all those links and after reading bug#240622 I get
the impression (may be wrong), that Gentoo Prefix _is_ what you're looking
for - as a _replacement_ for sagemath (and others):
It just seems necessary to isolate the math-only package(s) out of
sagemath and write the ebuild(s) which uses already existent Gentoo
package-definitions (=ebuilds) as dependencies.

(still may be wrong, but) HTH,
/haubi/
--
Michael Haubenwallner
Gentoo on a different level

Dag Sverre Seljebotn 03-29-2010 09:05 PM

Gentoo prefix with host system toolchain?
 
Michael Haubenwallner wrote:

Dag Sverre Seljebotn wrote:


There's lot of half-baked distribution solution available (sagemath.org,
femhub.org, enthought.com, pythonxy.com), but Gentoo seems like a better
choice for my project for several reasons.



After a short look on all those links and after reading bug#240622 I get
the impression (may be wrong), that Gentoo Prefix _is_ what you're looking
for - as a _replacement_ for sagemath (and others):
It just seems necessary to isolate the math-only package(s) out of
sagemath and write the ebuild(s) which uses already existent Gentoo
package-definitions (=ebuilds) as dependencies.


That's what I'm hoping, anyway.

The ebuilds won't be the problem, the ones I need are already in Gentoo
(and if not, François Bissey, who maintains a Sage overlay for Gentoo,
has agreed to help me get to know Gentoo).


The summary seems to be (and it's how others have put it as well):
- Gentoo prefix has the advantage of a much more mature package
distribution system (and community around it)
- The ones mentioned has (what is in the community percieved as) the
advantage of using LD_LIBRARY_PATH (being easily movable, bundleable)
and using the system compilers and libtool (so they feel less like an
"OS within an OS" and more like a software distribution mechanism -- one
could argue this is not a good argument though).


It seems I'm hoping for the best of both worlds, which I might not be
able to get (or, not without a lot of work, and I guess I should at
least gain much more Gentoo experience first).


Dag Sverre

Markus Duft 03-30-2010 05:19 AM

Gentoo prefix with host system toolchain?
 
On 03/29/10 23:05, Dag Sverre Seljebotn wrote:
> Michael Haubenwallner wrote:
>> Dag Sverre Seljebotn wrote:
>>
>>> There's lot of half-baked distribution solution available (sagemath.org,
>>> femhub.org, enthought.com, pythonxy.com), but Gentoo seems like a better
>>> choice for my project for several reasons.
>>>
>>
>> After a short look on all those links and after reading bug#240622 I get
>> the impression (may be wrong), that Gentoo Prefix _is_ what you're
>> looking
>> for - as a _replacement_ for sagemath (and others):
>> It just seems necessary to isolate the math-only package(s) out of
>> sagemath and write the ebuild(s) which uses already existent Gentoo
>> package-definitions (=ebuilds) as dependencies.
>>
> That's what I'm hoping, anyway.
>
> The ebuilds won't be the problem, the ones I need are already in Gentoo
> (and if not, François Bissey, who maintains a Sage overlay for Gentoo,
> has agreed to help me get to know Gentoo).
>
> The summary seems to be (and it's how others have put it as well):
> - Gentoo prefix has the advantage of a much more mature package
> distribution system (and community around it)
> - The ones mentioned has (what is in the community percieved as) the
> advantage of using LD_LIBRARY_PATH (being easily movable, bundleable)
> and using the system compilers and libtool (so they feel less like an
> "OS within an OS" and more like a software distribution mechanism -- one
> could argue this is not a good argument though).

just to chime in on the movability: i successfully moved whole prefix
installations to other directories, and they worked fine (hint: i used
chpathtool to adjust paths) the only limitation is, that the path has to
be the exact same length (or less long, but that one didn't work for me
yet, but that could be fixable). of course this is not really
convenient, but it should be possibly for you to create

"/home/user/prefix-with-long-path1"
and
"/cluster/disk25/user/prefix-short"

those would be exchangeable. Of course chpathtool is possibly still not
perfect, and i observed the one or the other segfault with certain files
(none of those lately though). However you're free to fix it ;)

Cheers, Markus

>
> It seems I'm hoping for the best of both worlds, which I might not be
> able to get (or, not without a lot of work, and I guess I should at
> least gain much more Gentoo experience first).
>
> Dag Sverre
>


All times are GMT. The time now is 07:48 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.