FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Alt

 
 
LinkBack Thread Tools
 
Old 02-07-2011, 11:35 PM
Perry Smith
 
Default AIX linking adventure

This is *not* ready for prime time but I thought I would mention it here.

I wrote a front end to AIX's ld command as well as a relink command I'm calling "rl". You can see them here:

https://github.com/pedz/aixbin

I've used this only for Ruby -- and it was in fact Ruby that caused me to do this. Their out of the box system does not work at all for AIX if the prefix directory is not the same as GCC's. WIth this, I can now use rvm on AIX and do "rvm install 1.9.1" and it will complete successfully without my fingers leaving my hands.

I mentioned this idea on this list before but it didn't fly very well. I'm not sure if it was completely understood.

This is more of a proof of concept at this stage. I'm going to putter around and try and use it on other packages.

"rl" could be enhanced to take "from" and "to" directories. If you wanted to change files installed in /some/old/directory and put them into /some/new/directory, rl could easily modified do that.

Enjoy
Perry
 
Old 02-08-2011, 07:49 AM
Michael Haubenwallner
 
Default AIX linking adventure

Hi Perry!

On 02/08/2011 01:35 AM, Perry Smith wrote:
> This is *not* ready for prime time but I thought I would mention it here.
>
> I wrote a front end to AIX's ld command as well as a relink command I'm calling "rl".

Actually, in Prefix we already do wrap AIX' ld in sys-devel/binutils-config
to get the runpaths (the -blibpath argument) right.

At the moment I'm trying another thing to get full "soname" support,
both with and without libtool, using Import Files. There is an RFC[1]
for the way I want to have shared libraries being created on AIX.

However, I'm still unsure if we actually should use the '# autoload' thingy,
but ignore the auto-dependency problem[2] instead.

[1] http://lists.gnu.org/archive/html/libtool/2011-01/msg00023.html
[2] http://archives.gentoo.org/gentoo-alt/msg_b7aec69c70e257206ac49f74a41c9af6.xml

Additionally, for packages not using libtool, there's another ld wrapper in
sys-devel/native-cctools waiting for checkin here to support the '-soname' flag
known on ELF (Linux), creating the shared libraries according to above way[1].
Eventually, this second wrapper would be merged into binutils-config's wrapper.

> You can see them here:
>
> https://github.com/pedz/aixbin

Commenting on 'Readme.markdown' found there:
Why do you move '-blibpath' arguments to '-L' arguments?
IMO, the other way round would make more sense actually.
Why do you need 'rl' (relink) at all?

> I've used this only for Ruby -- and it was in fact Ruby that caused me to do this.
> Their out of the box system does not work at all for AIX if the prefix directory is not the same as GCC's.
> WIth this, I can now use rvm on AIX and do "rvm install 1.9.1" and it will complete successfully without my fingers leaving my hands.

Haven't used/installed Ruby on any platform yet: What is 'rvm' ?

> I mentioned this idea on this list before but it didn't fly very well.
> I'm not sure if it was completely understood.

I fail to find that post you're referring to. Has it been within one
of the threads we already created together, and I've overlooked it?

> This is more of a proof of concept at this stage. I'm going to putter around and try and use it on other packages.

For packages not aware of AIX at all:
How would this require their build systems to be patched?

> "rl" could be enhanced to take "from" and "to" directories.
> If you wanted to change files installed in /some/old/directory and put them into /some/new/directory, rl could easily modified do that.

In which context do you need this use case?
And how would you do that on Linux?

/haubi/
--
Michael Haubenwallner
Gentoo on a different level
 
Old 02-08-2011, 05:36 PM
Perry Smith
 
Default AIX linking adventure

On Feb 8, 2011, at 2:49 AM, Michael Haubenwallner wrote:

> Hi Perry!
>
> On 02/08/2011 01:35 AM, Perry Smith wrote:
>> This is *not* ready for prime time but I thought I would mention it here.
>>
>> I wrote a front end to AIX's ld command as well as a relink command I'm calling "rl".
>
> Actually, in Prefix we already do wrap AIX' ld in sys-devel/binutils-config
> to get the runpaths (the -blibpath argument) right.
>
> At the moment I'm trying another thing to get full "soname" support,
> both with and without libtool, using Import Files. There is an RFC[1]
> for the way I want to have shared libraries being created on AIX.
>
> However, I'm still unsure if we actually should use the '# autoload' thingy,
> but ignore the auto-dependency problem[2] instead.
>
> [1] http://lists.gnu.org/archive/html/libtool/2011-01/msg00023.html
> [2] http://archives.gentoo.org/gentoo-alt/msg_b7aec69c70e257206ac49f74a41c9af6.xml

I am not on the libtool list. I did not read that thread yet but I will here shortly.

I need some "netiquette" help. Should I plop my thoughts / questions about that
thread into this thread or start a new thread? I get confused either way :-) (Maybe
I'm special that way.)

>
> Additionally, for packages not using libtool, there's another ld wrapper in
> sys-devel/native-cctools waiting for checkin here to support the '-soname' flag
> known on ELF (Linux), creating the shared libraries according to above way[1].
> Eventually, this second wrapper would be merged into binutils-config's wrapper.
>
>> You can see them here:
>>
>> https://github.com/pedz/aixbin
>
> Commenting on 'Readme.markdown' found there:
> Why do you move '-blibpath' arguments to '-L' arguments?
> IMO, the other way round would make more sense actually.
> Why do you need 'rl' (relink) at all?

For ruby, their libpath was a complete mistake. They did not include
the path to gcc's library. They did not have a -L
/path/to/where/they/would/install but they did have
-blibpath:/path/to/where/they/would/install:/usr/lib:/lib and somehow
thought that libpath was going to fix it. I guess it would have
if it wasn't for the fact that the executable could not find gcc's
library. And... the executable would never work in the directory
where it was built in. It would work only after the package was
installed (moved). I don't know how they were going to do
the "test" or "check" phase which I'm sure Ruby has.

I took their mistake as a symptom as a general practice. i.e.
the general population somehow thinks that libpath substitutes for
-L and it does not. I'm interested to hear of your experience but
I can't think of a time where open source understood what libpath
(inside the object) could/should be used for.

The Ruby guys did have a fairly reasonable fear. They did not want
to have just -L . and -L .., etc and not set the libpath because the -L .
adds that to the internal libpath and that creates what they called a
security risk. But, the problem is that the dot needs to be in the internal
libpath while the executable is in the directory it was built in.

The purpose for rt is after the install. The first link of ruby
leaves the path to libruby.so as relative (no path component). This
allows ruby to execute in the directory that it was built in (before rt is run)
as well as its final destination (after rt is run) because both dot and
the final destination directory are in the internal libpath.

This is going to be a general problem. You want the executables to
work before being installed. Often there are check or test processes
that use this feature. This implies that finding the libraries is going
to need to be done via relative paths.

But after ruby (the package) is installed, I wanted to fix the path
to libruby.so to the place that it was installed. I could not guess
where it was going to be installed during the "build" phase (the
first compile / link). I had to wait until it was installed and
then use libpath (which I had to fix for Ruby via adding the -L)
to find where libruby.so finally got installed at.

I also had a rather self imposed limitation to work around.
I am trying to get "rvm" to work and rvm thinks it knows how to
unpackage, configure, build, and install without any user
intervention. To get that to work, I had to hook in at the ld
point and then, after the fact, fix the resulting files after they
are installed.


>
>> I've used this only for Ruby -- and it was in fact Ruby that caused me to do this.
>> Their out of the box system does not work at all for AIX if the prefix directory is not the same as GCC's.
>> WIth this, I can now use rvm on AIX and do "rvm install 1.9.1" and it will complete successfully without my fingers leaving my hands.
>
> Haven't used/installed Ruby on any platform yet: What is 'rvm' ?

rvm is "Ruby Version Manager". rvm allows completely independent
versions of Ruby (1.8.7, 1.9.1, MacRuby, etc). Each version can have
its own independent sets of gems (called gemsets) so an application
can use its own version of Ruby and its own set of gems. I can set up
an environment on my development laptop and then fairly easily replicate
this environment on my staging server and production server. It solves
many problems I have when trying to have three or four Rails apps
on the same host. (sorry for the long winded reply).

>
>> I mentioned this idea on this list before but it didn't fly very well.
>> I'm not sure if it was completely understood.
>
> I fail to find that post you're referring to. Has it been within one
> of the threads we already created together, and I've overlooked it?

I *think* this is the post but it got munged:

http://archives.gentoo.org/gentoo-alt/msg_d16e4dce43974e6926eedf04a3b8fb36.xml

My archive is this (snipped out).

The time stamp is Tue, Dec 14, 2010 at 6:25 PM (to the gentoo-alt list)

> Also, the guy I'm talking to mentioned rtl_enable -s. I've not used it but it seems like it might make life a lot easier.
>
> You can play with it some but if you point it at a shared object, it creates a script, an import file, and an export file. Together, they will recreate the shared object. You can then edit the files, for example, adding absolute paths, if you want to change the header of the shared object file.
>
> I am not familiar with portage at all but I've seen that it patches various things like the configure script, etc. It might be easier / safer to leave the scripts alone (leave libtool do whatever it wants) and then use rtl_enable -s, edit the resulting files, and rebuild the shared object the way you want it.
>


>
>> This is more of a proof of concept at this stage. I'm going to putter around and try and use it on other packages.
>
> For packages not aware of AIX at all:
> How would this require their build systems to be patched?
>
>> "rl" could be enhanced to take "from" and "to" directories.
>> If you wanted to change files installed in /some/old/directory and put them into /some/new/directory, rl could easily modified do that.
>
> In which context do you need this use case?

With putting absolute paths into the objects, we are effectively nullifying the possible benefit of the LIBPATH environment. I think we
both agree that that benefit is questionable but others might like it. So if someone wants to move where I built a package from
/usr/local to /opt, the first step would be to change the internal absolute paths to each dependency. There would be other obstacles too
like any internal paths of the executables but often there are command line arguments to change those. e.g. you can give
emacs directions on where to find its lisp files but you can not magically tell the loader where to find emacs' dependent libraries.
A user once could via LIBPATH but that isn't going to work now with absolute internal paths. rl gives that possibly useful option back
to the user.

> And how would you do that on Linux?

I don't know squat about Linux. Part of my frustration is everyone appears to want to make AIX fit into the Linux way.

My objective is subtly but significantly different. I want to get open source packages onto AIX.

>
> /haubi/
> --
> Michael Haubenwallner
> Gentoo on a different level
>
 
Old 02-08-2011, 09:02 PM
Michael Haubenwallner
 
Default AIX linking adventure

Hi Perry!

On 02/08/11 19:36, Perry Smith wrote:
> On Feb 8, 2011, at 2:49 AM, Michael Haubenwallner wrote:
>> On 02/08/2011 01:35 AM, Perry Smith wrote:
>>> I wrote a front end to AIX's ld command as well as a relink command I'm calling "rl".
>> Actually, in Prefix we already do wrap AIX' ld in sys-devel/binutils-config
>> to get the runpaths (the -blibpath argument) right.
>> There is an RFC[1] for the way I want to have shared libraries being created on AIX.
>> [1] http://lists.gnu.org/archive/html/libtool/2011-01/msg00023.html
>
> I am not on the libtool list. I did not read that thread yet but I will here shortly.

You might consider subscribing to that Gentoo bug[1], its URL refers
to the libtool thread, and I'm usually adding progress there.

[1] http://bugs.gentoo.org/show_bug.cgi?id=213277

> I need some "netiquette" help. Should I plop my thoughts / questions about that
> thread into this thread or start a new thread? I get confused either way :-) (Maybe
> I'm special that way.)

Eventually we can sort out your concerns here (first).

>>> https://github.com/pedz/aixbin
>>
>> Commenting on 'Readme.markdown' found there:
>> Why do you move '-blibpath' arguments to '-L' arguments?
>> IMO, the other way round would make more sense actually.
>> Why do you need 'rl' (relink) at all?
>
> For ruby, their libpath was a complete mistake.

Ohw, seems there's need to fix your view on the libpath/runpath issue,
as everything you describe here actually is how it should be done:

Usually, compilers are installed as part of the host system into somewhere
like /usr, where /usr/lib usually is the search path for any installed
library at linktime (called "linkpath", set via "-L" linker flag), and
either "/usr/lib:/lib" or even "/lib:/usr/lib" is the search path at
runtime (called "runpath", set via "-blibpath" linker flag).

> They did not include the path to gcc's library.

Nobody but the compiler and/or the system should know the system's and
compiler's library directories. It is up to the compiler-, linker- and
loader-setup to find the system's and especially the compiler's libraries
at both linktime and runtime.

This exactly is the (major) reason why in Prefix we do wrap the linker
(as well as the compiler) to get gcc's and Prefix' (being the (sub)system)
library directories into both linkpath and runpath.

> They did not have a -L /path/to/where/they/would/install

The currently built package's libraries must not be linked from the target
installation directory. Imagine an old version of a library is already
installed there, and you want to upgrade the package - as is the usual
case with package managers (being either humans or software).

And if there are dependent libraries, they either are installed in system's
library directories the compiler/linker setup knows anyway, or the package
manager should add their locations via something like LDFLAGS or the like.

So while the new binaries must _not_ be /linked/ against the already installed
libraries provided by another instance of the package just built, ...

> but they did have -blibpath:/path/to/where/they/would/install:/usr/lib:/lib

..., at runtime (after installing) the libraries have to be /loaded/ from there.

One AIX specific thing here is that the AIX loader cannot be configured to
search additional directories, so the binaries have to contain the whole runpath.
While the linker does search /usr/lib and /lib at linktime, it will not add
these paths to the runpath when an explicit runpath is specified via -blibpath.
This is why "/usr/lib:/lib" is specified with -blibpath too.

> And... the executable would never work in the directory
> where it was built in. It would work only after the package was
> installed (moved). I don't know how they were going to do
> the "test" or "check" phase which I'm sure Ruby has.

They most likely set the LIBPATH environment variable to the local
library directory. This is one of the rare use cases where environment
variables like LD_LIBRARY_PATH, LIBPATH, SHLIB_PATH are useful.

> I took their mistake as a symptom as a general practice. i.e.
> the general population somehow thinks that libpath substitutes for
> -L and it does not. I'm interested to hear of your experience but
> I can't think of a time where open source understood what libpath
> (inside the object) could/should be used for.

Actually, the Ruby guys seem to have done it "right":
If you were using xlc (from /usr/bin), or even gcc from the
"AIX Toolbox for Linux applications", so both are "installed
with the host system", you might not have had any problem.

> The Ruby guys did have a fairly reasonable fear. They did not want
> to have just -L . and -L .., etc and not set the libpath because the -L .
> adds that to the internal libpath and that creates what they called a
> security risk.

The explicit '-blibpath' avoids recording the (relative) local library
paths passed via '-L' into the runpath:

*) Using buildtime paths (either relative or not) for -L, the linker is able to
find the new libraries (this is called "linktime"), before installation.
*) Using target paths (absolute only) for -blibpath, the loader is able to
find the installed library (this is called "runtime"), after installation.

> But, the problem is that the dot needs to be in the internal
> libpath while the executable is in the directory it was built in.

*) Using local paths (either relative or not) for LIBPATH, the loader is able to
find the new library (there is no such term "testtime"), before installation.

> The purpose for rt is after the install.

<snipped anything related to 'rt' (relink?), as it should be obsolete now>

> I also had a rather self imposed limitation to work around.
> I am trying to get "rvm" to work and rvm thinks it knows how to
> unpackage, configure, build, and install without any user
> intervention. To get that to work, I had to hook in at the ld
> point and then, after the fact, fix the resulting files after they
> are installed.

This actually is a not so uncommon problem:
It usually does not work that well using more than one package manager
at a time - portage and rvm in this case.

>>> I've used this only for Ruby -- and it was in fact Ruby that caused me to do this.
>>> Their out of the box system does not work at all for AIX if the prefix directory is not the same as GCC's.
>>> WIth this, I can now use rvm on AIX and do "rvm install 1.9.1" and it will complete successfully without my fingers leaving my hands.

Which directory (called "prefix") do you tell "rvm" to install to?
Have you tried this while having your Gentoo Prefix instance in your
environment (using /your/gentoo/prefix/startprefix)?

>> Haven't used/installed Ruby on any platform yet: What is 'rvm' ?
> rvm is "Ruby Version Manager". rvm allows completely independent
> versions of Ruby (1.8.7, 1.9.1, MacRuby, etc). Each version can have
> its own independent sets of gems (called gemsets) so an application
> can use its own version of Ruby and its own set of gems. I can set up
> an environment on my development laptop and then fairly easily replicate
> this environment on my staging server and production server. It solves
> many problems I have when trying to have three or four Rails apps
> on the same host. (sorry for the long winded reply).

Basically this also is what Gentoo Prefix is for - having multiple instances
of similar things set up on one machine. However, eventually it is possible
to have multiple Ruby versions within one instance of Gentoo Prefix (and even
Gentoo Linux) set up in some package-managed way (using portage as the
package manager instead of rvm).

>>> I mentioned this idea on this list before but it didn't fly very well.
>>> I'm not sure if it was completely understood.
>> I fail to find that post you're referring to. Has it been within one
>> of the threads we already created together, and I've overlooked it?
>
> I *think* this is the post but it got munged:
> http://archives.gentoo.org/gentoo-alt/msg_d16e4dce43974e6926eedf04a3b8fb36.xml
>
> My archive is this (snipped out).

Ok, found it in my archives too - for mailing list managers
it is easier to receive plain text mails, no html.

> The time stamp is Tue, Dec 14, 2010 at 6:25 PM (to the gentoo-alt list)
>
>> Also, the guy I'm talking to mentioned rtl_enable -s. I've not used it but it seems like it might make life a lot easier.

Still I do think we don't have any use for 'rtl_enable' when
we are able to /create/ corrent shared libraries firsthand.
rtl_enable is designed to "fix" shared objects for one's needs
when there is no sourcecode available.

>>> This is more of a proof of concept at this stage. I'm going to putter around and try and use it on other packages.
>>
>> For packages not aware of AIX at all:
>> How would this require their build systems to be patched?
>>
>>> "rl" could be enhanced to take "from" and "to" directories.

The above "rt" is a typo and should be "rl"?

>>> If you wanted to change files installed in /some/old/directory and put them into /some/new/directory, rl could easily modified do that.
>>
>> In which context do you need this use case?
>
> With putting absolute paths into the objects, we are effectively nullifying the possible benefit of the LIBPATH environment. I think we
> both agree that that benefit is questionable but others might like it. So if someone wants to move where I built a package from
> /usr/local to /opt, the first step would be to change the internal absolute paths to each dependency. There would be other obstacles too
> like any internal paths of the executables but often there are command line arguments to change those. e.g. you can give
> emacs directions on where to find its lisp files but you can not magically tell the loader where to find emacs' dependent libraries.
> A user once could via LIBPATH but that isn't going to work now with absolute internal paths. rl gives that possibly useful option back
> to the user.

Well, moving a binary package from /usr/local to /opt is a
completely different use case than the one above, which was:
compile, run from build directory, install (into /usr/local).

(replace "/usr/local/" by "/gentoo/prefix/usr/")

>> And how would you do that on Linux?
>
> I don't know squat about Linux.
> Part of my frustration is everyone appears to want to make AIX fit into the Linux way.

The linkpath/runpath problem isn't specific to Linux in any way, it is
an universal problem when managing packages - on /any/ operating system.

I did go through these /very/ same linkpath/runpath hell on Linux already,
and in Prefix we do the same linkpath/runpath trickery on /any/ platform.

Welcome to "rpath hell", "dll hell", "dependency hell", "distribution hell",
"<whatever> hell"!

The only problem AIX really does have is the lack of something like what is known as
"DT_SONAME" in ELF/SystemV-world[2], being immediately encoded into the shared objects.
The reason for this being a problem is not that ELF does have it, it is because
such a mechanism actually is necessary for managing packages.
However, it is possible to emulate the "DT_SONAME" thing with Import Files ...

[2] Linux, Solaris, (recent) HP-UX, *BSD, ...

> My objective is subtly but significantly different.
> I want to get open source packages onto AIX.

Same goal here.
However, Linux (as "the" open source platform these days) is "the"
reference platform open source packages most often are written for.

/haubi/
--
Michael Haubenwallner
Gentoo on a different level
 
Old 02-09-2011, 03:47 PM
Perry Smith
 
Default AIX linking adventure

Lets start over. I don't think I'm making any progress but I'll give it one last go.

Here is the output of dump -H of my installed ruby:

/usr/local/rvm/rubies/ruby-1.9.1-p378/bin/ruby:

***Loader Section***
Loader Header Information
VERSION# #SYMtableENT #RELOCent LENidSTR
0x00000001 0x00000553 0x0000008a 0x00000141

#IMPfilID OFFidSTR LENstrTBL OFFstrTBL
0x00000006 0x00008660 0x00006307 0x000087a1


***Import File Strings***
INDEX PATH BASE MEMBER
0 /usr/local/rvm/rubies/ruby-1.9.1-p378/lib:/usr/local/lib/gcc/powerpc-ibm-aix5.3.0.0/4.3.1:/usr/local/lib/gcc/powerpc-ibm-aix5.3.0.0/4.3.1/../../..:/usr/lib:/lib
1 librtl.a shr.o
2 /usr/lib libc.a shr.o
3 /usr/lib libpthread.a shr_comm.o
4 /usr/lib libpthread.a shr_xpg5.o
5 /usr/local/rvm/rubies/ruby-1.9.1-p378/lib libruby.so

Except for librtl.a, there are no relative paths anywhere. That is one of my objectives (and your's I believe).

It is using /usr/local/rvm/rubies/ruby-1.9.1-p378/lib/libruby.so . If I move that to somewhere else, like

/usr/local/rvm/rubies/ruby-1.9.1-p378/lib/other/libruby.so

ruby does not work and setting LIBPATH does not fix the problem. Thus, for an object (I'll use the word "object" to mean shared object or an executable) that is linked without only absolute paths, it will not work in the directory that it is built in. In the case of ruby, it builds the library and executable in the same pass. I think this is typical. If you plan to run "make test" or "make check" (different packages call it different things) before the install, you have a problem.

The way I solved that problem is to allow the object to have relative paths (or no path) before it is installed, do the "make check", then either fix the object just before install or fix it just after the install. I choose to fix it just after the install with "rl".

I don't see how you can have an object that has only absolute paths after it is installed *and* is able to be used before the install so that "make check" or "make test" can be run (except by some weird chroot tricks but no one wants to go down that path).

Lets pause here and see if we can come to an agreement on these two items.
1) objets have only absolute paths after they are installed
2) The "make check" must work before package is installed.


On Feb 8, 2011, at 4:02 PM, Michael Haubenwallner wrote:

> Hi Perry!
>
> On 02/08/11 19:36, Perry Smith wrote:
>> On Feb 8, 2011, at 2:49 AM, Michael Haubenwallner wrote:
>>> On 02/08/2011 01:35 AM, Perry Smith wrote:
>>>> I wrote a front end to AIX's ld command as well as a relink command I'm calling "rl".
>>> Actually, in Prefix we already do wrap AIX' ld in sys-devel/binutils-config
>>> to get the runpaths (the -blibpath argument) right.
>>> There is an RFC[1] for the way I want to have shared libraries being created on AIX.
>>> [1] http://lists.gnu.org/archive/html/libtool/2011-01/msg00023.html
>>
>> I am not on the libtool list. I did not read that thread yet but I will here shortly.
>
> You might consider subscribing to that Gentoo bug[1], its URL refers
> to the libtool thread, and I'm usually adding progress there.
>
> [1] http://bugs.gentoo.org/show_bug.cgi?id=213277
>
>> I need some "netiquette" help. Should I plop my thoughts / questions about that
>> thread into this thread or start a new thread? I get confused either way :-) (Maybe
>> I'm special that way.)
>
> Eventually we can sort out your concerns here (first).
>
>>>> https://github.com/pedz/aixbin
>>>
>>> Commenting on 'Readme.markdown' found there:
>>> Why do you move '-blibpath' arguments to '-L' arguments?
>>> IMO, the other way round would make more sense actually.
>>> Why do you need 'rl' (relink) at all?
>>
>> For ruby, their libpath was a complete mistake.
>
> Ohw, seems there's need to fix your view on the libpath/runpath issue,
> as everything you describe here actually is how it should be done:
>
> Usually, compilers are installed as part of the host system into somewhere
> like /usr, where /usr/lib usually is the search path for any installed
> library at linktime (called "linkpath", set via "-L" linker flag), and
> either "/usr/lib:/lib" or even "/lib:/usr/lib" is the search path at
> runtime (called "runpath", set via "-blibpath" linker flag).
>
>> They did not include the path to gcc's library.
>
> Nobody but the compiler and/or the system should know the system's and
> compiler's library directories. It is up to the compiler-, linker- and
> loader-setup to find the system's and especially the compiler's libraries
> at both linktime and runtime.
>
> This exactly is the (major) reason why in Prefix we do wrap the linker
> (as well as the compiler) to get gcc's and Prefix' (being the (sub)system)
> library directories into both linkpath and runpath.
>
>> They did not have a -L /path/to/where/they/would/install
>
> The currently built package's libraries must not be linked from the target
> installation directory. Imagine an old version of a library is already
> installed there, and you want to upgrade the package - as is the usual
> case with package managers (being either humans or software).
>
> And if there are dependent libraries, they either are installed in system's
> library directories the compiler/linker setup knows anyway, or the package
> manager should add their locations via something like LDFLAGS or the like.
>
> So while the new binaries must _not_ be /linked/ against the already installed
> libraries provided by another instance of the package just built, ...
>
>> but they did have -blibpath:/path/to/where/they/would/install:/usr/lib:/lib
>
> ..., at runtime (after installing) the libraries have to be /loaded/ from there.
>
> One AIX specific thing here is that the AIX loader cannot be configured to
> search additional directories, so the binaries have to contain the whole runpath.
> While the linker does search /usr/lib and /lib at linktime, it will not add
> these paths to the runpath when an explicit runpath is specified via -blibpath.
> This is why "/usr/lib:/lib" is specified with -blibpath too.
>
>> And... the executable would never work in the directory
>> where it was built in. It would work only after the package was
>> installed (moved). I don't know how they were going to do
>> the "test" or "check" phase which I'm sure Ruby has.
>
> They most likely set the LIBPATH environment variable to the local
> library directory. This is one of the rare use cases where environment
> variables like LD_LIBRARY_PATH, LIBPATH, SHLIB_PATH are useful.
>
>> I took their mistake as a symptom as a general practice. i.e.
>> the general population somehow thinks that libpath substitutes for
>> -L and it does not. I'm interested to hear of your experience but
>> I can't think of a time where open source understood what libpath
>> (inside the object) could/should be used for.
>
> Actually, the Ruby guys seem to have done it "right":
> If you were using xlc (from /usr/bin), or even gcc from the
> "AIX Toolbox for Linux applications", so both are "installed
> with the host system", you might not have had any problem.
>
>> The Ruby guys did have a fairly reasonable fear. They did not want
>> to have just -L . and -L .., etc and not set the libpath because the -L .
>> adds that to the internal libpath and that creates what they called a
>> security risk.
>
> The explicit '-blibpath' avoids recording the (relative) local library
> paths passed via '-L' into the runpath:
>
> *) Using buildtime paths (either relative or not) for -L, the linker is able to
> find the new libraries (this is called "linktime"), before installation.
> *) Using target paths (absolute only) for -blibpath, the loader is able to
> find the installed library (this is called "runtime"), after installation.
>
>> But, the problem is that the dot needs to be in the internal
>> libpath while the executable is in the directory it was built in.
>
> *) Using local paths (either relative or not) for LIBPATH, the loader is able to
> find the new library (there is no such term "testtime"), before installation.
>
>> The purpose for rt is after the install.
>
> <snipped anything related to 'rt' (relink?), as it should be obsolete now>
>
>> I also had a rather self imposed limitation to work around.
>> I am trying to get "rvm" to work and rvm thinks it knows how to
>> unpackage, configure, build, and install without any user
>> intervention. To get that to work, I had to hook in at the ld
>> point and then, after the fact, fix the resulting files after they
>> are installed.
>
> This actually is a not so uncommon problem:
> It usually does not work that well using more than one package manager
> at a time - portage and rvm in this case.
>
>>>> I've used this only for Ruby -- and it was in fact Ruby that caused me to do this.
>>>> Their out of the box system does not work at all for AIX if the prefix directory is not the same as GCC's.
>>>> WIth this, I can now use rvm on AIX and do "rvm install 1.9.1" and it will complete successfully without my fingers leaving my hands.
>
> Which directory (called "prefix") do you tell "rvm" to install to?
> Have you tried this while having your Gentoo Prefix instance in your
> environment (using /your/gentoo/prefix/startprefix)?
>
>>> Haven't used/installed Ruby on any platform yet: What is 'rvm' ?
>> rvm is "Ruby Version Manager". rvm allows completely independent
>> versions of Ruby (1.8.7, 1.9.1, MacRuby, etc). Each version can have
>> its own independent sets of gems (called gemsets) so an application
>> can use its own version of Ruby and its own set of gems. I can set up
>> an environment on my development laptop and then fairly easily replicate
>> this environment on my staging server and production server. It solves
>> many problems I have when trying to have three or four Rails apps
>> on the same host. (sorry for the long winded reply).
>
> Basically this also is what Gentoo Prefix is for - having multiple instances
> of similar things set up on one machine. However, eventually it is possible
> to have multiple Ruby versions within one instance of Gentoo Prefix (and even
> Gentoo Linux) set up in some package-managed way (using portage as the
> package manager instead of rvm).
>
>>>> I mentioned this idea on this list before but it didn't fly very well.
>>>> I'm not sure if it was completely understood.
>>> I fail to find that post you're referring to. Has it been within one
>>> of the threads we already created together, and I've overlooked it?
>>
>> I *think* this is the post but it got munged:
>> http://archives.gentoo.org/gentoo-alt/msg_d16e4dce43974e6926eedf04a3b8fb36.xml
>>
>> My archive is this (snipped out).
>
> Ok, found it in my archives too - for mailing list managers
> it is easier to receive plain text mails, no html.
>
>> The time stamp is Tue, Dec 14, 2010 at 6:25 PM (to the gentoo-alt list)
>>
>>> Also, the guy I'm talking to mentioned rtl_enable -s. I've not used it but it seems like it might make life a lot easier.
>
> Still I do think we don't have any use for 'rtl_enable' when
> we are able to /create/ corrent shared libraries firsthand.
> rtl_enable is designed to "fix" shared objects for one's needs
> when there is no sourcecode available.
>
>>>> This is more of a proof of concept at this stage. I'm going to putter around and try and use it on other packages.
>>>
>>> For packages not aware of AIX at all:
>>> How would this require their build systems to be patched?
>>>
>>>> "rl" could be enhanced to take "from" and "to" directories.
>
> The above "rt" is a typo and should be "rl"?
>
>>>> If you wanted to change files installed in /some/old/directory and put them into /some/new/directory, rl could easily modified do that.
>>>
>>> In which context do you need this use case?
>>
>> With putting absolute paths into the objects, we are effectively nullifying the possible benefit of the LIBPATH environment. I think we
>> both agree that that benefit is questionable but others might like it. So if someone wants to move where I built a package from
>> /usr/local to /opt, the first step would be to change the internal absolute paths to each dependency. There would be other obstacles too
>> like any internal paths of the executables but often there are command line arguments to change those. e.g. you can give
>> emacs directions on where to find its lisp files but you can not magically tell the loader where to find emacs' dependent libraries.
>> A user once could via LIBPATH but that isn't going to work now with absolute internal paths. rl gives that possibly useful option back
>> to the user.
>
> Well, moving a binary package from /usr/local to /opt is a
> completely different use case than the one above, which was:
> compile, run from build directory, install (into /usr/local).
>
> (replace "/usr/local/" by "/gentoo/prefix/usr/")
>
>>> And how would you do that on Linux?
>>
>> I don't know squat about Linux.
>> Part of my frustration is everyone appears to want to make AIX fit into the Linux way.
>
> The linkpath/runpath problem isn't specific to Linux in any way, it is
> an universal problem when managing packages - on /any/ operating system.
>
> I did go through these /very/ same linkpath/runpath hell on Linux already,
> and in Prefix we do the same linkpath/runpath trickery on /any/ platform.
>
> Welcome to "rpath hell", "dll hell", "dependency hell", "distribution hell",
> "<whatever> hell"!
>
> The only problem AIX really does have is the lack of something like what is known as
> "DT_SONAME" in ELF/SystemV-world[2], being immediately encoded into the shared objects.
> The reason for this being a problem is not that ELF does have it, it is because
> such a mechanism actually is necessary for managing packages.
> However, it is possible to emulate the "DT_SONAME" thing with Import Files ...
>
> [2] Linux, Solaris, (recent) HP-UX, *BSD, ...
>
>> My objective is subtly but significantly different.
>> I want to get open source packages onto AIX.
>
> Same goal here.
> However, Linux (as "the" open source platform these days) is "the"
> reference platform open source packages most often are written for.
>
> /haubi/
> --
> Michael Haubenwallner
> Gentoo on a different level
>
 
Old 02-09-2011, 09:33 PM
Michael Haubenwallner
 
Default AIX linking adventure

Hi Perry,

On 02/09/11 17:47, Perry Smith wrote:
> Lets start over. I don't think I'm making any progress but I'll give it one last go.
>
> Here is the output of dump -H of my installed ruby:
>
> /usr/local/rvm/rubies/ruby-1.9.1-p378/bin/ruby:
> ***Import File Strings***
> INDEX PATH BASE MEMBER
> 0 /usr/local/rvm/rubies/ruby-1.9.1-p378/lib:/usr/local/lib/gcc/powerpc-ibm-aix5.3.0.0/4.3.1:/usr/local/lib/gcc/powerpc-ibm-aix5.3.0.0/4.3.1/../../..:/usr/lib:/lib
> 1 librtl.a shr.o
> 2 /usr/lib libc.a shr.o
> 3 /usr/lib libpthread.a shr_comm.o
> 4 /usr/lib libpthread.a shr_xpg5.o
> 5 /usr/local/rvm/rubies/ruby-1.9.1-p378/lib libruby.so

Is this with our without your intervention in ruby's build process and/or toolchain setup?

> Except for librtl.a, there are no relative paths anywhere. That is one of my objectives (and your's I believe).

There is no "relative" path by the usual means of "relative" /anywhere/ here,
there are only "absolute" paths here - a "relative" path does not start with "/".
However, librtl.a is subject to "runpath search", while the others are not
due to "hardcoded" (unsure about this term) paths - but still "absolute"
paths everywhere.

Anyway - taking "relative" as "with runpath search", and "absolute" as
"without runpath search" or even "hardcoded" here now.

> It is using /usr/local/rvm/rubies/ruby-1.9.1-p378/lib/libruby.so .
> If I move that to somewhere else, like
> /usr/local/rvm/rubies/ruby-1.9.1-p378/lib/other/libruby.so
> ruby does not work and setting LIBPATH does not fix the problem.

Of course, due to the hardcoded paths no runpath search is performed.

> Thus, for an object (I'll use the word "object" to mean shared object or an executable)

Usually, "binary" is the term to be used here, but ok...

> that is linked without only absolute paths, it will not work in the directory that it is built in.

Let me grasp that: "without only absolute" paths "it will not work" where built.
Isn't this a typo and should read "*with* only absolute" ?

> In the case of ruby, it builds the library and executable in the same pass.
> I think this is typical.

Yep.

> If you plan to run "make test" or "make check" (different packages call it different things)
> before the install, you have a problem.

Only if the paths are hardcoded - but this is unusual.

> The way I solved that problem is to allow the object to have relative paths (or no path)
> before it is installed, do the "make check",

How exactly did you touch either the build process or the toolchain setup to "allow" that?

> then either fix the object just before install or fix it just after the install.
> I choose to fix it just after the install with "rl".

Same here: How to touch the build process or the toolchain setup to get "rl" used?

> I don't see how you can have an object that has only absolute paths after it is installed
> *and* is able to be used before the install so that "make check" or "make test" can be run
> (except by some weird chroot tricks but no one wants to go down that path).

Yep, that doesn't work.

> Lets pause here and see if we can come to an agreement on these two items.
> 1) objets have only absolute paths after they are installed

Still not clear to me:
Do you actually /prefer/ hardcoded paths (without runpath search)? Why?

Although runpath search is affected by LIBPATH, I'm just fine with that.
This is the outcome of using "-L" and "-l" linker flags together.

It is unusual to use hardcoded path (without runpath search), which is the
result of using /absolute/path/to/the/library.so instead of "-L" and "-l".

So when ruby's build system uses /absolute/path/to/the/library.so, you should
try to fix that to use "-L" and "-l", as well as "-blibpath" eventually instead.

> 2) The "make check" must work before package is installed.

Yep, agreed on this one.

Seems we have to agree on some terms to be used first.

What else to say:
When libtool knows for a platform that linking libraries results in hardcoded
paths, or there is no such thing like LIBPATH environment variable to temporarily
override the runpath, it actually /does/ "relink" the binaries during installation.
OTOH, when the /absolute/path/to/the/library.so found at linktime is hardcoded into
the resulting binary, installing into DESTDIR is broken - which is a prerequisite
for package managers like portage to work properly.

/haubi/
--
Michael Haubenwallner
Gentoo on a different level
 
Old 02-09-2011, 11:57 PM
Perry Smith
 
Default AIX linking adventure

On Feb 9, 2011, at 4:33 PM, Michael Haubenwallner wrote:Hi Perry,

On 02/09/11 17:47, Perry Smith wrote:
Lets start over. *I don't think I'm making any progress but I'll give it one last go.

Here is the output of dump -H of my installed ruby:

/usr/local/rvm/rubies/ruby-1.9.1-p378/bin/ruby:
**************************Import File Strings***
INDEX *PATH *************************BASE ***************MEMBER *************
0 *****/usr/local/rvm/rubies/ruby-1.9.1-p378/lib:/usr/local/lib/gcc/powerpc-ibm-aix5.3.0.0/4.3.1:/usr/local/lib/gcc/powerpc-ibm-aix5.3.0.0/4.3.1/../../..:/usr/lib:/lib ****************************************
1 ***********************************librtl.a ***********shr.o **************
2 *****/usr/lib *********************libc.a *************shr.o **************
3 *****/usr/lib *********************libpthread.a *******shr_comm.o *********
4 *****/usr/lib *********************libpthread.a *******shr_xpg5.o *********
5 *****/usr/local/rvm/rubies/ruby-1.9.1-p378/lib libruby.so *****************************

Is this with our without your intervention in ruby's build process and/or toolchain setup?

This is after my toolchain changes.

Except for librtl.a, there are no relative paths anywhere. *That is one of my objectives (and your's I believe).

There is no "relative" path by the usual means of "relative" /anywhere/ here,
there are only "absolute" paths here - a "relative" path does not start with "/".
However, librtl.a is subject to "runpath search", while the others are not
due to "hardcoded" (unsure about this term) paths - but still "absolute"
paths everywhere.

Anyway - taking "relative" as "with runpath search", and "absolute" as
"without runpath search" or even "hardcoded" here now.

Yea. *I need to use better terms. *but I think you are following me. *I'musing the same term to mean two different things.
In the 0'th entry above, there are no entries like "." (so in that case, that term "relative path"or "absolute path" is*correct.)
The 1st entry is what I was calling "relative" but as you point out, that isn't the right term."affected by run path search" is too long for me. :-)
The 2nd through 5th items are those I was calling "absolute" but really they should becalled "not affected by run path search". *Perhaps "fixed" in this case would work? *non-LIBPATH-able?

It is using /usr/local/rvm/rubies/ruby-1.9.1-p378/lib/libruby.so .
If I move that to somewhere else, like
/usr/local/rvm/rubies/ruby-1.9.1-p378/lib/other/libruby.so
ruby does not work and setting LIBPATH does not fix the problem.

Of course, due to the hardcoded paths no runpath search is performed.

Thus, for an object (I'll use the word "object" to mean shared object or an executable)

Usually, "binary" is the term to be used here, but ok...

that is linked without only absolute paths, it will not work in the directory that it is built in.

Let me grasp that: "without only absolute" paths "it will not work" where built.
Isn't this a typo and should read "*with* only absolute" ?

Yes. *typo. *sorry. *I really do go back and proof read my emails.

In the case of ruby, it builds the library and executable in the same pass.
I think this is typical.

Yep.

If you plan to run "make test" or "make check" (different packages call it different things)
before the install, you have a problem.

Only if the paths are hardcoded - but this is unusual.

The way I solved that problem is to allow the object to have relative paths (or no path)
before it is installed, do the "make check",

How exactly did you touch either the build process or the toolchain setup to "allow" that?

I put my ld in /usr/local/aixbin/ld and I put /usr/local/aixbin as the first element of PATH.So my ld gets called instead of the real ld. *My ld monkeys with things.
After everything is installed, I go back through with "rl". *I did this by hand at the present time.In the limited case that I'm experimenting with, everything is installed off in a directory thatis just the new ruby install. *I can do a find for the binary files and then call rl on them and itwill make the changes.

then either fix the object just before install or fix it just after the install.
I choose to fix it just after the install with "rl".

Same here: How to touch the build process or the toolchain setup to get "rl" used?

See above.

I don't see how you can have an object that has only absolute paths after it is installed
*and* is able to be used before the install so that "make check" or "make test" can be run
(except by some weird chroot tricks but no one wants to go down that path).

Yep, that doesn't work.

Lets pause here and see if we can come to an agreement on these two items.
1) objets have only absolute paths after they are installed

Still not clear to me:
Do you actually /prefer/ hardcoded paths (without runpath search)? Why?

Although runpath search is affected by LIBPATH, I'm just fine with that.
This is the outcome of using "-L" and "-l" linker flags together.

I thought this was one of your objectives based on the fact that java sets LIBPATHto something weird and it breaks everything.

It is unusual to use hardcoded path (without runpath search), which is the
result of using /absolute/path/to/the/library.so instead of "-L" and "-l".

So when ruby's build system uses /absolute/path/to/the/library.so, you should
try to fix that to use "-L" and "-l", as well as "-blibpath" eventually instead.

2) The "make check" must work before package is installed.

Yep, agreed on this one.

Seems we have to agree on some terms to be used first.

Yes. *
I like "fixed" for entries not affected by runtime search.
The other tact would*be to note that PATH is either empty or not.
"A path dependency with no path" or "a dependency with a path" ?
"binary" instead of "object" sounds better to me.

What else to say:
When libtool knows for a platform that linking libraries results in hardcoded
paths, or there is no such thing like LIBPATH environment variable to temporarily
override the runpath, it actually /does/ "relink" the binaries during installation.

I understood that part -- but
OTOH, when the /absolute/path/to/the/library.so found at linktime is hardcoded into
the resulting binary, installing into DESTDIR is broken - which is a prerequisite
for package managers like portage to work properly.

Didn't understand this part. *In particular, from the comma to the end ", installing ..."

/haubi/
--
Michael Haubenwallner
Gentoo on a different level
 
Old 02-10-2011, 08:06 AM
Michael Haubenwallner
 
Default AIX linking adventure

On 02/10/11 01:57, Perry Smith wrote:
>> On 02/09/11 17:47, Perry Smith wrote:
>>> Lets start over. I don't think I'm making any progress but I'll give it one last go.
>>> ***Import File Strings***
>>> INDEX PATH BASE MEMBER
>>> 0 /usr/local/rvm/rubies/ruby-1.9.1-p378/lib:/usr/local/lib/gcc/powerpc-ibm-aix5.3.0.0/4.3.1:/usr/local/lib/gcc/powerpc-ibm-aix5.3.0.0/4.3.1/../../..:/usr/lib:/lib
>>> 1 librtl.a shr.o
>>> 2 /usr/lib libc.a shr.o
>>> 3 /usr/lib libpthread.a shr_comm.o
>>> 4 /usr/lib libpthread.a shr_xpg5.o
>>> 5 /usr/local/rvm/rubies/ruby-1.9.1-p378/lib libruby.so
>> Is this with our without your intervention in ruby's build process and/or toolchain setup?
>
> This is after my toolchain changes.

How does this look like without your changes?

> The 1st entry is what I was calling "relative" but as you point out, that isn't the right term.
> "affected by run path search" is too long for me. :-)
>
> The 2nd through 5th items are those I was calling "absolute" but really they should be
> called "not affected by run path search". Perhaps "fixed" in this case would work?
> non-LIBPATH-able?

IIRC, "hardcoded" is what libtool uses for that kind of paths,
but I'm fine with "fixed" too.

>> How exactly did you touch either the build process or the toolchain setup to "allow" that?
>
> I put my ld in /usr/local/aixbin/ld and I put /usr/local/aixbin as the first element of PATH.
> So my ld gets called instead of the real ld. My ld monkeys with things.

Which compiler did you use - gcc or xlc?
Does xlc search for 'ld' using PATH?
Usually, gcc does not, you'll have to configure it to use "/usr/local/aixbin/ld".

>>> Lets pause here and see if we can come to an agreement on these two items.
>>> 1) objets have only absolute paths after they are installed
>>
>> Still not clear to me:
>> Do you actually /prefer/ hardcoded paths (without runpath search)? Why?
>>
>> Although runpath search is affected by LIBPATH, I'm just fine with that.
>> This is the outcome of using "-L" and "-l" linker flags together.
>
> I thought this was one of your objectives based on the fact that java sets LIBPATH
> to something weird and it breaks everything.

Nope. The problem with LIBPATH set by java was that there is nothing
like "soname". Iff my binary had searched for the /file/ "libiconv.so.2"
(either archive or not) instead of "libiconv.a(libiconv.so.2)", even with
LIBPATH set to "/usr/lib" it would not have found "/usr/lib/libiconv.a"
(without "libiconv.so.2" member), but continued along LIBPATH + runpath
until it found the "libiconv.so.2" file in its usual place.

>> Seems we have to agree on some terms to be used first.
>
> Yes.
>
> I like "fixed" for entries not affected by runtime search.

"fixed" = "hardcoded" (see above).

> The other tact would be to note that PATH is either empty or not.
>
> "A path dependency with no path" or "a dependency with a path" ?

Which "PATH" ("path") do you refer to here that can be either empty or not?
There is the "PATH" environment variable, which does not affect runpath search at all.
There is the "LIBPATH" environment variable, which does.
There is the encoded "hardcoded path" (="fixed") for one library, not allowing runpath search.
There is the encoded "runpath", used (after LIBPATH) for any library without a "hardcoded path".

> "binary" instead of "object" sounds better to me.

ok.

>> What else to say:
>> When libtool knows for a platform that linking libraries results in hardcoded
>> paths, or there is no such thing like LIBPATH environment variable to temporarily
>> override the runpath, it actually /does/ "relink" the binaries during installation.
>
> I understood that part -- but
>
>> OTOH, when the /absolute/path/to/the/library.so found at linktime is hardcoded into
>> the resulting binary, installing into DESTDIR is broken - which is a prerequisite
>> for package managers like portage to work properly.
>
> Didn't understand this part. In particular, from the comma to the end ", installing ..."

Ohw, ok - this is the core requirement for proper package managing:

Consider these raw build steps for an usual package, which does provide
"${prefix}/lib/library.so", and "${prefix}/bin/mybinary" linked against it:

1. Configure:
$ ./configure --prefix="/final"
2. Compile:
$ make
3. Install (to allow for a binary package):
$ make install DESTDIR="/image"
The important thing here is to /not/ copy anything to "/final" at the
build machine, but still the binaries copied to "/image/final" have
to be able to run when moved to "/final" by the package manager,
eventually on another machine with (ideally) identical setup.
4. Create the binary package:
$ cd "/image"
$ find "final" > "filelist-this-package-does-install"
$ tar cfj "binpackage.tar.bz2" "filelist-this-package-does-install" "final"
5. optionally ship the binary package to another host with (ideally) identical setup.
6. See if the binary package would fit:
$ tar xfj "binpackage.tar.bz2" "filelist-this-package-does-install"
$ for f in $(<"filelist-this-package-does-install")
do
if [[ -e /${f} ]]
then
do-we-update "${f}" && continue
echo "Installing this package conflicts with existing '/${f}'"
fi
done
7. Merge the binary package to the live filesystem:
$ cd "/"
$ tar xfj "binpackage.tar.bz2" "final"
8. Use the package:
$ "/final/bin/mybinary"

During step 3, the shared library gets installed as "/staging/final/lib/library.so".
Relinking "mybinary", the linker has to read that file, while encoding "/final/lib"
(without "/staging") into "mybinary" as the search path for "library.so".

Usually that is done this way, encoding the final "runpath":
$ ld -o "/staging/final/bin/mybinary" -L "/staging/final/lib" -lrary -blibpath:"/final/lib"

Because this won't work as expected, as the "hardcoded" path would contain "/staging":
$ ld -o "/staging/final/bin/mybinary" "/staging/final/lib/library.so"

Except... when an "Import File" is found at "library.so", referring to "#! /final/lib/...".

However, when "library.so" is a /standalone/ Import File,
it has to refer to something like "#! /final/lib/library.so.0",
so the loader actually can load a shared object rather than an Import File.

When "library.so" is an archive file /containing/ both the Import File "shr.imp" and the shared object "shr.o",
the Import File has to refer to "#! /final/lib/library.so(shr.o)".

For completion, "library.so.0" is called the "soname" in ELF world,
but that's a different (although related) topic.

/haubi/
--
Michael Haubenwallner
Gentoo on a different level
 
Old 02-10-2011, 01:14 PM
Perry Smith
 
Default AIX linking adventure

On Feb 10, 2011, at 3:06 AM, Michael Haubenwallner wrote:

>
> On 02/10/11 01:57, Perry Smith wrote:
>>> On 02/09/11 17:47, Perry Smith wrote:
>>>> Lets start over. I don't think I'm making any progress but I'll give it one last go.
>>>> ***Import File Strings***
>>>> INDEX PATH BASE MEMBER
>>>> 0 /usr/local/rvm/rubies/ruby-1.9.1-p378/lib:/usr/local/lib/gcc/powerpc-ibm-aix5.3.0.0/4.3.1:/usr/local/lib/gcc/powerpc-ibm-aix5.3.0.0/4.3.1/../../..:/usr/lib:/lib
>>>> 1 librtl.a shr.o
>>>> 2 /usr/lib libc.a shr.o
>>>> 3 /usr/lib libpthread.a shr_comm.o
>>>> 4 /usr/lib libpthread.a shr_xpg5.o
>>>> 5 /usr/local/rvm/rubies/ruby-1.9.1-p378/lib libruby.so
>>> Is this with our without your intervention in ruby's build process and/or toolchain setup?
>>
>> This is after my toolchain changes.
>
> How does this look like without your changes?

I can't recall 100%. I can go back and do another build. In particular, I can't recall
who had the problem if it was ruby itself or libruby.so that had the problem. In general,
there were no "hardcoded" elements and the 0th element was simply:

/usr/local/rvm/rubies/ruby-1.9.1-p378/lib:/usr/lib:/lib

Let me know if that is sufficient. If not, I can do another build without my mod's.

>
>> The 1st entry is what I was calling "relative" but as you point out, that isn't the right term.
>> "affected by run path search" is too long for me. :-)
>>
>> The 2nd through 5th items are those I was calling "absolute" but really they should be
>> called "not affected by run path search". Perhaps "fixed" in this case would work?
>> non-LIBPATH-able?
>
> IIRC, "hardcoded" is what libtool uses for that kind of paths,
> but I'm fine with "fixed" too.

Lets use existing terms if they exist.
hardcoded isn't too long. what is the opposite? non-hardcoded?

>
>>> How exactly did you touch either the build process or the toolchain setup to "allow" that?
>>
>> I put my ld in /usr/local/aixbin/ld and I put /usr/local/aixbin as the first element of PATH.
>> So my ld gets called instead of the real ld. My ld monkeys with things.
>
> Which compiler did you use - gcc or xlc?

gcc 4.3.1

> Does xlc search for 'ld' using PATH?

Good question. I don't know.

> Usually, gcc does not, you'll have to configure it to use "/usr/local/aixbin/ld".

collect2 (from gcc) calls ld. I'm using my "stock" gcc build and its working for me.

>
>>>> Lets pause here and see if we can come to an agreement on these two items.
>>>> 1) objets have only absolute paths after they are installed
>>>
>>> Still not clear to me:
>>> Do you actually /prefer/ hardcoded paths (without runpath search)? Why?
>>>
>>> Although runpath search is affected by LIBPATH, I'm just fine with that.
>>> This is the outcome of using "-L" and "-l" linker flags together.
>>
>> I thought this was one of your objectives based on the fact that java sets LIBPATH
>> to something weird and it breaks everything.
>
> Nope. The problem with LIBPATH set by java was that there is nothing
> like "soname". Iff my binary had searched for the /file/ "libiconv.so.2"
> (either archive or not) instead of "libiconv.a(libiconv.so.2)", even with
> LIBPATH set to "/usr/lib" it would not have found "/usr/lib/libiconv.a"
> (without "libiconv.so.2" member), but continued along LIBPATH + runpath
> until it found the "libiconv.so.2" file in its usual place.

Hmmm... ok. I had forgotten that detail.

>
>>> Seems we have to agree on some terms to be used first.
>>
>> Yes.
>>
>> I like "fixed" for entries not affected by runtime search.
>
> "fixed" = "hardcoded" (see above).
>
>> The other tact would be to note that PATH is either empty or not.
>>
>> "A path dependency with no path" or "a dependency with a path" ?
>
> Which "PATH" ("path") do you refer to here that can be either empty or not?

In the dump -H output above, there is a column called PATH. That PATH. I apologize for being
so consistently vague.

> There is the "PATH" environment variable, which does not affect runpath search at all.
> There is the "LIBPATH" environment variable, which does.
> There is the encoded "hardcoded path" (="fixed") for one library, not allowing runpath search.
> There is the encoded "runpath", used (after LIBPATH) for any library without a "hardcoded path".
>
>> "binary" instead of "object" sounds better to me.
>
> ok.
>
>>> What else to say:
>>> When libtool knows for a platform that linking libraries results in hardcoded
>>> paths, or there is no such thing like LIBPATH environment variable to temporarily
>>> override the runpath, it actually /does/ "relink" the binaries during installation.
>>
>> I understood that part -- but
>>
>>> OTOH, when the /absolute/path/to/the/library.so found at linktime is hardcoded into
>>> the resulting binary, installing into DESTDIR is broken - which is a prerequisite
>>> for package managers like portage to work properly.
>>
>> Didn't understand this part. In particular, from the comma to the end ", installing ..."
>
> Ohw, ok - this is the core requirement for proper package managing:
>
> Consider these raw build steps for an usual package, which does provide
> "${prefix}/lib/library.so", and "${prefix}/bin/mybinary" linked against it:
>
> 1. Configure:
> $ ./configure --prefix="/final"
> 2. Compile:
> $ make
> 3. Install (to allow for a binary package):
> $ make install DESTDIR="/image"
> The important thing here is to /not/ copy anything to "/final" at the
> build machine, but still the binaries copied to "/image/final" have
> to be able to run when moved to "/final" by the package manager,
> eventually on another machine with (ideally) identical setup.

Does the package (need to) work from the /image area or is this
just a temporary place used to create the tar file? In particular,
what about paths hard coded into the binaries? (Not the
paths in the binary's header but just paths hard coded in the code
like where to find my lisp files, etc. I know that *usually* there are
options to change them but there is still some default set at compile
time.

> 4. Create the binary package:
> $ cd "/image"
> $ find "final" > "filelist-this-package-does-install"
> $ tar cfj "binpackage.tar.bz2" "filelist-this-package-does-install" "final"
> 5. optionally ship the binary package to another host with (ideally) identical setup.
> 6. See if the binary package would fit:
> $ tar xfj "binpackage.tar.bz2" "filelist-this-package-does-install"
> $ for f in $(<"filelist-this-package-does-install")
> do
> if [[ -e /${f} ]]
> then
> do-we-update "${f}" && continue
> echo "Installing this package conflicts with existing '/${f}'"
> fi
> done
> 7. Merge the binary package to the live filesystem:
> $ cd "/"
> $ tar xfj "binpackage.tar.bz2" "final"
> 8. Use the package:
> $ "/final/bin/mybinary"
>
> During step 3, the shared library gets installed as "/staging/final/lib/library.so".
> Relinking "mybinary", the linker has to read that file, while encoding "/final/lib"
> (without "/staging") into "mybinary" as the search path for "library.so".

Shoot... you lost me. I think maybe /staging is the same as /image above?

>
> Usually that is done this way, encoding the final "runpath":
> $ ld -o "/staging/final/bin/mybinary" -L "/staging/final/lib" -lrary -blibpath:"/final/lib"
>
> Because this won't work as expected, as the "hardcoded" path would contain "/staging":
> $ ld -o "/staging/final/bin/mybinary" "/staging/final/lib/library.so"
>
> Except... when an "Import File" is found at "library.so", referring to "#! /final/lib/...".
>
> However, when "library.so" is a /standalone/ Import File,
> it has to refer to something like "#! /final/lib/library.so.0",
> so the loader actually can load a shared object rather than an Import File.
>
> When "library.so" is an archive file /containing/ both the Import File "shr.imp" and the shared object "shr.o",
> the Import File has to refer to "#! /final/lib/library.so(shr.o)".
>
> For completion, "library.so.0" is called the "soname" in ELF world,
> but that's a different (although related) topic.

Ok. I have more thoughts on this too. Lets not go there just yet. I'm still
lagging just a little.

One question: with each of your experiments, do you dump out the
binary's "header" with dump -H? The reason I ask is I would expect
linking with an import file basically just put that piece, the #!<whatever>
into the binary's header at link time and then what happens at run
time is the same as if you had linked against a library.

I remember toying with the import file inside the archive (and getting
frustrated at the lack of documentation) but I can't remember what
I found.
 
Old 02-10-2011, 04:26 PM
Michael Haubenwallner
 
Default AIX linking adventure

On 02/10/11 15:14, Perry Smith wrote:
>>>>> ***Import File Strings***
>>>>> INDEX PATH BASE MEMBER
>>>>> 0 /usr/local/rvm/rubies/ruby-1.9.1-p378/lib:/usr/local/lib/gcc/powerpc-ibm-aix5.3.0.0/4.3.1:/usr/local/lib/gcc/powerpc-ibm-aix5.3.0.0/4.3.1/../../..:/usr/lib:/lib
>>>>> 1 librtl.a shr.o
>>>>> 2 /usr/lib libc.a shr.o
>>>>> 3 /usr/lib libpthread.a shr_comm.o
>>>>> 4 /usr/lib libpthread.a shr_xpg5.o
>>>>> 5 /usr/local/rvm/rubies/ruby-1.9.1-p378/lib libruby.so
>>>> Is this with our without your intervention in ruby's build process and/or toolchain setup?
>>> This is after my toolchain changes.
>> How does this look like without your changes?
>
> In general, there were no "hardcoded" elements and the 0th element was simply:
>
> /usr/local/rvm/rubies/ruby-1.9.1-p378/lib:/usr/lib:/lib
>
> Let me know if that is sufficient. If not, I can do another build without my mod's.

Yes, this looks just fine IMO:
Each shared object should be found in a "dynamic" path, using "runpath" at Index 0.

>> IIRC, "hardcoded" is what libtool uses for that kind of paths,
>> but I'm fine with "fixed" too.
>
> Lets use existing terms if they exist.
> hardcoded isn't too long. what is the opposite? non-hardcoded?

IIRC a shared object being subject to "runpath" search does have a "dynamic" path.

>>> I put my ld in /usr/local/aixbin/ld and I put /usr/local/aixbin as the first element of PATH.
>>> So my ld gets called instead of the real ld. My ld monkeys with things.
>> Which compiler did you use - gcc or xlc?
> gcc 4.3.1
>> Does xlc search for 'ld' using PATH?
> Good question. I don't know.

For each patch designed to be submitted upstream, the xlc case has to be kept in mind.

>> Usually, gcc does not, you'll have to configure it to use "/usr/local/aixbin/ld".
>
> collect2 (from gcc) calls ld. I'm using my "stock" gcc build and its working for me.

I understand you've built gcc-4.3.1 yourself without any patches,
not configuring '--with-ld', right?
Actually I did have problems[1] to bootstrap Prefix on Fedora/RedHat, where
the host-gcc also does search for "ld" using ${PATH} environment variable.

[1] http://prefix-launcher.svn.sourceforge.net/viewvc/prefix-launcher?view=revision&revision=351

>>>> Although runpath search is affected by LIBPATH, I'm just fine with that.
>>>> This is the outcome of using "-L" and "-l" linker flags together.
>>> I thought this was one of your objectives based on the fact that java sets LIBPATH
>>> to something weird and it breaks everything.
>> Nope. The problem with LIBPATH set by java was that there is nothing
>> like "soname". Iff my binary had searched for the /file/ "libiconv.so.2"
>> (either archive or not) instead of "libiconv.a(libiconv.so.2)", even with
>> LIBPATH set to "/usr/lib" it would not have found "/usr/lib/libiconv.a"
>> (without "libiconv.so.2" member), but continued along LIBPATH + runpath
>> until it found the "libiconv.so.2" file in its usual place.
> Hmmm... ok. I had forgotten that detail.

Yes, a detail - but a really important one for package managers.

>>>> Seems we have to agree on some terms to be used first.
>>> Yes.
>>> I like "fixed" for entries not affected by runtime search.
>> "fixed" = "hardcoded" (see above).
>>> The other tact would be to note that PATH is either empty or not.
>>> "A path dependency with no path" or "a dependency with a path" ?
>> Which "PATH" ("path") do you refer to here that can be either empty or not?
>
> In the dump -H output above, there is a column called PATH. That PATH. I apologize for being
> so consistently vague.

Ah, ok. But what is a "path dependency with no path" ?

>>>> OTOH, when the /absolute/path/to/the/library.so found at linktime is hardcoded into
>>>> the resulting binary, installing into DESTDIR is broken - which is a prerequisite
>>>> for package managers like portage to work properly.
> Does the package (need to) work from the /image area or is this
> just a temporary place used to create the tar file? In particular,
> what about paths hard coded into the binaries? (Not the
> paths in the binary's header but just paths hard coded in the code
> like where to find my lisp files, etc. I know that *usually* there are
> options to change them but there is still some default set at compile
> time.

Nope, there is no need for anything to work from the /image area.
This is just a temporary location for the package manager to
identify/track/ship the files a package does install.

>> $ cd "/image"
>> During step 3, the shared library gets installed as "/staging/final/lib/library.so".
>> Relinking "mybinary", the linker has to read that file, while encoding "/final/lib"
>> (without "/staging") into "mybinary" as the search path for "library.so".
>
> Shoot... you lost me. I think maybe /staging is the same as /image above?

Yes, "/staging" is the same as "/image" - apparently I've missed
a few when replacing "/staging" by "/image". Sorry for confusion.
But that's not the reason you got lost, is it?

> One question: with each of your experiments, do you dump out the
> binary's "header" with dump -H?

Yes, at least usually.

> The reason I ask is I would expect
> linking with an import file basically just put that piece, the #!<whatever>
> into the binary's header at link time and then what happens at run
> time is the same as if you had linked against a library.

Basically yes.
The difference is that the values from "#! PATH/BASE(MEMBER)" are stored as-is
into the binary's "Import File Strings" (as PATH BASE MEMBER), independent of
how that Import File was specified on the linker's commandline, while for real
shared objects it does make a difference how to specify them there.

> I remember toying with the import file inside the archive (and getting
> frustrated at the lack of documentation) but I can't remember what
> I found.

Agreed, even with the ld(1) manpage it is quite hard to get an idea of how to do
it right. It took me years(!) of playing around/staring at that manpage/reading
online docs and howto's/sleepless nights, until I figured out how I want to do
"shared libraries" with "soname" (AIX does not have these terms) here.

/haubi/
--
Michael Haubenwallner
Gentoo on a different level
 

Thread Tools




All times are GMT. The time now is 11:45 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org