Compilers, linkers, 64-bits and multilib
 

Per subject, let's open a can full of worms.

For the last few weeks I've been working on 64-bits non-Linux targets.
sparc64-solaris at first appeared to be a piece of cake,
powerpc64-apple-darwin8 and x86_64-apple-darwin9 tricky because of
python, openssl, and most annoying odcctools not being 64-bits
safe-code. With x86_64-pc-solaris2.10 I've almost reached the end (5
packages pending, mostly trivial) but on the way fixing bugs in
packages, I found some horrible scenario forced upon us. Let's start
off that x86_64-pc-solaris2.10 needed patches to both gcc and binutils
because both do not recognise it as a valid target. By relatively
simple patches they seem to work reasonably ok, though.

Now the real problem is in the library search paths. When compiling
wget on Solaris 10/x86_64 the linker started to complain that there were
undefined symbols: intl_gettext. Shame on me, as I ignored those on
sparc64-sun-solaris2.10 (you only get them with USE="nls").
The message is weird, and doesn't appear on i386-pc-solaris2.10.
Further investigation shows that the linker pulls in
/lib/64/libintl.so.1 (or it's realpath
/lib/{amd64,sparcv9}/libintl.so.1) instead of libintl.so.8 as we have in
prefix. Linking with -lgettextlib results in a warning that conflicts
may arise since two libintl.so's are being included, as expected.
(gettextlib was probably because of libtool correctly linked against the
prefix libintl.so.8).

Further investigations (gcc .... -v -Wl,-t) revealed that gcc calls
collect2 nicely like ".../collect2 -L/usr/lib/amd64 -L/lib/amd64 ...",
which -- indeed -- pushes the host "64-bits libs" in front of any other
-L direction onto the linker. I considered that to be no good, and
quite explanatory for wget linking to /lib/amd64/libintl.so.1 by
default. After some searching, the MULTILIB_OSDIRNAMES variable
appeared to be causing this. For the m32 targets of gcc on Solaris it's
set to ".", for the m64 targets it's set to either amd64 or sparcv9.
Since "gcc -m32 .... -v -Wl,-t" revealed no -L instructions for /usr/lib
and /lib being passed onto collect2, I locally patched my GCC to have
also in the 64-bits case MULTILIB_OSDIRNAMES to be ".". This is where
the fun starts.

Under Linux, where I've always happily assumed everything is ok, gcc
injects -L/usr/lib/../lib64 and -L/lib/../lib64. This fits nicely with
MULTILIB_OSDIRNAMES set to "../lib64". It is no good though, so my
patch rips them out for Linux as well. Now I started compilling on
Solaris, and at the final xgcc call to link together a new stage of the
compiler, the GNU linker neatly segfaults. Before it does, it complains
that i386 objects aren't compatible with x86_64 ones. Looking into what
gcc passes on to collect2, indeed no -L/usr/lib/amd64 is passed, so yay,
that worked, but those paths /need/ to be searched in the end, of
course. So, work to do for the binutils-wrapper script, that's what we
have it for in the end, right?

Well, not entirely it seems.

First, we never ever injected -L/usr/lib{,64} and -L/lib{,64} into the
search path. Binutils folks apparently stick to the standard there, so
the 64-bits libpaths are not in the default search path. Right they
are, because neither does the runtime linker/kernel do so. Great. Then
why did my amd64 linux box at work run so well? On Fedora multi-lib,
/lib is the 32-bits dir, unlike under Gentoo, where /lib is always
linked to the native bits libdir. To all joy, there is no runpath to
/lib64 in bin/bash, and yet ldd happily shows it links against
/lib64/libc.so.6. Conclusion, the runtime linker/kernel *does* use
lib64 on Fedora at least for a 64-bits object. Getting sweaty yet? I
do.

My GCC on the Fedora box nicely passes /usr/lib/../lib64 as first
argument to collect2. The ldwrapper adds -rpath=${EPREFIX}/usr/lib64.
The compiler checks for includes in ${EPREFIX}/usr/include first.
* compiler sees correct headers
* linker sees uncorrect libs when resolving symbols
* kernel sees correct libs, because of run_path directions in the object
It probably never failed for me so bad, since Fedora is quite up-to-date
with their stuff these days, so I never had clashes. This is scary of
course.

On Darwin, no -L paths are ever passed to collect2, but we would have
noticed it there much sooner. This is where the multilib story comes
in. The search order we basically need is: ${EPREFIX}-libs, /-libs. On
Darwin no lib64, lib/64, or lib/sparc64 exists. Indeed, they use "FAT"
objects, sort of archives where the kernel extracts the object for the
arch it needs. Neat or not, it means on Darwin there is only "lib" as
directory. We pass ${EPREFIX}-libs in the ldwrapper, and get /-libs
from the linker and runtime linker's defaults. So Darwin is correct in
this case, as any other 32-bits target. GCC doesn't pass -L/usr/lib as
it has been told that's the default anyway.

I implemented the 64-bits native targets for Solaris and Darwin as
entire prefixes just with 64-bits code and a 64-bits defaulting
compiler. I just didn't think of it actually, but I just use
${EPREFIX}/{usr,}/lib for the libraries. E.g. no multilib profiles.
For inside the prefix this all works well, however with my patched gcc,
the 64-bits system libraries are no longer seen by the linker, as
neither gcc nor the ldwrapper give -L directions to them for platforms
where they are not /lib. Only since Darwin has FAT objects, it works
fine there. Not for Solaris and Linux. Well, Linux, depends on the
vendor. Would work for Gentoo, not for Fedora.

In prefix, I copied an amd64 profile, and it is a multilib-profile. I
don't know exactly why, but at the time I did it, I thought that was
good. From a desktop perspective multilib is good for Linux users, as
e.g. acroread doesn't come in a 64-bits version. However, should that
be our concern? Anyway, the ldwrapper adds:
ADDLIBDIR("/usr/lib64");
ADDLIBDIR("/usr/lib");
ADDLIBDIR("/lib64");
ADDLIBDIR("/lib");
For the prefix, and at runtime those libs are added if they exist on the
system the wrapper runs on. Now for most platforms a bunch of these are
not necessary. Darwin will never use the 64 variants, 32-bits installs
neither. But why do we use lib64 at all? If we have a compiler that
produces 64-bits code, shouldn't the prefix be fully 64-bits as well?
The ldwrapper has to be made smarter for cross-compiling anyway, but
shouldn't we just nuke lib64? ... or should we think about a "sane" way
to overlay a 32-bits prefix with a 64-bits one? Personally I'm against
that, better keep it clean, and add paths from both prefixes to your
path.

In any case, the ldwrapper will have to add /usr/lib64 and friends in
case the target system is one that uses those...


--
Fabian Groffen
Gentoo on a different level
--
gentoo-alt@gentoo.org mailing list