FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 10-11-2012, 09:35 AM
"Gregory M. Turner"
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

On 10/10/2012 9:14 PM, Mike Frysinger wrote:

On Wednesday 10 October 2012 23:37:26 Gregory M. Turner wrote:

(1) is worse than (2), but it does have some quasi-legitimate usages.
For example, prefix bootstrap does this (or used to), as do many of the
crossdev-wrapper scripts. I've also resorted to such usage, myself,
when repairing a prefix after I've broken its compiler. These cases
don't really seem completely "correct" or "incorrect" -- about the best
I can say for them it is that they are mostly efficacious, but prone to
problems.


and no, crossdev doesn't do this. it is properly sysrooted.


You are right. I guess I was thinking of this:

http://tinderbox.x86.dev.gentoo.org/misc/cmerge,

Which does use -L but looks pretty ancient. I see now that crossdev's
equivalent hacks up the linker scripts -- which seems as "correct" as
one could reasonably hope for... pretty sexy.



further, you cannot do multilib with users putting -L<system libdir> into
their LDFLAGS. they just forced a single ABI and any other will simply fail.
the toolchain's compiler driver knows exactly where to find its own libraries.
if it doesn't, it's broken, and it should be fixed.


ACK


So technically speaking, LDFLAGS="-L. ${LDFLAGS}" or similar is not a
complete abomination. Still... I don't like it.


it's significantly better than the proposed code which tries to parse the
meaning of various flags and insert it at the "right" point. i can't see it
being useful at all. it's a recipe for fragility and being unmaintainable.


I think I meant the above differently than you interpreted it: I meant
that solving library-search-path selection in autotools steps and/or
Makefile.in would make the question of how to manipulate LDFLAGS
academic, and seems more elegant. Apparently you agree:



If the Makefiles are building against libraries expected to be in
${PWD}, it seems to me that the Makefiles should know to look there
automatically.


yes, so why is the ebuild specifying -L. for it ? the package should already
have a -L to the right place, and if it wants to make sure it gets used before
the user's LDFLAGS, it should show up in the linking before it (or have the
build system prepend the value to LDFLAGS if it's appending currently).


If we decide that manipulating LDFLAGS is correct, whether we do this
using my "parser" or LDFLAGS="-L. ${LDFLAGS}" isn't particularly important.


My point above was that coming to that decision in the first place
suggests some underlying issue exists that arguably ought to be fixed
instead.



i'm not sure why this applies to the larger build system. i can understand
that the python build itself might not be putting the -L. in a place where a
user's misconfigured LDFLAGS will cause a problem.


This only pertains to dev-lang/python AFAIK... there may be others
lurking out there somewhere, but I'm not too worried about them. The
use-cases I have in mind are prefix bootstrap and system-recovery when
gcc breaks.


In general, if python breaks, folks can end up in pretty bad shape due
to chicken-and-egg issues with portage. That's the only reason I've
spent so much effort trying to think this fairly obscure corner-case
through.


-gmt

P.S.: After poking around, the first commit where this appeared is this
one:
http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/dev-lang/python/python-2.5.1.ebuild?hideattic=0&r1=1.4&r2=1.5&view=patch,
a mere 5.5 years ago by uberlord.


The cvs ChangeLog references bug 177426 which doesn't seem particularly
illuminating. It also says:


export LDFLAGS=-L. so we link modules correctly on FreeBSD
and possibly other systems where python2.5 isn't installed yet.

Which at least seems to confirm that this was originally a problem not
of search path ordering, but of complete failure to search ${S} in the
first place.


Meanwhile, I've looked into the patches a bit. In regular Gentoo
nothing special happens to LDFLAGS, but in prefix, there is special
handling that suggests some avenues of attack. Before I do anything
further, I want to figure out if it works right without LDPATH
manipulation on linux, and, if so, whether I can't generalize whatever
mechanism allows that to happen.
 
Old 10-11-2012, 03:50 PM
Mike Frysinger
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

On Thursday 11 October 2012 05:35:21 Gregory M. Turner wrote:
> On 10/10/2012 9:14 PM, Mike Frysinger wrote:
> > On Wednesday 10 October 2012 23:37:26 Gregory M. Turner wrote:
> >> (1) is worse than (2), but it does have some quasi-legitimate usages.
> >> For example, prefix bootstrap does this (or used to), as do many of the
> >> crossdev-wrapper scripts. I've also resorted to such usage, myself,
> >> when repairing a prefix after I've broken its compiler. These cases
> >> don't really seem completely "correct" or "incorrect" -- about the best
> >> I can say for them it is that they are mostly efficacious, but prone to
> >> problems.
> >
> > and no, crossdev doesn't do this. it is properly sysrooted.
>
> You are right. I guess I was thinking of this:
>
> http://tinderbox.x86.dev.gentoo.org/misc/cmerge,
>
> Which does use -L but looks pretty ancient. I see now that crossdev's
> equivalent hacks up the linker scripts -- which seems as "correct" as
> one could reasonably hope for... pretty sexy.

by default it doesn't. with the elibtoolize hacks and the drive for .pc files,
this has been largely mitigated. although the latest libtool has a --with-
sysroot option that needs investigating.

> >> If the Makefiles are building against libraries expected to be in
> >> ${PWD}, it seems to me that the Makefiles should know to look there
> >> automatically.
> >
> > yes, so why is the ebuild specifying -L. for it ? the package should
> > already have a -L to the right place, and if it wants to make sure it
> > gets used before the user's LDFLAGS, it should show up in the linking
> > before it (or have the build system prepend the value to LDFLAGS if it's
> > appending currently).
>
> If we decide that manipulating LDFLAGS is correct, whether we do this
> using my "parser" or LDFLAGS="-L. ${LDFLAGS}" isn't particularly important.

it's not particularly important, but on one hand, the LDFLAGS parsing logic
should not be in the tree ever. on the other, i can stomach much smaller one-
off hacks like prepending -L. to LDFLAGS if the maintainers of the python
ebuild really really want to add it.

i'd still lobby for rejecting of invalid LDFLAGS settings and fixing things at
the "right" place.

> In general, if python breaks, folks can end up in pretty bad shape due
> to chicken-and-egg issues with portage. That's the only reason I've
> spent so much effort trying to think this fairly obscure corner-case
> through.

that's really only a problem in the bootstrap case. if an existing system
can't install python, then it isn't like they're screwed since they already
have a working python installed.

> Meanwhile, I've looked into the patches a bit. In regular Gentoo
> nothing special happens to LDFLAGS, but in prefix, there is special
> handling that suggests some avenues of attack. Before I do anything
> further, I want to figure out if it works right without LDPATH
> manipulation on linux, and, if so, whether I can't generalize whatever
> mechanism allows that to happen.

my [limited] understanding of the prefix compiler is that they wrap things with
a custom shell script to inject -L flags behind the back of the compiler at the
very last possible minute and point to the right place.
-mike
 
Old 10-11-2012, 08:39 PM
"Gregory M. Turner"
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

On 10/11/2012 8:50 AM, Mike Frysinger wrote:

On Thursday 11 October 2012 05:35:21 Gregory M. Turner wrote:

On 10/10/2012 9:14 PM, Mike Frysinger wrote:

it's not particularly important, but on one hand, the LDFLAGS parsing logic
should not be in the tree ever.


I've no major attachment to it. Took all of five minutes to code up
reading the ld manpage, and as you pointed out, it's probably solving a
non-problem.



on the other, i can stomach much smaller one-
off hacks like prepending -L. to LDFLAGS if the maintainers of the python
ebuild really really want to add it.


If fixing the python builds proves too onerous then this is what I'll
end up filing a bug for.



my [limited] understanding of the prefix compiler is that they wrap things with
a custom shell script to inject -L flags behind the back of the compiler at the
very last possible minute and point to the right place.


Pretty fuzzy on this myself. Whatever binutils-config does works so
well I've never had to look into it very deeply


Above, however, I'm referring to the prefix-specific cpython patch-sets,
where I was "sure" I'd seen patches to project LDFLAGS into setup.py's
compiler invocation tables. But it seems I was mistaken. Perhaps it
was in the main gentoo-patches after all... or maybe I just need more
coffee...


Anyhow one thing I have figured out is how things can work correctly on
Linux wihtout -L.: on Linux, the python plugins aren't actually linked
against libpython.so!


With any luck, this explains what's going on and suggests that platforms
linking python modules against libpython.so just need to wedge -L${S}
somewhere in configure.in, possibly do-able via sed script (unless we
end up having to mess with setup.py, in which case, things may get a bit
more complicated).


-gmt
 
Old 10-11-2012, 09:40 PM
Marien Zwart
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

I'm going to do something potentially rude and comment on this without
having read the entire thread.

On Thu, Oct 11, 2012 at 10:39 PM, Gregory M. Turner <gmt@malth.us> wrote:
> Anyhow one thing I have figured out is how things can work correctly on
> Linux wihtout -L.: on Linux, the python plugins aren't actually linked
> against libpython.so!

Python can be built with and without a shared library
(libpythonx.y.so). If the shared library is built the interpreter
executable is linked against it (of course), and normally so are the
extension modules. This is useful because it makes the linker do the
right thing for applications that embed python (and end up loading
libpythonx.y via dlopen). If the shared library is not built extension
modules cannot link against it, and the python executable itself
exports the relevant symbols. So on a normal shared python build
extensions should end up linked against libpythonx.y.so.

However, at least one popular distro and presumably most of its
derivatives (Debian) has a system python built without the shared
library, plus the shared library built separately, for use by
applications that embed python. I am not entirely sure why they still
do this (if I understand correctly it was originally done for
performance reasons, but I'm not sure if they've checked those are
still valid). Their extension modules cannot link to libpythonx.y:
their python interpreter executable provides the same symbols
libpythonx.y provides, so loading such an extension module into the
regular interpreter would result in collisions.

The reason I mention this is Gentoo's python is built with the shared
library, so on Gentoo your extensions *should* end up linked to
libpythonx.y (if they use a more or less standard build system, at
least). If yours are not I'm curious what happened, and would
appreciate it if you gave me some more information off-list (marienz
on freenode or email).

--
Marien Zwart.
 
Old 10-12-2012, 11:03 AM
"Gregory M. Turner"
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

On 10/11/2012 2:40 PM, Marien Zwart wrote:

I'm going to do something potentially rude and comment on this without
having read the entire thread.

On Thu, Oct 11, 2012 at 10:39 PM, Gregory M. Turner <gmt@malth.us> wrote:

Anyhow one thing I have figured out is how things can work correctly on
Linux wihtout -L.: on Linux, the python plugins aren't actually linked
against libpython.so!


Python can be built with and without a shared library
(libpythonx.y.so). If the shared library is built the interpreter
executable is linked against it (of course), and normally so are the
extension modules. This is useful because it makes the linker do the
right thing for applications that embed python (and end up loading
libpythonx.y via dlopen). If the shared library is not built extension
modules cannot link against it, and the python executable itself
exports the relevant symbols. So on a normal shared python build
extensions should end up linked against libpythonx.y.so.

However, at least one popular distro and presumably most of its
derivatives (Debian) has a system python built without the shared
library, plus the shared library built separately, for use by
applications that embed python. I am not entirely sure why they still
do this (if I understand correctly it was originally done for
performance reasons, but I'm not sure if they've checked those are
still valid). Their extension modules cannot link to libpythonx.y:
their python interpreter executable provides the same symbols
libpythonx.y provides, so loading such an extension module into the
regular interpreter would result in collisions.

The reason I mention this is Gentoo's python is built with the shared
library, so on Gentoo your extensions *should* end up linked to
libpythonx.y (if they use a more or less standard build system, at
least). If yours are not I'm curious what happened, and would
appreciate it if you gave me some more information off-list (marienz
on freenode or email).


Thanks a ton for this info, that explains a lot. Actually, got this
notion testing outside of Gentoo, building from a vanilla python 3.3.0
tarball on a Fedora box. I wanted to see what happens in a completely
"Gentoo-free" environment so as to get some kind of baseline
understanding of how upstream is propagating stuff through the Makefiles
to setup.py.


Probably, I screwed up ./configure (I tried to approximate Gentoo's
arguments), and requested a non-shlib build. Guess it would have been
clearer if I'd mentioned that in my post -- sorry if I worried you


I do, of course, intend to try it on my main Gentoo ~amd64 -- if I ever
see it fail to do as you describe, I'll let you know, Marien.


My i686 prefix is up to date and I can confirm that it is linking
against libpython there.


But... wow. In my almost-vanilla i686 cross-prefix (hosted by my
only-moderately-rice multilib ~amd64 workstation), in
=dev-lang/python-2.7.3-r2, the modules are linked like so:


i686-pc-linux-gnu-gcc -pthread -shared -Wl,-O1
-L${EPREFIX}/lib -L${EPREFIX}/usr/lib
-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64
-L.
-Wl,-O1
-L${EPREFIX}/lib -L${EPREFIX}/usr/lib
-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64
-L.
-fno-strict-aliasing
-O2 -march=i686 -pipe -m32
-fwrapv -DNDEBUG
-I. -IInclude -I./Include
-I${EPREFIX}/usr/include
build/temp.linux-i686-2.7${EPREFIX}/var/tmp/portage/dev-lang/python-2.7.3-r2/work/Python-2.7.3/Modules/mathmodule.o
build/temp.linux-i686-2.7${EPREFIX}/var/tmp/portage/dev-lang/python-2.7.3-r2/work/Python-2.7.3/Modules/_math.o
-L${EPREFIX}/lib -L${EPREFIX}/usr/lib
-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64
-L.
-lm -lpython2.7
-o build/lib.linux-i686-2.7/math.so

Which is just shameful. In this case, I'm pretty sure all those
-L{${EPREFIX},}{/usr,}/$(libdir) clauses are not coming from me:


[PREFIX] $ emerge --info --verbose | grep /lib
sys-devel/libtool: 2.4.2::gentoo_prefix
COLLISION_IGNORE="/lib/modules/* *.py[co]"
PKG_CONFIG_PATH="${EPREFIX}/usr/lib/pkgconfig:$( :;
)${EPREFIX}/usr/share/pkgconfig"
PORTAGE_BIN_PATH="${EPREFIX}/usr/lib/portage/bin"
PORTAGE_PYM_PATH="${EPREFIX}/usr/lib/portage/pym"
UNINSTALL_IGNORE="/lib/modules/*"

(snippets edited for readability obv).

Regarding the gcc-as-linker invocation above:

First, something puts in all kinds of inappropriate amd64 multilib paths
(this ends up being harmless as wrong-arch libraries get rejected at
link-time and treated as non-matches for -lclauses... still, WTF?).


Secondly, something puts the built-in ld search paths into the
command-line ld search path (a practice which has been roundly
disparaged in this thread, and correctly so, IMO).


Thirdly, prefix x86-linux also clearly suffers from the original problem
I was seeing in cygwin, and which inspired this thread (putting -L.
after $(libdir) paths and therefore linking against the wrong
libpython.so). I never suspected this applied to every prefix python
ebuild, but that's what the above seems to suggest... not cool.


Plus, it repeats itself like a stuttering Makefool -- but that's pretty
standard and only ever seems to bother me


Perhaps, this is a consequence of mine being a cross-prefix, somehow.
That's the only thing significantly off-the-beaten-path thing about it.
Even if so, I'd like amd64->i686 cross-prefixes to work and don't see
why they shouldn't.


The only good news I can report is that changing

append-ldflags "-L."

to

export LDFLAGS="-L. ${LDFLAGS}"

does seem to fix the third issue, above.

Looks like, at least for prefix, it's going to take more than the quick
configure patch I was hoping for, if we're to make things fully right.
We'll see soon enough about ~amd64 linux, and I'm spinning up a
freebsd90 vm to kick around as well.


-gmt
 
Old 10-14-2012, 08:49 AM
"Gregory M. Turner"
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

On 10/12/2012 4:03 AM, Gregory M. Turner wrote:

First, something puts in all kinds of inappropriate amd64 multilib paths
(this ends up being harmless as wrong-arch libraries get rejected at
link-time and treated as non-matches for -lclauses... still, WTF?).

Secondly, something puts the built-in ld search paths into the
command-line ld search path (a practice which has been roundly
disparaged in this thread, and correctly so, IMO).

Thirdly, prefix x86-linux also clearly suffers from the original problem
I was seeing in cygwin, and which inspired this thread (putting -L.
after $(libdir) paths and therefore linking against the wrong
libpython.so). I never suspected this applied to every prefix python
ebuild, but that's what the above seems to suggest... not cool.


Just wanted to update this thread with latest current triage and
prognosis data for this patient:


Regarding etiology, for "Firstly" and "Secondly" above: it seems that
these are by design (which is not to say that they should be left: in
each case, the "design" is pretty bad and ought to be fixed, I think).


My hope is that "Firstly" and "Secondly" are needed only to accommodate
prefix bootstrap, in which case we can fix them by detecting whether a
prefix bootstrap is ongoing or not, and conditioning these hacks on that
(improving "Firstly" to not be braindead would be a nice touch, although
doing so in a way that doesn't break anything may be prohibitively
difficult or beyond my abilities).


"Thirdly" has been addressed ad nauseam in this thread and will be
solved by prepending the LDFLAG rather than appending, or, preferably,
by patching autotools (but only if I can find a simple, low-maintenance
approach that is likely to work without building any new per-platform
matrices or case-statements).


Note that, even without this third fix, absent a PEBKAC placement of
"-L${EPREFIX}/usr/$(libdir)" into LDFLAGS, either:


o it's a prefix bootstrap, in which case there probably is no
${EPREFIX}/usr/$(libdir)/libpython${PV}.so, or

o it's /not/ a prefix bootstrap, in which case the above-mentioned
fixes would prevent the "-L${EPREFIX}/usr/$(libdir)" from
getting into LDFLAGS, thus preventing the problem from arising.

So by fixing "Firstly" and "Secondly," we have at least brought things
sufficiently in line with expectations that we are only fixing
user-level configuration mistakes instead of our own, as originally
expected. **


I'm feeling ready, at this point, to file bugs for these issues; once I
do, I'll post bug#'s into this thread and plan to take all off this
chatter off-list, unless something more comes up that seems to require
community review.


Thanks for everyone's help figuring this stuff out,

-gmt

** I suppose a third possibility is that someone is doing an in-place
"re-bootstrap" or restarting a bootstrap build that failed mid-merge,
but ... doesn't matter, it'll be academic once we fix the bug.
 
Old 10-15-2012, 04:29 AM
Mike Frysinger
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

On Sunday 14 October 2012 04:49:28 Gregory M. Turner wrote:
> "Thirdly" has been addressed ad nauseam in this thread and will be
> solved by prepending the LDFLAG rather than appending, or, preferably,
> by patching autotools (but only if I can find a simple, low-maintenance
> approach that is likely to work without building any new per-platform
> matrices or case-statements).

i'm fairly certain this isn't autotools. i've poked around the python build
system before in the past and while it uses autoconf to do platform tests, it
doesn't use automake/libtool. make is used to bootstrap python, and then they
descend into a horrible hack of a custom build system written in python. i
suspect much of the pain you're seeing is coming from that last part.
-mike
 
Old 10-15-2012, 08:35 AM
"Gregory M. Turner"
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

On 10/14/2012 9:29 PM, Mike Frysinger wrote:

On Sunday 14 October 2012 04:49:28 Gregory M. Turner wrote:

"Thirdly" has been addressed ad nauseam in this thread and will be
solved by prepending the LDFLAG rather than appending, or, preferably,
by patching autotools (but only if I can find a simple, low-maintenance
approach that is likely to work without building any new per-platform
matrices or case-statements).


i'm fairly certain this isn't autotools. i've poked around the python build
system before in the past and while it uses autoconf to do platform tests, it
doesn't use automake/libtool. make is used to bootstrap python, and then they
descend into a horrible hack of a custom build system written in python. i
suspect much of the pain you're seeing is coming from that last part.
-mike


I guess I should say "in src_prepare" rather than "in autotools."

Specifically, I was thinking some kind of configure.in patch might be
good, since configure.in seems to churn less than Makefile.in, a good
thing if I want to produce a one-size-fits-all patch.


And, yeah, setup.py is definitely behind all this yucky.

Python clearly has an amazing community, so I hate to say anything
negative... but I sometimes wish they would "build" less and "buy" more.


Anyhow, as everyone knows, bitching and moaning about FOSS is pointless
so I'll stop there and spend my time fixing stuff instead


-gmt
 
Old 10-15-2012, 05:47 PM
Mike Frysinger
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

On Monday 15 October 2012 04:35:09 Gregory M. Turner wrote:
> On 10/14/2012 9:29 PM, Mike Frysinger wrote:
> > On Sunday 14 October 2012 04:49:28 Gregory M. Turner wrote:
> >> "Thirdly" has been addressed ad nauseam in this thread and will be
> >> solved by prepending the LDFLAG rather than appending, or, preferably,
> >> by patching autotools (but only if I can find a simple, low-maintenance
> >> approach that is likely to work without building any new per-platform
> >> matrices or case-statements).
> >
> > i'm fairly certain this isn't autotools. i've poked around the python
> > build system before in the past and while it uses autoconf to do
> > platform tests, it doesn't use automake/libtool. make is used to
> > bootstrap python, and then they descend into a horrible hack of a custom
> > build system written in python. i suspect much of the pain you're
> > seeing is coming from that last part. -mike
>
> And, yeah, setup.py is definitely behind all this yucky.
>
> Python clearly has an amazing community, so I hate to say anything
> negative... but I sometimes wish they would "build" less and "buy" more.

build systems are hard to get right. python is in the situation where the
setups they care about mostly work and people generally aren't complaining,
but it's more through a hack effort than doing it right which means all the
other cases they haven't considered break horribly. cross-compiling for
example has never worked correctly out of the box.
-mike
 

Thread Tools




All times are GMT. The time now is 06:00 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org