FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 10-06-2012, 02:50 AM
"Gregory M. Turner"
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

----[ "eclass/flag-o-matic.eclass" ]----->8----->
--- PORTAGE/eclass/flag-o-matic.eclass
+++ OVERLAY/eclass/flag-o-matic.eclass
@@ -117,6 +117,42 @@
return 0
}

+# @FUNCTION: prepend-ldpath
+# @USAGE: <path>
+# @DESCRIPTION:
+# Place the specified ldpath into LDFLAGS before any options which could
+# add additional paths to ld's search path. Specifically, it will place
+# the new <path> "/foo" into LDFLAGS as "-L/foo" just before the first
+# occurance matching any of the globs: '-L*', '-T', '--library-path*',
+# and '--script*', but not matching any of the globs: '-Tbss=*',
+# '-Tdata=*', '-Ttext=*', and '-Ttext-segment=*'. If no such match is
+# found, then this is equivalent to "append-ldflags -L<path>".
+prepend-ldpath() {
+ local new=()
+ local f
+ local done=no
+ for f in ${LDFLAGS} ; do
+ case "${f}" in
+ -Tbss=*|-Tdata=*|-Ttext=*|-Ttext-segment=*)
+ new+=( "${f}" )
+ ;;
+ -L*|-T*|--library-path*|--script*)
+ if [[ ${done} == yes ]] ; then
+ new+=( "${f}" )
+ else
+ new+=( "-L${1}" "${f}" )
+ done=yes
+ fi
+ ;;
+ *)
+ new+=( "${f}" )
+ ;;
+ esac
+ done
+ [[ ${done} == no ]] && new+=( "-L${1}" )
+ export LDFLAGS="${new[*]}"
+}
+
# @FUNCTION: filter-lfs-flags
# @DESCRIPTION:
# Remove flags that enable Large File Support.
<-----8<-----

I think my code is probably fine, or if it's buggy, so be it. A prior
question is: does this have sufficient utility?


I "need" (not exactly... keep reading) something like this for python.
Currently python*.ebuild contains:


append-ldflags "-L."

which is required in order to pull in the newly-rebuilt
libpython${VERSION}.so at ebuild-time, instead of the system version
(perhaps it's only so on obscure platforms?).


The problem is, LDFLAGS may already have some library paths coming in
from the environment. In this case, we end up with something like:


LDFLAGS="-Wl,--engage-warp-engines -L/random/prefix/usr/lib -L."

and python goes ahead links everything against, i.e.:

/random/prefix/usr/lib/libpython3.2.so,

or whatever is the version in question, so we achieve precisely nothing,
effecting the outcome we are trying to prevent (the problem manifests in
subtle, hard-to-diagnose ways, unfortunately).


In general, when we say append-ldflags -Lsomething, we may or may not
mean "always use something/libfoo, not $(libdir)/libfoo to link." In
the case where we do mean "always", append-ldflags is a broken approach.


An alternative would be to just replace

append-ldflags "-L."

with

export LDFLAGS="-L. ${LDFLAGS}".

however, there are problems with this:

First, it's aesthetically annoying that this spreads out the "-L*"
arguments into two different "zones" in LDFLAGS. No big deal, but it
definitely doesn't make visually scanning the logs any easier.


Second, although I'm not aware of any ld arguments which act as
"modifiers" to "-L" arguments (except the other -L arguments and
ldscript arguments, which the above patch scans for), ld does use
order-dependent patterns for other arguments.


For example: "-lfoo -( -lbar -lbaz -)" is not the same thing as "-(
-lfoo -lbar -) -lbaz". So order-dependencies might emerge in some
future binutils... in that case we'd still probably need to extend the
case statement above to include the new order-affecting arguments, but
at least we'd have a place in which to do so.


Third, although the meaning of -L* options may not be affected by other
arguments in an order-dependant manner, the reverse is not so. For
example, see --mri-script, --script (which both can affect and be
affected by the command-line library search path in an order-dependent
manner!), -rpath-link (SunOS only), etc...


One could certainly argue that, in practice, points two and three amount
to hopelessly obscure nitpicking and that nobody in their right mind
should be relying on this type of stuff in their LDFLAGS.


I'm not sure I'd state it quite so strongly as that, (after all these
might come just from profile.bashrc and be targeted to a particular
ebuild) but justification is clearly on the "thin" side.


Indeed, if we're going to worry about side effects, it's not entirely
clear that what what my patch does is safe. For example, if the
environment supplied "-L/foo/bar --script baz", and python for some
reason had a "baz" file in "${S}", then some kind of breakage would
surely happen when we did


prepend-ldpath "."

Also, we might also legitimately worry that the presence of this
function in flag-o-matic will somehow "encourage" people to do this
more, and, I'd have to agree that it's a pretty shitty approach to
getting gcc to find your generated libraries (perhaps I could answer
this concern by means of some kind of "don't use unless you must"
disclaimer, or by grafting this code into python.eclass during configure()?)


Gross as it may be, it's the approach we've adopted in dev-lang/python*,
a critically important, mandatory package, and it's broken. It's
definitely not just an academic problem, at least not for my
cygwin-overlay where it makes it impossible to toggle USE=threads for
this package. Presumably the same goes for BSD and any other platforms
relying on


So we either need to fix the tactics or the strategy -- personally I'm a
bit reluctant to mess with the latter, which, I guess, is how I end up
with the above, despite some misgivings.


-gmt
 
Old 10-06-2012, 07:47 AM
"Gregory M. Turner"
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

My god, I am a horrible self-editor. Sorry. Please ignore the magnum
opus above and allow me to try again.


In dev-lang/python*, we use

append-ldflags '-L.'

to ensure linking is performed against the built libpython.so in-tree,
rather than than in the one in $(libdir). But, this doesn't work if
LDFLAGS contains "-L$(libdir)".


We could try to fix this like:

export LDFLAGS="-L. ${LDFLAGS}"

or so. That would cover 99.9% of the cases out there. But very rarely,
indiscriminately placing our '-L.' before every other clause in LDFLAGS
might cause an unanticipated side-effect.


The flag-o-matic patch in my previous post analyses LDPATH and finds the
first statement that might possibly cause the ld command-line
library-search-path to be impacted, and inserts the "-L." just before
that statement, hopefully achieving the desired result with the least
possible side-effects.


I wonder: do people feel this distinction is meaningful enough to
justify actually making it? Or am I just being an anal-retentive nut-job?


If you tried to read my first post, thanks for not offing yourself like
the guys who get stuck sitting next to Ted Striker in _Airplane!_


-gmt
 
Old 10-06-2012, 08:31 AM
Fabian Groffen
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

On 06-10-2012 00:47:57 -0700, Gregory M. Turner wrote:
> My god, I am a horrible self-editor. Sorry. Please ignore the magnum
> opus above and allow me to try again.
>
> In dev-lang/python*, we use
>
> append-ldflags '-L.'
>
> to ensure linking is performed against the built libpython.so in-tree,
> rather than than in the one in $(libdir). But, this doesn't work if
> LDFLAGS contains "-L$(libdir)".
>
> We could try to fix this like:
>
> export LDFLAGS="-L. ${LDFLAGS}"
>
> or so. That would cover 99.9% of the cases out there. But very rarely,
> indiscriminately placing our '-L.' before every other clause in LDFLAGS
> might cause an unanticipated side-effect.
>
> The flag-o-matic patch in my previous post analyses LDPATH and finds the
> first statement that might possibly cause the ld command-line
> library-search-path to be impacted, and inserts the "-L." just before
> that statement, hopefully achieving the desired result with the least
> possible side-effects.

I think it would make more sense in this case to just add one more patch
to Python, such that the build-system just inserts it. I recall it's
necessary at least on FreeBSD, Darwin, Solaris, but I don't recall any
more why it works/worked on Linux fine.


--
Fabian Groffen
Gentoo on a different level
 
Old 10-06-2012, 03:13 PM
Duncan
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

Gregory M. Turner posted on Sat, 06 Oct 2012 00:47:57 -0700 as excerpted:

> If you tried to read my first post, thanks for not offing yourself like
> the guys who get stuck sitting next to Ted Striker in _Airplane!_

It was worth it just for this:

>> LDFLAGS="-Wl,--engage-warp-engines -L/random/prefix/usr/lib -L."

=:^)

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
 
Old 10-09-2012, 09:26 PM
Mike Frysinger
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

On Saturday 06 October 2012 03:47:57 Gregory M. Turner wrote:
> My god, I am a horrible self-editor. Sorry. Please ignore the magnum
> opus above and allow me to try again.
>
> In dev-lang/python*, we use
>
> append-ldflags '-L.'
>
> to ensure linking is performed against the built libpython.so in-tree,
> rather than than in the one in $(libdir). But, this doesn't work if
> LDFLAGS contains "-L$(libdir)".

well, some notes here ...

for users: putting -L<some-system-path> is wrong. it changes the meaning of
"system paths" when it comes to searching for linking by elevating it to "user
specified path". imo, you should be fixing the source rather than the symptom.

for packagers: using -L$(libdir) is almost always wrong and pointless. the
toolchain itself knows the proper system path to search for libraries to link
against, and when you cross-compile, the --libdir you specify to configure is
not going to be valid when dereferenced on the build system.

> We could try to fix this like:
>
> export LDFLAGS="-L. ${LDFLAGS}"
>
> or so. That would cover 99.9% of the cases out there. But very rarely,
> indiscriminately placing our '-L.' before every other clause in LDFLAGS
> might cause an unanticipated side-effect.

i would go with this if you want to support these people doing it wrong
-mike
 
Old 10-10-2012, 10:47 PM
"Gregory M. Turner"
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

On 10/6/2012 1:31 AM, Fabian Groffen wrote:

On 06-10-2012 00:47:57 -0700, Gregory M. Turner wrote:

In dev-lang/python*, we use

append-ldflags '-L.'

to ensure linking is performed against the built libpython.so in-tree,
rather than than in the one in $(libdir). But, this doesn't work if
LDFLAGS contains "-L$(libdir)".

We could try to fix this like:

export LDFLAGS="-L. ${LDFLAGS}"

or so. That would cover 99.9% of the cases out there. But very rarely,
indiscriminately placing our '-L.' before every other clause in LDFLAGS
might cause an unanticipated side-effect.

I think it would make more sense in this case to just add one more patch
to Python, such that the build-system just inserts it. I recall it's
necessary at least on FreeBSD, Darwin, Solaris, but I don't recall any
more why it works/worked on Linux fine.


I was thinking along these lines as well. As it turned out, however, I
was not able to find an automagical, PMS-compliant, non-crazy way to do
it that resolved my test-case (more on that below), without effectively
hooking econf().


Literally hooking a core API seems like a Pandora's box better left
unopened, so this is what I came up with (note: the fpectl stuff was
also in my overlay and the patches got entangled -- rather than editing
the hunk by hand, I just left it. It's orthogonal to this issue,
however, and for present purposes can be ignored):


------>8----->
--- PORTAGE/dev-lang/python/python-2.7.3-r2.ebuild
+++ OVERLAY/dev-lang/python/python-2.7.3-r2.ebuild
@@ -218,13 +255,6 @@
# http://bugs.python.org/issue15506
export ac_cv_path_PKG_CONFIG=$(tc-getPKG_CONFIG)

- # Set LDFLAGS so we link modules with -lpython2.7 correctly.
- # Needed on FreeBSD unless Python 2.7 is already installed.
- # Please query BSD team before removing this!
- # On AIX this is not needed, but would record '.' as runpath.
- [[ ${CHOST} == *-aix* ]] ||
- append-ldflags "-L."
-
local dbmliborder
if use gdbm; then
dbmliborder+="${dbmliborder:+:}gdbm"
@@ -248,11 +278,18 @@
&& myconf="${myconf} --enable-framework=${EPREFIX}/usr/lib"
|| myconf="${myconf} --enable-shared"

+ if [[ ${CHOST} == *-cygwin* ]] ; then
+ fpeconfig="--without-fpectl"
+ myconf="${myconf} ac_cv_func_bind_textdomain_codeset=yes"
+ else
+ fpeconfig="--with-fpectl"
+ fi
+
# note: for a framework build we need to use ucs2 because OSX
# uses that internally too:
# http://bugs.python.org/issue763708
- OPT="" econf
- --with-fpectl
+ OPT="" cpython_econf
+ ${fpeconfig}
$(use_enable ipv6)
$(use_with threads)
$( (use wide-unicode && use !aqua) && echo "--enable-unicode=ucs4"
|| echo "--enable-unicode=ucs2")

--- PORTAGE/eclass/python.eclass
+++ OVERLAY/eclass/python.eclass
@@ -401,6 +401,64 @@
fi
}

+# see http://thread.gmane.org/gmane.linux.gentoo.devel/80633/focus=80635
+_python_prepend_cwd_ldpath() {
+ local new=()
+ local f
+ local done=no
+ for f in ${LDFLAGS} ; do
+ case "${f}" in
+ -Tbss=*|-Tdata=*|-Ttext=*|-Ttext-segment=*)
+ new+=( "${f}" )
+ ;;
+ -L*|-T*|--library-path*|--script*)
+ if [[ ${done} == yes ]] ; then
+ new+=( "${f}" )
+ elif [[ "${f}" == "-L." ]] ; then
+ # if it's somehow already there, don't duplicate it
+ new+=( "-L." )
+ done=yes
+ else
+ new+=( "-L." "${f}" )
+ done=yes
+ fi
+ ;;
+ *)
+ new+=( "${f}" )
+ ;;
+ esac
+ done
+ [[ ${done} == no ]] && new+=( "-L." )
+ export LDFLAGS="${new[*]}"
+}
+
+# @FUNCTION: cpython_econf
+# @DESCRIPTION:
+# econf() substitute for use in dev-lang/python ebuilds
+#
+# On some platforms, it is neccesary to prepend "-L." to ldpath before
+# proceeding with python's configure process. Using cpython_econf()
+# instead of econf() will ensure that this is taken care of correctly
+# before python's configure step can proceed. This is not needed for
+# any python ebuilds other than dev-lang/python; any other ebuild
+# calling this function will receive an error.
+cpython_econf() {
+ if [[ "${CATEGORY}/${PN}" != "dev-lang/python" ]] ; then
+ die "cpython_econf can only be used by dev-lang/python ebuilds"
+ fi
+ # econf will enforce ${EBUILD_PHASE} requirements, so we don't bother.
+
+ # Set LDFLAGS so we link modules with -lpython2.7 correctly.
+ # Needed on FreeBSD unless Python 2.7 is already installed.
+ # Even if python is already installed, linking against the old
+ # python will cause problems, i.e., when toggling USE="threads".
+ # Also needed on cygwin. Please query BSD team before removing this!
+ # On AIX this is not needed, and would record '.' as runpath.
+ [[ ${CHOST} == *-aix* ]] || _python_prepend_cwd_ldpath
+
+ econf "$@"
+}
+
# @FUNCTION: python_pkg_setup
# @DESCRIPTION:
# Perform sanity checks and initialize environment.
<-----8<-----

Some notes about the above:

Of course we would presumably want to make a similar change in all of
the dev-lang/python ebuilds to the one above.


We could do this 100% automagically in python_pkg_setup, which was my
initial plan. But my test-case involves LDPATH being "scrubbed" in
every phase by profile.bashrc. After python_pkg_setup() changed LDPATH,
it just broke again in subsequent phases (at least, I think that's what
would have happened -- I never tested it). Yes, that's a GIGO problem;
however, it's debatable whether Gentoo or profile.bashrc is the one
putting the "G)arbage I)n". The most robust route is to insert our
LDFLAG just before econf runs -- that's how the current implementation
does it, after all, and we potentially create a regression if we move
this logic to pkg_setup.


Also, I decided to leave my internal API as "_python_foo," although
today, I'd say "__python_foo" is more correct. python.eclass is chock
full of "_python_foo" API's, so I decided it was probably cleaner to
follow the existing patterns for now, rather than change one lone API
while the rest remain unchanged.


As for cpython_econf -- is it OK that it doesn't begin with "python"?
Or should it perhaps be "python_cpython_econf?"


Thanks for your input,

-gmt
 
Old 10-11-2012, 03:28 AM
Mike Frysinger
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

On Wednesday 10 October 2012 18:47:46 Gregory M. Turner wrote:
> + if [[ ${CHOST} == *-cygwin* ]] ; then
> + fpeconfig="--without-fpectl"

just re-use myconf. this is what it's for.

> + myconf="${myconf} ac_cv_func_bind_textdomain_codeset=yes"

just export it:
export ac_cv_func_bind_textdomain_codeset=yes

> As for cpython_econf -- is it OK that it doesn't begin with "python"?
> Or should it perhaps be "python_cpython_econf?"

i think it's all a giant hack for a problem i have yet to see described where
the user wasn't doing something wrong. it's one thing to add LDFLAGS="-L.
${LDFLAGS}" to support their dumbness, but it's a completely different ballgame
to write an entire flag parser (which i'm going to assume isn't even really
necessary or covers all possible angles).

another alternative is to modify the python build system so that it uses the
full path to the library directly rather than searching for it via -L ...
-mike
 
Old 10-11-2012, 03:37 AM
"Gregory M. Turner"
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

On 10/9/2012 2:26 PM, Mike Frysinger wrote:

On Saturday 06 October 2012 03:47:57 Gregory M. Turner wrote:

My god, I am a horrible self-editor. Sorry. Please ignore the magnum
opus above and allow me to try again.

In dev-lang/python*, we use

append-ldflags '-L.'

to ensure linking is performed against the built libpython.so in-tree,
rather than than in the one in $(libdir). But, this doesn't work if
LDFLAGS contains "-L$(libdir)".


well, some notes here ...

for users: putting -L<some-system-path> is wrong. it changes the meaning of
"system paths" when it comes to searching for linking by elevating it to "user
specified path". imo, you should be fixing the source rather than the symptom.

for packagers: using -L$(libdir) is almost always wrong and pointless. the
toolchain itself knows the proper system path to search for libraries to link
against, and when you cross-compile, the --libdir you specify to configure is
not going to be valid when dereferenced on the build system.


Agreed on these points. Not until we combine the doubly broken
circumstances of (1) User specified -L/usr/$(libdir) in LDPATH and (2)
an ebuild that puts -L. into LDPATH to fix Makefile recipes that appear
more-or-less broken, do we get this conflict. This leads to a certain
odor of "unsupportable circumstances" which, I must admit, makes me
slightly uncomfortable trying to fix it.


Breaking this down:

(1) is worse than (2), but it does have some quasi-legitimate usages.
For example, prefix bootstrap does this (or used to), as do many of the
crossdev-wrapper scripts. I've also resorted to such usage, myself,
when repairing a prefix after I've broken its compiler. These cases
don't really seem completely "correct" or "incorrect" -- about the best
I can say for them it is that they are mostly efficacious, but prone to
problems.


As for (2)... meh. LDFLAGS, to my thinking, means something like: "Use
these guidelines and parameters for linking, in preference to the
default guidelines that the build process would normally use." LDFLAGS
coming from make.conf have a slightly different meaning, I guess, but
the gist of it is the same.


So technically speaking, LDFLAGS="-L. ${LDFLAGS}" or similar is not a
complete abomination. Still... I don't like it.


If the Makefiles are building against libraries expected to be in
${PWD}, it seems to me that the Makefiles should know to look there
automatically.


In the particular case of Gentoo's cpython, I have to admit I'm
concerned that if I start mucking around with the Makefiles/autotool
scaffolding, I'm going to break stuff. But maybe it's worth taking that
risk if it allows us to do things more correctly going forward... I
could at least take a crack at it and see what people think of the results.


-gmt
 
Old 10-11-2012, 03:50 AM
Diego Elio Pettenò
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

On 10/10/2012 20:37, Gregory M. Turner wrote:
>
> If the Makefiles are building against libraries expected to be in
> ${PWD}, it seems to me that the Makefiles should know to look there
> automatically.

Using the -L . -lfoo is a very bad style in general. Just think what
happen if you're trying to link to a libutil.a

--
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/
 
Old 10-11-2012, 04:14 AM
Mike Frysinger
 
Default eclass/flag-o-matic.eclass: prepend-ldpath

On Wednesday 10 October 2012 23:37:26 Gregory M. Turner wrote:
> (1) is worse than (2), but it does have some quasi-legitimate usages.
> For example, prefix bootstrap does this (or used to), as do many of the
> crossdev-wrapper scripts. I've also resorted to such usage, myself,
> when repairing a prefix after I've broken its compiler. These cases
> don't really seem completely "correct" or "incorrect" -- about the best
> I can say for them it is that they are mostly efficacious, but prone to
> problems.

they're broken. and no, crossdev doesn't do this. it is properly sysrooted.

further, you cannot do multilib with users putting -L<system libdir> into
their LDFLAGS. they just forced a single ABI and any other will simply fail.
the toolchain's compiler driver knows exactly where to find its own libraries.
if it doesn't, it's broken, and it should be fixed.

> So technically speaking, LDFLAGS="-L. ${LDFLAGS}" or similar is not a
> complete abomination. Still... I don't like it.

it's significantly better than the proposed code which tries to parse the
meaning of various flags and insert it at the "right" point. i can't see it
being useful at all. it's a recipe for fragility and being unmaintainable.

> If the Makefiles are building against libraries expected to be in
> ${PWD}, it seems to me that the Makefiles should know to look there
> automatically.

yes, so why is the ebuild specifying -L. for it ? the package should already
have a -L to the right place, and if it wants to make sure it gets used before
the user's LDFLAGS, it should show up in the linking before it (or have the
build system prepend the value to LDFLAGS if it's appending currently).

> In the particular case of Gentoo's cpython, I have to admit I'm
> concerned that if I start mucking around with the Makefiles/autotool
> scaffolding, I'm going to break stuff. But maybe it's worth taking that
> risk if it allows us to do things more correctly going forward... I
> could at least take a crack at it and see what people think of the results.

i'm not sure why this applies to the larger build system. i can understand
that the python build itself might not be putting the -L. in a place where a
user's misconfigured LDFLAGS will cause a problem.

but if you want the right answer, it's to reject broken user LDFLAGS. we
don't support wrong things like -fPIC or -fPIE in global CFLAGS. i don't
think we should be wasting time on LDFLAGS like this either.
-mike
 

Thread Tools




All times are GMT. The time now is 10:41 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org