FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 04-03-2010, 12:09 PM
Maciej Mrozowski
 
Default More reliable hiding preserved libraries

On Saturday 03 of April 2010 12:56:04 Fabian Groffen wrote:
> Is it known why this does happen exactly? When a lib is kept because it
> is still used, only its soname + what the soname points to should be
> kept. That would mean the lib can no longer be found during linking,
> unless you add some trickery (or does GNU ld do something "handy" out of
> the box right here?). So for example:
>
> % ls
> usr/lib/libfoo.so -> libfoo.so.2.1
> usr/lib/libfoo.so.2 -> libfoo.so.2.1
> usr/lib/libfoo.so.2.1
>
> % scanelf --soname usr/lib/libfoo.so
> libfoo.so.2 usr/lib/libfoo.so
>
> what should kept preserved is:
>
> usr/lib/libfoo.so.2 -> libfoo.so.2.1
> usr/lib/libfoo.so.2.1
>
> because trying to link to libfoo using `gcc -o bar -lfoo bar.c` should
> (in theory and on some platforms at least) fail.

It doesn't matter, as 'broken' build system may alphabetically find library by
file name, and link to this library by full path.

On Saturday 03 of April 2010 13:13:00 Michał Górny wrote:
> > 2. During "emerge", unset environment variable corresponding to said
> > preserved library directory - orphans are no longer located.
> Wouldn't that cause failure when the toolkit relies on a 'hidden'
> preserved library?

It would indeed. Now when I think about it, moving stuff to preserved library
dir could be just done - provided it's possible - along with fixing/setting
DT_RPATH's in reverse runtime dependencies. This way no system-wide
LIBRARY_PATH's would be necessary.
Is it possible? Mike?

On Saturday 03 of April 2010 13:33:16 Gilles Dartiguelongue wrote:
> > There is opt-out suggestion[2], unfortunately it does not provide any
> > info how exactly it's supposed to be achieved. As far as portage/pkgcore
> > is concerned, maybe - as Brian Harring suggested - sandbox could be used
> > to somehow "hide" preserved libraries or preserved library directory
> > from ebuild environment (preserved library directory a'ka "purgatory" -
> > libs could be moved there when considered orphaned).
> that sounds nice, it would allow us to more easily spot
> packages/upstreams doing it wrong (maybe that would work for packages
> linking to themselves too btw)

Keeping preserved libraries in separate location (in "purgatory" or dumping
place) is just a method for making implementation possibly easier (or possible
at all), with nice side effects though.

--
regards
MM
 
Old 04-03-2010, 12:16 PM
Fabian Groffen
 
Default More reliable hiding preserved libraries

On 03-04-2010 14:09:42 +0200, Maciej Mrozowski wrote:
> > because trying to link to libfoo using `gcc -o bar -lfoo bar.c` should
> > (in theory and on some platforms at least) fail.
>
> It doesn't matter, as 'broken' build system may alphabetically find
> library by file name, and link to this library by full path.

Shouldn't we fix that buildsystem then? Do you have an example of a
package/buildsystem that does that?

> On Saturday 03 of April 2010 13:13:00 Michał Górny wrote:
> > > 2. During "emerge", unset environment variable corresponding to said
> > > preserved library directory - orphans are no longer located.
> > Wouldn't that cause failure when the toolkit relies on a 'hidden'
> > preserved library?
>
> It would indeed. Now when I think about it, moving stuff to preserved library
> dir could be just done - provided it's possible - along with fixing/setting
> DT_RPATH's in reverse runtime dependencies. This way no system-wide
> LIBRARY_PATH's would be necessary.
> Is it possible? Mike?

No, unless you somehow make sure you reserve space for this, by e.g.
setting a bogus rpath entry at buildtime. If you want to go that route,
you probably want to look at the Prefix' binutils-config wrapper that
already calls the linker with added rpath arguments. Afterwards you can
use chrpath to set it to the correct location. Will get messy with the
vdb though, but if Portage's doing it, it can probably be dealt with.


--
Fabian Groffen
Gentoo on a different level
 
Old 04-03-2010, 06:51 PM
Tiziano Mller
 
Default More reliable hiding preserved libraries

Am Samstag, den 03.04.2010, 12:38 +0200 schrieb Maciej Mrozowski:
> Problem
>
> ..is known, let me summarize briefly.
>
> Uninstalling packages providing libraries, without checking reverse runtime
> dependencies of those packages leaves their dependencies unsatisfied (packages
> with broken executables and/or shared libs).
> Some package managers try their best not to remove said libraries, yet
> allowing packages to be removed.
> Those orphaned libraries cause problems[1] as build systems of some other
> packages being (re)installed afterwards pick them up and abuse those orphaned
> libraries. (we don't like orphans abused, we prefer them... "gone").
>
> Solution
>
> Now, I suppose there are some ideas how to make orphaned libraries not go in a
> way. Basically then need to be available for system, but hidden for "emerge".
>
> There is opt-out suggestion[2], unfortunately it does not provide any info how
> exactly it's supposed to be achieved. As far as portage/pkgcore is concerned,
> maybe - as Brian Harring suggested - sandbox could be used to somehow "hide"
> preserved libraries or preserved library directory from ebuild environment
> (preserved library directory a'ka "purgatory" - libs could be moved there when
> considered orphaned).
>
> Opt-in suggestion is as follows:
> 1. Use some library path (read by ld loader from environment) and export it
> globally to environment pointing it to preserved library directory.
> 2. During "emerge", unset environment variable corresponding to said preserved
> library directory - orphans are no longer located.
> Attached patch for glibc (2.11, but should apply to any other glibc around)
> and uClibc (this one is not tested but should work as well) that makes ld
> loader aware of LD_GENTOO_PRESERVED_LIBRARY_PATH variable.
>
> (LD_LIBRARY_PATH would work as well, but it's being used widely so cannot be
> safely mangled.. on the second though it can - one could filter out preserved
> library paths from it during "emerge").
>
> Thoughts?

Don't fix the hack. Remove the preserve libs "feature", make the PMs
check for rdeps per default before unmerging things. Slot libraries
where needed, slot dep operators (EAPI 4) will help. And if that doesn't
work out we need a separate var to give the PM a hint when API/ABI
breakages happen (such that the PM knows when to re-install the rev
deps).


--
Tiziano Mller
Gentoo Linux Developer
Areas of responsibility:
Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor
E-Mail : dev-zero@gentoo.org
GnuPG FP : F327 283A E769 2E36 18D5 4DE2 1B05 6A63 AE9C 1E30
 
Old 04-03-2010, 09:05 PM
Maciej Mrozowski
 
Default More reliable hiding preserved libraries

On Saturday 03 of April 2010 14:16:14 Fabian Groffen wrote:
> Shouldn't we fix that buildsystem then? Do you have an example of a
> package/buildsystem that does that?
"We" already do, the thing is that maybe we don't have to.
https://bugs.gentoo.org/show_bug.cgi?id=240323
From top of my head: python having issues with sys-libs/db as well as some
packages with readline.

> > It would indeed. Now when I think about it, moving stuff to preserved
> > library dir could be just done - provided it's possible - along with
> > fixing/setting DT_RPATH's in reverse runtime dependencies. This way no
> > system-wide LIBRARY_PATH's would be necessary.
> > Is it possible? Mike?
> No, unless you somehow make sure you reserve space for this, by e.g.
> setting a bogus rpath entry at buildtime. If you want to go that route,
> you probably want to look at the Prefix' binutils-config wrapper that
> already calls the linker with added rpath arguments. Afterwards you can
> use chrpath to set it to the correct location. Will get messy with the
> vdb though, but if Portage's doing it, it can probably be dealt with.
Sounds messy indeed, what about hardened/SELinux/AppArmor/whatever - do they
allow such DT_RPATH operations? It should be probably also restricted for
binary-only packages.

On Saturday 03 of April 2010 20:51:43 Tiziano Mller wrote:
> Don't fix the hack. Remove the preserve libs "feature", make the PMs
> check for rdeps per default before unmerging things.
This will only prevent creating orphans of uninstalled libraries, what about
upgraded ones when SOVERSION has been bumped (the most common case)? Besides I
can already imagine PMS-related discussion regarding "make the PMs check for
rdeps per default before unmerging things" - thx but no thx.

> Slot libraries where needed, slot dep operators (EAPI 4) will help.
Again, you suggest to SLOT every library that happened to bump SOVERSION. It's
unrealistic. Besides library should be slotted when it's API changes, for
source based distributions it's not needed for ABI changes - let's not confuse
those two. Also excessive slotting increases probability of breaking library
discovery mechanisms in various build systems (not everyone uses pkg-config).

> And if that doesn't work out we need a separate var to give the PM a hint
when API/ABI breakages happen (such that the PM knows when to re-install the
rev deps).
It needs PMS amended and thus GLEP. Please submit a GLEP item for this if you
see it fit.
Anyway, as explained in OT, it's not a problem of package manager dependencies
system but issue with broken/not smart build systems - no dependency tree
magic will solve this issue.

--
regards
MM
 
Old 04-04-2010, 03:33 PM
Tiziano Mller
 
Default More reliable hiding preserved libraries

Am Samstag, den 03.04.2010, 23:05 +0200 schrieb Maciej Mrozowski:
> On Saturday 03 of April 2010 14:16:14 Fabian Groffen wrote:
> > Shouldn't we fix that buildsystem then? Do you have an example of a
> > package/buildsystem that does that?
> "We" already do, the thing is that maybe we don't have to.
> https://bugs.gentoo.org/show_bug.cgi?id=240323
> From top of my head: python having issues with sys-libs/db as well as some
> packages with readline.
>
> > > It would indeed. Now when I think about it, moving stuff to preserved
> > > library dir could be just done - provided it's possible - along with
> > > fixing/setting DT_RPATH's in reverse runtime dependencies. This way no
> > > system-wide LIBRARY_PATH's would be necessary.
> > > Is it possible? Mike?
> > No, unless you somehow make sure you reserve space for this, by e.g.
> > setting a bogus rpath entry at buildtime. If you want to go that route,
> > you probably want to look at the Prefix' binutils-config wrapper that
> > already calls the linker with added rpath arguments. Afterwards you can
> > use chrpath to set it to the correct location. Will get messy with the
> > vdb though, but if Portage's doing it, it can probably be dealt with.
> Sounds messy indeed, what about hardened/SELinux/AppArmor/whatever - do they
> allow such DT_RPATH operations? It should be probably also restricted for
> binary-only packages.
>
> On Saturday 03 of April 2010 20:51:43 Tiziano Mller wrote:
> > Don't fix the hack. Remove the preserve libs "feature", make the PMs
> > check for rdeps per default before unmerging things.
> This will only prevent creating orphans of uninstalled libraries, what about
> upgraded ones when SOVERSION has been bumped (the most common case)?
I addressed this in the next phrase.

> Besides I
> can already imagine PMS-related discussion regarding "make the PMs check for
> rdeps per default before unmerging things" - thx but no thx.
This is not related to PMS. Paludis for example does it already with the
current EAPIs.

>
> > Slot libraries where needed, slot dep operators (EAPI 4) will help.
> Again, you suggest to SLOT every library that happened to bump SOVERSION. It's
> unrealistic. Besides library should be slotted when it's API changes, for
> source based distributions it's not needed for ABI changes - let's not confuse
> those two. Also excessive slotting increases probability of breaking library
> discovery mechanisms in various build systems (not everyone uses pkg-config).
I know that slotting can cause problems. But forcing build systems to
use specific versions of libraries is much easier than "shadowing"
libraries present in library search dirs, especially when those libs are
not even tracked by the PM.

>
> > And if that doesn't work out we need a separate var to give the PM a hint
> when API/ABI breakages happen (such that the PM knows when to re-install the
> rev deps).
> It needs PMS amended and thus GLEP.
Wrong, we don't have GLEPs for >90% of the PMS changes going into EAPI-3
or 4.

> Please submit a GLEP item for this if you
> see it fit.
> Anyway, as explained in OT, it's not a problem of package manager dependencies
> system but issue with broken/not smart build systems - no dependency tree
> magic will solve this issue.
It is a problem which can _only_ be solved at the PM level. You have to
tell the PM when API breakages happen. Either by slotting the lib or by
something else. Using guesswork to determine whether or not a library
should be removed may be a temporary solution. Remember: you're
partially ignoring a users request since he wanted the package to be
removed - and we all know how things like "protect the user from
himself" end up.




--
Tiziano Mller
Gentoo Linux Developer
Areas of responsibility:
Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor
E-Mail : dev-zero@gentoo.org
GnuPG FP : F327 283A E769 2E36 18D5 4DE2 1B05 6A63 AE9C 1E30
 
Old 04-05-2010, 06:16 AM
Maciej Mrozowski
 
Default More reliable hiding preserved libraries

On Sunday 04 of April 2010 17:33:17 Tiziano Mller wrote:

>> Besides I
>> can already imagine PMS-related discussion regarding "make the PMs check
for rdeps per default before unmerging things" - thx but no thx.
> This is not related to PMS. Paludis for example does it already with the
> current EAPIs.
So how does Paludis handle those issues?
(Read my comments about "emerge" atomicy below)

> It is a problem which can _only_ be solved at the PM level. You have to
> tell the PM when API breakages happen. Either by slotting the lib or by
> something else.
^^^^^^^^^^^^^^^
This is important - as slotting is not a practical solution. What is it then?

> Using guesswork to determine whether or not a library
> should be removed may be a temporary solution. Remember: you're
> partially ignoring a users request since he wanted the package to be
> removed - and we all know how things like "protect the user from
> himself" end up.

Unconditionally removing libraries (instead of preserving them) and making
their reverse runtime dependencies reinstalled is unacceptable because
"emerge" process involving multiple packages is not atomic. Simple as that.
Is this what you suggest? Correct me if I'm wrong:
1. Users wants to uninstall or reinstall package, we let him do it provided
reverse runtime dependencies are satisfied afterwards. Let's say he wants to
upgrade expat.
2. Expat SOVERSION changed meanwhile but package was not SLOTtted and runtime
reverse deps will still be satisfied when we upgrade.
3. Expat has been upgraded sucessfully,
4a. "emerge" discovers reverse runtime dependencies are broken and it starts
to rebuild them, then it bails out due to error ld: libexpat.so.<sth> not
found. Because step 3 cannot be rolled back (no atomicy) - game over.
or
4b. "emerge does not discover those and does nothing. python is broken so
emerge cannot be used anymore. Game over

--
regards
MM
 
Old 04-05-2010, 06:44 AM
Brian Harring
 
Default More reliable hiding preserved libraries

On Mon, Apr 05, 2010 at 08:16:42AM +0200, Maciej Mrozowski wrote:
> Unconditionally removing libraries (instead of preserving them) and making
> their reverse runtime dependencies reinstalled is unacceptable because
> "emerge" process involving multiple packages is not atomic. Simple as that.
> Is this what you suggest? Correct me if I'm wrong:
> 1. Users wants to uninstall or reinstall package, we let him do it provided
> reverse runtime dependencies are satisfied afterwards. Let's say he wants to
> upgrade expat.
> 2. Expat SOVERSION changed meanwhile but package was not SLOTtted and runtime
> reverse deps will still be satisfied when we upgrade.
> 3. Expat has been upgraded sucessfully,
> 4a. "emerge" discovers reverse runtime dependencies are broken and it starts
> to rebuild them, then it bails out due to error ld: libexpat.so.<sth> not
> found. Because step 3 cannot be rolled back (no atomicy) - game over.
> or
> 4b. "emerge does not discover those and does nothing. python is broken so
> emerge cannot be used anymore. Game over

This is called 'nondeterministic resolution'- known issue also w/
proposals of that sort.

Pretty much everytime someone proposes it as a solution, it gets
smacked down by most folk since an emerge -p invocation that is a
single pkg upgade shouldn't be able to go rebuild your entire world.

The alternative is a slotted ABI var- basically a counter (although it
doesn't have to be) w/in ebuilds themselves to indicate if they're
carrying a new ABI from upstream for that slotting. For example,
you've got EXPAT merged w/ ABI=2, version 2.0. version 2.0.1, for
whatever reason, breaks ABI- thus v2.0.1 in the tree is ABI=2.0.1 (or
3, as said it's an arbitrary value).

Via that, the resolver can see that a rebuild is necessary and plan a
rebuild of all consumers (whether NEEDED based or revdep). Note
preserve-lib would be rather useful here- specifically holding onto
the intermediate lib while doing rebuilding. This however breaks down
a bit when the ABI change is in reverse of normal versioning.

Usually whenever I've floated this idea, it's gotten smacked down by
devs due to the extra work it may entail- someone doing an experiment
on this would be rather useful (someone with a tinderbox
specifically).

~harring
 
Old 04-05-2010, 01:27 PM
Tiziano Mller
 
Default More reliable hiding preserved libraries

Am Sonntag, den 04.04.2010, 23:44 -0700 schrieb Brian Harring:
> On Mon, Apr 05, 2010 at 08:16:42AM +0200, Maciej Mrozowski wrote:
> > Unconditionally removing libraries (instead of preserving them) and making
> > their reverse runtime dependencies reinstalled is unacceptable because
> > "emerge" process involving multiple packages is not atomic. Simple as that.
> > Is this what you suggest? Correct me if I'm wrong:
> > 1. Users wants to uninstall or reinstall package, we let him do it provided
> > reverse runtime dependencies are satisfied afterwards. Let's say he wants to
> > upgrade expat.
> > 2. Expat SOVERSION changed meanwhile but package was not SLOTtted and runtime
> > reverse deps will still be satisfied when we upgrade.
> > 3. Expat has been upgraded sucessfully,
> > 4a. "emerge" discovers reverse runtime dependencies are broken and it starts
> > to rebuild them, then it bails out due to error ld: libexpat.so.<sth> not
> > found. Because step 3 cannot be rolled back (no atomicy) - game over.
> > or
> > 4b. "emerge does not discover those and does nothing. python is broken so
> > emerge cannot be used anymore. Game over
>
> This is called 'nondeterministic resolution'- known issue also w/
> proposals of that sort.
>
> Pretty much everytime someone proposes it as a solution, it gets
> smacked down by most folk since an emerge -p invocation that is a
> single pkg upgade shouldn't be able to go rebuild your entire world.
>
> The alternative is a slotted ABI var- basically a counter (although it
> doesn't have to be) w/in ebuilds themselves to indicate if they're
> carrying a new ABI from upstream for that slotting. For example,
> you've got EXPAT merged w/ ABI=2, version 2.0. version 2.0.1, for
> whatever reason, breaks ABI- thus v2.0.1 in the tree is ABI=2.0.1 (or
> 3, as said it's an arbitrary value).
>
> Via that, the resolver can see that a rebuild is necessary and plan a
> rebuild of all consumers (whether NEEDED based or revdep). Note
> preserve-lib would be rather useful here- specifically holding onto
> the intermediate lib while doing rebuilding.
No, it doesn't help since you may have the same problems some people try
to solve in this thread.

> This however breaks down
> a bit when the ABI change is in reverse of normal versioning.
How so? Such a var should just specify the ABI and the PM only has to
check whether it changed from one PVR to the other. The "how" is
completely irrelevant.


--
Tiziano Mller
Gentoo Linux Developer
Areas of responsibility:
Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor
E-Mail : dev-zero@gentoo.org
GnuPG FP : F327 283A E769 2E36 18D5 4DE2 1B05 6A63 AE9C 1E30
 
Old 04-05-2010, 01:38 PM
Tiziano Mller
 
Default More reliable hiding preserved libraries

Am Montag, den 05.04.2010, 08:16 +0200 schrieb Maciej Mrozowski:
> On Sunday 04 of April 2010 17:33:17 Tiziano Mller wrote:
>
> >> Besides I
> >> can already imagine PMS-related discussion regarding "make the PMs check
> for rdeps per default before unmerging things" - thx but no thx.
> > This is not related to PMS. Paludis for example does it already with the
> > current EAPIs.
> So how does Paludis handle those issues?
> (Read my comments about "emerge" atomicy below)
>
> > It is a problem which can _only_ be solved at the PM level. You have to
> > tell the PM when API breakages happen. Either by slotting the lib or by
> > something else.
> ^^^^^^^^^^^^^^^
> This is important - as slotting is not a practical solution. What is it then?
>
> > Using guesswork to determine whether or not a library
> > should be removed may be a temporary solution. Remember: you're
> > partially ignoring a users request since he wanted the package to be
> > removed - and we all know how things like "protect the user from
> > himself" end up.
>
> Unconditionally removing libraries (instead of preserving them) and making
> their reverse runtime dependencies reinstalled is unacceptable because
> "emerge" process involving multiple packages is not atomic. Simple as that.
A side mark: preserving libs may work in a number of cases, but there
are a lot of (possible) examples where the rdeps (and/or the preserved
libs) are nevertheless broken or become useless (think of: ui-files,
plugins, dlopen'ed libs, etc.).

Please forget your atomicity. It's a really nice idea (and I'd like to
see it too) but not possible now or in the near future.


> Is this what you suggest? Correct me if I'm wrong:
> 1. Users wants to uninstall or reinstall package, we let him do it provided
> reverse runtime dependencies are satisfied afterwards. Let's say he wants to
> upgrade expat.
> 2. Expat SOVERSION changed meanwhile but package was not SLOTtted and runtime
> reverse deps will still be satisfied when we upgrade.
> 3. Expat has been upgraded sucessfully,
> 4a. "emerge" discovers reverse runtime dependencies are broken and it starts
> to rebuild them, then it bails out due to error ld: libexpat.so.<sth> not
> found. Because step 3 cannot be rolled back (no atomicy) - game over.
> or
In that case you most probably have a problem in your dep-tree (either
unspecified deps or a dep-resolver bug)

--
Tiziano Mller
Gentoo Linux Developer
Areas of responsibility:
Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor
E-Mail : dev-zero@gentoo.org
GnuPG FP : F327 283A E769 2E36 18D5 4DE2 1B05 6A63 AE9C 1E30
 
Old 04-05-2010, 06:09 PM
Brian Harring
 
Default More reliable hiding preserved libraries

On Mon, Apr 05, 2010 at 03:27:34PM +0200, Tiziano MMMller wrote:
> > Via that, the resolver can see that a rebuild is necessary and plan a
> > rebuild of all consumers (whether NEEDED based or revdep). Note
> > preserve-lib would be rather useful here- specifically holding onto
> > the intermediate lib while doing rebuilding.
>
> No, it doesn't help since you may have the same problems some people try
> to solve in this thread.

Might I suggest in the future purning down what you're responding to,
to just that blurb? At first read of your response I thought you were
arguing against the ABI var itself (which didn't make a helluva lot of
sense).

Meanwhile, yes, using preserve lib there still has some issues- the
reason I mentioned it possibly being held onto is that if we're
talking about something like openssl, having your system be horked for
the intervening period isn't a great thing thus it may be useful as an
option at the least.

> > This however breaks down
> > a bit when the ABI change is in reverse of normal versioning.
> How so? Such a var should just specify the ABI and the PM only has to
> check whether it changed from one PVR to the other. The "how" is
> completely irrelevant.

That comment is in reference to if preserve-lib is still enabled for
the rebuild.

~harring
 

Thread Tools




All times are GMT. The time now is 10:52 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org