FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 06-26-2011, 02:59 AM
Stuart Longland
 
Default Thoughts about broken package handling

Hi all,

I've been busy for the past month or two, busy updating some of my
systems. In particular, the Yeeloong I have, hasn't seen attention in a
very long time. Soon as I update one part however, I find some swath of
packages break because of a soname change, anything Python-related stops
working because of a move from Python 2.6 to 2.7, or Perl gets updated.

Currently we have three packages that handle this separately:
- revdep-rebuild (handles packages broken by soname changes, etc)
- python-updater (handles Python module rebuilds after upgrading Python)
- perl-cleaner (handles Perl module rebuilds after upgrading Perl)

My bugbear at the moment, is often a package is broken for more than one
reason in my situation, and I find myself having to manhandle the
package lists generated by the above three, building each package
one-by-one, until I manage to rebuild them all.

Or sometimes a package being rebuilt by revdep-rebuild fails because of
a Python module, I'll manually merge that module, then play another
round of Russian Roulette to see which package gets shot down next.

Issues are complicated further when revdep-rebuild or whatever tool,
passes the list to Portage, and it fails to calculate dependencies... I
just had one before where revdep-rebuild failed because there were no
ebuilds to satisfy:

sys-devel/gcc:i686-pc-linux-gnu-4.4.5

I've worked around this by picking up the list generated by
revdep-rebuild (in /var/cache/... ), and using a bash while read loop to
pass each package individually to emerge for building.

How well is this cleanup trio working? It works, but I think it could
improve.

The thing I see is that all three are fixing essentially the same
problem: package breakage due to a change in the dependencies. I think
there is scope for a single package, or better yet, Portage extension,
that handles all three cases.

Concept:

Tool will be written in separate modules to handle:
- ELF soname change breakage
- Python module updates
- Perl module updates
- other checks that can cause broken packages...

Each check is run in order, generating a list of packages that should be
rebuilt.

Having generated this list, it is then evaluated to sort the candidate
packages into a suitable order for rebuilding.

This is then passed to the package manager... three modes for rebuilds:
- All-in-one-hit rebuild: What the tools presently do now.
- One-by-one rebuild: For each package in the list, build each one
individually... useful if Portage coughs up an error otherwise
- Dump the list: allows people to handle it with their own tools

I might see if I can rough something up, but that's what I'm thinking
of. It has been an irritation for me for quite some time.

Thoughts,
--
Stuart Longland (aka Redhatter, VK4MSL) .'.
Gentoo Linux/MIPS Cobalt and Docs Developer '.'` :
. . . . . . . . . . . . . . . . . . . . . . .'.'
http://dev.gentoo.org/~redhatter :.'

I haven't lost my mind...
...it's backed up on a tape somewhere.
 
Old 06-26-2011, 04:23 AM
Duncan
 
Default Thoughts about broken package handling

Stuart Longland posted on Sun, 26 Jun 2011 12:59:05 +1000 as excerpted:

> Currently we have three packages that handle this separately:
> - revdep-rebuild (handles packages broken by soname changes, etc)
> - python-updater (handles Python module rebuilds after upgrading Python)
> - perl-cleaner (handles Perl module rebuilds after upgrading Perl)
>
> My bugbear at the moment, is often a package is broken for more than one
> reason in my situation, and I find myself having to manhandle the
> package lists generated by the above three, building each package
> one-by-one, until I manage to rebuild them all.

I've gone thru that once on my netbook, and will likely be doing it again
every 4-8 months, as I don't keep it as updated as my workstation (which
I try to update weekly to daily).

At 6-8 months it's doable, but requires patience... Much beyond that and
doing a new stage install might well be easier.

> Issues are complicated further when [portage] fails to calculate
> dependencies [due to python breakage, etc]...

> The thing I see is that all three are fixing essentially the same
> problem: package breakage due to a change in the dependencies. I think
> there is scope for a single package, or better yet, Portage extension,
> that handles all three cases.
>
> Concept:
>
> Tool will be written in separate modules to handle:
> - ELF soname change breakage
> - Python module updates
> - Perl module updates
> - other checks that can cause broken packages...

> [The combined list] is then passed to the package manager...

> three modes for rebuilds:
> - All-in-one-hit [current]
> - One-by-one [if portage chokes on the big list]
> - Dump the list: allows people to handle it with their own tools
>
> I might see if I can rough something up, but that's what I'm thinking
> of. It has been an irritation for me for quite some time.
>
> Thoughts,

I'm sure most users will find this VERY useful. I know I will, in no
small part because while I've integrated revdep-rebuild into my regular
update routine, the perl and python rebuilders don't get run as
regularly. If there was a single tool that could scan all three sets,
plus be modular enough to expand to others as necessary...

That's even for routine updates. For the longer term updates, the second
and third modes would be a HUGE help, particularly as they could allow
etc-updates or other config and other not-automatically-package-install-
triggered updates at critical points as well, something the first mode
doesn't really handle.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
 
Old 06-26-2011, 05:44 AM
Benedikt Böhm
 
Default Thoughts about broken package handling

On Sun, Jun 26, 2011 at 4:59 AM, Stuart Longland <redhatter@gentoo.org> wrote:
> - revdep-rebuild (handles packages broken by soname changes, etc)

solved by preserved-libs in portage-2.2

> - python-updater (handles Python module rebuilds after upgrading Python)
> - perl-cleaner (handles Perl module rebuilds after upgrading Perl)

these just exist because python and perl ebuilds are horribly broken.
take a look at RUBY_TARGETS or PHP_TARGETS for an example of how to do
it right. this would also fix all the failures that python and perl
introduce to binary packages.

-Bene
 
Old 06-26-2011, 05:53 AM
Kent Fredric
 
Default Thoughts about broken package handling

On 26 June 2011 17:44, Benedikt Böhm <hollow@gentoo.org> wrote:
> these just exist because python and perl ebuilds are horribly broken.
> take a look at RUBY_TARGETS or PHP_TARGETS for an example of how to do
> it right. this would also fix all the failures that python and perl
> introduce to binary packages.

Perl doesn't slot and is presently far too complex to slot it
reliably. Ideally we could have Perl slotted, but the effort involved
is huge at present. If you're willing to contribute patches to solve
this problem you're welcome, but until then, perl-cleaner is really
the best we can do.


--
Kent

perl -e* "print substr( "edrgmaM* SPA NOcomil.ic@tfrken", $_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz
 
Old 06-26-2011, 06:51 AM
Kent Fredric
 
Default Thoughts about broken package handling

On 26 June 2011 14:59, Stuart Longland <redhatter@gentoo.org> wrote:
> Hi all,
>
> I've been busy for the past month or two, busy updating some of my
> systems. *In particular, the Yeeloong I have, hasn't seen attention in a
> very long time. *Soon as I update one part however, I find some swath of
> packages break because of a soname change, anything Python-related stops
> working because of a move from Python 2.6 to 2.7, or Perl gets updated.
>

I have a system I use/developed which tries to improve consistency,
but it greatly increases the amount of compile/test runs that get done
in the process, but for my cases that's ok because I *want* things to
be rebuilt needlessly "just to make sure they can still be built".

<Massive rant follows>

I make some assumptions:

Take a timestamp.

Record all packages that need to be updated or reinstalled due to USE changes.

Assume, that there is a chance that any direct dependent of those
packages may become broken by this update, either at install time, or
during runtime ( ie: .so breakages, ABI breakages, $LANGUAGE breakages
etc )

Consider that all these packages were installed before that timestamp.

You can then mostly assume all packages installed after that timestamp
are built with the consideration of the changes that have occurred in
their dependencies.

For the most part, this principle appears to cover a very large range
of scenarios, and is somewhat like a proactive "revdep-rebuild" that
assumes that every new install/upgrade breaks everything that uses it.

To this end, I have conjured a very Naïve toolset which does what I want it to.

https://github.com/kentfredric/treebuilder

Note a lot of the paths are hardcoded in the source, and its only
really on Github for educational value.

Workflow proceeds as follows: ( nb: I /do/ use paludis, so I'll be
using its terms for the sake of accuracy )

./mk_timestamp.sh # this sets our timestamp values.

echo >| rebuild.txt # start over the list of things that need rebuilding

Then I sync portage and overlays. ( I have a custom script that calls
cave sync as well as doing a few other tasks )

{

then I do a "deep update of system including new uses" but not execute it.

cave resolve -c system

And I record each updated/reinstalled package line-by-line in rebuild.txt

Then perform it:

cave resolve -c system --continue-on-failure if-independent -x

}

and then repeat that with 'world'.

Then I run ./sync.sh , which works out all the "Still old" packages,
computes some simple regexps' and emits 'rebuilds.out', which is a
list of packages that "Might be broken" which I'll reinstall "just in
case".

VERY Often this is completely needless, ie: -r bumps to kdelibs
triggers me rebuilding everything in KDE, -r bumps to perl causes me
rebuilding every perl module in existence, and everything that uses
perl ( including everything in KDE incidentally , as well as GHC and a
few other very large nasties ).

Once this list is complete, there are 2 approaches I generally take,

1. If the list is small enough, I'll pass the whole thing to cave/paludis.

cave resolve -c -1 $( cat /root/rebuilder/rebuilds.out )

and note record any significant changes ( ie: new uses/updates of
dependendents for things that are "orphans" for whatever reason )

and then

cave resolve -c -1 $( cat /root/rebuilder/rebuilds.out ) -x
--continue-on-failure if-independent

or

2. If this list looks a bit large, I'll pass the things to reinstall
randomly in groups

dd if=/dev/urandom of=/tmp/rand count=1024 bs=1024 # generate
random file for 'shuf' to produce random but repeatable sort.

cave resolve -c -1 $( shuf --random-source=/tmp/rand -n 200
/root/rebuilder/rebuilds.out )

again, noting updates/etc

cave resolve -c -1 $( shuf --random-source=/tmp/rand -n 200
/root/rebuilder/rebuilds.out ) -x --continue-on-failure
if-independent


After each build run, sync.sh is re-run, updating the list of things
which this code still considers "broken", and then I continue the
build/build/build/sync pattern until the build contains all items in
rebuilds.out, and they are all either failing or skipped.

At this point, all the files still listed in rebuilds.out are deemed
"somewhat broken". This list is then concatentated with "brokens.out"
and the process is repeated until all results fail/skip.

then brokens.out and rebuilds.out are concatentated together and
replace broken.txt , which is a list of "things to check later".

at this point I consider that "merging things are as good as its going
to get", and the entire process starts over, update the timestamp,
sync portage.

Over time, broken.txt adapts itself, growing and shrinking as new
things start being broken due to dependencies, or being resolved due
to fixes entering portage.

Long story short, all of the above is mostly insanity, but its
reasonably successfull. And after all, I am an insane kinda guy =)

--
Kent

perl -e* "print substr( "edrgmaM* SPA NOcomil.ic@tfrken", $_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz
 
Old 06-26-2011, 11:48 AM
Thomas Sachau
 
Default Thoughts about broken package handling

chrome://messenger/locale/messengercompose/composeMsgs.properties:
> Hi all,
>
> I've been busy for the past month or two, busy updating some of my
> systems. In particular, the Yeeloong I have, hasn't seen attention in a
> very long time. Soon as I update one part however, I find some swath of
> packages break because of a soname change, anything Python-related stops
> working because of a move from Python 2.6 to 2.7, or Perl gets updated.
>
> Currently we have three packages that handle this separately:
> - revdep-rebuild (handles packages broken by soname changes, etc)
> - python-updater (handles Python module rebuilds after upgrading Python)
> - perl-cleaner (handles Perl module rebuilds after upgrading Perl)

I am thinking about a solution for those similar to current ruby idea and already implemented for
cross-compilation in my multilib-portage branch of portage. The very short version:

Set the needed details in the ebuilds, where needed, in case of revdep-rebuild, either adjust the
SLOT var for each change requiring a rebuild of depending packages or using some new var, e.g.
API_SLOT for this. Ebuilds depending on packages like python or perl should define the range of
versions they support.

Now portage generates a (use_expanded) list of USE flags for depending packages, e.g. for a package
depending on python-2.6 and 2.7 it adds something like PYTHON_DEPEND="pyhon26 python27" to the list
of USE flags. If there is only one dependency installed (like perl or changing libs), this could be
a hidden USE flag.

When the dependency is now updated, the USE flags will change, so in case of portage, a --newuse
will catch those changes and shows those packages in the list of packages, that need to be emerged
again.

In case of slotted dependencies (like python, ruby or php), this would also allow the user to define
per package, if he wants support for one or more slots of e.g. python.
 
Old 06-26-2011, 11:48 AM
Thomas Sachau
 
Default Thoughts about broken package handling

chrome://messenger/locale/messengercompose/composeMsgs.properties:
> Hi all,
>
> I've been busy for the past month or two, busy updating some of my
> systems. In particular, the Yeeloong I have, hasn't seen attention in a
> very long time. Soon as I update one part however, I find some swath of
> packages break because of a soname change, anything Python-related stops
> working because of a move from Python 2.6 to 2.7, or Perl gets updated.
>
> Currently we have three packages that handle this separately:
> - revdep-rebuild (handles packages broken by soname changes, etc)
> - python-updater (handles Python module rebuilds after upgrading Python)
> - perl-cleaner (handles Perl module rebuilds after upgrading Perl)

I am thinking about a solution for those similar to current ruby idea and already implemented for
cross-compilation in my multilib-portage branch of portage. The very short version:

Set the needed details in the ebuilds, where needed, in case of revdep-rebuild, either adjust the
SLOT var for each change requiring a rebuild of depending packages or using some new var, e.g.
API_SLOT for this. Ebuilds depending on packages like python or perl should define the range of
versions they support.

Now portage generates a (use_expanded) list of USE flags for depending packages, e.g. for a package
depending on python-2.6 and 2.7 it adds something like PYTHON_DEPEND="pyhon26 python27" to the list
of USE flags. If there is only one dependency installed (like perl or changing libs), this could be
a hidden USE flag.

When the dependency is now updated, the USE flags will change, so in case of portage, a --newuse
will catch those changes and shows those packages in the list of packages, that need to be emerged
again.

In case of slotted dependencies (like python, ruby or php), this would also allow the user to define
per package, if he wants support for one or more slots of e.g. python.
 
Old 06-26-2011, 12:27 PM
Stuart Longland
 
Default Thoughts about broken package handling

On 06/26/11 15:44, Benedikt Böhm wrote:
> On Sun, Jun 26, 2011 at 4:59 AM, Stuart Longland <redhatter@gentoo.org> wrote:
>> - revdep-rebuild (handles packages broken by soname changes, etc)
>
> solved by preserved-libs in portage-2.2

Hmmm, except that portage-2.2 isn't stable yet... indeed it isn't even
out of alpha yet. Not going to unleash that on my production systems.

The reason why some of these things get out of date to this extent is
because I'm in the "it ain't broke, don't fix it" camp much of the time,
and so outside the kernel and a few other applications, I just keep
things plodding along as they are.

I then get bitten a little when I go to update. Unstable portage is
kinda an iffy area when dealing with production machines... experimental
boxes, no worries... but not on machines I depend on for work.

>> - python-updater (handles Python module rebuilds after upgrading Python)
>> - perl-cleaner (handles Perl module rebuilds after upgrading Perl)
>
> these just exist because python and perl ebuilds are horribly broken.
> take a look at RUBY_TARGETS or PHP_TARGETS for an example of how to do
> it right. this would also fix all the failures that python and perl
> introduce to binary packages.

Perhaps there is room for improving things there... that's a more
long-term solution however. This will require some careful forethought.

Modifying the above tools though, to spit out a list of packages,
shouldn't be much of a change... and then making a tool that can collate
this information and merge it (revdep-rebuild has this code already)
shouldn't be that much of a burden to maintain in the short term.
--
Stuart Longland (aka Redhatter, VK4MSL) .'.
Gentoo Linux/MIPS Cobalt and Docs Developer '.'` :
. . . . . . . . . . . . . . . . . . . . . . .'.'
http://dev.gentoo.org/~redhatter :.'

I haven't lost my mind...
...it's backed up on a tape somewhere.
 
Old 06-26-2011, 12:27 PM
Stuart Longland
 
Default Thoughts about broken package handling

On 06/26/11 15:44, Benedikt Böhm wrote:
> On Sun, Jun 26, 2011 at 4:59 AM, Stuart Longland <redhatter@gentoo.org> wrote:
>> - revdep-rebuild (handles packages broken by soname changes, etc)
>
> solved by preserved-libs in portage-2.2

Hmmm, except that portage-2.2 isn't stable yet... indeed it isn't even
out of alpha yet. Not going to unleash that on my production systems.

The reason why some of these things get out of date to this extent is
because I'm in the "it ain't broke, don't fix it" camp much of the time,
and so outside the kernel and a few other applications, I just keep
things plodding along as they are.

I then get bitten a little when I go to update. Unstable portage is
kinda an iffy area when dealing with production machines... experimental
boxes, no worries... but not on machines I depend on for work.

>> - python-updater (handles Python module rebuilds after upgrading Python)
>> - perl-cleaner (handles Perl module rebuilds after upgrading Perl)
>
> these just exist because python and perl ebuilds are horribly broken.
> take a look at RUBY_TARGETS or PHP_TARGETS for an example of how to do
> it right. this would also fix all the failures that python and perl
> introduce to binary packages.

Perhaps there is room for improving things there... that's a more
long-term solution however. This will require some careful forethought.

Modifying the above tools though, to spit out a list of packages,
shouldn't be much of a change... and then making a tool that can collate
this information and merge it (revdep-rebuild has this code already)
shouldn't be that much of a burden to maintain in the short term.
--
Stuart Longland (aka Redhatter, VK4MSL) .'.
Gentoo Linux/MIPS Cobalt and Docs Developer '.'` :
. . . . . . . . . . . . . . . . . . . . . . .'.'
http://dev.gentoo.org/~redhatter :.'

I haven't lost my mind...
...it's backed up on a tape somewhere.
 
Old 06-26-2011, 12:34 PM
Stuart Longland
 
Default Thoughts about broken package handling

On 06/26/11 21:48, Thomas Sachau wrote:
> I am thinking about a solution for those similar to current ruby idea and already implemented for
> cross-compilation in my multilib-portage branch of portage. The very short version:
>
> Set the needed details in the ebuilds, where needed, in case of revdep-rebuild, either adjust the
> SLOT var for each change requiring a rebuild of depending packages or using some new var, e.g.
> API_SLOT for this. Ebuilds depending on packages like python or perl should define the range of
> versions they support.
>
> Now portage generates a (use_expanded) list of USE flags for depending packages, e.g. for a package
> depending on python-2.6 and 2.7 it adds something like PYTHON_DEPEND="pyhon26 python27" to the list
> of USE flags. If there is only one dependency installed (like perl or changing libs), this could be
> a hidden USE flag.
>
> When the dependency is now updated, the USE flags will change, so in case of portage, a --newuse
> will catch those changes and shows those packages in the list of packages, that need to be emerged
> again.
>
> In case of slotted dependencies (like python, ruby or php), this would also allow the user to define
> per package, if he wants support for one or more slots of e.g. python.
>
>

This sounds pretty good on the surface... the devil as always is in the
details, and I'll have to have a look. I take it though, this would be
exclusively tackling the domains of Python and Perl modules? Or does it
also tackle ABI breakage of other packages?

Sounds like the SLOT variable could get quite unwieldy where several
SLOT-ed packages contribute to a package.
--
Stuart Longland (aka Redhatter, VK4MSL) .'.
Gentoo Linux/MIPS Cobalt and Docs Developer '.'` :
. . . . . . . . . . . . . . . . . . . . . . .'.'
http://dev.gentoo.org/~redhatter :.'

I haven't lost my mind...
...it's backed up on a tape somewhere.
 

Thread Tools




All times are GMT. The time now is 08:50 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org