FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian dpkg

 
 
LinkBack Thread Tools
 
Old 07-05-2011, 04:42 PM
Jonathan Nieder
 
Default dpkg BUG install order problem & suggestion to add...

Hi,

John D. Hendrickson and Sara Darnell wrote:

> I've always had problems especially during upgrade (last was sarge
> to sqeeze) and wished to help.

Thanks for the offer of help. There are always problems with upgrades
"skipping a few releases" like this, since package maintainers do not
tend to think about or test upgrade paths other than Debian N ->
N+1. In particular, often there will be a package or two that assumes
packages have already been upgraded to the version from the previous
release without declaring specific dependencies in that vein.

So the usual advice is to upgrade one release at a time:

sarge ->
etch ->
lenny ->
squeeze

Of course if there is some simple way to allow shortcuts (e.g., some
tool to perform multiple upgrades after a person types a single
command?) then that could be very interesting.

For clarity, could you give an example of problems you ran into?
I.e., what command did you run, and what bad behavior resulted?

[...]
> My suggestion is NOT to replace what is there but to add a method.
> The method checks the final install decisions. It easily shows what
> order they should be done in AND whether they go out of bounds.

Perhaps you are looking into changing the upgrade order to avoid
running into temporary conflicts and other dependency problems? If so,
yes, that sounds very welcome, though it probably has more to do with
higher-level tools like apt, aptitude, and cupt than dpkg (and alas,
it wouldn't be enough to address the problem described above).

> I have a working example of what to do that mostly uses existing
> unix tools.

Well, what are you waiting for?

Thanks for writing, and good luck.
Jonathan


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110705164208.GB28864@elie">http://lists.debian.org/20110705164208.GB28864@elie
 
Old 07-05-2011, 07:58 PM
"John D. Hendrickson and Sara Darnell"
 
Default dpkg BUG install order problem & suggestion to add...

debian-dpkg@lists.debian.orgJohn D. Hendrickson and Sara Darnell wrote:

(to jonathan - repost - I did reply incorrectly)


Jonathan Nieder wrote:



John D. Hendrickson and Sara Darnell wrote:





I've always had problems especially during upgrade (last was sarge
to sqeeze) and wished to help.


Thanks for the offer of help. There are always problems with upgrades
"skipping a few releases" like this, since package maintainers do not
tend to think about or test upgrade paths other than Debian N ->
N+1. In particular, often there will be a package or two that assumes
packages have already been upgraded to the version from the previous
release without declaring specific dependencies in that vein.

So the usual advice is to upgrade one release at a time:

sarge ->
etch ->
lenny ->
squeeze


You are right. The extra work takes as much time as saved I presume
But good advice.


While the problems is worst when upgrading - it comes up now and then
when doing any install change that tags a large number of packages.





Of course if there is some simple way to allow shortcuts (e.g., some
tool to perform multiple upgrades after a person types a single
command?) then that could be very interesting.



What order? Dependancy order. tsort. It can be done I did it. You
can build it 1x for all packages then check against it. (there are
side-track details about when and why do rebuild the list more than 1x)




For clarity, could you give an example of problems you ran into?
I.e., what command did you run, and what bad behavior resulted?


Sure. debian.org documentation release-notes (upgrading). Says to expect
glitches.


# apt-get upgrade
begins reasonable - some pkgs fail so...
# apt-get -f install
sometimes manual fixing is necessary
usu. just force file overwrite
# dpkg --configure --pending
# apt-get upgrade


--> Here is where we see large install lists develop that want to install out of dependency order,
very different from origional list. Happens after upgrades need to recalculate, the pkg "grading",
what the list to install is, the etc.


Again I'd mentioned it's likely more a problem during upgrade but not totally upgrade specific.





[...]

My suggestion is NOT to replace what is there but to add a method.
The method checks the final install decisions. It easily shows what
order they should be done in AND whether they go out of bounds.


Perhaps you are looking into changing the upgrade order to avoid
running into temporary conflicts and other dependency problems? If so,
yes, that sounds very welcome, though it probably has more to do with
higher-level tools like apt, aptitude, and cupt than dpkg (and alas,
it wouldn't be enough to address the problem described above).


I have a working example of what to do that mostly uses existing
unix tools.


Well, what are you waiting for?


I don't wanna just dump code only I can read



(I can send the code but it's a makefile (to keep table build parts up to date), uses mostly awk,
etc, and mostly table building, some simple lookup scripts like rdeps, etc (in awk))



*** So what I'm saying is installer(s) sometimes choose to try to install higher
*** dependencies before lower dependencies, again the order.


upgrade also limits the number of items attempted (see above,
apt-get upgrade v. dist-upgrade). However packages which depend on allot (ie, gnome)
end up in the "limited list" and the limited list it seems grows.



The thought is this. Take all dependencies and sort them in one list.
Really once have to do it 1x.


THEN. Whenever (install, upgrade, anytime) has a list it wishes to
install / remove it can order that list before attempting by the
dependency sorted list. Since the deps are already sorted ordering the
packages doesn't even require sorting but just list-list order.


I tried tsorting all pkgs it works (ie, libc6 is low on list,
zope-tinytables high).


The "/var/lib/available" list itself could be in tsorted order. I don't
see the order in it myself.


YES, there are details. Build the list once or every time (fast pc
only, 20min. takes a while on a pentium). There is a detail about ORs
(which I ignore since what installer next chooses as "a good grade"
changes - meanining you MIGHT have tried to install both ORs when your
done: true story! )


Ok. Hope you understood that enough. If you like I can send example
script or post it somewhere.


(prepare data) | tsort | tac > ordered list

when looking up, given any pkg name, you get a depth into the list (so
ordering any given list is a cinch)




Thanks for writing, and good luck.
Jonathan



Well thank you!

John Hendrickson



Once again I'll say I also love these pkg managers. I realize there is a need for what they do even
if my method is added. Even if "perfect dependency order" happens for a particular install list or
is even automatic: there is still the issues of deciding which to pick for the user which debian's
pkgs suites are simply tops at doing !


have fun

John Hendrickson


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4E136CD1.7090708@cox.net">http://lists.debian.org/4E136CD1.7090708@cox.net
 
Old 07-05-2011, 08:24 PM
"John D. Hendrickson and Sara Darnell"
 
Default dpkg BUG install order problem & suggestion to add...

Jonathan Nieder wrote:

Hi,

John D. Hendrickson and Sara Darnell wrote:




I have a working example of what to do that mostly uses existing
unix tools.


Well, what are you waiting for?

Thanks for writing, and good luck.
Jonathan


It's preliminary - to test it see it work - it's not a c++ patch [yet].

Also I've watched lists.debian.org an not seen people post awk or makefiles only patches. So I do
not know if "attatch" is right, or what is right if you want to see code.


The sorted list size is only 250 kilobytes text (pkg #s only) and it may be an answer only to use
the method to order entries in "/var/lib/dpkg/available" just once per release (so orig. collected
order is already tsorted).


I have a makefile to make dep tables (several intermediate tables) and finally tsort the result
(make the list). And then some scripts which show / sort more, lookup, deps (given any pkg, sort
list of req. pkgs against tsort list, as discussed), rdeps. etc. It can be done with mathematical
suites but tsort is a unix tool and making dep. tables is easy: shouldn't depend on a gui math suite.


It works to show it works and anything further I'll wait for feedback of the thesis. (that checking
any given install list against a tsorted dependency list of all pkgs is a good idea)


So just say what you'd like to see. I don't want to dump scripts in the mail channel unless
requested because I don't see others doing it and it's probably not a good practice in general.


John


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4E137309.2010905@cox.net">http://lists.debian.org/4E137309.2010905@cox.net
 
Old 07-08-2011, 05:44 PM
"John D. Hendrickson and Sara Darnell"
 
Default dpkg BUG install order problem & suggestion to add...

debian-dpkg@lists.debian.org


Jonathan Nieder wrote:



John D. Hendrickson and Sara Darnell wrote:



I have a working example of what to do that mostly uses existing
unix tools.


Well, what are you waiting for?


The problem seems to be when allot of packages need to be installed (makes unreasonable choices) and
it begins to deteriorate after initial install failures. Soon the "solver" is trying to install
small desktop apps before base components (every time I upgrade).


I looked at the dpkg ant aptitude sources. It seems there's allot of maybe's involved "grading"
(not that I'm complaining it makes good lists in a thin env.) (extended discussion). The "grade"
method is great for finding a wishlist.


I only say the resulting list is likely unordered and this is likely to cause problems during large
install / uninstall "runs".


My thesis, still preliminary, is ONLY TO RE-ORDER the given list dpkg will attempt, only optionally,
nothing else. It should reduce errors and no effect anything, not even triggers, at all.


SEE PROPOSAL BELOW. (it shows the make tsort-out but not utilities I have that go with it)

(the simple lookup utilities to things like show all possible deps and ordre them against the
tsorted list (easy), etc)


Thanks hope you enjoy discussion,

John Hendrickson


# -----------------------------------------------------
# FILE: Makefile
# DESC: generates tsort-out
# USAGE: 'make'
# or make [ show | clean | realclean ]
# -----------------------------------------------------

# below is mostly simple table making & lookup. several
# tables can be a bear to tend but make makes it easy

# due to make syntax sh script needs $$ instead of $

SHELL = /bin/sh

DEPENDENCIES= available pack prov dep ppd prov-flat prov-idx prov-flat
pP-dup pP-dep num-dep-1 num-dep-2 pP-dep-2 num-dep-done tsort-in

# -----------------------------------------------------

LINES = `wc -l pack | sed -e 's/ .*$$//'`

# -----------------------------------------------------
# main file : depends on other files
# before being able to create it

tsort-out : ${DEPENDENCIES}
@echo 'making tsort-in';
tsort tsort-in 2>>dep-loops | tac > tsort-out

# tsort-out : tsort-in # skips optionals, ie, ppd

# -----------------------------------------------------

.PHONY : show
show : tsort-out
show-all | sort -s -k 3,3 | less -S

.PHONY : install
install :
@echo "preliminary - nothing to install"

.PHONY : clean
clean :
@echo "cleaning intermediate files (can still use scripts)"
rm dep dep-idx dep-loops growth num-dep-1 num-dep-2
pP-dep pP-dep-2 pP-dup ppd prov prov-flat
tsort-in pass.log 2>> /dev/null ; /bin/true

.PHONY : realclean
realclean : clean
@echo "cleaning all tables ('make all' needed to use scripts)"
rm ${DEPENDENCIES} 2>> /dev/null ; /bin/true

.PHONY : all
all : tsort-out

available :
@echo "no ./available, using softlink"
ln -s /var/lib/dpkg/available .

pack prov dep : available
@echo 'making pack prov dep';
@rm pack prov dep 2>>/dev/null ; awk 'BEGIN{OFS=""} /^Pa/{if(ln!=0){print n pdhold dhold >> "dep"
}; ln=0; dhold=""; pdhold=""; ++n; print n " " $$2 >> "pack"} /^Pro/{sub(/Provides: /,"");
gsub(/,/,""); print n " " $$0 >> "prov"} /^Dep/{ln=n; gsub(/Depends: /,""); gsub(/([^)]*)/,"");
gsub(/,/,""); gsub(/|/,""); dhold=" "$$0} /^Pre/{ln=n; gsub(/Pre-Depends: /,"");
gsub(/([^)]*)/,""); gsub(/,/,""); gsub(/|/,""); pdhold=" "$$0} {if(d==1 && pre==0) print n" "hold
>> "dep"; pre=0}' available


ppd : available
@echo 'making ppd';
@awk 'BEGIN{FS="
"}; /^Pa/{print $$0}; /^Pri/{gsub(/,/,""); print $$0}; /^Sec/{print $$0};
/^Pro/{print $$0}; /^Dep/{gsub(/([^)]*)/,""); gsub(/,/,""); gsub(/|/,""); print $$0};
/^Pre/{gsub(/([^)]*)/,""); gsub(/,/,""); gsub(/|/,""); print $$0}; /^$$/{print "Index: "
++n"
"}' available > ppd


prov-flat : prov
@echo 'making prov-flat';
@awk '{x=2; while($$x != ""){print $$1 " " $$x; ++x}}' prov > prov-flat

prov-idx : prov-flat
@echo 'making prov-idx';
@cat prov-flat | sed -e 's/^.* //' | sort | uniq | awk -v l=${LINES} '{print NR+l " " $$1}' > prov-idx

pP-dup : prov-idx prov-flat
@echo 'making pP-dup';
@awk 'BEGIN{while(getline <"prov-idx"){x[$$2]=$$1}} ; {print x[$$2] " " $$1}' prov-flat | sort > pP-dup

pP-dep : pP-dup
@echo 'making pP-dep';
@awk 'BEGIN{ORS=""} ; {if(last == $$1){print " " $$2} else {if(NR>1)print "
"; print $$1" "$$2;}
last=$$1;}' pP-dup > pP-dep


num-dep-1 : pack prov-idx dep
@echo 'making num-dep-1';
@awk 'BEGIN{ORS=""; while(getline <"pack"){x[$$2]=$$1}; while(getline <"prov-idx"){xx[$$2]=$$1}};
{print $$1 ; y=2; while(y <= NF){str=x[$$y]; if(str=="") str=xx[$$y]; print " " str; ++y}; print
"
"}' dep | sed -e 's/0[[:space:]]*||//g ; s/||[[:space:]]*0//g' > num-dep-1


num-dep-2 : num-dep-1
@echo 'making num-dep-2';
@awk '{if($$1>FNR){while(FNR<$$1){print FNR " " FNR; ++FNR}}; if($$2==""){print $$1 " " $$1; next}
else print $$0}' num-dep-1 > num-dep-2


# NOTE
# fix Provides-Depends loops now, right time to sweep that
# could just do until no provs, but does all
# sweep zo L-R, insert new unique as found at zo at nj (non-recursive)
# don't keep a Provides after once de-referencing
# leave all in order found (inserts zi in zo)
# if(jj<=lines) means is not in dep table so try prov table
# don't try combining tables AWK searches funny: really

pP-dep-2 : num-dep-2 pP-dep
@echo 'making pP-dep-2';
@rm pass.log 2>>/dev/null ; awk -v rn=40000 -v lines=${LINES} 'BEGIN { ORS=""; while(getline
<"num-dep-2") x1[$$1]=$$0; while(getline <"pP-dep") x2[$$1]=$$0 } { if($$1==rn) print "#"; if(NF==2
&& $$1==$$2){print $$1" "$$1"
"; next} delete zi; delete zo; delete zb; delete zy; zb[$$1]=1;
zy[$$1]=1; zf=0; nf=NF; split($$0,zo); nj=1; while(++nj<=nf) { jj=zo[nj]; if(jj==0 ||
zb[jj]==1){if($$1==rn) print jj"-->"zo[nj]"<--"zb[jj]; zo[nj]=""; continue} zb[jj]=1; zy[jj]=1;
if($$1==rn) print "%"nj; if(jj<=lines){len=split(x1[jj], str)}else{zo[nj]=""; len=split(x2[jj],
str)} if($$1==rn) print " L="len" jj="jj; k=1; while(++k<=len) { if(zy[str[k]]==0){zy[str[k]]=1;
zi[++zf]=str[k]; if($$1==rn) print "+"} if($$1==rn) print "
rec "$$1"
nf "nf"
nj "nj"
jj
"jj"
k "k"
strk "str[k]"
" } if($$1==rn){print "
zo "; n=0; while(++n<=nf) print zo[n]" ";
print "
zi "; n=0; while(++n<=zf) print zi[n]" "} if(zf>0){n=nf+1; while(--n>nj) zo[n+zf]=zo[n]}
n=0; while(++n<=zf) zo[nj+n]=zi[n]; nf+=zf; if($$1==rn){print "
zo "; n=0; while(++n<=nf) print
zo[n]" "; print "
zi "; n=0; while(++n<=zf) print zi[n]" "; print "
"} delete zi; zf=0; nf2=nf;
if(nj>=lines){print "
"$$0" err: pass#>"lines", see pass.log
" >>"/dev/stderr"; break} } print
"
"$$1" pass "nj >>"pass.log"; k=0; n=0; while(++n<=nf) if(zo[n]!=""){print zo[n] " "; ++k}
if(k==1) print $$1; print "
"; if($$1==rn) exit }' pP-dep > pP-dep-2


num-dep-done : num-dep-2 pP-dep-2
@echo 'making num-dep-done';
@rm pass.log 2>>/dev/null ; awk -v rn=40000 -v lines=${LINES} 'BEGIN { ORS=""; while(getline
<"num-dep-2") x1[$$1]=$$0; while(getline <"pP-dep-2") x2[$$1]=$$0 } { if($$1==rn) print "#";
if(NF==2 && $$1==$$2){print $$1" "$$1"
"; next} delete zi; delete zo; delete zb; delete zy;
zb[$$1]=1; zy[$$1]=1; zf=0; nf=NF; split($$0,zo); nj=1; while(++nj<=nf) { jj=zo[nj]; if(jj==0 ||
zb[jj]==1){if($$1==rn) print jj"-->"zo[nj]"<--"zb[jj]; zo[nj]=""; continue} zb[jj]=1; zy[jj]=1;
if($$1==rn) print "%"nj; if(jj<=lines){len=split(x1[jj], str)}else{zo[nj]=""; len=split(x2[jj],
str)} if($$1==rn) print " L="len" jj="jj; k=1; while(++k<=len) { if(zy[str[k]]==0){zy[str[k]]=1;
zi[++zf]=str[k]; if($$1==rn) print "+"} if($$1==rn) print "
rec "$$1"
nf "nf"
nj "nj"
jj
"jj"
k "k"
strk "str[k]"
" } if($$1==rn){print "
zo "; n=0; while(++n<=nf) print zo[n]" ";
print "
zi "; n=0; while(++n<=zf) print zi[n]" "} if(zf>0){n=nf+1; while(--n>nj) zo[n+zf]=zo[n]}
n=0; while(++n<=zf) zo[nj+n]=zi[n]; nf+=zf; if($$1==rn){print "
zo "; n=0; while(++n<=nf) print
zo[n]" "; print "
zi "; n=0; while(++n<=zf) print zi[n]" "; print "
"} delete zi; zf=0; nf2=nf;
if(nj>=lines){print "
"$$0" err: pass#>"lines", see pass.log
" >>"/dev/stderr"; break} } print
"
"$$1" pass "nj >>"pass.log"; k=0; n=0; while(++n<=nf) if(zo[n]!=""){print zo[n] " "; ++k}
if(k==1) print $$1; print "
"; if($$1==rn) exit }' num-dep-2 > num-dep-done


tsort-in : num-dep-done
@echo 'making tsort-in';
awk '{n=1; while(++n<=NF){print $$1 " " $$n}}' num-dep-done > tsort-in













--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4E17421A.9010906@cox.net">http://lists.debian.org/4E17421A.9010906@cox.net
 
Old 07-08-2011, 05:57 PM
"John D. Hendrickson and Sara Darnell"
 
Default dpkg BUG install order problem & suggestion to add...

debian-dpkg@lists.debian.org


Jonathan Nieder wrote:



John D. Hendrickson and Sara Darnell wrote:



I have a working example of what to do that mostly uses existing
unix tools.


Well, what are you waiting for?


excuse me - let me include "show-all" incase anyone wishes to view tsort-out.


#!/bin/sh
# FILE: show-all
# DESC: works only after tsort-out is generated
#
if [ "$1" == "-h" ] || [ "$1" == "--help" ] ; then
echo "DESCR: show-all packages in tsorted dep order w/ some info"
echo "USAGE: show-all File"
echo "NOTES: File is tsort-in, a dep sorted numberic list"
exit
fi

if [ "$1" == "-f" ] ; then shift 1 ; file="$1" ; else file="tsort-out" ; fi

if [ ! -f ppd ] ; then
available="available"
if [ ! -f $available ] ; then
available="/var/lib/dpkg/available"
fi
awk 'BEGIN{FS="
"}; /^Pa/{print $0}; /^Pri/{gsub(/,/,""); print $0}; /^Sec/{print $0}; /^Pro/{print
$0}; /^Dep/{gsub(/([^)]*)/,""); gsub(/,/,""); gsub(/|/,""); print $0}; /^$/{print ++n"
"}'
$available > ppd

once=1
fi

awk 'BEGIN {ORS=""; pri["req"]=1; pri["imp"]=2; pri["sta"]=3; pri["opt"]=4; pri["ext"]=5; RS="

";
n=1; while(getline <"ppd"){x2[n]=$2; x3[n]=substr($4,1,3); x4[n]=substr($6,1,3); ++n}; RS="
"}; {
print $1 " " x3[$1] " " pri[x3[$1]] " " x4[$1] " " x2[$1] "
" }' $file


if [ $once ] ; then rm ppd ; fi


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4E174517.8020601@cox.net">http://lists.debian.org/4E174517.8020601@cox.net
 
Old 07-08-2011, 06:06 PM
Jonathan Nieder
 
Default dpkg BUG install order problem & suggestion to add...

John D. Hendrickson and Sara Darnell wrote:

> I looked at the dpkg ant aptitude sources. It seems there's allot
> of maybe's involved "grading"

I'm afraid I have no idea what you're talking about, sorry.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110708180640.GA3187@elie">http://lists.debian.org/20110708180640.GA3187@elie
 
Old 07-08-2011, 06:08 PM
"John D. Hendrickson and Sara Darnell"
 
Default dpkg BUG install order problem & suggestion to add...

Jonathan Nieder wrote:

John D. Hendrickson and Sara Darnell wrote:


I looked at the dpkg ant aptitude sources. It seems there's allot
of maybe's involved "grading"


I'm afraid I have no idea what you're talking about, sorry.



tsort-out is a list of all packages in avail sorted by dependancy (as listed by maintainer). easy !


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4E1747B0.3020608@cox.net">http://lists.debian.org/4E1747B0.3020608@cox.net
 

Thread Tools




All times are GMT. The time now is 05:31 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org