FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 06-13-2011, 12:48 PM
Joachim Breitner
 
Default Merging three direcories in postinst

Hi,

in the Haskell packaging we have made a mistake that we want to
work-around without having to sourcefully upload 230 packages.
Eventually, we will have to do these uploads anyways (with the next
major ghc release), so the work-around will not be in place forever.

Here is the situation:

A haskell library consists of a -dev and a -doc package. The -doc
package contains a .haddock file which is placed
under /usr/lib/ghc-$(GHC_VERSION)/haddock/. The -dev part contains a
package description which, among other things, has a reference to the
path. So far so good.

Now -dev is arc:any and -doc is arch:all. A lot of packages were built
with ghc-7.0.2, and, after the upload of ghc-7.0.3, binNMUed. This has
the effect that the .haddock file lies in /usr/lib/ghc-7.0.2/haddock but
the package registry refers to /usr/lib/ghc-7.0.3/haddock and the index
is broken.

The proper thing that should have been done is to install stuff in a
directory not containing the ghc version, e.g. /usr/lib/haddock. This is
what we want to do in the future.

Nevertheless, we want to be able to use the previous builds. I am
considering to do the following:

In ghc’s postinst script, I’d do approximately this:

for dir in /usr/lib/ghc-7.0.2/haddock /usr/lib/ghc-7.0.3/haddock
do
if test -d $dir
then
mv $dir/* /usr/lib/haddock
rmdir $dir
ln -s ../haddock $dir
fi
done

But before doing so, I’d like to get comments as to what can go wrong.
Especially, does dpkg mind if I replace something that dpkg thinks is a
proper directory by a symbolic link? Will it still cleanly remove files
that were installed by libghc-foo-doc into /usr/lib/ghc-7.0.3/haddock/
when libghc-foo-doc is removed? Or is there a neater way?

Greetings,
Joachim


--
Joachim "nomeata" Breitner
Debian Developer
nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
 
Old 06-14-2011, 05:31 PM
Ian Jackson
 
Default Merging three direcories in postinst

Joachim Breitner writes ("Merging three direcories in postinst"):
> But before doing so, I?d like to get comments as to what can go wrong.
> Especially, does dpkg mind if I replace something that dpkg thinks is a
> proper directory by a symbolic link?

Thanks for asking.

No, dpkg will not mind. It will follow the link.

> Will it still cleanly remove files that were installed by
> libghc-foo-doc into /usr/lib/ghc-7.0.3/haddock/ when libghc-foo-doc
> is removed?

Yes, it will. It will also (try to) remove the symlink when it thinks
/usr/lib/ghc-7.0.3 becomes empty.

However, dpkg will not know that
/usr/lib/ghc-7.0.2/haddock/foo.haddock (in old libghc-foo-doc)
and
/usr/lib/haddock/foo.haddock (in old libghc-foo-doc)
are the same file. This will not normally be a problem as dpkg will
ihappily "overwrite" the "unexpected" other file, but you should be
aware of it.

> for dir in /usr/lib/ghc-7.0.2/haddock /usr/lib/ghc-7.0.3/haddock
> do
> if test -d $dir
> then
> mv $dir/* /usr/lib/haddock
> rmdir $dir
> ln -s ../haddock $dir
> fi
> done

This code is buggy because it will always trigger since test -d will
be true for a symlink to a directory.

Also you might want to think about what you want to happen if the same
file is found in two or more of
/usr/lib/haddock
/usr/lib/ghc-7.0.2/haddock
/usr/lib/ghc-7.0.3/haddock

In general I think you have a tenable approach here. But you should
do thorough testing; particularly, you should do testing with
"dpkg -i" and "dpkg --remove" in arbitrary orders, and you should do
testing of what happens if your fixup script falls over halfway
through (eg, put a sleep in it and ^Z then kill -9 the whole dpkg
process stack).

It is important that the sh code you write is idempotent and the
dependencies are all correct. I suggest you come back here (or to
debian-dpkg) and ask for another review when you have a more concrete
implementation.

Regards,
Ian.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 19959.39647.971076.700668@chiark.greenend.org.uk"> http://lists.debian.org/19959.39647.971076.700668@chiark.greenend.org.uk
 
Old 06-14-2011, 06:07 PM
Joachim Breitner
 
Default Merging three direcories in postinst

Hi Ian,

Am Dienstag, den 14.06.2011, 18:31 +0100 schrieb Ian Jackson:
> Joachim Breitner writes ("Merging three direcories in postinst"):
> > Will it still cleanly remove files that were installed by
> > libghc-foo-doc into /usr/lib/ghc-7.0.3/haddock/ when libghc-foo-doc
> > is removed?
>
> Yes, it will. It will also (try to) remove the symlink when it thinks
> /usr/lib/ghc-7.0.3 becomes empty.

Great. This solves one of my worries (how to get rid of the symlinks
when they are not needed any more).

> However, dpkg will not know that
> /usr/lib/ghc-7.0.2/haddock/foo.haddock (in old libghc-foo-doc)
> and
> /usr/lib/haddock/foo.haddock (in old libghc-foo-doc)
> are the same file. This will not normally be a problem as dpkg will
> ihappily "overwrite" the "unexpected" other file, but you should be
> aware of it.

Because the file name contains the package name, there can never be a
case of two installed packages installing files to paths that look
different but are the same after symlinking.

> > for dir in /usr/lib/ghc-7.0.2/haddock /usr/lib/ghc-7.0.3/haddock
> > do
> > if test -d $dir
> > then
> > mv $dir/* /usr/lib/haddock
> > rmdir $dir
> > ln -s ../haddock $dir
> > fi
> > done
>
> This code is buggy because it will always trigger since test -d will
> be true for a symlink to a directory.

Thanks for the pointer. It was only pseudo-pseudo-code at this point, of
course.

> Also you might want to think about what you want to happen if the same
> file is found in two or more of
> /usr/lib/haddock
> /usr/lib/ghc-7.0.2/haddock
> /usr/lib/ghc-7.0.3/haddock
>
> In general I think you have a tenable approach here. But you should
> do thorough testing; particularly, you should do testing with
> "dpkg -i" and "dpkg --remove" in arbitrary orders, and you should do
> testing of what happens if your fixup script falls over halfway
> through (eg, put a sleep in it and ^Z then kill -9 the whole dpkg
> process stack).
>
> It is important that the sh code you write is idempotent and the
> dependencies are all correct. I suggest you come back here (or to
> debian-dpkg) and ask for another review when you have a more concrete
> implementation.

I am a bit worried about getting the removal part right, and of course
there are so many possible failures.

Maybe it is cleaner to just dpkg-divert the .haddock files to the
location in /usr/lib/haddock? I thought I need to access them in the old
locations as well, but the path mentioned is in an arch:any, so it will
be correct (/usr/lib/haddock) after the obligatory round of binNMUs when
the next ghc version is installed.

But since dpkg-divert cannot divert directories, I’d either have to
collect the full list of broken files from the Contents files and
hard-code them, or do it via a trigger, which also seems to be overkill
for a work-around.

I guess I’ll give the postinst-symlink-approach a shot.

Thanks for the advice,
Joachim


--
Joachim "nomeata" Breitner
Debian Developer
nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
 
Old 06-15-2011, 12:35 PM
Ian Jackson
 
Default Merging three direcories in postinst

Joachim Breitner writes ("Re: Merging three direcories in postinst"):
> I am a bit worried about getting the removal part right, and of course
> there are so many possible failures.

It is somewhat tricky, yes. The best thing to do is to try to write
idempotent scripts that always move "bad" states to "better" ones.
And get review :-).

> Maybe it is cleaner to just dpkg-divert the .haddock files to the
> location in /usr/lib/haddock? I thought I need to access them in the old
> locations as well, but the path mentioned is in an arch:any, so it will
> be correct (/usr/lib/haddock) after the obligatory round of binNMUs when
> the next ghc version is installed.

No, I would strongly recommend against attempting to do this with
dpkg-divert. The results will be very unpleasant and the number of
semi-broken corner cases will be enormous.

> But since dpkg-divert cannot divert directories, I?d either have to
> collect the full list of broken files from the Contents files and
> hard-code them, or do it via a trigger, which also seems to be overkill
> for a work-around.

I don't think you need a trigger for this. The code you write should
be fast and can run in the single core package postinst.

> I guess I?ll give the postinst-symlink-approach a shot.

Do come back when you have more final-looking code and I'll see if I
can find bugs in it :-).

Ian.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 19960.42746.953470.615751@chiark.greenend.org.uk"> http://lists.debian.org/19960.42746.953470.615751@chiark.greenend.org.uk
 
Old 06-15-2011, 08:28 PM
Joachim Breitner
 
Default Merging three direcories in postinst

Hi Ian,

Am Mittwoch, den 15.06.2011, 13:35 +0100 schrieb Ian Jackson:
> > I guess I?ll give the postinst-symlink-approach a shot.
>
> Do come back when you have more final-looking code and I'll see if I
> can find bugs in it :-).

I implemented it and did some tests. I did not test aborting it halfway,
but I’m confident that it is ok – the moving of files can be continued.
Only an abortion between rmdir and ln would be bad.

The postinst is here:
http://anonscm.debian.org/darcs/pkg-haskell/ghc/ghc-doc.postinst.in

Unless you see major problems there, I’ll upload it tomorrow.

@DHG: There is no hurry in getting ghc-7.0.4 in. Therefore, I suggest we
do this change with 7.0.3, upload to unstable, and wait until this fix
is in testing.

Greetings,
Joachim

--
Joachim "nomeata" Breitner
Debian Developer
nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
 
Old 06-17-2011, 11:31 AM
Ian Jackson
 
Default Merging three direcories in postinst

Joachim Breitner writes ("Re: Merging three direcories in postinst"):
> Am Mittwoch, den 15.06.2011, 13:35 +0100 schrieb Ian Jackson:
> > > I guess I?ll give the postinst-symlink-approach a shot.
> >
> > Do come back when you have more final-looking code and I'll see if I
> > can find bugs in it :-).
>
> I implemented it and did some tests. I did not test aborting it halfway,
> but I?m confident that it is ok ? the moving of files can be continued.
> Only an abortion between rmdir and ln would be bad.
>
> The postinst is here:
> http://anonscm.debian.org/darcs/pkg-haskell/ghc/ghc-doc.postinst.in

Are you sure this is the version you tested ? Your tests don't seem
to have been very comprehensive ...

I see this:
find $path -maxdepth 1 -mindepth 1 -print0 | xargs -0 -r mv /usr/lib/ghc-doc/haddock/
which doesn't look like it will do what you want!

Ian.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 19963.15104.534406.280177@chiark.greenend.org.uk"> http://lists.debian.org/19963.15104.534406.280177@chiark.greenend.org.uk
 
Old 06-17-2011, 02:38 PM
Joachim Breitner
 
Default Merging three direcories in postinst

Hi,

Am Freitag, den 17.06.2011, 12:31 +0100 schrieb Ian Jackson:
> Joachim Breitner writes ("Re: Merging three direcories in postinst"):
> > Am Mittwoch, den 15.06.2011, 13:35 +0100 schrieb Ian Jackson:
> > > > I guess I?ll give the postinst-symlink-approach a shot.
> > >
> > > Do come back when you have more final-looking code and I'll see if I
> > > can find bugs in it :-).
> >
> > I implemented it and did some tests. I did not test aborting it halfway,
> > but I?m confident that it is ok ? the moving of files can be continued.
> > Only an abortion between rmdir and ln would be bad.
> >
> > The postinst is here:
> > http://anonscm.debian.org/darcs/pkg-haskell/ghc/ghc-doc.postinst.in
>
> Are you sure this is the version you tested ? Your tests don't seem
> to have been very comprehensive ...
>
> I see this:
> find $path -maxdepth 1 -mindepth 1 -print0 | xargs -0 -r mv /usr/lib/ghc-doc/haddock/
> which doesn't look like it will do what you want!

Strange. The "-t" after mv got missing. But it was working at some
point... guess that was when it was still the "mv" command, and then
this command line was not checked in all cases.

I’ll add it, retry my tests, and come back.

BTW, this line (with "-t" after mv") ought to do the same as
mv $path/* /usr/lib/ghc-doc/haddock/
without failing when $path is empty. Is there a nicer way than find and
xargs?

Greetings,
Joachim

--
Joachim "nomeata" Breitner
Debian Developer
nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
 
Old 06-17-2011, 02:58 PM
Ian Jackson
 
Default Merging three direcories in postinst

Joachim Breitner writes ("Re: Merging three direcories in postinst"):
> BTW, this line (with "-t" after mv") ought to do the same as
> mv $path/* /usr/lib/ghc-doc/haddock/
> without failing when $path is empty. Is there a nicer way than find and
> xargs?

Use bash and set -o nullglob, perhaps. You'd have to faff about a bit
because mv won't like an empty argument list.

Ian.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 19963.27536.972193.484933@chiark.greenend.org.uk"> http://lists.debian.org/19963.27536.972193.484933@chiark.greenend.org.uk
 
Old 06-17-2011, 03:07 PM
Joachim Breitner
 
Default Merging three direcories in postinst

Hi again,

Am Freitag, den 17.06.2011, 16:38 +0200 schrieb Joachim Breitner:
> > Are you sure this is the version you tested ? Your tests don't seem
> > to have been very comprehensive ...
> >
> > I see this:
> > find $path -maxdepth 1 -mindepth 1 -print0 | xargs -0 -r mv /usr/lib/ghc-doc/haddock/
> > which doesn't look like it will do what you want!
>
> Strange. The "-t" after mv got missing. But it was working at some
> point... guess that was when it was still the "mv" command, and then
> this command line was not checked in all cases.
>
> I’ll add it, retry my tests, and come back.

allright, added the -t that should have been there and re-tested various
order of installing and purging ghc-doc and libghc-xhtml-doc. Thinks
work smoothly, with one drawback:

If ghc-doc is removed before libghc-xhtml-doc, /usr/lib/ghc-doc/haddock/
is left over as an empty directory. Because ghc-doc cannot remove it, as
the .haddock file from libghc-xhtml-doc resides in it. And when
libghc-xhtml-doc is removed, dpkg only removes the .haddock file and
symlink that used to be the directory it was in.

Theoretically, the correct solution is to undo the symlinking and moving
the files back when ghc-doc is removed. But this would require
remembering where the files came from and does not seem to be worth the
trouble and risk of breakage. If later ghc-doc is re-installed, things
are fine again. Also, once all libghc-*-doc packages move their -doc
files to the proper location, the problem will disappear automatically.

Current code at
http://anonscm.debian.org/darcs/pkg-haskell/ghc/ghc-doc.postinst.in

Greetings,
Joachim
--
Joachim "nomeata" Breitner
Debian Developer
nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
 
Old 06-17-2011, 03:25 PM
Ian Jackson
 
Default Merging three direcories in postinst

Joachim Breitner writes ("Re: Merging three direcories in postinst"):
> If ghc-doc is removed before libghc-xhtml-doc, /usr/lib/ghc-doc/haddock/
> is left over as an empty directory. Because ghc-doc cannot remove it, as
> the .haddock file from libghc-xhtml-doc resides in it. And when
> libghc-xhtml-doc is removed, dpkg only removes the .haddock file and
> symlink that used to be the directory it was in.

If you
remove ghc-doc
remove libghc-xhtml-doc
purge ghc-doc
does it work ?

One option might be to make the ghc-doc postrm purge check that the
directory is empty and bomb out if it isn't. But I'm not sure whether
this cure (blocking purge of a package with no configuration etc.)
isn't worse than the disease (a leftover cruft empty directory).

> Theoretically, the correct solution is to undo the symlinking and moving
> the files back when ghc-doc is removed. But this would require
> remembering where the files came from and does not seem to be worth the
> trouble and risk of breakage. If later ghc-doc is re-installed, things
> are fine again. Also, once all libghc-*-doc packages move their -doc
> files to the proper location, the problem will disappear automatically.

No, I think that is definitely not the right thing to do. It will be
very fragile, as you say.

> Current code at
> http://anonscm.debian.org/darcs/pkg-haskell/ghc/ghc-doc.postinst.in

That looks plausible.

Thanks,
Ian.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 19963.29184.276850.298224@chiark.greenend.org.uk"> http://lists.debian.org/19963.29184.276850.298224@chiark.greenend.org.uk
 

Thread Tools




All times are GMT. The time now is 06:46 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org