FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian dpkg

 
 
LinkBack Thread Tools
 
Old 02-14-2012, 01:28 PM
Ian Jackson
 
Default Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

Guillem Jover writes ("Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)"):
> On Mon, 2012-02-13 at 22:43:04 -0800, Russ Allbery wrote:
> > * The binNMU process is changed to add the binNMU changelog entry to an
> > arch-qualified file (changelog.Debian.arch, probably). We need to
> > figure out what this means if the package being binNMU'd has a
> > /usr/share/doc/<package> symlink to another package, though; it's not
> > obvious what to do here.
>
> This requires IMO multitude of hacks when the simplest and obvious
> arch-qualified pkgname solves this cleanly, and allows debhelper to
> automatically deal with it. And for tools to just change where they
> always look for those files in the M-A:same case regardless of the
> package being binNMUed or not.

I agree that it would be nice to always arch-qualify the changelog
filename. But that would involve a lot of changes to
changelog-reading tools which we perhaps don't want to do right now.

Note that even if we decide to always arch-qualify, we will still have
lots of old packages so all changelog-reading tools will need to look
in both places.

For most changelog-reading tools it won't be very troublesome if they
accidentally don't spot a binNMU entry. So Russ's proposal is a good
step towards your proposal. And if we decide we don't need to go all
the way then it's good enough for now.

> This still does not solve the other issues I listed, namely binNMUs
> have to be performed in lock-step, more complicated transitions /
> upgrades.

I don't think I see where this is coming from. Are you talking about
variation in gzip output ? Given the evidence we've seen here, in
practice I think that is not going to be a problem. Certainly it
won't demand that binNMUs be performed in lock-step.

> So this is still pretty much unconvincing, and seems like clinging
> into the refcnt'ing “solution” while it makes things overall more
> complicated, will introduce inconsistency and incertainty to
> maintainers, needs way more global changes to keep it going, etc.

I think the refcounting approach is very worthwhile because it
eliminates unnecessary work (by human maintainers) in many simple
cases.

Ian.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20282.28586.577528.890135@chiark.greenend.org.uk"> http://lists.debian.org/20282.28586.577528.890135@chiark.greenend.org.uk
 
Old 02-14-2012, 01:40 PM
Josselin Mouette
 
Default Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

Le lundi 13 fvrier 2012 22:43 -0800, Russ Allbery a crit :
> There's been a lot of discussion of this, but it seems to have been fairly
> inconclusive. We need to decide what we're doing, if anything, for wheezy
> fairly soon, so I think we need to try to drive this discussion to some
> concrete conclusions.

Thank you very much for your constructive work.

> 3. Generated documentation. Here's where I think refcounting starts
> failing.

So we need to move a lot of documentation generated with gtk-doc or
doxygen from -dev packages to -doc packages. But it really seems an
acceptable tradeoff between the amount of work required and the
cleanness of the solution.

> Does this seem comprehensive to everyone? Am I missing any cases?

Are there any cases of configuration files in /etc that vary across
architectures? Think of stuff like ld.so.conf, where some plugins or
library path is coded in a configuration file.

--
.'`. Josselin Mouette
: :' :
`. `'
`-


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 1329230441.3297.378.camel@pi0307572">http://lists.debian.org/1329230441.3297.378.camel@pi0307572
 
Old 02-14-2012, 01:40 PM
Josselin Mouette
 
Default Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

Le lundi 13 fvrier 2012 22:43 -0800, Russ Allbery a crit :
> There's been a lot of discussion of this, but it seems to have been fairly
> inconclusive. We need to decide what we're doing, if anything, for wheezy
> fairly soon, so I think we need to try to drive this discussion to some
> concrete conclusions.

Thank you very much for your constructive work.

> 3. Generated documentation. Here's where I think refcounting starts
> failing.

So we need to move a lot of documentation generated with gtk-doc or
doxygen from -dev packages to -doc packages. But it really seems an
acceptable tradeoff between the amount of work required and the
cleanness of the solution.

> Does this seem comprehensive to everyone? Am I missing any cases?

Are there any cases of configuration files in /etc that vary across
architectures? Think of stuff like ld.so.conf, where some plugins or
library path is coded in a configuration file.

--
.'`. Josselin Mouette
: :' :
`. `'
`-


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 1329230441.3297.378.camel@pi0307572">http://lists.debian.org/1329230441.3297.378.camel@pi0307572
 
Old 02-14-2012, 01:43 PM
Jakub Wilk
 
Default Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

* Raphael Hertzog <hertzog@debian.org>, 2012-02-14, 14:17:
dpkg-buildpackage --binary-version <ver> --binary-changelog 'foo' could
create debian/changelog.build with the given changelog version and
changelog entry.


dpkg-parsechangelog could be taught to read debian/changelog.build
before debian/changelog so that dpkg-parsechangelog continues to do the
right thing (when called from debian/rules).


And dh_installchangelogs can be taught to install
debian/changelog.build as
/usr/share/doc/<foo>/changelog.Debian.build-$arch.


dpkg-buildpackage would clean up debian/changelog.build if it wasn't
passed the proper option. dpkg-source would learn to not include it in
generated source packages, too.


This looks like rather appealing to me. What do you think?


Yes, it does look appealing. But...

Are we sure than no existing package uses debian/changelog.build for
their own purposes?


Are we sure that all existing packages (and helpers) that parse
debian/changelog use dpkg-parsechangelog?


--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120214144341.GA3346@jwilk.net">http://lists.debian.org/20120214144341.GA3346@jwilk.net
 
Old 02-14-2012, 01:43 PM
Jakub Wilk
 
Default Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

* Raphael Hertzog <hertzog@debian.org>, 2012-02-14, 14:17:
dpkg-buildpackage --binary-version <ver> --binary-changelog 'foo' could
create debian/changelog.build with the given changelog version and
changelog entry.


dpkg-parsechangelog could be taught to read debian/changelog.build
before debian/changelog so that dpkg-parsechangelog continues to do the
right thing (when called from debian/rules).


And dh_installchangelogs can be taught to install
debian/changelog.build as
/usr/share/doc/<foo>/changelog.Debian.build-$arch.


dpkg-buildpackage would clean up debian/changelog.build if it wasn't
passed the proper option. dpkg-source would learn to not include it in
generated source packages, too.


This looks like rather appealing to me. What do you think?


Yes, it does look appealing. But...

Are we sure than no existing package uses debian/changelog.build for
their own purposes?


Are we sure that all existing packages (and helpers) that parse
debian/changelog use dpkg-parsechangelog?


--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120214144341.GA3346@jwilk.net">http://lists.debian.org/20120214144341.GA3346@jwilk.net
 
Old 02-14-2012, 02:13 PM
Raphael Hertzog
 
Default Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

Hi,

On Tue, 14 Feb 2012, Guillem Jover wrote:
> > * All packages that want to be multiarch: same have to move all generated
> > documentation into a separate package unless the maintainer has very
> > carefully checked that the generated documentation will be byte-for-byte
> > identical even across minor updates of the documentation generation
> > tools and when run at different times.
>
> If packages have to be split anyway to cope with the other cases, then
> the number of new packages which might not be needed otherwise will be
> even smaller than the predicted amount, at which point it makes even
> less sense to support refcnt'ing.

Why are you so opposed to the refcnt'ing?

It's not such a big deal to maintain this feature in dpkg. And even if the
current implementation is not perfect, it can be improved later when dpkg
will store by itself checksums of provided files.

To me it looks like you don't like refcnt'ing and you're trying to find
some reasons to make it unacceptable.

> It also requires maintainers to carefully consider if the (doc, etc)
> toolchains will generate predictible ouput.

If the maintainer has to install files in non-standard path (because of
the need to arch-qualify it), it will also need maintainers to carefully
consider how to ensure that this move doesn't break anything.

It's not a white/black situation. You're trading one potential problem for
another. And the differing files are likely to be much more easy to spot
than other behaviour changes that might be implied by the move of some
files to arch qualified paths.

> Your proposal still requires papering over the other corner-cases.

Can you be explicit about which corner cases you're referring to ?

> This still does not solve the other issues I listed, namely binNMUs
> have to be performed in lock-step

Can you explain why? If the binnmu changelog is in a arch-specific file,
then we're free to bin-nmu packages separately.

dpkg must just ensure that all "M-A: same" packages have the same source
version (instead of the binary version as currently).

>, more complicated transitions / upgrades.

We have no experience on this. It's a bit early to say whether those
constraints are going to be problematic or not.

> And introduces different solutions for different problems, while my
> proposal is generic for all cases.

There's nothing like a generic solution. You still have to decide whether
you move files to a -common package or if you arch qualify them and keep
them in the M-A: same package. And in both cases, you have to evaluate the
implications, in terms of package installation ordering in one case, in
terms of modifications to do to properly support the arch-qualified files
in the other one.

While it may sound like "cleaner" from a theoretical point of view, I'm
not convinced that it's better than the approach outlined by Russ.

Also you completely ignore the fact that what you're proposing is an
important change for multi-arch packages that have already been converted
both in Debian and in Ubuntu. You're pushing back the work to package
maintainers when there's not reason to not deal with this at the build
infrastructure level.

To reduce some of the downsides associated to compressed files in M-A:
same packages, we could/should investigate how to not compress files
in such packages instead of duplicating them needlessly.

> So this is still pretty much unconvincing, and seems like clinging
> into the refcnt'ing “solution” while it makes things overall more
> complicated, will introduce inconsistency and incertainty to
> maintainers, needs way more global changes to keep it going, etc.

This is not a fair characterization of the situation. IMO "Global changes" are
better than "lots of maintainers having to do busy-work splitting their
packages".

You see inconsistency in Russ's proposal but you don't see
inconsistency/incertainty when you change the standard location of
changelog files.

And the "more complicated", it might be true at the dpkg level, but I
don't believe that it's true from the maintainers points of view.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Pre-order a copy of the Debian Administrator's Handbook and help
liberate it: http://debian-handbook.info/liberation/


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120214151318.GA14915@rivendell.home.ouaza.com">h ttp://lists.debian.org/20120214151318.GA14915@rivendell.home.ouaza.com
 
Old 02-14-2012, 02:13 PM
Raphael Hertzog
 
Default Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

Hi,

On Tue, 14 Feb 2012, Guillem Jover wrote:
> > * All packages that want to be multiarch: same have to move all generated
> > documentation into a separate package unless the maintainer has very
> > carefully checked that the generated documentation will be byte-for-byte
> > identical even across minor updates of the documentation generation
> > tools and when run at different times.
>
> If packages have to be split anyway to cope with the other cases, then
> the number of new packages which might not be needed otherwise will be
> even smaller than the predicted amount, at which point it makes even
> less sense to support refcnt'ing.

Why are you so opposed to the refcnt'ing?

It's not such a big deal to maintain this feature in dpkg. And even if the
current implementation is not perfect, it can be improved later when dpkg
will store by itself checksums of provided files.

To me it looks like you don't like refcnt'ing and you're trying to find
some reasons to make it unacceptable.

> It also requires maintainers to carefully consider if the (doc, etc)
> toolchains will generate predictible ouput.

If the maintainer has to install files in non-standard path (because of
the need to arch-qualify it), it will also need maintainers to carefully
consider how to ensure that this move doesn't break anything.

It's not a white/black situation. You're trading one potential problem for
another. And the differing files are likely to be much more easy to spot
than other behaviour changes that might be implied by the move of some
files to arch qualified paths.

> Your proposal still requires papering over the other corner-cases.

Can you be explicit about which corner cases you're referring to ?

> This still does not solve the other issues I listed, namely binNMUs
> have to be performed in lock-step

Can you explain why? If the binnmu changelog is in a arch-specific file,
then we're free to bin-nmu packages separately.

dpkg must just ensure that all "M-A: same" packages have the same source
version (instead of the binary version as currently).

>, more complicated transitions / upgrades.

We have no experience on this. It's a bit early to say whether those
constraints are going to be problematic or not.

> And introduces different solutions for different problems, while my
> proposal is generic for all cases.

There's nothing like a generic solution. You still have to decide whether
you move files to a -common package or if you arch qualify them and keep
them in the M-A: same package. And in both cases, you have to evaluate the
implications, in terms of package installation ordering in one case, in
terms of modifications to do to properly support the arch-qualified files
in the other one.

While it may sound like "cleaner" from a theoretical point of view, I'm
not convinced that it's better than the approach outlined by Russ.

Also you completely ignore the fact that what you're proposing is an
important change for multi-arch packages that have already been converted
both in Debian and in Ubuntu. You're pushing back the work to package
maintainers when there's not reason to not deal with this at the build
infrastructure level.

To reduce some of the downsides associated to compressed files in M-A:
same packages, we could/should investigate how to not compress files
in such packages instead of duplicating them needlessly.

> So this is still pretty much unconvincing, and seems like clinging
> into the refcnt'ing “solution” while it makes things overall more
> complicated, will introduce inconsistency and incertainty to
> maintainers, needs way more global changes to keep it going, etc.

This is not a fair characterization of the situation. IMO "Global changes" are
better than "lots of maintainers having to do busy-work splitting their
packages".

You see inconsistency in Russ's proposal but you don't see
inconsistency/incertainty when you change the standard location of
changelog files.

And the "more complicated", it might be true at the dpkg level, but I
don't believe that it's true from the maintainers points of view.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Pre-order a copy of the Debian Administrator's Handbook and help
liberate it: http://debian-handbook.info/liberation/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120214151318.GA14915@rivendell.home.ouaza.com">h ttp://lists.debian.org/20120214151318.GA14915@rivendell.home.ouaza.com
 
Old 02-14-2012, 02:27 PM
Raphael Hertzog
 
Default Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

On Tue, 14 Feb 2012, Jakub Wilk wrote:
> Are we sure than no existing package uses debian/changelog.build for
> their own purposes?

No, but with debian/changelog.dpkg-build we should be safe.

> Are we sure that all existing packages (and helpers) that parse
> debian/changelog use dpkg-parsechangelog?

No, but I would consider anything else as a bug and we would notice
relatively quickly (we could even do a full rebuild to try to verify
pro-actively).

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Pre-order a copy of the Debian Administrator's Handbook and help
liberate it: http://debian-handbook.info/liberation/


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120214152757.GC14915@rivendell.home.ouaza.com">h ttp://lists.debian.org/20120214152757.GC14915@rivendell.home.ouaza.com
 
Old 02-14-2012, 02:27 PM
Raphael Hertzog
 
Default Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

On Tue, 14 Feb 2012, Jakub Wilk wrote:
> Are we sure than no existing package uses debian/changelog.build for
> their own purposes?

No, but with debian/changelog.dpkg-build we should be safe.

> Are we sure that all existing packages (and helpers) that parse
> debian/changelog use dpkg-parsechangelog?

No, but I would consider anything else as a bug and we would notice
relatively quickly (we could even do a full rebuild to try to verify
pro-actively).

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Pre-order a copy of the Debian Administrator's Handbook and help
liberate it: http://debian-handbook.info/liberation/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120214152757.GC14915@rivendell.home.ouaza.com">h ttp://lists.debian.org/20120214152757.GC14915@rivendell.home.ouaza.com
 
Old 02-14-2012, 03:40 PM
Guillem Jover
 
Default Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

On Tue, 2012-02-14 at 14:28:58 +0000, Ian Jackson wrote:
> Guillem Jover writes ("Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)"):
> > On Mon, 2012-02-13 at 22:43:04 -0800, Russ Allbery wrote:
> > > * The binNMU process is changed to add the binNMU changelog entry to an
> > > arch-qualified file (changelog.Debian.arch, probably). We need to
> > > figure out what this means if the package being binNMU'd has a
> > > /usr/share/doc/<package> symlink to another package, though; it's not
> > > obvious what to do here.
> >
> > This requires IMO multitude of hacks when the simplest and obvious
> > arch-qualified pkgname solves this cleanly, and allows debhelper to
> > automatically deal with it. And for tools to just change where they
> > always look for those files in the M-A:same case regardless of the
> > package being binNMUed or not.
>
> I agree that it would be nice to always arch-qualify the changelog
> filename. But that would involve a lot of changes to
> changelog-reading tools which we perhaps don't want to do right now.

I've never proposed to arch-qualify the filename for the stuff under
/usr/share/doc/pkgname/, I've proposed to arch-qualify the pkgname in
the path (/usr/share/doc/pkgname:arch/), but only for M-A:same packages,
which are the only ones needing the disambiguation. This is how dpkg
handles pkgname output, or how it stores their data in the db too.

And it should be easy to ask a multiarch enabled dpkg-query for example
to normalize the pkgname output to be used on those paths, or otherwise
do it by hand:

if M-A == same
pkgname:arch
else
pkgname

> Note that even if we decide to always arch-qualify, we will still have
> lots of old packages so all changelog-reading tools will need to look
> in both places.

> For most changelog-reading tools it won't be very troublesome if they
> accidentally don't spot a binNMU entry. So Russ's proposal is a good
> step towards your proposal. And if we decide we don't need to go all
> the way then it's good enough for now.

How many tools are there that actually read the binary package changelog
file anyway? I only know of packages.d.o. Any other tool reading from
the installed path, cannot really rely on it being present at all
anyway, per policy.

And in addition, binNMU split changelogs are going to be there forever,
and as such their possible double locations. While the possible double
location for M-A:same packages using pkgname:arch qualified pathnames
would only be temporary and disappear once the packages have been rebuilt
with a new debhelper which automatically installs them in the correct
place.

> > So this is still pretty much unconvincing, and seems like clinging
> > into the refcnt'ing “solution” while it makes things overall more
> > complicated, will introduce inconsistency and incertainty to
> > maintainers, needs way more global changes to keep it going, etc.
>
> I think the refcounting approach is very worthwhile because it
> eliminates unnecessary work (by human maintainers) in many simple
> cases.

As I mentioned in Riku's reply, the amount of packages that would need
splitting that would otherwise not be needed should be even less than
before (which was predicted at around 700), also as I mentioned there
too, nothing prevents us from arch-qualifying paths (with Debian arch
or multiarch triplet depending on the case) if that's more convenient
or safer (as per your essential data example), and is what we've been
doing anyway for arch-indep data shipped in arch:any packages all along.
Given the amount of hacks or special casing piling up to make refcnt'ing
workable, when all that's really needed is a one time handling (or a
possible additional change for already converted packages, for things
that debhelper might not be able to handle) of moving qualifying paths
or splitting into new packages, it really does not seem worth it, no.

regards,
guillem


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120214164015.GA27571@gaara.hadrons.org">http://lists.debian.org/20120214164015.GA27571@gaara.hadrons.org
 

Thread Tools




All times are GMT. The time now is 04:57 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org