Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Debian dpkg (http://www.linux-archive.org/debian-dpkg/)
-   -   BinNMU changelog handling for Multi-Arch: same packages (http://www.linux-archive.org/debian-dpkg/682324-binnmu-changelog-handling-multi-arch-same-packages.html)

Jonathan Nieder 07-11-2012 07:35 AM

BinNMU changelog handling for Multi-Arch: same packages
 
Hi Raphael,

Raphael Hertzog wrote:

> I know that in the long term you're in favor of moving the changelog in
> the package metadata and I agree with this plan. But IMO we must find
> an interim solution in the mean time.
>
> Here's a suggestion. Please share your thoughts:
>
> 1/ we modify dh_installchangelog to strip the binary-only changelog entry
> for Multi-Arch: same packages
[...]
> 2/ we modify dpkg to allow co-installation of M-A: same packages which share the
> same source version regardless of the binary version
>
> 3/ we modify sbuild to add the required "binary-only=yes" in the binNMU
> changelog entries. Here's a sample header line:
>
> ftplib (3.1-1-9+b1) unstable; urgency=low, binary-only=yes

(2) and (3) sound like very good things.

Wouldn't (1) be throwing away information, unless the stripped
information goes into another file?

Making the stripped info go into another file sounds fine to me.

A crazier possibility is teaching the unpack procedure to treat
/usr/share/doc/<package>/changelog.Debian.gz specially, collecting the
binary-only changelog entries and merging them in a single file, but
that's a pretty severe layering violation and it would not be easy to
find which entries are no longer relevant when shrinking the set of
installed arches for a package.

Thanks,
Jonathan


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120711073537.GA2006@burratino

Raphael Hertzog 07-11-2012 11:44 AM

BinNMU changelog handling for Multi-Arch: same packages
 
On Wed, 11 Jul 2012, Jonathan Nieder wrote:
> > 1/ we modify dh_installchangelog to strip the binary-only changelog entry
> > for Multi-Arch: same packages
>
> Wouldn't (1) be throwing away information, unless the stripped
> information goes into another file?
>
> Making the stripped info go into another file sounds fine to me.

Yes, that's another possibility but this changelog entry is not so
important that it has to be in the .deb file IMO. The information
is already in the .changes file, and thus in the corresponding build
logs.

Both approaches are fine for me (i.e. throwing the entry away or
diverting it to another file).

> A crazier possibility is teaching the unpack procedure to treat
> /usr/share/doc/<package>/changelog.Debian.gz specially, collecting the
> binary-only changelog entries and merging them in a single file, but
> that's a pretty severe layering violation and it would not be easy to
> find which entries are no longer relevant when shrinking the set of
> installed arches for a package.

Indeed, that's a no-no, way too crazy.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Get the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120711114439.GD11359@rivendell.home.ouaza.com">h ttp://lists.debian.org/20120711114439.GD11359@rivendell.home.ouaza.com

Russ Allbery 07-11-2012 04:10 PM

BinNMU changelog handling for Multi-Arch: same packages
 
Raphael Hertzog <hertzog@debian.org> writes:
> On Wed, 11 Jul 2012, Jonathan Nieder wrote:

>> Wouldn't (1) be throwing away information, unless the stripped
>> information goes into another file?

>> Making the stripped info go into another file sounds fine to me.

> Yes, that's another possibility but this changelog entry is not so
> important that it has to be in the .deb file IMO. The information
> is already in the .changes file, and thus in the corresponding build
> logs.

> Both approaches are fine for me (i.e. throwing the entry away or
> diverting it to another file).

I don't think it's mandatory that we save the information, but it
certainly would be nice. I'd like apt-listchanges to show me the binary
NMU changelog. I've used that information before on the local system, and
that's a lot more convenient than going to the web and trying to do
archeology on what happened.

Saving the binary NMU changelog in a separate file feels like the right
solution to me.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 87bojm8ejo.fsf@windlord.stanford.edu">http://lists.debian.org/87bojm8ejo.fsf@windlord.stanford.edu

Raphael Hertzog 07-11-2012 06:45 PM

BinNMU changelog handling for Multi-Arch: same packages
 
On Wed, 11 Jul 2012, Russ Allbery wrote:
> I don't think it's mandatory that we save the information, but it
> certainly would be nice. I'd like apt-listchanges to show me the binary
> NMU changelog. I've used that information before on the local system, and
> that's a lot more convenient than going to the web and trying to do
> archeology on what happened.
>
> Saving the binary NMU changelog in a separate file feels like the right
> solution to me.

The right *temporary* solution then. Remember that this was meant as an
intermediary solution until the full changelog (with the bin-nmu entry)
is just integrated in the package medata (control.tar).

But it's true that it might require more than a cycle until we're there so
it's probably best to not drop this information, even if it's only a
temporary measure.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Get the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120711184515.GG14405@rivendell.home.ouaza.com">h ttp://lists.debian.org/20120711184515.GG14405@rivendell.home.ouaza.com

Julien Cristau 07-11-2012 07:23 PM

BinNMU changelog handling for Multi-Arch: same packages
 
On Wed, Jul 11, 2012 at 09:23:05 +0200, Raphael Hertzog wrote:

> I know that in the long term you're in favor of moving the changelog in
> the package metadata and I agree with this plan. But IMO we must find
> an interim solution in the mean time.
>
Whatever solution ends up being chosen in the end (whether it's dropping
the binNMU changelog, moving it to a separate file, or moving the whole
changelog away, I don't hugely care), it's too late to make these
changes for wheezy IMO.

Cheers,
Julien

Russ Allbery 07-12-2012 12:01 AM

BinNMU changelog handling for Multi-Arch: same packages
 
Raphael Hertzog <hertzog@debian.org> writes:

> The right *temporary* solution then. Remember that this was meant as an
> intermediary solution until the full changelog (with the bin-nmu entry)
> is just integrated in the package medata (control.tar).

Oh, yes, absolutely agreed. Sorry for not making that clear. Putting the
changelog in the package metadata makes a ton of sense. In fact, if we
could also do that with the copyright file, that would eliminate nearly
all of our issues with linked doc directories and would simplify a whole
currently-contentious area of Policy, replacing it with a much simpler
debate about the right interface to view those files for installed
packages.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 87394xn8yz.fsf@windlord.stanford.edu">http://lists.debian.org/87394xn8yz.fsf@windlord.stanford.edu

Russ Allbery 07-12-2012 12:15 AM

BinNMU changelog handling for Multi-Arch: same packages
 
Julien Cristau <jcristau@debian.org> writes:
> On Wed, Jul 11, 2012 at 09:23:05 +0200, Raphael Hertzog wrote:

>> I know that in the long term you're in favor of moving the changelog in
>> the package metadata and I agree with this plan. But IMO we must find
>> an interim solution in the mean time.

> Whatever solution ends up being chosen in the end (whether it's dropping
> the binNMU changelog, moving it to a separate file, or moving the whole
> changelog away, I don't hugely care), it's too late to make these
> changes for wheezy IMO.

My gut instinct is to agree. Given the incomplete multiarch conversion,
it seems like we should just do a final consistency binNMU on all affected
packages right before the wheezy release so that they all match (I assume
it's possible to do that? if not, we could do a sourceful upload/NMU
through testing-proposed-updates) and call it good enough for wheezy.
Stable updates are unlikely to have this problem, since I believe binNMUs
are very rare inside stable.

Doing new feature and design work in dpkg at this point in the release
cycle doesn't seem like a good idea.

I think using the separate file approach makes sense for wheezy+1 if the
dpkg maintainers don't think that the move to package metadata will be
done in time.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 87mx35ltrt.fsf@windlord.stanford.edu">http://lists.debian.org/87mx35ltrt.fsf@windlord.stanford.edu

Neil Williams 07-12-2012 04:17 AM

BinNMU changelog handling for Multi-Arch: same packages
 
On Wed, 11 Jul 2012 17:01:24 -0700
Russ Allbery <rra@debian.org> wrote:

> Raphael Hertzog <hertzog@debian.org> writes:
>
> > The right *temporary* solution then. Remember that this was meant as an
> > intermediary solution until the full changelog (with the bin-nmu entry)
> > is just integrated in the package medata (control.tar).

Please don't put that into the files used to prepare content for dpkg
-s and apt-cache - that output is large enough as it is. A single step
like this could seriously compromise package handling on low resource
machines and push apt passed it's memory mapping limits again even on
more powerful machines.

> Oh, yes, absolutely agreed. Sorry for not making that clear. Putting the
> changelog in the package metadata makes a ton of sense. In fact, if we
> could also do that with the copyright file, that would eliminate nearly
> all of our issues with linked doc directories and would simplify a whole
> currently-contentious area of Policy, replacing it with a much simpler
> debate about the right interface to view those files for installed
> packages.

... and that would be even worse if not isolated from the status and
cache information at the point where it is unpacked.

Wherever the data lives inside the .deb is not the problem.

Where the data gets cached, copied, listed and parsed is likely to be a
major problem.

--


Neil Williams
=============
http://www.linux.codehelp.co.uk/

Guillem Jover 07-12-2012 05:06 AM

BinNMU changelog handling for Multi-Arch: same packages
 
Sigh, again...

On Wed, 2012-07-11 at 09:23:05 +0200, Raphael Hertzog wrote:
> On Wed, 20 Jun 2012, Guillem Jover wrote:
> > I'll be doing a first push today. The remaning things I'll be finishing
> > up next are at least the strings cleanup left out from the 1.16.4
> > release, the cross-multiarch patches, part of the changelog binNMU
> > solution, and some other multiarch related improvements.
>
> So it looks like that the "part of the changelog binNMU solution"
> was just the possibility to tag a changelog entry "binary-only"
> with a keyword.

> But that doesn't solve the release team's problem of having to schedule
> bin-nmus for all arches for Multi-Arch: same.

No, the part of the solution was to create the needed user and program
infrastructure interfaces to retrieve the metadata files from the db in a
future-proof way. That's the new commands «dpkg-query --control-list pkg»
and «dpkg-query --control-show pkg file», which should eventually replace
the previous semi-private «dpkg-query --control-path pkg [file]».

There's no other changes required from the dpkg side. To get changelog
(and possibly copyright files) as package metadata, I think the only
remaining things that would need to happen if the project agreed that's
the right path would be:

* Change apt-listchanges to use «dpkg-deb -I pkg changelog» to try to
get the package changelog.
* Change the “website” to use «dpkg-deb -I pkg changelog» to try to
get the package changelog (and possibly copyright).
* Change policy to allow packages to ship changelogs (and possibly
copyright) as package metadata instead of «/use/share/doc/<pkg>/».
* Change lintian per the above.
* Change dh_installchangelogs to install the changelog in the DEBIAN/
dir instaed of «/use/share/doc/<pkg>/».
* Progressively change any remaning package not using debhelper to
store these under DEBIAN/.

? If there's any program showing changelog files from installed paths
switch them to use «dpkg-query --control-show pkg changelog».

> I know that in the long term you're in favor of moving the changelog in
> the package metadata and I agree with this plan. But IMO we must find
> an interim solution in the mean time.

First, I don't see why we _must_, it's been a known limitation of the
spec for a long time and as Julien said now it's probably too late
anyway, there's always the possibility for a last sourceful upload
before the release, and I've said before I think a solution to this
should really not be rushed...

*But* if something needed to be done, I keep failing to see the point
in temporary hacks which imply, as much if not more work (as it needs
to be reverted back and switched to the new scheme) or wrongness,
instead of just going for the metadata solution...

> Here's a suggestion. Please share your thoughts:
>
> 1/ we modify dh_installchangelog to strip the binary-only changelog entry
> for Multi-Arch: same packages
>
> Some rough shell code to show the logic:
>
> if dpkg-parsechangelog | grep -q "^Binary-Only: yes"; then
> perl -i -ne '$found++ if /^S/; print if $found >= 2;' $changelog
> fi

For packages not using debhelper this would need to be duplicated all
over the place, to later on having to be reverted.

> 2/ we modify dpkg to allow co-installation of M-A: same packages which share the
> same source version regardless of the binary version

As I've said before, this right here seems unacceptable. This implies
at least:

* loosing the binNMU changelog entry, with a version in the changelog
not matching the one on the dpkg db (in possibly both directions).
* making installed file contents flip-flop depending on what package
got installed last.
* making dpkg unable to detect different generated file contents on
different binary rebuilds.

> 3/ we modify sbuild to add the required "binary-only=yes" in the binNMU
> changelog entries. Here's a sample header line:
>
> ftplib (3.1-1-9+b1) unstable; urgency=low, binary-only=yes

This could be done regardless if the buildd people agree to it, and that
was the idea when I added this.

guillem


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120712050630.GB11646@gaara.hadrons.org">http://lists.debian.org/20120712050630.GB11646@gaara.hadrons.org

Raphael Hertzog 07-12-2012 06:59 AM

BinNMU changelog handling for Multi-Arch: same packages
 
On Thu, 12 Jul 2012, Guillem Jover wrote:
> Sigh, again...

There's no need to "sigh" every time I try to discuss something with
you... I'm not trying to impose anything but just wanted to get a clearer
picture of the situation. And this imposes some communication between us.
Sorry for that.

> On Wed, 2012-07-11 at 09:23:05 +0200, Raphael Hertzog wrote:
> > So it looks like that the "part of the changelog binNMU solution"
> > was just the possibility to tag a changelog entry "binary-only"
> > with a keyword.
>
> No, the part of the solution was to create the needed user and program
> infrastructure interfaces to retrieve the metadata files from the db in a
> future-proof way. That's the new commands «dpkg-query --control-list pkg»
> and «dpkg-query --control-show pkg file», which should eventually replace
> the previous semi-private «dpkg-query --control-path pkg [file]».

Oh, ok. That makes sense. Now we have the correct interface in wheezy so
that other programs can start to rely on it in wheezy + 1.

> There's no other changes required from the dpkg side. To get changelog
> (and possibly copyright files) as package metadata, I think the only
> remaining things that would need to happen if the project agreed that's
> the right path would be:
>
> * Change apt-listchanges to use «dpkg-deb -I pkg changelog» to try to
> get the package changelog.
> * Change the “website” to use «dpkg-deb -I pkg changelog» to try to
> get the package changelog (and possibly copyright).
> * Change policy to allow packages to ship changelogs (and possibly
> copyright) as package metadata instead of «/use/share/doc/<pkg>/».
> * Change lintian per the above.
> * Change dh_installchangelogs to install the changelog in the DEBIAN/
> dir instaed of «/use/share/doc/<pkg>/».
> * Progressively change any remaning package not using debhelper to
> store these under DEBIAN/.

We should then probably open a bug against debian-policy. I'm doing it
now. We can then discuss there whether we need to find a solution
so that we're never in situation where neither the direct access nor the
command can be trusted to work. For the /usr/doc transition we created
symlinks for this purpose but it has been a painful transition due to
this.

Discussion shall continue in #681289.

> > I know that in the long term you're in favor of moving the changelog in
> > the package metadata and I agree with this plan. But IMO we must find
> > an interim solution in the mean time.
>
> First, I don't see why we _must_, it's been a known limitation of the

Right, I should have said "IMO we should find".

> > 2/ we modify dpkg to allow co-installation of M-A: same packages which share the
> > same source version regardless of the binary version
>
> As I've said before, this right here seems unacceptable. This implies
> at least:
>
> * loosing the binNMU changelog entry, with a version in the changelog
> not matching the one on the dpkg db (in possibly both directions).
> * making installed file contents flip-flop depending on what package
> got installed last.
> * making dpkg unable to detect different generated file contents on
> different binary rebuilds.

There's a mis-understanding here. I was not telling to drop the check
that ensures that the files are identical between the various arches.
Instead I'm just saying that we must change the check that ensures that
all package are at the same version to use the source version and not the
binary version.

And this is required even if we move the changelog to control.tar.
Otherwise the release team will still have to bin-nmu all arches at the
same time, which is what they would like to avoid.

> > 3/ we modify sbuild to add the required "binary-only=yes" in the binNMU
> > changelog entries. Here's a sample header line:
> >
> > ftplib (3.1-1-9+b1) unstable; urgency=low, binary-only=yes
>
> This could be done regardless if the buildd people agree to it, and that
> was the idea when I added this.

Filed a bug for this.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Get the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120712065938.GB22032@rivendell.home.ouaza.com">h ttp://lists.debian.org/20120712065938.GB22032@rivendell.home.ouaza.com


All times are GMT. The time now is 12:25 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.