Moving changelog and copyright files to control tarball, or merging control and data tarballs ?
 

Hi!

Just to give some context and background, this started with the
discussion of binNMUs and multi-arch [0], related to file conflicts
due to the coinstallability of multiple M-A:same instances, and while
initially a wild idea to possibly fix that specific case, it seemed
to me it was generic enough and which would make other parts of the
system easier and more structured.

[0] <http://lists.debian.org/debian-dpkg/2011/09/msg00029.html>

On Sat, 2012-07-14 at 09:48:35 +0900, Charles Plessy wrote:
> in http://bugs.debian.org/681289, Raphaël proposed that the Changelog and
> copyright should be package metadata.

A policy proposal (prompted by [1]) which was IMO premature (as usual),
that I was planning on bringing up later on to debian-devel before
even considering proposing such policy change... Now I guess we'll end
up with this being discussed in several places.

[1] <http://lists.debian.org/debian-dpkg/2012/07/msg00026.html>

> I personally like the idea, but I note
> that last time it was introduced on debian-devel, it was not consensual
> (http://lists.debian.org/20437.51932.971859.384176@chiark.greenend.org.uk), so
> I answer to your proposal on debian-devel to restart the discussion there.

Debian usually operates by rough consensus, which means not every one
has to agree, as long as mostly everyone does. And as I mentioned in
my reply, there didn't appear much convincing arguments in that mail
you point to.

> As the logic applied to the changelog and copyright files can be applied to
> most other files produced by the packager, like NEWS.Debian, it looks like the
> trend would be to have them installed in /var/lib/dpkg rather than
> /usr/share/doc.

I already mentioned this in [2] and [3], I think it only makes sense to
consider as metadata machine parseable files, and ones closely related
to the packaging system, not any maintainer created files such as ones
specific for installation helpers for example. As such NEWS.Debian
*might* indeed make some sense to put there too, but not many more if
at all, and they would need careful consideration.

[2] <http://lists.debian.org/debian-devel/2012/06/msg00314.html>
[3] <http://lists.debian.org/debian-devel/2012/06/msg00410.html>

> This is somewhat equivalent of a change in the binary package format, as
> packages will need to pre-depend on a recent version of dpkg,

No, it's not really equivalent, and no, they will not need to Pre-Depend
on a recent dpkg, any dpkg version will handle those files as control
files automatically, and do the right thing. Also the fact that those
files were in «/usr/share/doc/» before means no program can truly rely
on them being present as per policy those paths can be removed by the
admin. The only new interfaces that I introduced in dpkg 1.16.5 specific
to this (dpkg-query --control-list and --control-show) are for users
and programs to be able to easily get those files w/o having to rely
on the internal database paths.

> and programs not getting the metadata through dpkg will need to be
> corrected.

Also mentioned in [2] and [3].

And just to clarify, when I've been referring to this as metadata,
I've not been referring to say a new field in the status file, just as
those files as additional files in the db, like the md5sums shipped by
packages and generated at build time.

> Pushing the logic further, I wonder if that suggests that the Debian binary
> package format could be simplified to be a simple tarball with the metadata
> in /var/lib/dpkg, instead of the current format with a data and a control
> tarballs joined together in the 'ar' format.

That would not simplify it at all, in fact it would make it way more
complex, not to mention it would require bumping the .deb version
format to 3.0.

> This would not prevent dpkg to store the metadata somewhere else than
> /var/lib/dpkg later, as it would be easy to intercept the files and treat
> them appropriately, similar to what is being done with files in
> usr/share/doc with multi-arch packages. Also, once using a simple
> tarball as binary package, arbitrary files become trivial to extract
> quickly, or perhaps even to update (like for translations).

This would require to hardcode several path locations in dpkg just to
make them generic again internally, which does not make any sense to
me. Also dpkg does not currently have any internal knowledge about
magic paths, and there's no special handling of multi-arch paths for
«/usr/share/doc/».

regards,
guillem


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120714070029.GA9061@gaara.hadrons.org">http://lists.debian.org/20120714070029.GA9061@gaara.hadrons.org

Moving changelog and copyright files to control tarball, or merging control and data tarballs ?
 

On 07/14/2012 08:48 AM, Charles Plessy wrote:
> Pushing the logic further, I wonder if that suggests that the Debian binary
> package format could be simplified to be a simple tarball with the metadata
> in /var/lib/dpkg, instead of the current format with a data and a control
> tarballs joined together in the 'ar' format.
>
> This would not prevent dpkg to store the metadata somewhere else than
> /var/lib/dpkg later, as it would be easy to intercept the files and treat them
> appropriately, similar to what is being done with files in usr/share/doc with
> multi-arch packages. Also, once using a simple tarball as binary package,
> arbitrary files become trivial to extract quickly, or perhaps even to update
> (like for translations).
>
> Have a nice day,
>
Are you writing that ar is a bad choice, and that tar would
be better? I read ar it was better than tar because containing
an index. Was I misslead by wrong web sites?

Cheers,

Thomas


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 5001F1E5.9040901@debian.org">http://lists.debian.org/5001F1E5.9040901@debian.org

Moving changelog and copyright files to control tarball, or merging control and data tarballs ?
 

On 07/14/2012 08:48 AM, Charles Plessy wrote:
> Pushing the logic further, I wonder if that suggests that the Debian binary
> package format could be simplified to be a simple tarball with the metadata
> in /var/lib/dpkg, instead of the current format with a data and a control
> tarballs joined together in the 'ar' format.
>
> This would not prevent dpkg to store the metadata somewhere else than
> /var/lib/dpkg later, as it would be easy to intercept the files and treat them
> appropriately, similar to what is being done with files in usr/share/doc with
> multi-arch packages. Also, once using a simple tarball as binary package,
> arbitrary files become trivial to extract quickly, or perhaps even to update
> (like for translations).
>
> Have a nice day,
>
Are you writing that ar is a bad choice, and that tar would
be better? I read ar it was better than tar because containing
an index. Was I misslead by wrong web sites?

Cheers,

Thomas


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 5001F1E5.9040901@debian.org">http://lists.debian.org/5001F1E5.9040901@debian.org

Moving changelog and copyright files to control tarball, or merging control and data tarballs ?
 

On 14/07/12 01:48, Charles Plessy wrote:
> This would not prevent dpkg to store the metadata somewhere else than
> /var/lib/dpkg later, as it would be easy to intercept the files and treat them
> appropriately, similar to what is being done with files in usr/share/doc with
> multi-arch packages.

/usr/share/doc in multi-arch packages isn't special: any
arch-independent file can be shared between libfoo:i386 and libfoo:amd64
(or whatever) with refcounting, as long as it's guaranteed to be
byte-for-byte identical in all architectures.

(Other uses for refcounting include Lintian overrides in /usr/share,
arch-indep headers in /usr/include, and tables in /usr/share that aren't
large enough to justify a libfoo-common or libfoo-data package,
such as /usr/share/libaudio2/AuErrorDB and
/usr/share/libgphoto2/2.4.14/konica/english.)

S


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 5001FB60.9090103@debian.org">http://lists.debian.org/5001FB60.9090103@debian.org

Moving changelog and copyright files to control tarball, or merging control and data tarballs ?
 

On Sat, Jul 14, 2012 at 09:00:29AM +0200, Guillem Jover wrote:
> Hi!
>
> Just to give some context and background, this started with the
> discussion of binNMUs and multi-arch [0], related to file conflicts
> due to the coinstallability of multiple M-A:same instances, and while
> initially a wild idea to possibly fix that specific case, it seemed
> to me it was generic enough and which would make other parts of the
> system easier and more structured.
>
> [0] <http://lists.debian.org/debian-dpkg/2011/09/msg00029.html>
>
> On Sat, 2012-07-14 at 09:48:35 +0900, Charles Plessy wrote:
> > Pushing the logic further, I wonder if that suggests that the Debian binary
> > package format could be simplified to be a simple tarball with the metadata
> > in /var/lib/dpkg, instead of the current format with a data and a control
> > tarballs joined together in the 'ar' format.

One thing that is being done with package "metadata" is that it is extracted
before the package proper into a temporary location and used there.
As such as a minimum the /var/lib/dpkg/ you propse should be first in the
tarball so only a minimum of data needs to be uncompressed to extract the
metadata.

And you would have to always intercept those files all the time since
they need to go to a temporary place. So as Guillem says:

> That would not simplify it at all, in fact it would make it way more
> complex, not to mention it would require bumping the .deb version
> format to 3.0.

But what I wanted to say is that one thing that is being done with
package "metadata" is that it is extracted before the package proper
into a temporary location and used there.

Dpkg uses the control file and maintainer scripts (preinst, prerm
sometimes). But other tools need access to data too, like
apt-listchanges. In that light I too think it makes sense to move the
NEWS.Debian and changelog files to the control tarball so they can be
accessed easier and faster.

> > This would not prevent dpkg to store the metadata somewhere else than
> > /var/lib/dpkg later, as it would be easy to intercept the files and treat
> > them appropriately, similar to what is being done with files in
> > usr/share/doc with multi-arch packages. Also, once using a simple
> > tarball as binary package, arbitrary files become trivial to extract
> > quickly, or perhaps even to update (like for translations).
>
> This would require to hardcode several path locations in dpkg just to
> make them generic again internally, which does not make any sense to
> me. Also dpkg does not currently have any internal knowledge about
> magic paths, and there's no special handling of multi-arch paths for
> «/usr/share/doc/».
>
> regards,
> guillem

I'm not opposed to merging the control and data tarballs, neigther am I
for it. I don't quite see a reason for it.

But putting the metadata into /var/lib/dpkg is indeed a bad idea, esspecially
with the idea/plans to change the internal storage of those files in dpkg. By
having the files in the package already in place you put knowledge of the
dpkg internal DB into each and every deb and any change will be a nightmare.

Instead, if you want to play more with the idea of a single tarball, think
about having the metadata in /DEBIAN/ instead. That is where they are
already during building of debs so build tools would not need changes (other
than dpkg itself) And you would have a single hardcoded path location to
work from. Just my 2c.

MfG
Goswin


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120717135508.GD23876@frosties