FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian dpkg

 
 
LinkBack Thread Tools
 
Old 04-24-2011, 09:44 PM
A Mennucc
 
Default Debdelta and Streaming Package Installation for dpkg/APT

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dear Ishan, and also in CC Michael and debian-dpkg,

here is a reply to a message you sent me long ago; this message also
referenced a message in debian-dpkg [1], so I am CC-ing them as well.

The message, in short, was:

Il 06/01/2011 14:55, Ishan Jayawardena ha scritto:
> I came across the idea of streaming package installation[2] for
> dpkg/APT in Debian's wiki page for last year's summer of code. I
> found it interesting. I wrote to the debian-dpkg list about my
> interest and got two replies from them. In the replies, there was a
> mention about exploring the possibilities of using debdeltas in the
> installation process to make it faster.

Yes, there is a way (and it is actually not very difficult, at least
on the 'debdelta' side).

Let me summarize. When 'debdelta-upgrade' (or 'debpatch') recreates a
deb, one step is reassembling the data.tar part inside it; this part
moreover is compressed (gzip, bzip2 or lately lzma). This
'reassembling and compressing' takes time (both for CPU and for HD),
and is moreover quite useless, since, in short time, 'apt' will call
'dpkg -i' that decompresses and reopens the data.tar in the deb.

It is then reasonable to collapse this two parts, and this would
possibly speed up the upgrade a bit.

Here is my idea. When 'debdelta-upgrade' is called in upgrading a
package 'foobar' it currently creates 'foobar_2.deb'. By an
appropriate cmdline switch, instead of creating a 'foobar_2.deb' , it
would directly save all of its file to the filesystem, and it would
add an extension to all the file names, making sure that no file name
would conflict (=overwrite) with a preexisting file on the filesystem
; then it would create a file 'foobar_2.deb_unpacked' , that would be
just a text file similar to the usual control file, but specifying
also the extension used, and possibly the list of unpacked files.

I could change debdelta to accomplish that, it would not be a huge change.

Someone should help instead in changing 'dpkg' so that it would be
able to install starting from 'foobar_2.deb_unpacked'. And change APT
so that it would interact with 'debdelta' to create the
'foobar_2.deb_unpacked' files, and pass them to dpkg .

Note that the above idea overlaps a lot with [2].

a.

ps: for sake of brevity and clarity, I am skipping a lot of details: I
preferred to give the whole picture first.

Links:
[1] http://lists.debian.org/debian-dpkg/2011/01/msg00008.html
[2] http://wiki.debian.org/SummerOfCode2010/StreamingPackageInstall

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk20mbkACgkQ9B/tjjP8QKQoXQCcDCIGjPJzonYvQiTo9sLgg3Qo
1xMAniKKvv9rcZlOVNlm1CQBPuQ+p/Ge
=ei/T
-----END PGP SIGNATURE-----


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4DB499B9.6090104@debian.org">http://lists.debian.org/4DB499B9.6090104@debian.org
 
Old 04-25-2011, 05:28 AM
Guillem Jover
 
Default Debdelta and Streaming Package Installation for dpkg/APT

Hi!

On Sun, 2011-04-24 at 23:44:25 +0200, A Mennucc wrote:
> here is a reply to a message you sent me long ago; this message also
> referenced a message in debian-dpkg [1], so I am CC-ing them as well.

Ah perfect, had in mind talking about the apt/debdelta integration
GSoC proposal [0] with you guys, but have not had the time to check
several details, anyway here it is, still with some facts not checked.

[0] <http://wiki.debian.org/SummerOfCode2011/AptDebdeltaIntegration>

> Let me summarize. When 'debdelta-upgrade' (or 'debpatch') recreates a
> deb, one step is reassembling the data.tar part inside it; this part
> moreover is compressed (gzip, bzip2 or lately lzma). This
> 'reassembling and compressing' takes time (both for CPU and for HD),
> and is moreover quite useless, since, in short time, 'apt' will call
> 'dpkg -i' that decompresses and reopens the data.tar in the deb.

The data.tar does not need to be recompressed for dpkg to be able to
install it. But my main problem right now is that I didn't find clear
documentation of the “debdelta” file format, the closest thing that I
found was the debpatch [1] example file in the debdelta package.
Something that I found undesirable is that it seems to require executing
a shell script included in the debdelta package to regenerate the data?

[1] /usr/share/debdelta/debpatch.sh

> It is then reasonable to collapse this two parts, and this would
> possibly speed up the upgrade a bit.

Yeah, AFAIUI the debdelta side seems to be similar in nature to
dpkg-split (specifically the --auto command), so I think it might make
more sense to integreate that part into dpkg instead of apt. At least
the retrieving part would still need to be integrated into apt though.

> Here is my idea. When 'debdelta-upgrade' is called in upgrading a
> package 'foobar' it currently creates 'foobar_2.deb'. By an
> appropriate cmdline switch, instead of creating a 'foobar_2.deb' , it
> would directly save all of its file to the filesystem, and it would
> add an extension to all the file names, making sure that no file name
> would conflict (=overwrite) with a preexisting file on the filesystem
> ; then it would create a file 'foobar_2.deb_unpacked' , that would be
> just a text file similar to the usual control file, but specifying
> also the extension used, and possibly the list of unpacked files.

Well, doesn't it have to verify the reconstructed package matches the
expected one? Anyway I don't quite like the idea, it would imply
offloading some of the dpkg unpacking logic out of dpkg, or just
duplicating it inside dpkg itself to deal with unpacking from tar
and from the file system itself and to rely safely on the file metadata
from the binary package.The control files would also need to be
preserved somewhere, etc.

thanks,
guillem


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110425052856.GA15797@gaara.hadrons.org">http://lists.debian.org/20110425052856.GA15797@gaara.hadrons.org
 
Old 04-25-2011, 07:47 AM
A Mennucc
 
Default Debdelta and Streaming Package Installation for dpkg/APT

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

hi again,

Il 25/04/2011 07:28, Guillem Jover ha scritto:
> The data.tar does not need to be recompressed for dpkg to be able to
> install it.
(that is true)
> But my main problem right now is that I didn't find clear
> documentation of the “debdelta” file format, the closest thing that I
> found was the debpatch [1] example file in the debdelta package.
(unfortunately the problem is that, from 2005 to ~two months ago, I
was the only one working on debdelta.. so the documentation is mostly
in my head...)
> Something that I found undesirable is that it seems to require executing
> a shell script included in the debdelta package to regenerate the data?
yes. Why would that be undesirable? Shell scripts are simple, well
known, and very powerful. (Indeed they are often used also for
"postinst" etc etc in deb packages...). Using shell scripts for
patches was a design decision in 2005 that I am quite happy of: it
enabled me to ameliorate the patches a lot in the following years,
without ever changing the delta internal format.

>> It is then reasonable to collapse this two parts, and this would
>> possibly speed up the upgrade a bit.
>
> Yeah, AFAIUI the debdelta side seems to be similar in nature to
> dpkg-split (specifically the --auto command), so I think it might make
> more sense to integreate that part into dpkg instead of apt. At least
> the retrieving part would still need to be integrated into apt though.
>
don't know about this; I had a different idea in mind; I will investigate.
>> Here is my idea. When 'debdelta-upgrade' is called in upgrading a
>> package 'foobar' it currently creates 'foobar_2.deb'. By an
>> appropriate cmdline switch, instead of creating a 'foobar_2.deb' , it
>> would directly save all of its file to the filesystem, and it would
>> add an extension to all the file names, making sure that no file name
>> would conflict (=overwrite) with a preexisting file on the filesystem
>> ; then it would create a file 'foobar_2.deb_unpacked' , that would be
>> just a text file similar to the usual control file, but specifying
>> also the extension used, and possibly the list of unpacked files.
>
> Well, doesn't it have to verify the reconstructed package matches the
> expected one?
yes; I would make sure that debdelta does that (in an internal pipe).
> Anyway I don't quite like the idea, it would imply
> offloading some of the dpkg unpacking logic out of dpkg, or just
> duplicating it inside dpkg itself to deal with unpacking from tar
> and from the file system itself and to rely safely on the file metadata
> from the binary package.
yes that second one would be my idea. Well, in all ways we think of it
, since there is an overlap we may want to eliminate, then either we
bring some logic of debdelta into dpkg, or some logic of dpkg into
debdelta...
> The control files would also need to be
> preserved somewhere, etc.
yes.

a.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk21JyMACgkQ9B/tjjP8QKSyBQCdHoUYPxuzG1GF6bVbtkV0Bvwa
dLoAoIthGSG/SGPdQUrhnk57ovUjROa3
=9U6T
-----END PGP SIGNATURE-----


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4DB52723.9050800@debian.org">http://lists.debian.org/4DB52723.9050800@debian.org
 
Old 05-06-2011, 08:59 AM
Guillem Jover
 
Default Debdelta and Streaming Package Installation for dpkg/APT

Hi!

On Mon, 2011-04-25 at 09:47:47 +0200, A Mennucc wrote:
> Il 25/04/2011 07:28, Guillem Jover ha scritto:
> > But my main problem right now is that I didn't find clear
> > documentation of the “debdelta” file format, the closest thing that I
> > found was the debpatch [1] example file in the debdelta package.

> (unfortunately the problem is that, from 2005 to ~two months ago, I
> was the only one working on debdelta.. so the documentation is mostly
> in my head...)

Sure. I think just documenting the file format, and nothing else, would
be enough to get a clear idea of how all this works, and at least for
me how to try to integrate it, and where. It would not need to be too
exhaustive though.

> > Something that I found undesirable is that it seems to require executing
> > a shell script included in the debdelta package to regenerate the data?

> yes. Why would that be undesirable? Shell scripts are simple, well
> known, and very powerful. (Indeed they are often used also for
> "postinst" etc etc in deb packages...). Using shell scripts for
> patches was a design decision in 2005 that I am quite happy of: it
> enabled me to ameliorate the patches a lot in the following years,
> without ever changing the delta internal format.

While I can understand that it makes changing the format easier, and
it was probably the right tool for fast prototyping, it also implies
the file cannot be automatically inspected/verified/handled w/o
implicitly trusting it. Which I don't think it's a good property for
a file format.

The case you mention about maintainer scripts is not equivalent, as
they are only run on installation/etc, and not on say dpkg-deb
extraction (say -I, -c, -f, etc), or when joining split packages
with dpkg-split.

> > Anyway I don't quite like the idea, it would imply
> > offloading some of the dpkg unpacking logic out of dpkg, or just
> > duplicating it inside dpkg itself to deal with unpacking from tar
> > and from the file system itself and to rely safely on the file metadata
> > from the binary package.

> yes that second one would be my idea. Well, in all ways we think of it
> , since there is an overlap we may want to eliminate, then either we
> bring some logic of debdelta into dpkg, or some logic of dpkg into
> debdelta...

So, my first instinct would be to bring that logic into dpkg (or a new
dpkg-debdelta or whatever), instead of offloading it, but as mentioned
earlier I don't think I've enough understanding of how debdelta works
internally to know how the integration could happen, or if that would
be the best way to do it.

thanks,
guillem


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110506085932.GA1597@gaara.hadrons.org">http://lists.debian.org/20110506085932.GA1597@gaara.hadrons.org
 

Thread Tools




All times are GMT. The time now is 10:46 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org