Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Debian dpkg (http://www.linux-archive.org/debian-dpkg/)
-   -   Streaming Package Installation for dpkg/APT (http://www.linux-archive.org/debian-dpkg/472514-streaming-package-installation-dpkg-apt.html)

Ishan Jayawardena 01-05-2011 03:25 AM

Streaming Package Installation for dpkg/APT
 
Hi,

I would like to know about the streaming package installation for
dpkg/APT. I read about this from last year's summer of code ideas list
of Debian [1], and found it interesting. I also found that it had not
been taken by any of the applicants, and, therefore, I would like to
work on it this summer.

Is there any ongoing development related to that idea? There is a
description given in [1] and apart from that, are there any concerns
of it? I would like to know your ideas and suggestions about it, to
proceed. Please let me know if you have something to share with me,
I'm looking forward to your feedback.


[1] http://wiki.debian.org/SummerOfCode2010/StreamingPackageInstall

Thank you.
--
Regards,
Ishan Jayawardena.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: AANLkTikew9zuJPLwMdaMxLJE6690_PGmkXT9kHs4Ws9T@mail .gmail.com">http://lists.debian.org/AANLkTikew9zuJPLwMdaMxLJE6690_PGmkXT9kHs4Ws9T@mail .gmail.com

Guillem Jover 01-05-2011 05:01 AM

Streaming Package Installation for dpkg/APT
 
Hi!

On Wed, 2011-01-05 at 09:55:29 +0530, Ishan Jayawardena wrote:
> I would like to know about the streaming package installation for
> dpkg/APT. I read about this from last year's summer of code ideas list
> of Debian [1], and found it interesting. I also found that it had not
> been taken by any of the applicants, and, therefore, I would like to
> work on it this summer.

You might want to talk with the people involved in that proposal, as I
don't think/remeber it ever being discussed on this list. CCed them now.
CCed Lars too which AFAIR mentioned something like this to me at some
point?

> Is there any ongoing development related to that idea?

I don't know of any.

> There is a description given in [1] and apart from that, are there
> any concerns of it? I would like to know your ideas and suggestions
> about it, to proceed. Please let me know if you have something to
> share with me, I'm looking forward to your feedback.

Michael and Simon might be able to fill the blanks.

About concerns, the one that comes to mind immediately is that dpkg
treats the packages as the basic units of operation, when invoked it
first parses the control files for all provided packages, and then
operates on them, reordering if needed, bailing out if dependencies
cannot be satisfied, breaking cycles, etc. If the packages are not on
disk, and they are streamed to dpkg, then it might not be able to
operate properly. Which might not be an unsurmountable issue, but then
I've not thought this through too much...

Something which I guess would speed up the installation process could
be to just make apt download the packages in self-contained batches,
which can be unpacked/configured independently. This would also not
really need any change in dpkg AFAICS. This way the installation
process could start sooner than having to wait for the whole thing to
get downloaded. It does not remove the need to store those batched
packages on disk, but still.

> [1] http://wiki.debian.org/SummerOfCode2010/StreamingPackageInstall

thanks,
guillem


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110105060107.GA586@gaara.hadrons.org">http://lists.debian.org/20110105060107.GA586@gaara.hadrons.org

Lars Wirzenius 01-05-2011 09:46 PM

Streaming Package Installation for dpkg/APT
 
On ke, 2011-01-05 at 07:01 +0100, Guillem Jover wrote:
> Something which I guess would speed up the installation process could
> be to just make apt download the packages in self-contained batches,
> which can be unpacked/configured independently. This would also not
> really need any change in dpkg AFAICS. This way the installation
> process could start sooner than having to wait for the whole thing to
> get downloaded. It does not remove the need to store those batched
> packages on disk, but still.

I can't look up the URLs for this, but when I worked for Canonical we
discussed something like this at one UDS, and there should be blueprints
and wiki pages on the Ubuntu sites for this. Some searching should turn
them up.

>From memory, what we came up with was basically what Guillem hints at:

* apt will order its downloads in "installation order"
* whenever apt has a self-contained batch, it will feed them to dpkg
* while dpkg runs, apt will continue to download things in the
background

Further, we discussed the possibility of doing some of the dpkg
installation phases in parallel, even while waiting for the rest of a
batch to be downloaded: for example, unpacking might be possible already
at that time. This is more error prone and more complicated, though.

Related to these discussions we also discussed the possibility of
speeding up downloads by using debdeltas. debdelta seems to work quite
well, and it might be a good idea for Debian to adopt it officially.

--
Blog/wiki/website hosting with ikiwiki (free for free software):
http://www.branchable.com/


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 1294267575.2953.42.camel@havelock.lan">http://lists.debian.org/1294267575.2953.42.camel@havelock.lan

Ishan Jayawardena 01-06-2011 05:04 AM

Streaming Package Installation for dpkg/APT
 
Hi,

Guillem and Lars, thank you very much for your detailed replies. They
are really informative and encouraging.

>From your replies what I learned was that, the idea is quite complex
and instead of streaming, we can think of other ways of speeding up
the installation process, which has taken my idea to a new direction.
As you have pointed out, I will also look into the possibilities of
using self-contained batches and debdeltas with apt/dpkg in the
speeding up process.

But still, the streaming idea also looks interesting to me. I will
study apt/dpkg functionality and streaming technologies in more detail
and try to come up with a suggestion on it. What I feel is, we can use
a tree like structure to resolve the dependencies and use it in the
streaming process in some way to achieve our goal. I am not sure if
it's doable or not. But I would like to develop the idea on top of
this. I will let you know if I could come up with something
interesting.

I really appreciate your help.

Thank you.

On 1/6/11, Lars Wirzenius <liw@liw.fi> wrote:
> On ke, 2011-01-05 at 07:01 +0100, Guillem Jover wrote:
>> Something which I guess would speed up the installation process could
>> be to just make apt download the packages in self-contained batches,
>> which can be unpacked/configured independently. This would also not
>> really need any change in dpkg AFAICS. This way the installation
>> process could start sooner than having to wait for the whole thing to
>> get downloaded. It does not remove the need to store those batched
>> packages on disk, but still.
>
> I can't look up the URLs for this, but when I worked for Canonical we
> discussed something like this at one UDS, and there should be blueprints
> and wiki pages on the Ubuntu sites for this. Some searching should turn
> them up.
>
> From memory, what we came up with was basically what Guillem hints at:
>
> * apt will order its downloads in "installation order"
> * whenever apt has a self-contained batch, it will feed them to dpkg
> * while dpkg runs, apt will continue to download things in the
> background
>
> Further, we discussed the possibility of doing some of the dpkg
> installation phases in parallel, even while waiting for the rest of a
> batch to be downloaded: for example, unpacking might be possible already
> at that time. This is more error prone and more complicated, though.
>
> Related to these discussions we also discussed the possibility of
> speeding up downloads by using debdeltas. debdelta seems to work quite
> well, and it might be a good idea for Debian to adopt it officially.
>
> --
> Blog/wiki/website hosting with ikiwiki (free for free software):
> http://www.branchable.com/
>
>


--
Regards,
Ishan Jayawardena.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: AANLkTimt5fMfp0cVh_6jSnhYErD=5vWU1dDa3oUV2hb9@mail .gmail.com">http://lists.debian.org/AANLkTimt5fMfp0cVh_6jSnhYErD=5vWU1dDa3oUV2hb9@mail .gmail.com


All times are GMT. The time now is 03:49 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.