FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 04-02-2008, 09:44 PM
Hendrik Sattler
 
Default Adding lzma to dpkg's Pre-Depends

Am Mittwoch 02 April 2008 schrieb Mike Bird:
> On Wed April 2 2008 01:52:39 Steve Langasek wrote:
> > On Wed, Apr 02, 2008 at 10:24:43AM +0200, Lucas Nussbaum wrote:
> > > The results are a bit misleading, because they compare the absolute
> > > gain.
> >
> > It's the absolute size savings that counts - that's what's going to have
> > the biggest impact on mirror size, mirroring time, CD set size, and
> > download times for users.
>
> Lucas made a good point. Better to save 20% on ten 10MB packages than
> saving 10% on one 100MB package.
>
> Although we can probably have both, modulo decompression RAM issues.

Well, oo.org probably needs the same amount of RAM to actually run, so it's
not an issue, there. That not the case for the libgl1-mesa-dri but probably
for programs that use OpenGL . So the consideration should always include
what effect this actually has for the users of that package.
No question that for Oo.org this is pure gain.

HS


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 04-04-2008, 01:20 PM
Julian Andres Klode
 
Default Adding lzma to dpkg's Pre-Depends

Lucas Nussbaum wrote:
> On 02/04/08 at 01:52 -0700, Steve Langasek wrote:
>> On Wed, Apr 02, 2008 at 10:24:43AM +0200, Lucas Nussbaum wrote:
>>>> So of course besides OOo on there we also find the kernel packages. We
>>>> wouldn't have to use lzma for the kernels though, if that would raise the
>>>> minimum memory requirements for servers, or lzma could be selectively
>>>> enabled on a per-flavor or per-arch basis as appropriate.
>>> The results are a bit misleading, because they compare the absolute
>>> gain.
>> No, why would that be misleading? You don't want the overhead of lzma
>> compression for a 10-fold reduction in the size of a package that's already
>> in the bottom 5% of packages by absolute size.
>
> The overhead (time and memory to compress/uncompress) is likely to be a
> function of the data size. I'm just saying that it's not that simple.
> For example, in the case of OOo, you save 27% of the size, but multiply
> by 4 the time to compress.
>
> If instead, you use lzma for ttf-arabeyes, smbclient,
> gnome-system-monitor, tomboy, gdb, and language-pack-gnome-en-base, you
> save the same size (10.5M in that case, 10.2M in openoffice's case), but
> you only compress/decompress 43M of uncompressed data in this case,
> while you touch 112M in the OO case. So it's likely that the overhead is
> more important in the OO case.
>
> I'm just saying that, if we don't use lzma for all packages, it's not
> straightforward to decide for which packages to use it. It's probably a
> function of absolute saving, saving ratio, popcon, etc.
I suggest to use lzma by default on the amd64 architecture, because AFAIK
there are no slow amd64 machines. That means that there is no disadvantage.

The i386 architecture may be used by smaller machines, with less power.
Therefore, we can use lzma there if the application uses much memory,
because we can expect that the memory needed to unpack the package is
available.

The ia64 platform could maybe also use lzma by default, but I have only
limited knowledge about it.

I strongly recommend not to use lzma on embedded architectures.
--
Julian Andres Klode, Fellow of the Free Software Foundation Europe
Debian Maintainer | Developer | Ubuntu Member

try Debian: http://www.debian.org/ | my site: http://jak-linux.org/
jabber: juliank@jabber.org | IRC: juliank (FreeNode, OFTC)
languages: German | English
 
Old 04-05-2008, 03:30 PM
The Fungi
 
Default Adding lzma to dpkg's Pre-Depends

On Fri, Apr 04, 2008 at 03:20:10PM +0200, Julian Andres Klode wrote:
[...]
> I strongly recommend not to use lzma on embedded architectures.

I suppose that depends on how it's used, but Wikipedia's article on
the algorithm even goes so far as to state:

"Small code size and relatively low memory overhead, particularly
with smaller dictionary lengths, make the LZMA decompression
algorithm well-suited to embedded applications."

http://en.wikipedia.org/wiki/Lempel-Ziv-Markov_algorithm

And yes, I understand Wikipedia is a far from reputable source, but
correlate with 7-zip's 7z format page which says:

"The LZMA compression algorithm is very suitable for embedded
applications."

http://www.7-zip.org/7z.html

And on their SDK page:

"LZMA provides a high compression ratio and very fast
decompression, so it is very suitable for embedded applications."

http://www.7-zip.org/sdk.html

Again, objective statement or marketing zealotry? As an embedded
platform use-case example, here's a page talking about (among other
things) work in progress back in 2006 (no idea if it came to
fruition) using LZMA for JFFS2 filesystem compression on OpenWRT:

http://bitsum.com/openwiking/owbase/ow.asp?Embedded_Linux_Compression_Optimizations

I understand that decreasing the dictionary length has implications
on the compression ratio, but it would be interesting to see some
numbers indicating to what degree this is the case (and roughly
where the break-even is against whatever gzip compression level the
autobuilders currently use for packages). Perhaps it's worth
considering adjusting this parameter downward for smaller arch all
packages and binary packages built for typically lower-resource
architectures (m68k, mipsel, arm, et cetera), if there's still an
all-around performance gain to be had over gzip.
--
{ IRL(Jeremy_Stanley); PGP(9E8DFF2E4F5995F8FEADDC5829ABF7441FB84657);
SMTP(fungi@yuggoth.org); IRC(fungi@irc.yuggoth.org#ccl); ICQ(114362511);
AIM(dreadazathoth); YAHOO(crawlingchaoslabs); FINGER(fungi@yuggoth.org);
MUD(fungi@katarsis.mudpy.org:6669); WWW(http://fungi.yuggoth.org/); }


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 04-09-2008, 03:18 AM
Guillem Jover
 
Default Adding lzma to dpkg's Pre-Depends

Hi,

On Wed, 2008-04-02 at 14:01:16 +1000, Anthony Towns wrote:
> On Tue, Apr 01, 2008 at 08:05:06AM +0300, Guillem Jover wrote:
> > As per policy 3.5 I'm bringing this up here. I'd like to add lzma to
> > dpkg's Pre-Depends, so that we can use lzma compressed packages after
> > lenny w/o having to add an lzma Pre-Depends on each .deb package
> > compressed that way.
>
> Hrm. Alternatively, the packages _do_ pre-depend on lzma though; and
> you're aiming to avoid that by making lzma Essential:yes -- in the same
> way packages that pre-depend on perl or bash don't need an explicit
> dependency.

Well not really Essential, it's going to be pulled like that yes, but
other derivatives, might want to disable it. And I agree with Chris that
it makes sense for dpkg to Pre-Depend on lzma as it's the one calling
it, and that's an internal implementation detail, in case there's a
liblzma in the future and we'd switch to using it, packages should not
require to be changed.

> libz and libbz2 are both included statically;

Switching to dynamic linking is something I've been pondering for some
time, but that's independent of this discussion, and something not to
be changed at this point of the release anyway.

> would it make more sense to do the same thing with the lzma (ie,
> copy the binary into /usr/lib/dpkg), instead of making lzma
> essential?

The lzma package is tiny, it could be halved as I pointed out in the
bug report (should probably just file bug reports on lzma for that).
That would imply an installed size of about 150 KiB, 100 of those
are just the lzma binary. In that case I don't really see the gain
in having mostly the package there but w/o it being handled by the
packaging system.

regards,
guillem


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 07-23-2008, 02:46 AM
Guillem Jover
 
Default Adding lzma to dpkg's Pre-Depends

Hi,

Summary of the thread starting at:

<http://lists.debian.org/debian-devel/2008/04/msg00003.html>

The only valid concerns rised in that thread were from Anthony, and I
think I gave reasons to dismiss them (found below).

The rest were concerning policy decisions on when to use lzma
compression, which are not relevant to this particular decision.

So, I'd like to ask the Release Team, if it would be fine with you to
add such Pre-Depends for the next dpkg upload targetting lenny.


On Wed, 2008-04-09 at 06:18:36 +0300, Guillem Jover wrote:
> On Wed, 2008-04-02 at 14:01:16 +1000, Anthony Towns wrote:
> > On Tue, Apr 01, 2008 at 08:05:06AM +0300, Guillem Jover wrote:
> > > As per policy 3.5 I'm bringing this up here. I'd like to add lzma to
> > > dpkg's Pre-Depends, so that we can use lzma compressed packages after
> > > lenny w/o having to add an lzma Pre-Depends on each .deb package
> > > compressed that way.
> >
> > Hrm. Alternatively, the packages _do_ pre-depend on lzma though; and
> > you're aiming to avoid that by making lzma Essential:yes -- in the same
> > way packages that pre-depend on perl or bash don't need an explicit
> > dependency.
>
> Well not really Essential, it's going to be pulled like that yes, but
> other derivatives, might want to disable it. And I agree with Chris that
> it makes sense for dpkg to Pre-Depend on lzma as it's the one calling
> it, and that's an internal implementation detail, in case there's a
> liblzma in the future and we'd switch to using it, packages should not
> require to be changed.
>
> > libz and libbz2 are both included statically;
>
> Switching to dynamic linking is something I've been pondering for some
> time, but that's independent of this discussion, and something not to
> be changed at this point of the release anyway.
>
> > would it make more sense to do the same thing with the lzma (ie,
> > copy the binary into /usr/lib/dpkg), instead of making lzma
> > essential?
>
> The lzma package is tiny, it could be halved as I pointed out in the
> bug report (should probably just file bug reports on lzma for that).
> That would imply an installed size of about 150 KiB, 100 of those
> are just the lzma binary. In that case I don't really see the gain
> in having mostly the package there but w/o it being handled by the
> packaging system.

I filed that bug and it was fixed some time ago, lzma in testing and
unstable uses around 130 KiB on i386. So I don't think the size
argument should really be a problem now.

thanks,
guillem


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 07-29-2008, 07:28 PM
Andreas Barth
 
Default Adding lzma to dpkg's Pre-Depends

* Guillem Jover (guillem@debian.org) [080723 05:08]:
> Summary of the thread starting at:
>
> <http://lists.debian.org/debian-devel/2008/04/msg00003.html>
>
> The only valid concerns rised in that thread were from Anthony, and I
> think I gave reasons to dismiss them (found below).
>
> The rest were concerning policy decisions on when to use lzma
> compression, which are not relevant to this particular decision.
>
> So, I'd like to ask the Release Team, if it would be fine with you to
> add such Pre-Depends for the next dpkg upload targetting lenny.

I currently don't see the strict reasoning why dpkg should pre-depend on
lzma.


> On Wed, 2008-04-09 at 06:18:36 +0300, Guillem Jover wrote:
> > On Wed, 2008-04-02 at 14:01:16 +1000, Anthony Towns wrote:
> > > On Tue, Apr 01, 2008 at 08:05:06AM +0300, Guillem Jover wrote:
> > > > As per policy 3.5 I'm bringing this up here. I'd like to add lzma to
> > > > dpkg's Pre-Depends, so that we can use lzma compressed packages after
> > > > lenny w/o having to add an lzma Pre-Depends on each .deb package
> > > > compressed that way.
> > >
> > > Hrm. Alternatively, the packages _do_ pre-depend on lzma though; and
> > > you're aiming to avoid that by making lzma Essential:yes -- in the same
> > > way packages that pre-depend on perl or bash don't need an explicit
> > > dependency.
> >
> > Well not really Essential, it's going to be pulled like that yes, but
> > other derivatives, might want to disable it. And I agree with Chris that
> > it makes sense for dpkg to Pre-Depend on lzma as it's the one calling
> > it, and that's an internal implementation detail, in case there's a
> > liblzma in the future and we'd switch to using it, packages should not
> > require to be changed.

What advantage would we (as in Debian) have if dpkg pre-depends on lzma,
instead of the packages pre-depending on lzma?


If there isn't an real and strong advantage, I'd rather think it's
better to not do it. And, BTW, if dpkg pre-depends on lzma, lzma is
basically essential.


Cheers,
Andi
--
http://home.arcor.de/andreas-barth/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 07-29-2008, 08:16 PM
Steve Langasek
 
Default Adding lzma to dpkg's Pre-Depends

On Tue, Jul 29, 2008 at 09:28:01PM +0200, Andreas Barth wrote:
> > On Wed, 2008-04-09 at 06:18:36 +0300, Guillem Jover wrote:
> > > On Wed, 2008-04-02 at 14:01:16 +1000, Anthony Towns wrote:
> > > > On Tue, Apr 01, 2008 at 08:05:06AM +0300, Guillem Jover wrote:
> > > > > As per policy 3.5 I'm bringing this up here. I'd like to add lzma to
> > > > > dpkg's Pre-Depends, so that we can use lzma compressed packages after
> > > > > lenny w/o having to add an lzma Pre-Depends on each .deb package
> > > > > compressed that way.

> > > > Hrm. Alternatively, the packages _do_ pre-depend on lzma though; and
> > > > you're aiming to avoid that by making lzma Essential:yes -- in the same
> > > > way packages that pre-depend on perl or bash don't need an explicit
> > > > dependency.

> > > Well not really Essential, it's going to be pulled like that yes, but
> > > other derivatives, might want to disable it. And I agree with Chris that
> > > it makes sense for dpkg to Pre-Depend on lzma as it's the one calling
> > > it, and that's an internal implementation detail, in case there's a
> > > liblzma in the future and we'd switch to using it, packages should not
> > > require to be changed.

> What advantage would we (as in Debian) have if dpkg pre-depends on lzma,
> instead of the packages pre-depending on lzma?

If dpkg internalizes the lzma support (by static linking, dynamic linking,
or depending on the lzma binary), and packages which use lzma pre-depend on
the correct version of dpkg, then the pre-dependency on dpkg is transitional
and can go away after a release cycle.

If dpkg doesn't internalize the lzma support, then this pre-depends never
goes away (which is ugly), and means additional complexity in calculating
upgrades indefinitely since at any point a user may for the first time be
installing a package that needs lzma.

What is fundamentally different about lzma that it should be handled
differently than gzip and bzip2?

(BTW, from my POV, dpkg Pre-Depends lzma / dpkg Pre-Depends liblzma / dpkg
statically links liblzma are practically equivalent as long as we can assume
that the lzma package itself is maintained in an appropriate manner.
Whichever one of these options is chosen will pull the new code into base
and have roughly the same effect on the install size.)

> If there isn't an real and strong advantage, I'd rather think it's
> better to not do it. And, BTW, if dpkg pre-depends on lzma, lzma is
> basically essential.

It's essential for the purposes of calculating the Essential: yes closure,
and not essential in the sense that packages which use its functionality
independently still need to depend on it directly.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slangasek@ubuntu.com vorlon@debian.org


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 07-29-2008, 10:44 PM
Ian Jackson
 
Default Adding lzma to dpkg's Pre-Depends

Steve Langasek writes ("Re: Adding lzma to dpkg's Pre-Depends"):
> On Tue, Jul 29, 2008 at 09:28:01PM +0200, Andreas Barth wrote:
> > What advantage would we (as in Debian) have if dpkg pre-depends on lzma,
> > instead of the packages pre-depending on lzma?
>
> If dpkg internalizes the lzma support (by static linking, dynamic linking,
> or depending on the lzma binary), and packages which use lzma pre-depend on
> the correct version of dpkg, then the pre-dependency on dpkg is transitional
> and can go away after a release cycle.

No, unless the dpkg.deb binary package were to actually _contain_ the
lzma code, dpkg would have to Pre-Depend on it forever (or it would
have to be made Essential). Dynamic linking or expecting to use the
lzma binary from another package would not suffice.

> What is fundamentally different about lzma that it should be handled
> differently than gzip and bzip2?

lzma is much more of a minority interest than gzip. We do not expect
ever to transition the entire archive to use gzip. The approach taken
with bzip2 was a mistake and should not be repeated.

> > If there isn't an real and strong advantage, I'd rather think it's
> > better to not do it. And, BTW, if dpkg pre-depends on lzma, lzma is
> > basically essential.
>
> It's essential for the purposes of calculating the Essential: yes closure,
> and not essential in the sense that packages which use its functionality
> independently still need to depend on it directly.

Making a package Essential in that sense is very expensive and should
not be done without a good reason.

The last time we had this conversation it appeared that the only
reason why Pre-Depends in each package wasn't favoured was because our
package building tools were not capable of including that dependency
in a sufficiently automated way. I think that is a really poor
reason.

Ian.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 07-30-2008, 12:36 AM
Steve Langasek
 
Default Adding lzma to dpkg's Pre-Depends

On Tue, Jul 29, 2008 at 11:44:30PM +0100, Ian Jackson wrote:
> Steve Langasek writes ("Re: Adding lzma to dpkg's Pre-Depends"):
> > On Tue, Jul 29, 2008 at 09:28:01PM +0200, Andreas Barth wrote:
> > > What advantage would we (as in Debian) have if dpkg pre-depends on lzma,
> > > instead of the packages pre-depending on lzma?

> > If dpkg internalizes the lzma support (by static linking, dynamic linking,
> > or depending on the lzma binary), and packages which use lzma pre-depend on
> > the correct version of dpkg, then the pre-dependency on dpkg is transitional
> > and can go away after a release cycle.

> No, unless the dpkg.deb binary package were to actually _contain_ the
> lzma code, dpkg would have to Pre-Depend on it forever (or it would
> have to be made Essential). Dynamic linking or expecting to use the
> lzma binary from another package would not suffice.

"The pre-dependency on dpkg", not "the pre-dependency on lzma".

> > What is fundamentally different about lzma that it should be handled
> > differently than gzip and bzip2?

> lzma is much more of a minority interest than gzip. We do not expect
> ever to transition the entire archive to use gzip. The approach taken
> with bzip2 was a mistake and should not be repeated.

That's a new argument I haven't heard before. *Why* was it a mistake to do
this the way it was done with bzip2?

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slangasek@ubuntu.com vorlon@debian.org


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-03-2008, 01:24 AM
Adeodato Simó
 
Default Adding lzma to dpkg's Pre-Depends

* Guillem Jover [Wed, 23 Jul 2008 05:46:26 +0300]:

Hello, Guillem.

> So, I'd like to ask the Release Team, if it would be fine with you to
> add such Pre-Depends for the next dpkg upload targetting lenny.

Yes, it's fine with us. Please let us know when the upload is done.

Cheers,

--
Adeodato Simó dato at net.com.org.es
Debian Developer adeodato at debian.org

Listening to: Tamara - Un poco de dolor


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 

Thread Tools




All times are GMT. The time now is 04:26 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org