FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 08-11-2011, 06:30 PM
Adam Borowski
 
Default The archive now supports xz compression

On Thu, Aug 11, 2011 at 05:12:36PM +0200, Ansgar Burchardt wrote:
> Hi,
>
> The archive software now accepts packages using xz for compression in
> addition to gzip and bzip2 for both source and binary packages.

Hurray!

> please only use xz (or bzip2 for that matter) if your
> package really profits from its usage (for example, it provides a
> significant space saving). While those methods may compress better they
> often use more CPU time to do so and a very small decrease in package
> size is hardly worth the extra effort placed on slower systems.

This is very bad advice.

Do you remember my joke package "goodbye" less than two weeks ago? If you
compared the optimized[1] debhelper-less dpkg-less code to standard "slow"
debhelper, you lose 2.6 seconds per package[2].

In that time you can compress 8MB (and decompress that in 0.09s).

Thus, if you care about amounts of CPU time that small, you can as well go
through the whole archive replacing debhelper with "goodbye", for all
packages smaller than that 8MB.

The cost is roughly linear, too -- so compressing a 20KB package costs about
nothing.

And what do we gain in return? A massive decrease of the archive size --
both disk space and bandwidth. Regular packages compress twice as much with
xz as with gzip, with uncompressible data in the mix the average was 2/3 for
amd64 CD1.


Thus, I'd strongly recommend just compressing everything with xz, on all
architectures. Preferably, as a default in dpkg-dev.


> Think of both user systems and the Debian buildds which will waste more
> time - an especially bad problem on slower architectures.

The gain is especially meaningful for slower architectures, as they tend to
have less disk space and slower network links (arm tends to be used in
phones). No extra memory is needed -- decompression is not done in parallel
with memory-hungry stages of dpkg's work. The decompression, merely 2.5
times slower than with gzip, is a tiny fraction of what dpkg takes.


> Please remember that packages in the base system[1] (and dependencies)
> *must* currently use gzip as otherwise debootstrap will be unable to
> install a system.
>
> 1: Meaning everything with Priority: required.

I'm not as strongly opinionated here, but I guess decreasing the size of d-i
images would be a huge win as well.



Meow!

[1]. Three calls of system() could be avoided for little effort and two more
for substantial effort -- and all of them could be turned into execve(), but
otherwise, it's pretty much as fast as you can get. With package build time
below a single disk seek, further optimizations are pointless. The point of
"goodbye" was abuse of policy and buzzword compliancy, not speed.

[2]. Debhelper costs are surprisingly constant, even between a "copy a
single file" package and full-blown autoconf+C ones.

--
1KB // Yo momma uses IPv4!


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110811183010.GA27546@angband.pl">http://lists.debian.org/20110811183010.GA27546@angband.pl
 
Old 08-11-2011, 07:19 PM
Raphael Hertzog
 
Default The archive now supports xz compression

Hi,

On Thu, 11 Aug 2011, Adam Borowski wrote:
> Thus, I'd strongly recommend just compressing everything with xz, on all
> architectures. Preferably, as a default in dpkg-dev.

I am very much in favor of this as well but after having discussed this
at debconf with Colin Watson and Joey Hess, I'm not going to change this
immediately.

As Ansgar mentionned, it creates a new requirement: for debootstrap to work
xz must be present and it's currently not present in debian-installer.

So changing the default in dpkg-dev would either require to hardcode
gzip for all "base" packages or accept xz as a new dependency for
debootstrap.

The set of "base" package is not clearly defined, it's roughly "all
required packages and their dependencies" but this can't be really
automated and can theoretically change at the whims of ftpmasters
since the priorities are under the control of ftpmasters.

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=634248

Since hardcoding gzip for base packages seems to be a bit brittle,
we need to work towards allowing xz usage in debian-installer and accept
it as a dependency for deboostrap on non-Debian systems (I don't think
it's a big issue, xz is portable and is more and more widely used).

I guess that a first step is to have an xz udeb...

Would you like to drive a release goal to have all packages of DVD1
use XZ compressions ?

1/ Ensure XZ support is integrated in debian-installer (and deboostrap
fixed to add it as a dependency).
2/ Wait until we update dpkg to use XZ by default.
3/ Ensure all packages useful in DVD1 are updated or bin-nmued
before the freeze.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
▶ http://RaphaelHertzog.fr (Français)


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110811191955.GH6038@rivendell.home.ouaza.com">ht tp://lists.debian.org/20110811191955.GH6038@rivendell.home.ouaza.com
 
Old 08-11-2011, 07:52 PM
Philipp Kern
 
Default The archive now supports xz compression

On 2011-08-11, Adam Borowski <kilobyte@angband.pl> wrote:
>> Think of both user systems and the Debian buildds which will waste more
>> time - an especially bad problem on slower architectures.
> The gain is especially meaningful for slower architectures, as they tend to
> have less disk space and slower network links (arm tends to be used in
> phones). No extra memory is needed -- decompression is not done in parallel
> with memory-hungry stages of dpkg's work. The decompression, merely 2.5
> times slower than with gzip, is a tiny fraction of what dpkg takes.

It takes a lot longer to compress on slower architectures (i.e. on the
buildds), though. You could've built a whole package in that time. (Resorting
to your style of argument.)

>> Please remember that packages in the base system[1] (and dependencies)
>> *must* currently use gzip as otherwise debootstrap will be unable to
>> install a system.
>>
>> 1: Meaning everything with Priority: required.
>
> I'm not as strongly opinionated here, but I guess decreasing the size of d-i
> images would be a huge win as well.

You cannot debootstrap it anymore. I can hardly call that a win. (So we'd
need to wait a few releases for that.)

> [1]. Three calls of system() could be avoided for little effort and two more
> for substantial effort -- and all of them could be turned into execve(), but
> otherwise, it's pretty much as fast as you can get. With package build time
> below a single disk seek, further optimizations are pointless. The point of
> "goodbye" was abuse of policy and buzzword compliancy, not speed.

It compiles the same script multiple times during build.

Kind regards
Philipp Kern


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: slrnj48coe.a7f.trash@kelgar.0x539.de">http://lists.debian.org/slrnj48coe.a7f.trash@kelgar.0x539.de
 
Old 08-11-2011, 10:55 PM
Colin Watson
 
Default The archive now supports xz compression

On Thu, Aug 11, 2011 at 09:19:55PM +0200, Raphael Hertzog wrote:
> As Ansgar mentionned, it creates a new requirement: for debootstrap to work
> xz must be present and it's currently not present in debian-installer.

The main thing I consider to be difficult is that putting xz-compressed
packages in the base system requires people on foreign systems to have
an xz decompressor available. To a large extent this is outside our
control.

> Since hardcoding gzip for base packages seems to be a bit brittle,
> we need to work towards allowing xz usage in debian-installer and accept
> it as a dependency for deboostrap on non-Debian systems (I don't think
> it's a big issue, xz is portable and is more and more widely used).

Can you quantify that? I don't have hard numbers for the non-Debian
systems where people report running debootstrap; perhaps you do ...

> I guess that a first step is to have an xz udeb...

No. busybox already has an unxz applet, so if we choose to do this then
we should enable that applet in busybox-udeb, not add a separate udeb.

--
Colin Watson [cjwatson@debian.org]


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110811225511.GA20617@riva.dynamic.greenend.org.u k">http://lists.debian.org/20110811225511.GA20617@riva.dynamic.greenend.org.u k
 
Old 08-12-2011, 12:50 PM
Raphael Hertzog
 
Default The archive now supports xz compression

On Thu, 11 Aug 2011, Colin Watson wrote:
> > Since hardcoding gzip for base packages seems to be a bit brittle,
> > we need to work towards allowing xz usage in debian-installer and accept
> > it as a dependency for deboostrap on non-Debian systems (I don't think
> > it's a big issue, xz is portable and is more and more widely used).
>
> Can you quantify that? I don't have hard numbers for the non-Debian
> systems where people report running debootstrap; perhaps you do ...

Nope, sorry. I was referring to things like GNOME shipping only .tar.xz.
I mean they would not take such a decision if getting an xz decompressor
was a pain on many systems.

ftp.gnu.org also ships .tar.xz nowadays.

> > I guess that a first step is to have an xz udeb...
>
> No. busybox already has an unxz applet, so if we choose to do this then
> we should enable that applet in busybox-udeb, not add a separate udeb.

Ok, then maybe a first step is to verify if deboostrap + busybox's unxz
work correctly together to setup a debian chroot with all base packages
recompiled with xz.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
▶ http://RaphaelHertzog.fr (Français)


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110812125032.GE23627@rivendell.home.ouaza.com">h ttp://lists.debian.org/20110812125032.GE23627@rivendell.home.ouaza.com
 
Old 08-12-2011, 01:16 PM
Colin Watson
 
Default The archive now supports xz compression

On Fri, Aug 12, 2011 at 02:50:32PM +0200, Raphael Hertzog wrote:
> On Thu, 11 Aug 2011, Colin Watson wrote:
> > Can you quantify that? I don't have hard numbers for the non-Debian
> > systems where people report running debootstrap; perhaps you do ...
>
> Nope, sorry. I was referring to things like GNOME shipping only .tar.xz.
> I mean they would not take such a decision if getting an xz decompressor
> was a pain on many systems.

I would love to have such faith. :-)

--
Colin Watson [cjwatson@debian.org]


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110812131622.GA5677@riva.dynamic.greenend.org.uk ">http://lists.debian.org/20110812131622.GA5677@riva.dynamic.greenend.org.uk
 
Old 08-14-2011, 03:19 PM
Joey Hess
 
Default The archive now supports xz compression

Raphael Hertzog wrote:
> Nope, sorry. I was referring to things like GNOME shipping only .tar.xz.
> I mean they would not take such a decision if getting an xz decompressor
> was a pain on many systems.

There is a large distance between systems on which users are likely to
build gnome from scratch (which involves installing many dependencies
anyway), and systems on which users may want to run debootstrap. The
latter can include embedded systems that don't have a compiler and are
not of a common architecture, which makes xz difficult to install.

It would be best if debootstrap's dependencies remained limited to
things that are commonly enabled in busybox.

--
see shy jo
 
Old 08-14-2011, 07:19 PM
Lucas Nussbaum
 
Default The archive now supports xz compression

On 11/08/11 at 19:52 +0000, Philipp Kern wrote:
> On 2011-08-11, Adam Borowski <kilobyte@angband.pl> wrote:
> >> Think of both user systems and the Debian buildds which will waste more
> >> time - an especially bad problem on slower architectures.
> > The gain is especially meaningful for slower architectures, as they tend to
> > have less disk space and slower network links (arm tends to be used in
> > phones). No extra memory is needed -- decompression is not done in parallel
> > with memory-hungry stages of dpkg's work. The decompression, merely 2.5
> > times slower than with gzip, is a tiny fraction of what dpkg takes.
>
> It takes a lot longer to compress on slower architectures (i.e. on the
> buildds), though. You could've built a whole package in that time. (Resorting
> to your style of argument.)

Wouldn't it be better to get more buildds for those archs, then?
That would be a totally appropriate use of Debian money...

Of course, it might require finding more buildd maintainers. But I must
admit that I have no idea what buildd admins spend time on, and how it's
possible to help them. For example, if we tried to have more identical
buildds, instead of just one of each model, would it reduce the workload
of buildd admins significantly?

Lucas


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110814191902.GA3932@xanadu.blop.info">http://lists.debian.org/20110814191902.GA3932@xanadu.blop.info
 
Old 08-14-2011, 08:52 PM
Philipp Kern
 
Default The archive now supports xz compression

On Sun, Aug 14, 2011 at 09:19:02PM +0200, Lucas Nussbaum wrote:
> On 11/08/11 at 19:52 +0000, Philipp Kern wrote:
> > On 2011-08-11, Adam Borowski <kilobyte@angband.pl> wrote:
> > >> Think of both user systems and the Debian buildds which will waste more
> > >> time - an especially bad problem on slower architectures.
> > > The gain is especially meaningful for slower architectures, as they tend to
> > > have less disk space and slower network links (arm tends to be used in
> > > phones). No extra memory is needed -- decompression is not done in parallel
> > > with memory-hungry stages of dpkg's work. The decompression, merely 2.5
> > > times slower than with gzip, is a tiny fraction of what dpkg takes.
> > It takes a lot longer to compress on slower architectures (i.e. on the
> > buildds), though. You could've built a whole package in that time. (Resorting
> > to your style of argument.)
> Wouldn't it be better to get more buildds for those archs, then?
> That would be a totally appropriate use of Debian money...
>
> Of course, it might require finding more buildd maintainers. But I must
> admit that I have no idea what buildd admins spend time on, and how it's
> possible to help them. For example, if we tried to have more identical
> buildds, instead of just one of each model, would it reduce the workload
> of buildd admins significantly?

Replying more generally than the problem at hand, given that you did
the same.

The problem on those arches is finding stable buildds. We want boards
that support more RAM that they normally do (e.g. on MIPS) or some
with SATA disks and good throughput (e.g. on ARM).

So you get to use development boards, which you can only get hosted by
the company doing them (e.g. for ARM), or which are not stable unless
heavily patched (because they normally use a custom kernel by the
vendor, e.g. for MIPS). And then there are the boards which were just
buggy CPU-wise (older Loongson ones). Andi Barth bought some, but
it's unhelpful if you're able to crash them as a normal user due to a
pipelining bug.

The situation is getting better for those machines we have. DSA would
like to run Debian kernels on them, but it seems to be hard. Hence
you get to run some kernels with specific patchsets to be as near to a
released kernel as possible, with some extensions to let the hardware
run stable (or boot at all).

Yes, there is a point where you don't want to add more buildds instead
of getting faster ones. There's a bunch of locking in the wanna-build
database, which used to be much worse, but which is still present in
some way. There's the overhead of managing the machines for DSA.
But the addition of four(?) additional builders hosted at ARM, which
were identical, was great. Especially if they are all set up alike,
you can just put up a clusterssh session to handle them. It's just
that we now lack a bit of redundancy because all of them are hosted at
the same location…

Kind regards
Philipp Kern
 
Old 08-14-2011, 09:52 PM
Lars Wirzenius
 
Default The archive now supports xz compression

On Sun, Aug 14, 2011 at 09:19:02PM +0200, Lucas Nussbaum wrote:
> Of course, it might require finding more buildd maintainers. But I must
> admit that I have no idea what buildd admins spend time on, and how it's
> possible to help them.

"A life in the day of a buildd maintainer" would not be a bad
continuation to the IRC Q&A sessions we've already had.

--
Freedom-based blog/wiki/web hosting: http://www.branchable.com/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110814215211.GA26583@havelock.liw.fi">http://lists.debian.org/20110814215211.GA26583@havelock.liw.fi
 

Thread Tools




All times are GMT. The time now is 03:12 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org