FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian dpkg

 
 
LinkBack Thread Tools
 
Old 08-29-2012, 07:51 AM
Guillem Jover
 
Default use xz compression for Debian package by default

Hi!

On Tue, 2012-08-28 at 12:10:18 +0900, Hideki Yamane wrote:
> In DebConf12, I talked about xz compression for Debian packages(*).
> Now I'll talk about next step, suggestion for use xz with with result
> from some experiment.

> ------------------------------------------------------------------------------
> conclusion (half)
> ------------------------------------------------------------------------------
> We should use xz compression instead of bzip2 at least. bzip is harmful for
> compressing debian package, so should drop it from support to check easier.

Removing bzip2 uncompression support is not an option, there's packages
on the wild compressed with it that should remain extractable from
dpkg-deb. Removing compression support could be considered, but then I
think right now bzip2 availability is more widespread than xz, so some
might want to use it for just that reason, even if it might be slower
or produce bigger packages.

> ------------------------------------------------------------------------------
> conclusion (rest)
> ------------------------------------------------------------------------------
> I recommend to use xz ***by default*** (with appropriate option) on not only
> i386/amd64 but on ANY architectures. Increasing extract time can be ignore by
> decreasing download time and its only part of installation as Mike Hommey
> suggested "I/O is still more time consuming than CPU", and nothing worse than
> high cpu usage.
>
> We know some packages are better to use gzip, but it's an exception. Using xz
> is best choice for rest 99.99% of packages. We can deal with such exception
> by specifying gzip for that (e.g. openclipart-png).

I thought this was already the consensus, and the only dissenting
opinion was that the base system should still be using gzip so that
foreign non-Debian systems can unpack it w/o requiring to build or
install xz beforehand.

Given the recent flurry of several packaging helpers and packages
switching to use xz, I think for jessie what makes most sense is
to switch all base packages to explicitly compress with gzip and then
switch dpkg-deb to default to xz; which I think would have made more
sense for wheezy too, but it seems too late now, given that non-base
packages have already been switched, because it might imply reverting
stuff and having to modify all base right now.

So if there's still consensus on this by then, I'll be switching
the default dpkg-deb compression to xz, *after* all base has been
switched to gzip. I've already queued a tiny patch for 1.17.x that
allows changing the default dpkg-deb compressor when building dpkg.
I've also some code already which avoids at least two copies of the
data.tar through pipes, which should speed up the unpacking.

> *) tiny pseudo code
>
> arch=`dpkg-architecture -qDEB_HOST_ARCH`
>
> if [ arch = arm | armel | armhf | aarch64 ] // maybe
> set on_arch --arm
> elsif [ arch = powerpc | ppc64 | powerpcspe ] // maybe
> set on_arch --powerpc
> elsif [ arch = sparc | sparc64 ] // maybe
> set on_arch --sparc
> elsif [ arch = ia64 ]
> set on_arch --ia64
> elsif [ arch = i386 | amd64 ]
> set --x86
> fi

I don't think this is a good idea, and I'm not really planning on
making the dpkg-deb compression code conditional on the being built
package architecture.

In any case, thanks for the testing and comparisons!

thanks,
guillem


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120829075114.GA2876@gaara.hadrons.org">http://lists.debian.org/20120829075114.GA2876@gaara.hadrons.org
 
Old 08-29-2012, 12:35 PM
 
Default use xz compression for Debian package by default

On Aug 29, Guillem Jover <guillem@debian.org> wrote:

> I thought this was already the consensus, and the only dissenting
> opinion was that the base system should still be using gzip so that
> foreign non-Debian systems can unpack it w/o requiring to build or
> install xz beforehand.
I am not sure if there was a consensus that we really need to care so
much about systems where xz is not installed by default.

> Given the recent flurry of several packaging helpers and packages
> switching to use xz, I think for jessie what makes most sense is
> to switch all base packages to explicitly compress with gzip and then
This looks like a lot of work for a dubious gain.

--
ciao,
Marco
 
Old 08-29-2012, 12:35 PM
 
Default use xz compression for Debian package by default

On Aug 29, Guillem Jover <guillem@debian.org> wrote:

> I thought this was already the consensus, and the only dissenting
> opinion was that the base system should still be using gzip so that
> foreign non-Debian systems can unpack it w/o requiring to build or
> install xz beforehand.
I am not sure if there was a consensus that we really need to care so
much about systems where xz is not installed by default.

> Given the recent flurry of several packaging helpers and packages
> switching to use xz, I think for jessie what makes most sense is
> to switch all base packages to explicitly compress with gzip and then
This looks like a lot of work for a dubious gain.

--
ciao,
Marco
 
Old 08-29-2012, 02:01 PM
Jon Dowland
 
Default use xz compression for Debian package by default

On Tue, Aug 28, 2012 at 12:56:47PM +0200, Vincent Lefevre wrote:
> Before wondering whether PNG files should have an additional
> compression level, is there any reason why a better PNG compression
> isn't used in the first place? For instance, "optipng -o9" tries
> various parameters and keeps the best one.

So recompress upstream PNGs and suffix +dfsg to the source version? There might
be some disadvantages to this. If you are using VCS to manage the package, you
are probably carrying the upstream PNGs in that already, so there's an
appreciable increase in repository size to carry the optimized PNGs too.
Another approach could be to inject optipng into the build process and treat
the outputs like compiler output (packed into binary packages, thrown away on
clean), but then repeated builds could be CPU-expensive. Perhaps getting
upstream to carry better-compressed PNGs in their next release is a good idea.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120829140143.GA22300@debian
 
Old 08-29-2012, 02:17 PM
Dmitrijs Ledkovs
 
Default use xz compression for Debian package by default

On 29/08/12 15:01, Jon Dowland wrote:
> On Tue, Aug 28, 2012 at 12:56:47PM +0200, Vincent Lefevre wrote:
>> Before wondering whether PNG files should have an additional
>> compression level, is there any reason why a better PNG compression
>> isn't used in the first place? For instance, "optipng -o9" tries
>> various parameters and keeps the best one.
>
> So recompress upstream PNGs and suffix +dfsg to the source version? There might
> be some disadvantages to this. If you are using VCS to manage the package, you
> are probably carrying the upstream PNGs in that already, so there's an
> appreciable increase in repository size to carry the optimized PNGs too.
> Another approach could be to inject optipng into the build process and treat
> the outputs like compiler output (packed into binary packages, thrown away on
> clean), but then repeated builds could be CPU-expensive. Perhaps getting
> upstream to carry better-compressed PNGs in their next release is a good idea.
>
>

Better to create a dh_ plugin to do this. I think there is something
like that already floating around for PNGs, at least on Ubuntu the
pkgbinarymangler does re-compress all PNGs.

I don't think it's worth +dfsg, and CPU cycles will only be wasted once
on the maintainer side, since most of PNGs are in arch:all packages anyway.

--
Regards,
Dmitrijs.
 
Old 08-29-2012, 02:47 PM
Jon Dowland
 
Default use xz compression for Debian package by default

On Wed, Aug 29, 2012 at 03:17:15PM +0100, Dmitrijs Ledkovs wrote:
> I don't think it's worth +dfsg, and CPU cycles will only be wasted once
> on the maintainer side, since most of PNGs are in arch:all packages anyway.

I used to hack on the games-thumbnails package a bit, which ran optipng as part
of the build stage and removed the results as part of the clean stage.¹ It used
to be quite a pain to run 'debuild' over and over, if you were making small,
iterative packaging changes, because the 'clean' target would be run. I suppose
I could have simply ran the build stage by hand repeatedly instead.

¹ It would appear that the package no longer does the remove stage, from a
cursory glance at the rules file these days. This is no doubt a pragmatic
decision, but one that might conflict with people's build expectations.)


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120829144701.GB22300@debian
 
Old 08-29-2012, 03:12 PM
"Didier 'OdyX' Raboud"
 
Default use xz compression for Debian package by default

Le mercredi, 29 août 2012 16.01:43, Jon Dowland a écrit :
> On Tue, Aug 28, 2012 at 12:56:47PM +0200, Vincent Lefevre wrote:
> > Before wondering whether PNG files should have an additional
> > compression level, is there any reason why a better PNG compression
> > isn't used in the first place? For instance, "optipng -o9" tries
> > various parameters and keeps the best one.
>
> So recompress upstream PNGs and suffix +dfsg to the source version?

No. Such changes belong to the binary package building.

> Another approach could be to inject optipng into the build
> process and treat the outputs like compiler output (packed into binary
> packages, thrown away on clean), but then repeated builds could be
> CPU-expensive.

That's of course the right place to do it.

We are building a binary distribution which main characteristic is to have the
packages built _rarely_. As such, a useful but CPU-expensive operation is
always worth it. And if the package building infrastructure comes in your way
by enforcing unneeded repeated builds, then the problem resides in the package
building infrastructure. GNU make²'s it easy to avoid reconstructing files,
UseThatâ„¢ !

OdyX


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 201208291712.15564.odyx@debian.org">http://lists.debian.org/201208291712.15564.odyx@debian.org
 
Old 08-29-2012, 03:22 PM
Steve McIntyre
 
Default use xz compression for Debian package by default

Marco wrote:
>On Aug 29, Guillem Jover <guillem@debian.org> wrote:
>
>> I thought this was already the consensus, and the only dissenting
>> opinion was that the base system should still be using gzip so that
>> foreign non-Debian systems can unpack it w/o requiring to build or
>> install xz beforehand.
>I am not sure if there was a consensus that we really need to care so
>much about systems where xz is not installed by default.

People have worried about it, but I think the consensus from DebConf
is that we don't want to be hampered in our own development by
considering external users *that* much. We {have added,will add} code
to debootstrap to check if xz is available and fail with a useful
error message if it's not found.

>> Given the recent flurry of several packaging helpers and packages
>> switching to use xz, I think for jessie what makes most sense is
>> to switch all base packages to explicitly compress with gzip and then
>This looks like a lot of work for a dubious gain.

Agreed.

--
Steve McIntyre, Cambridge, UK. steve@einval.com
Google-bait: http://www.debian.org/CD/free-linux-cd
Debian does NOT ship free CDs. Please do NOT contact the mailing
lists asking us to send them to you.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: E1T6k60-0004jq-Q3@mail.einval.com">http://lists.debian.org/E1T6k60-0004jq-Q3@mail.einval.com
 
Old 08-29-2012, 04:11 PM
Joey Hess
 
Default use xz compression for Debian package by default

Steve McIntyre wrote:
> People have worried about it, but I think the consensus from DebConf
> is that we don't want to be hampered in our own development by
> considering external users

How are "external users" different from "users"?

--
see shy jo
 
Old 08-29-2012, 04:24 PM
Steve McIntyre
 
Default use xz compression for Debian package by default

On Wed, Aug 29, 2012 at 12:11:02PM -0400, Joey Hess wrote:
>Steve McIntyre wrote:
>> People have worried about it, but I think the consensus from DebConf
>> is that we don't want to be hampered in our own development by
>> considering external users
>
>How are "external users" different from "users"?

Sorry, badly worded. People using debootstrap outside of Debian and
other distros or other expected places. Especially in hostile
environments - IIRC Solaris was mentioned as an example. While we like
these people to be able to use Debian's tools like debootstrap, it
should not be a priority for us to hand-hold them if that causes too
much trouble and effort.

--
Steve McIntyre, Cambridge, UK. steve@einval.com
< sladen> I actually stayed in a hotel and arrived to find a post-it
note stuck to the mini-bar saying "Paul: This fridge and
fittings are the correct way around and do not need altering"


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120829162417.GK32286@einval.com">http://lists.debian.org/20120829162417.GK32286@einval.com
 

Thread Tools




All times are GMT. The time now is 10:00 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org