FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 09-27-2012, 08:58 AM
Tobias Klausmann
 
Default ship app-arch/pbzip2 instead of app-arch/bzip2

Hi!

On Wed, 26 Sep 2012, Ch*-Thanh Christopher Nguyễn wrote:
> A different question is whether in the cases where parallel bzip2 makes
> sense, is it really the best solution? xz is outperforming bzip2's
> compression ratio for large files (for an informal comparison, see bug
> 434350). And xz is faster at decompression, which offsets the parallel
> advantage to some degree.

As for the performance side of things:
http://blog.i-no.de/archives/2008/05/08/index.html#e2008-05-08T16_35_13.txt

Yes, that's four years old and needs to be redone with modern
implementations (and machines). Will do so this weekend (I hope).

Regards,
Tobias

--
printk (KERN_ALERT "You are screwed! " ...);
linux-2.6.6/arch/i386/kernel/efi.c
 
Old 09-27-2012, 09:23 AM
Piotr Szymaniak
 
Default ship app-arch/pbzip2 instead of app-arch/bzip2

On Wed, Sep 26, 2012 at 01:43:27PM -0700, Matt Turner wrote:
> On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol <mikemol@gmail.com> wrote:
> > A few months ago, I filed bug 423651 to ask that bzip2 on the install
> > media be replaced with
> > pbzip2. It was closed a short while later, telling me that it'd
> > involve changing what's kept in @system, and that had to be discussed
> > here, rather than in a bug report.
>
> If we're going to ship a parallel bzip2 implementation, it should be
> lbzip2 and not pbzip2.
>
> lbzip2 can decompress bz2 archives with multiple threads that haven't
> been compressed with lbzip2/pbzip2.

Afair I'm using PORTAGE_BZIP2_COMMAND with lbzip2 and it works fine.
Also some time ago I've changed a bit the (famous?) stage4 backup
script from g-wiki to support parallel gz/bz2 implementations (simple
check if there pbzip2/lbzip2/foobar installed and if yes, use it instead
of normal gzip/bzip2).

Maybe portage should be like my stage4 mod? If it finds some parallel
(de)compressor it should use it, if not fallback to standard gzip/bzip2?


Piotr Szymaniak.
--
- Jeden hamburger na dziesiec ci zaszkodzi. Jeden moj stary przyjaciel
to sprawdzil. Zjadal dziewiec hamburgerow i byl idealnie zdrowy, a gdy
probowal zjesc dziesiatego, z miejsca dostawal torsji.
-- Graham Masterton, "Night Warriors"
 
Old 09-27-2012, 09:48 AM
Ulrich Mueller
 
Default ship app-arch/pbzip2 instead of app-arch/bzip2

>>>>> On Wed, 26 Sep 2012, Michael Mol wrote:

> A few months ago, I filed bug 423651 to ask that bzip2 on the
> install media be replaced with pbzip2. It was closed a short while
> later, telling me that it'd involve changing what's kept in @system,
> and that had to be discussed here, rather than in a bug report.

We need to be careful when we replace such standard tools. Often the
replacement isn't completely compatible. For example, pbzip2 suffers
from the same bug as pigz [1] when it encounters a zero-padded tarball:

$ echo foo | bzip2 | dd conv=sync 2>/dev/null | pbzip2 -d
foo
pbzip2: *ERROR during BZ2_bzDecompress - trailing garbage: ret=4; block=0; seq=0; isLastInSeq=1; avail_in=472
Terminator thread: premature exit requested - quitting...
$ echo $?
1

The same command line as above but with bzip2 -d will return a good
exit status.

Ulrich

[1] <https://bugs.gentoo.org/show_bug.cgi?id=417657#c17>
 
Old 09-27-2012, 03:09 PM
Florian Philipp
 
Default ship app-arch/pbzip2 instead of app-arch/bzip2

Am 27.09.2012 09:22, schrieb Florian Philipp:
> Am 26.09.2012 23:53, schrieb Michael Mol:
>> On Wed, Sep 26, 2012 at 5:27 PM, Florian Philipp <lists@binarywings.net> wrote:
>>> Am 26.09.2012 22:43, schrieb Matt Turner:
>>>> On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol <mikemol@gmail.com> wrote:
>>>>> A few months ago, I filed bug 423651 to ask that bzip2 on the install
>>>>> media be replaced with
>>>>> pbzip2. It was closed a short while later, telling me that it'd
>>>>> involve changing what's kept in @system, and that had to be discussed
>>>>> here, rather than in a bug report.
>>>>
>>>> If we're going to ship a parallel bzip2 implementation, it should be
>>>> lbzip2 and not pbzip2.
>>>>
>>>> lbzip2 can decompress bz2 archives with multiple threads that haven't
>>>> been compressed with lbzip2/pbzip2.
>>>>
>>>
>>> This seems relevant, especially comment 12ff:
>>> https://bugs.gentoo.org/show_bug.cgi?id=309683
>>>
>>> For further anecdotal evidence: I've used pbzip2 with USE="symlink" for
>>> several months now and never had trouble with it. Checking out lbzip2
>>> now. I noticed it doesn't install a bunzip2 symlink.
>>
>> Piotr Szymaniak asked me about lbzip2, and I bounced the question over
>> to my friend. He didn't investigate it deeply; it crashed (OOM or
>> something else, I don't know) when he tried it on a large file. Could
>> have been from 2GB to 2TB, from what he has laying around. I don't
>> know; I didn't get that one in writing.
>>
>> But if it proves to be stable for small and very large files, I'd have
>> no complaint.
>>
>
> I just encountered this:
>
> bzip2 -c </srv/qemu/hpwin.img >/dev/null
> bzip2:
> /var/tmp/portage/app-arch/lbzip2-2.2/work/lbzip2-2.2/src/encode.c:794:
> generate_initial_trees: Assertion `a < b' failed.
>
> Something in that file is upsetting lbzip2. I'm investigating.

Okay, reported and (hopefully) fixed in
https://bugs.gentoo.org/show_bug.cgi?id=436382

In my infinite confidence in my own coding and testing skills I suggest
copying the proposed patch to /etc/portage/patches/app-arch/lbzip2-2.2
before trying out lbzip2. ;-)

Regards,
Florian Philipp
 

Thread Tools




All times are GMT. The time now is 07:52 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org