On Mon, Dec 22, 2008 at 01:10:04PM +0200, Volkan YAZICI wrote:
> Hi,
>
> Are there any compression tools in the official Debian GNU/Linux
> repositories that can take the benefit of multiple cores/cpus?
Could you please giva a usage scenario?
Is the task really CPU bound and not i/o bound?
--
Tzafrir Cohen | tzafrir@jabber.org | VIM is
http://tzafrir.org.il | | a Mutt's
tzafrir@cohens.org.il | | best
ICQ# 16849754 | | friend
--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
12-22-2008, 11:48 AM
Volkan YAZICI
Parallel GZIP
On Mon, 22 Dec 2008, Tzafrir Cohen <tzafrir@cohens.org.il> writes:
> Could you please giva a usage scenario?
Compressing a database backup file of size 300GB.
> Is the task really CPU bound and not i/o bound?
"top -d 1" tells me that CPU usage is 800%. (BTW, have you ever met with
an input file that is not CPU bound, but I/O bound -- discarding
specifically generated worst-case scenario inputs.)
Regards.
--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
12-22-2008, 11:53 AM
"Eugene V. Lyubimkin"
Parallel GZIP
Volkan YAZICI wrote:
> Hi,
>
> Are there any compression tools in the official Debian GNU/Linux
> repositories that can take the benefit of multiple cores/cpus?
pbzip2?
--
Eugene V. Lyubimkin aka JackYF, JID: jackyf.devel(maildog)gmail.com
Ukrainian C++ Developer, Debian Maintainer, APT contributor
12-22-2008, 12:16 PM
Osamu Aoki
Parallel GZIP
Hi,
On Mon, Dec 22, 2008 at 02:48:41PM +0200, Volkan YAZICI wrote:
> On Mon, 22 Dec 2008, Tzafrir Cohen <tzafrir@cohens.org.il> writes:
> > Could you please giva a usage scenario?
>
> Compressing a database backup file of size 300GB.
>
> > Is the task really CPU bound and not i/o bound?
>
> "top -d 1" tells me that CPU usage is 800%. (BTW, have you ever met with
> an input file that is not CPU bound, but I/O bound -- discarding
> specifically generated worst-case scenario inputs.)
gzip, bzip2, and lzma seem quite CPU intensive as I saw too.
For my local archiving of not-so-big ata, I just tar/cpio them w/o
compression. Lazy me.
You may wish to try lzop which is optimized for speed.
--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
12-22-2008, 01:25 PM
"Douglas A. Tutty"
Parallel GZIP
On Mon, Dec 22, 2008 at 11:37:34AM +0000, Tzafrir Cohen wrote:
> On Mon, Dec 22, 2008 at 01:10:04PM +0200, Volkan YAZICI wrote:
> >
> > Are there any compression tools in the official Debian GNU/Linux
> > repositories that can take the benefit of multiple cores/cpus?
>
> Could you please giva a usage scenario?
>
> Is the task really CPU bound and not i/o bound?
Installing on a new dual PII-450 box from a local apt-cacher repository.
The box was an HP NetServer LPr with hardware RAID. Yes, the install
was compute bound according to top. It would have been nice if there
was a drop-in replacement for gzip that dpkg could have used that would
use both CPUs.
Doug.
--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
12-22-2008, 01:51 PM
"Thomas Preud'homme"
Parallel GZIP
The Monday 22 December 2008 13:53:43 Eugene V. Lyubimkin, you wrote*:
> Volkan YAZICI wrote:
> > Hi,
> >
> > Are there any compression tools in the official Debian GNU/Linux
> > repositories that can take the benefit of multiple cores/cpus?
>
> pbzip2?
I profit of this thread to ask you if you see amelioration with pbzip2. I made
2 tests on file of 50 MB and the result if approximately the same time as
bzip2 (a little much longer with pbzip2 !!!) but a CPU twice time more used.
I have a core 2 duo and time give me a usage of almost 200% with pbzip2
Thomas Preud'homme wrote:
> The Monday 22 December 2008 13:53:43 Eugene V. Lyubimkin, you wrote :
>> Volkan YAZICI wrote:
>>> Hi,
>>>
>>> Are there any compression tools in the official Debian GNU/Linux
>>> repositories that can take the benefit of multiple cores/cpus?
>> pbzip2?
>
>
> I profit of this thread to ask you if you see amelioration with pbzip2. I made
> 2 tests on file of 50 MB and the result if approximately the same time as
> bzip2 (a little much longer with pbzip2 !!!) but a CPU twice time more used.
> I have a core 2 duo and time give me a usage of almost 200% with pbzip2
>
> Anyone has better results ?
Yes, see following:
jackyf@work:~/temp$ time tar -cjf temp1.tar.bz2 debian
real 0m18.830s
user 0m18.317s
sys 0m0.268s
jackyf@work:~/temp$ time bash -c "tar -cf - debian | pbzip2 -c > temp2.tar.bz2"
real 0m10.494s
user 0m19.557s
sys 0m0.488s
10 is much less than 18.
--
Eugene V. Lyubimkin aka JackYF, JID: jackyf.devel(maildog)gmail.com
Ukrainian C++ Developer, Debian Maintainer, APT contributor
12-22-2008, 02:17 PM
Ron Johnson
Parallel GZIP
On 12/22/08 09:13, Eugene V. Lyubimkin wrote:
Thomas Preud'homme wrote:
The Monday 22 December 2008 13:53:43 Eugene V. Lyubimkin, you wrote :
Volkan YAZICI wrote:
Hi,
Are there any compression tools in the official Debian GNU/Linux
repositories that can take the benefit of multiple cores/cpus?
pbzip2?
I profit of this thread to ask you if you see amelioration with pbzip2. I made
2 tests on file of 50 MB and the result if approximately the same time as
bzip2 (a little much longer with pbzip2 !!!) but a CPU twice time more used.
I have a core 2 duo and time give me a usage of almost 200% with pbzip2
Anyone has better results ?
Yes, see following:
jackyf@work:~/temp$ time tar -cjf temp1.tar.bz2 debian
real 0m18.830s
user 0m18.317s
sys 0m0.268s
jackyf@work:~/temp$ time bash -c "tar -cf - debian | pbzip2 -c > temp2.tar.bz2"
real 0m10.494s
user 0m19.557s
sys 0m0.488s
10 is much less than 18.
That might be because much of the data is already cached. Each of
those commands should be run 3 times to get good numbers.
--
Ron Johnson, Jr.
Jefferson LA USA
I like my women like I like my coffee - purchased at above-market
rates from eco-friendly organic farming cooperatives in Latin America.
--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
12-22-2008, 02:18 PM
"Thomas Preud'homme"
Parallel GZIP
The Monday 22 December 2008 16:13:17 Eugene V. Lyubimkin, you wrote*:
> Thomas Preud'homme wrote:
> > The Monday 22 December 2008 13:53:43 Eugene V. Lyubimkin, you wrote :
> >> Volkan YAZICI wrote:
> >>> Hi,
> >>>
> >>> Are there any compression tools in the official Debian GNU/Linux
> >>> repositories that can take the benefit of multiple cores/cpus?
> >>
> >> pbzip2?
> >
> > I profit of this thread to ask you if you see amelioration with pbzip2. I
> > made 2 tests on file of 50 MB and the result if approximately the same
> > time as bzip2 (a little much longer with pbzip2 !!!) but a CPU twice time
> > more used. I have a core 2 duo and time give me a usage of almost 200%
> > with pbzip2
> >
> > Anyone has better results ?
>
> Yes, see following:
>
> jackyf@work:~/temp$ time tar -cjf temp1.tar.bz2 debian
>
> real 0m18.830s
> user 0m18.317s
> sys 0m0.268s
> jackyf@work:~/temp$ time bash -c "tar -cf - debian | pbzip2 -c >
> temp2.tar.bz2"
>
> real 0m10.494s
> user 0m19.557s
> sys 0m0.488s
>
> 10 is much less than 18.
Ok so sometime it works well. It would be great to know when it'll be quicker
and when it will not. What kind of files are in your debian directory ?
Regards,
Thomas Preud'homme
--
Why debian : http://www.debian.org/intro/why_debian
Parallel GZIP
