FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian User

 
 
LinkBack Thread Tools
 
Old 08-22-2012, 02:56 PM
lina
 
Default compressor

On Wednesday 22,August,2012 10:50 PM, Jon Dowland wrote:
> On Wed, Aug 22, 2012 at 04:24:19PM +0200, Ralf Mardorf wrote:
>> On Wed, 2012-08-22 at 16:15 +0200, Jerome BENOIT wrote:
>>> xz: tar Jcf
$ tar --version
tar (GNU tar) 1.14
no -J options.

On desktop it's tar (GNU tar) 1.26 can support the .tar.xz

Thanks,

>>
>> I'm using a distro that packages with xz.
>>
>> I'm sure that there never was a big difference between
>> "gz: tar zcf" and "bzip2: tar jcf" for the length of the files, but the
>> time for packing and unpacking does differ very much.
>
> Indeed, bz2 is very slow. But it can be much better at compressing:
>
> linux-3.6-rc2.tar.bz2 16-Aug-2012 22:01 78M
> linux-3.6-rc2.tar.gz 16-Aug-2012 22:01 99M
> linux-3.6-rc2.tar.xz 16-Aug-2012 22:01 65M
>
>


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 5034F30E.6080707@gmail.com">http://lists.debian.org/5034F30E.6080707@gmail.com
 
Old 08-22-2012, 03:43 PM
Camaleón
 
Default compressor

On Wed, 22 Aug 2012 21:40:36 +0800, lina wrote:

> Basically which compressor is the most efficient one.

Ha, that's like asking "what do clouds smell like"? >:-)

> I got 2T data, basically won't get a chance to use in future, but still
> need to keep there at least for the next two years just in case.
>
> so I tried the xz, but xz not support the directory? or maybe I don't
> know how to compress the directory.

This may help:

http://unix.stackexchange.com/questions/28976/how-to-xz-a-directory-with-tar-using-maximum-compression

> now tried the .bz2, but still very large.

For my usual data backup I use tar and bz2 though I don't expect to compress
too much but I need the best interop and compatibility.

Anyway, what kind of data is that heap of 2 TiB?

Multimedia binary files (image, video, audio...) use their own
compression algorithms so you are not going to get any noticeable gain
when compressing these kind of files.

For general purpose data compression, Wikipedia has a good list:

http://en.wikipedia.org/wiki/Comparison_of_file_archivers

And also some info on benchmarks:

http://en.wikipedia.org/wiki/Lossless_compression_benchmarks#Lossless_compressi on_benchmarks

Greetings,

--
Camaleón


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/k12ums$6m0$10@ger.gmane.org
 
Old 08-22-2012, 04:12 PM
lina
 
Default compressor

On Wednesday 22,August,2012 11:43 PM, Camaleón wrote:
> On Wed, 22 Aug 2012 21:40:36 +0800, lina wrote:
>
>> Basically which compressor is the most efficient one.
>
> Ha, that's like asking "what do clouds smell like"? >:-)
>
>> I got 2T data, basically won't get a chance to use in future, but still
>> need to keep there at least for the next two years just in case.
>>
>> so I tried the xz, but xz not support the directory? or maybe I don't
>> know how to compress the directory.
>
> This may help:
>
> http://unix.stackexchange.com/questions/28976/how-to-xz-a-directory-with-tar-using-maximum-compression
>
>> now tried the .bz2, but still very large.
>
> For my usual data backup I use tar and bz2 though I don't expect to compress
> too much but I need the best interop and compatibility.
>
> Anyway, what kind of data is that heap of 2 TiB?
Thanks. those data files accumulated there for 2 years, today just do
the clear up.
>
> Multimedia binary files (image, video, audio...) use their own
> compression algorithms so you are not going to get any noticeable gain
> when compressing these kind of files.
>
> For general purpose data compression, Wikipedia has a good list:
>
> http://en.wikipedia.org/wiki/Comparison_of_file_archivers
>
> And also some info on benchmarks:
>
> http://en.wikipedia.org/wiki/Lossless_compression_benchmarks#Lossless_compressi on_benchmarks
>
Thanks again for those links.

Warm regards,
> Greetings,
>


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 503504FD.9010509@gmail.com">http://lists.debian.org/503504FD.9010509@gmail.com
 
Old 08-22-2012, 04:52 PM
Gary Dale
 
Default compressor

On 22/08/12 12:12 PM, lina wrote:

On Wednesday 22,August,2012 11:43 PM, Camaleón wrote:

On Wed, 22 Aug 2012 21:40:36 +0800, lina wrote:


Basically which compressor is the most efficient one.

Ha, that's like asking "what do clouds smell like"?>:-)


I got 2T data, basically won't get a chance to use in future, but still
need to keep there at least for the next two years just in case.

so I tried the xz, but xz not support the directory? or maybe I don't
know how to compress the directory.

This may help:

http://unix.stackexchange.com/questions/28976/how-to-xz-a-directory-with-tar-using-maximum-compression


now tried the .bz2, but still very large.

For my usual data backup I use tar and bz2 though I don't expect to compress
too much but I need the best interop and compatibility.

Anyway, what kind of data is that heap of 2 TiB?

Thanks. those data files accumulated there for 2 years, today just do
the clear up.

Multimedia binary files (image, video, audio...) use their own
compression algorithms so you are not going to get any noticeable gain
when compressing these kind of files.

For general purpose data compression, Wikipedia has a good list:

http://en.wikipedia.org/wiki/Comparison_of_file_archivers

And also some info on benchmarks:

http://en.wikipedia.org/wiki/Lossless_compression_benchmarks#Lossless_compressi on_benchmarks


Thanks again for those links.

Warm regards,

Greetings,



I find that .lzma does a pretty good job and isn't too slow.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Archive: 50350E3D.1030607@rogers.com">http://lists.debian.org/50350E3D.1030607@rogers.com
 
Old 08-22-2012, 10:44 PM
Bob Proulx
 
Default compressor

Jon Dowland wrote:
> Ralf Mardorf wrote:
> > Jerome BENOIT wrote:
> > > xz: tar Jcf
> >
> > I'm using a distro that packages with xz.
> >
> > I'm sure that there never was a big difference between
> > "gz: tar zcf" and "bzip2: tar jcf" for the length of the files, but the
> > time for packing and unpacking does differ very much.
>
> Indeed, bz2 is very slow. But it can be much better at compressing:
>
> linux-3.6-rc2.tar.bz2 16-Aug-2012 22:01 78M
> linux-3.6-rc2.tar.gz 16-Aug-2012 22:01 99M
> linux-3.6-rc2.tar.xz 16-Aug-2012 22:01 65M

I think lzip is worthy enough that it should have a mention too. It
has gotten less attention than xz and that is sad since it is a nice
free software tool. I recompressed that file using lzip for this
comparison.

$ ls -lhog linux-3.6-rc2.tar.gz
-rw-rw-r-- 1 100M Aug 22 15:31 linux-3.6-rc2.tar.gz

$ ls -lhog linux-3.6-rc2.tar.lz
-rw-rw-r-- 1 66M Aug 22 15:54 linux-3.6-rc2.tar.lz

Bob
 
Old 08-23-2012, 09:13 AM
Gaël DONVAL
 
Default compressor

Le mercredi 22 août 2012 à 12:52 -0400, Gary Dale a écrit :
> I find that .lzma does a pretty good job and isn't too slow.

My 2 cents:

LZMA/LZMA2 is indeed a good choice if you want best compression: it
should work with almost anything (except already compressed streams such
as videos, images, sounds, obviously).

For text files however, you could try the PPMd algorithm which performs
equally well as LZMA but for a fraction of memory/time LZMA uses. But it
sucks for almost everything else than text.

A warning: xz -9e might increase a LOT the compression time for almost
NO size reduction compared to the defaults. Also keep in mind that the
compression algorithm uses more memory when the compression level
increases: for big datasets, you will run out of memory if you are too
greedy.





--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 1345713219.4552.8.camel@p76-nom-gd.cnrs-imn.fr">http://lists.debian.org/1345713219.4552.8.camel@p76-nom-gd.cnrs-imn.fr
 
Old 08-23-2012, 10:24 AM
Jon Dowland
 
Default compressor

On Wed, Aug 22, 2012 at 04:44:38PM -0600, Bob Proulx wrote:
> Jon Dowland wrote:
> > linux-3.6-rc2.tar.bz2 78M
> > linux-3.6-rc2.tar.gz 99M
> > linux-3.6-rc2.tar.xz 65M
> linux-3.6-rc2.tar.lz 66M
>
> I think lzip is worthy enough that it should have a mention too. It
> has gotten less attention than xz and that is sad since it is a nice
> free software tool. I recompressed that file using lzip for this
> comparison.

Thanks for the data (mashed/reformatted into quote above). I copied the
listings from the kernel.org archives, so the choice of compression types
was theirs (although I hadn't heard of lzip, thanks!)


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120823102436.GI10344@debian
 
Old 08-23-2012, 10:26 AM
Jon Dowland
 
Default compressor

On Wed, Aug 22, 2012 at 03:43:24PM +0000, Camaleón wrote:
> On Wed, 22 Aug 2012 21:40:36 +0800, lina wrote:
>
> > Basically which compressor is the most efficient one.
>
> Ha, that's like asking "what do clouds smell like"? >:-)

Remember to run your chosen compression algorithm at least twice!


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120823102607.GJ10344@debian
 
Old 08-23-2012, 12:24 PM
lina
 
Default compressor

On Thursday 23,August,2012 06:26 PM, Jon Dowland wrote:
> On Wed, Aug 22, 2012 at 03:43:24PM +0000, Camaleón wrote:
>> On Wed, 22 Aug 2012 21:40:36 +0800, lina wrote:
>>
>>> Basically which compressor is the most efficient one.
>>
>> Ha, that's like asking "what do clouds smell like"? >:-)
>
> Remember to run your chosen compression algorithm at least twice!

Sorry, here you mean,

once tar -Jcf a.tar.xz a

again
tar -Jcf a.tar.xz a.tar.xz
?

>
>


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 50362116.5010301@gmail.com">http://lists.debian.org/50362116.5010301@gmail.com
 
Old 08-23-2012, 08:26 PM
Bob Proulx
 
Default compressor

Jon Dowland wrote:
> Bob Proulx wrote:
> > Jon Dowland wrote:
> > > linux-3.6-rc2.tar.bz2 78M
> > > linux-3.6-rc2.tar.gz 99M
> > > linux-3.6-rc2.tar.xz 65M
> > linux-3.6-rc2.tar.lz 66M
> >
> > I think lzip is worthy enough that it should have a mention too. It
> > has gotten less attention than xz and that is sad since it is a nice
> > free software tool. I recompressed that file using lzip for this
> > comparison.
>
> Thanks for the data (mashed/reformatted into quote above). I copied the
> listings from the kernel.org archives, so the choice of compression types
> was theirs (although I hadn't heard of lzip, thanks!)

There is a problem with the mashing and reformatting. It makes lzip
appear to be 66M against xz being 65M and so xz is better, right? But
wait the above says that gz is 99M. But ls says 100M. So the listed
sizes are not 100% correct. So 66M is true if 100M is true. But it
seems that something was truncating down to 99M and so perhaps that
65M is actually 66M? In which case xz and lz were actually the same
for that sample. Or perhaps if they count 65M as true for xy then
perhaps it should be 65M for lz too?

I think you see the problem. I don't really know from the above data
whether xz or lz is the same or worse or better.

I didn't go and download the linux-3.6-rc2.tar.xz file to see what
size it actually should be listed as. I probably should have. But I
didn't have the time.

It would be better to look at the long byte counts for this type of
comparison.

Bob
 

Thread Tools




All times are GMT. The time now is 12:39 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org