FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian User

 
 
LinkBack Thread Tools
 
Old 01-18-2010, 04:20 PM
Leonardo Canducci
 
Default rsync: different target size

I'm using rsync -aHS to backup some stuff (mostly jpgs and docs from
my home) to an external usb hard drive (same ext3 fs).
After a backup I ran du -s to get a fast check on size and found
source and target to be slightly different.
Even using du -cb or du-cbk size doesn't match. So what's wrong?!

I've noticed some dir size doesn't match:
leo@zazzero:~$ ls -ld /media/toshiba-docs/foto/d50/
drwxr-xr-x 43 leo leo 4096 6 gen 11:17 /media/toshiba-docs/foto/d50/
leo@zazzero:~$ ls -ld /share/foto/d50/
drwxr-xr-x 43 leo leo 69632 6 gen 11:17 /share/foto/d50/

I'd like the size of the backup to be exactly the same and check sync
result with du.

BTW, is there some better fast check I could do to test rsync behavior?

Thanks!
--
Leonardo Canducci


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-18-2010, 08:13 PM
Johannes Wiedersich
 
Default rsync: different target size

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Leonardo Canducci wrote:
> I'm using rsync -aHS to backup some stuff (mostly jpgs and docs from
> my home) to an external usb hard drive (same ext3 fs).

[snip]

> I'd like the size of the backup to be exactly the same and check sync
> result with du.
>
> BTW, is there some better fast check I could do to test rsync behavior?

If your data and the backup are important to you, the question is not,
whether your check is fast, but if it is accurate. I'd suggest you use
the -cnv options of rsync to check for differences between source and
target. It will display all different files (ie. files with different
checksum) without modifying anything.

FWIW, why do you use the -S option and are you aware of the consequences:

/---man rsync---
-S, --sparse
Try to handle sparse files efficiently so they take up
less space on the destination. Conflicts with
--inplace because it’s not possible to overwrite data in a
sparse fashion.
---------------

- --
Johannes

Three nations have not officially adopted the International System
of Units as their primary or sole system of measurement: Burma,
Liberia, and the United States.

http://en.wikipedia.org/wiki/Si_units
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAktUzwwACgkQC1NzPRl9qEUJiwCeNGsfNf9wWC RDYwnftk8wdTS1
Y2kAnAoR0m+u5IJuZ70CLnJWy9adTPLq
=MpRH
-----END PGP SIGNATURE-----


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-19-2010, 04:03 AM
Rick Thomas
 
Default rsync: different target size

On Jan 18, 2010, at 12:20 PM, Leonardo Canducci wrote:


I'm using rsync -aHS to backup some stuff (mostly jpgs and docs from
my home) to an external usb hard drive (same ext3 fs).
After a backup I ran du -s to get a fast check on size and found
source and target to be slightly different.
Even using du -cb or du-cbk size doesn't match. So what's wrong?!

I've noticed some dir size doesn't match:
leo@zazzero:~$ ls -ld /media/toshiba-docs/foto/d50/
drwxr-xr-x 43 leo leo 4096 6 gen 11:17 /media/toshiba-docs/foto/d50/
leo@zazzero:~$ ls -ld /share/foto/d50/
drwxr-xr-x 43 leo leo 69632 6 gen 11:17 /share/foto/d50/

I'd like the size of the backup to be exactly the same and check sync
result with du.

BTW, is there some better fast check I could do to test rsync
behavior?


Thanks!
--
Leonardo Canducci


Probably you have directories that have grown and shrunk on the source
filesystem. They will still have the space allocated (just the actual
directory -- not the files in it) for the file-name entries that were
deleted. When you transfer them to the target (in this case the USB
hard drive), the directories are rebuilt from scratch, so they don't
have space allocated for the deleted file-names.


If you want to use file sizes as a check on the operation of rsync
(not the best check, but it will catch some kinds of errors) you could
do something like this


( cd source ; find . -type f -print0 | sort -z | xargs ls -s ) > /tmp/
source-stuff
( cd target ; find . -type f -print0 | sort -z | xargs ls -s ) > /tmp/
target-stuff

diff /tmp/target-stuff /tmp/source-stuff

you can use something like "md5sum" in place of "ls -s" if you want a
more industrial strength check...


Enjoy!

Rick


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-19-2010, 09:57 AM
Leonardo Canducci
 
Default rsync: different target size

2010/1/18 Johannes Wiedersich <johannes@physik.blm.tu-muenchen.de>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Leonardo Canducci wrote:
>> I'm using rsync -aHS to backup some stuff (mostly jpgs and docs from
>> my home) to an external usb hard drive (same ext3 fs).
>
> [snip]
>
>> I'd like the size of the backup to be exactly the same and check sync
>> result with du.
>>
>> BTW, is there some better fast check I could do to test rsync behavior?
>
> If your data and the backup are important to you, the question is not,
> whether your check is fast, but if it is accurate. I'd suggest you use
> the -cnv options of rsync to check for differences between source and
> target. It will display all different files (ie. files with different
> checksum) without modifying anything.

I just wanted to get a fast (not accurate) check with some other tool.
I knew about -c switch but I read it's really slow and my backup is
30Gb.
>
> FWIW, why do you use the -S option and are you aware of the consequences:
>
> /---man rsync---
> * * * -S, --sparse
> * * * * * * *Try to handle sparse files efficiently so they take up
> * * * * * * *less *space *on *the *destination. * Conflicts *with
> * * * * * * *--inplace because it’s not possible to overwrite data in a
> * * * * * * *sparse fashion.

I read the man page and I usually use plain rsync -a. I was told to
try -aHS on a chat after having the same problem with rsync -a.
--
Leonardo Canducci


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-19-2010, 09:58 AM
Leonardo Canducci
 
Default rsync: different target size

2010/1/19 Rick Thomas <rbthomas@pobox.com>:
>
> On Jan 18, 2010, at 12:20 PM, Leonardo Canducci wrote:
>
>> I'm using rsync -aHS to backup some stuff (mostly jpgs and docs from
>> my home) to an external usb hard drive (same ext3 fs).
>> After a backup I ran du -s to get a fast check on size and found
>> source and target to be slightly different.
>> Even using du -cb or du-cbk size doesn't match. So what's wrong?!
>>
>> I've noticed some dir size doesn't match:
>> leo@zazzero:~$ ls -ld /media/toshiba-docs/foto/d50/
>> drwxr-xr-x 43 leo leo 4096 *6 gen 11:17 /media/toshiba-docs/foto/d50/
>> leo@zazzero:~$ ls -ld /share/foto/d50/
>> drwxr-xr-x 43 leo leo 69632 *6 gen 11:17 /share/foto/d50/
>>
>> I'd like the size of the backup to be exactly the same and check sync
>> result with du.
>>
>> BTW, is there some better fast check I could do to test rsync behavior?
>>
>> Thanks!
>> --
>> Leonardo Canducci
>
> Probably you have directories that have grown and shrunk on the source
> filesystem. *They will still have the space allocated (just the actual
> directory -- not the files in it) for the file-name entries that were
> deleted. *When you transfer them to the target (in this case the USB hard
> drive), the directories are rebuilt from scratch, so they don't have space
> allocated for the deleted file-names.

That's it!
>
> If you want to use file sizes as a check on the operation of rsync (not the
> best check, but it will catch some kinds of errors) you could do something
> like this
>
> ( cd source ; find . -type f -print0 | sort -z | xargs ls -s ) >
> /tmp/source-stuff
> ( cd target ; find . -type f -print0 | sort -z | xargs ls -s ) >
> /tmp/target-stuff
> diff /tmp/target-stuff /tmp/source-stuff
>
> you can use something like "md5sum" in place of "ls -s" if you want a more
> industrial strength check...

how does that compare to rsync -cn? Is it faster? safer?

Thanks!
--
Leonardo Canducci


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-19-2010, 10:13 AM
Jon Dowland
 
Default rsync: different target size

On Mon, Jan 18, 2010 at 06:20:27PM +0100, Leonardo Canducci
wrote:
> I'm using rsync -aHS to backup some stuff (mostly jpgs and
> docs from my home) to an external usb hard drive (same
> ext3 fs).
snip
> I've noticed some dir size doesn't match:
snip

Ext3 has a limitation in the way it stores directory
entries: the space allocated for a directories contents can
grow but not shrink. You can test this by creating a new
directory, adding a single file, checking the size of the
directory, then creating a large number of files ("for i in
`seq 1 100000000`; do mktemp --tmpdir=<testdir>& done" or
similar): check the size again. Then remove all the files,
re-check.

Thus checking the size of the source and destination folders
is not useful if the rate of change/growth for the folders
will differ (which for an active partition and a relatively
inactive backup partition will always be the case). The most
reliable (imho) way to ensure that you have an accurate copy
is an rsync dry run:

rsync -van --delete ./src ./dest

(or similar). If it says it will make no changes, you're ok.


--
Jon Dowland
 
Old 01-19-2010, 12:06 PM
Rick Thomas
 
Default rsync: different target size

On Jan 19, 2010, at 5:58 AM, Leonardo Canducci wrote:


( cd source ; find . -type f -print0 | sort -z | xargs ls -s ) >
/tmp/source-stuff
( cd target ; find . -type f -print0 | sort -z | xargs ls -s ) >
/tmp/target-stuff
diff /tmp/target-stuff /tmp/source-stuff

you can use something like "md5sum" in place of "ls -s" if you want
a more

industrial strength check...


how does that compare to rsync -cn? Is it faster? safer?


It's probably about the same speed as "rsync -cn", but it doesn't
depend on rsync. I was thinking: if you don't trust rsync, then use
something else to generate the checksums. But if you trust rsync, why
bother with double-checking in the first place?


Rick


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-19-2010, 02:02 PM
Johannes Wiedersich
 
Default rsync: different target size

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Rick Thomas wrote:
> I was thinking: if you don't trust rsync, then use something
> else to generate the checksums. But if you trust rsync, why bother with
> double-checking in the first place?

Because it could be that rsync works fine, but there are some problems
with the hardware -- like bad blocks on the disk.

If you don't trust rsync you should use some other tool.

FWIW my 300GB take a few hours to check; on reasonably recent hardware
30 GB should take minutes, ie. less than an hour or so to verify.

- --
Johannes

Three nations have not officially adopted the International System
of Units as their primary or sole system of measurement: Burma,
Liberia, and the United States.

http://en.wikipedia.org/wiki/Si_units
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAktVyXwACgkQC1NzPRl9qEWQFQCeN2Nc0HNpL5 33xQq15FhyNTsv
na0AnRK8zTd/bSGlwkbyTujG7lmoEHtJ
=huTF
-----END PGP SIGNATURE-----


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 

Thread Tools




All times are GMT. The time now is 11:55 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org