FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian User

 
 
LinkBack Thread Tools
 
Old 08-10-2008, 06:12 AM
Brian Wells
 
Default rsync backup to ext3-formatted usb flash drive?

Hi. I'm using svn-fast-backup (in the subversion-tools package) to make
rsync backups of my subversion repository. The place I'm backing up to
is on an ext3-formatted usb flash drive, and I'm wondering if this is a
wise thing to do.

I know that flash drive blocks wear out after a certain number of
writes, and rsync uses hard links to avoid duplicate backup files. My
question is, does this mean that the inodes' reference counts will be
changed frequently enough that it might wear out that block of the flash
drive any time soon? (I currently backup about 900 files one or two
times a day.) If so, is ext3 capable of using another block for the
inode, or will I lose the ability to read/write some files completely?

Thanks,
Brian


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-11-2008, 06:21 AM
Rick Thomas
 
Default rsync backup to ext3-formatted usb flash drive?

On Aug 10, 2008, at 2:12 AM, Brian Wells wrote:

Hi. I'm using svn-fast-backup (in the subversion-tools package) to
make
rsync backups of my subversion repository. The place I'm backing
up to
is on an ext3-formatted usb flash drive, and I'm wondering if this
is a

wise thing to do.

I know that flash drive blocks wear out after a certain number of
writes, and rsync uses hard links to avoid duplicate backup files. My
question is, does this mean that the inodes' reference counts will be
changed frequently enough that it might wear out that block of the
flash

drive any time soon? (I currently backup about 900 files one or two
times a day.) If so, is ext3 capable of using another block for the
inode, or will I lose the ability to read/write some files completely?

Thanks,
Brian



Here are some basic facts about USB flash drives:


The underlying NAND flash RAM is guaranteed for 100K writes (older
technology) or 1M (more recent). After that it's retention time
starts to decrease, sometimes to as little as a few weeks, and with
continued use eventually to complete failure. The retention time of
a new, fully-functional NAND flash RAM is hundreds of years.


USB flash drives have controller chips in the drive that, among other
things, talk USB protocol to the host. One of the other things they
do is try to distribute the writes over the available blocks,
extending the lifetime of the device beyond the raw lifetime of the
underlying NAND flash. But the firmware in the controller usually
assumes you are using a FAT file-system. If, for example, you use
EXT3 rather than FAT, you will probably confuse it. I don't know
what the effect will be on device life, but I'm guessing it will
probably be something like a return to the behavior of the raw NAND
flash.


Write blocks are large (4-8K or more). If you update one inode in a
block, you have to read the entire block, update the inode and write
the entire block back again. Each write to an inode causes an
effective write to all the inodes in the same block. So there's
roughly a 64x multiplier (8Kbytes/block divided by 128bytes/inode =
64 inodes/block) in the number of writes. Thus any given single
inode is good for roughly 16K updates before it wears out.


Let's suppose your rsync backup causes a small number of updates for
each inode it touches. Call it 10, conservatively (it very well
could be as low as 1 or 2 -- I don't know enough of the details of
how rsync works at the filesystem level). That gives you about 1600
rsync runs before you start to wear out the inodes. At twice a day,
you can safely use your USB flash drive for 800 days, or between 2
and 3 years.


So here's what I'd do. Use your USB flash drive for a year, then
retire it to the long-term archives and replace it with a new one.
Let that be one of those things you do on or near your birthday, like
schedule a checkup with your Doctor. Also: dismount the flash drive,
force an fsck, and remount it, every time you do a backup. This will
read and sanity-check every inode on the drive, without doing any
writes. If the fsck shows random errors, replace the drive because
it's probably starting to wear out.


Sound reasonable?


Rick



--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-11-2008, 08:25 PM
Brian Wells
 
Default rsync backup to ext3-formatted usb flash drive?

On Mon, 2008-08-11 at 02:21 -0400, Rick Thomas wrote:
> On Aug 10, 2008, at 2:12 AM, Brian Wells wrote:
>
> > Hi. I'm using svn-fast-backup (in the subversion-tools package) to
> > make
> > rsync backups of my subversion repository. The place I'm backing
> > up to
> > is on an ext3-formatted usb flash drive, and I'm wondering if this
> > is a
> > wise thing to do.
> >
> > I know that flash drive blocks wear out after a certain number of
> > writes, and rsync uses hard links to avoid duplicate backup files. My
> > question is, does this mean that the inodes' reference counts will be
> > changed frequently enough that it might wear out that block of the
> > flash
> > drive any time soon? (I currently backup about 900 files one or two
> > times a day.) If so, is ext3 capable of using another block for the
> > inode, or will I lose the ability to read/write some files completely?
> >
> > Thanks,
> > Brian
>
>
> Here are some basic facts about USB flash drives:
>
>
> The underlying NAND flash RAM is guaranteed for 100K writes (older
> technology) or 1M (more recent). After that it's retention time
> starts to decrease, sometimes to as little as a few weeks, and with
> continued use eventually to complete failure. The retention time of
> a new, fully-functional NAND flash RAM is hundreds of years.
>
> USB flash drives have controller chips in the drive that, among other
> things, talk USB protocol to the host. One of the other things they
> do is try to distribute the writes over the available blocks,
> extending the lifetime of the device beyond the raw lifetime of the
> underlying NAND flash. But the firmware in the controller usually
> assumes you are using a FAT file-system. If, for example, you use
> EXT3 rather than FAT, you will probably confuse it. I don't know
> what the effect will be on device life, but I'm guessing it will
> probably be something like a return to the behavior of the raw NAND
> flash.
>
> Write blocks are large (4-8K or more). If you update one inode in a
> block, you have to read the entire block, update the inode and write
> the entire block back again. Each write to an inode causes an
> effective write to all the inodes in the same block. So there's
> roughly a 64x multiplier (8Kbytes/block divided by 128bytes/inode =
> 64 inodes/block) in the number of writes. Thus any given single
> inode is good for roughly 16K updates before it wears out.
>
> Let's suppose your rsync backup causes a small number of updates for
> each inode it touches. Call it 10, conservatively (it very well
> could be as low as 1 or 2 -- I don't know enough of the details of
> how rsync works at the filesystem level). That gives you about 1600
> rsync runs before you start to wear out the inodes. At twice a day,
> you can safely use your USB flash drive for 800 days, or between 2
> and 3 years.
>
> So here's what I'd do. Use your USB flash drive for a year, then
> retire it to the long-term archives and replace it with a new one.
> Let that be one of those things you do on or near your birthday, like
> schedule a checkup with your Doctor. Also: dismount the flash drive,
> force an fsck, and remount it, every time you do a backup. This will
> read and sanity-check every inode on the drive, without doing any
> writes. If the fsck shows random errors, replace the drive because
> it's probably starting to wear out.
>
> Sound reasonable?
>
>
> Rick
>
>

It does sound reasonable, with one sticking point: a lot of space in the
flash drive will never be used. I should have mentioned that; despite
having 900 files, the repository is currently only about 5 MB, and the
drive itself is 4 GB.

I'm looking at an alternative utility, svn-backup-dumps, in the same
package. I could do ~800 full backups, or many more incremental backups
with a full backup here and there, before running out of space. That
would only write most blocks once or twice, and once it filled up I
could erase old backups with only one write and reuse it, right? (If I
do this, should I reformat to FAT so the controllers won't be confused?)

Thanks,
Brian



--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-11-2008, 09:28 PM
Ted Hilts
 
Default rsync backup to ext3-formatted usb flash drive?

Brian Wells wrote:

On Mon, 2008-08-11 at 02:21 -0400, Rick Thomas wrote:


On Aug 10, 2008, at 2:12 AM, Brian Wells wrote:


Hi. I'm using svn-fast-backup (in the subversion-tools package) to
make
rsync backups of my subversion repository. The place I'm backing
up to
is on an ext3-formatted usb flash drive, and I'm wondering if this
is a

wise thing to do.

I know that flash drive blocks wear out after a certain number of
writes, and rsync uses hard links to avoid duplicate backup files. My
question is, does this mean that the inodes' reference counts will be
changed frequently enough that it might wear out that block of the
flash

drive any time soon? (I currently backup about 900 files one or two
times a day.) If so, is ext3 capable of using another block for the
inode, or will I lose the ability to read/write some files completely?

Thanks,
Brian


Here are some basic facts about USB flash drives:


The underlying NAND flash RAM is guaranteed for 100K writes (older
technology) or 1M (more recent). After that it's retention time
starts to decrease, sometimes to as little as a few weeks, and with
continued use eventually to complete failure. The retention time of
a new, fully-functional NAND flash RAM is hundreds of years.


USB flash drives have controller chips in the drive that, among other
things, talk USB protocol to the host. One of the other things they
do is try to distribute the writes over the available blocks,
extending the lifetime of the device beyond the raw lifetime of the
underlying NAND flash. But the firmware in the controller usually
assumes you are using a FAT file-system. If, for example, you use
EXT3 rather than FAT, you will probably confuse it. I don't know
what the effect will be on device life, but I'm guessing it will
probably be something like a return to the behavior of the raw NAND
flash.


Write blocks are large (4-8K or more). If you update one inode in a
block, you have to read the entire block, update the inode and write
the entire block back again. Each write to an inode causes an
effective write to all the inodes in the same block. So there's
roughly a 64x multiplier (8Kbytes/block divided by 128bytes/inode =
64 inodes/block) in the number of writes. Thus any given single
inode is good for roughly 16K updates before it wears out.


Let's suppose your rsync backup causes a small number of updates for
each inode it touches. Call it 10, conservatively (it very well
could be as low as 1 or 2 -- I don't know enough of the details of
how rsync works at the filesystem level). That gives you about 1600
rsync runs before you start to wear out the inodes. At twice a day,
you can safely use your USB flash drive for 800 days, or between 2
and 3 years.


So here's what I'd do. Use your USB flash drive for a year, then
retire it to the long-term archives and replace it with a new one.
Let that be one of those things you do on or near your birthday, like
schedule a checkup with your Doctor. Also: dismount the flash drive,
force an fsck, and remount it, every time you do a backup. This will
read and sanity-check every inode on the drive, without doing any
writes. If the fsck shows random errors, replace the drive because
it's probably starting to wear out.


Sound reasonable?


Rick





It does sound reasonable, with one sticking point: a lot of space in the
flash drive will never be used. I should have mentioned that; despite
having 900 files, the repository is currently only about 5 MB, and the
drive itself is 4 GB.

I'm looking at an alternative utility, svn-backup-dumps, in the same
package. I could do ~800 full backups, or many more incremental backups
with a full backup here and there, before running out of space. That
would only write most blocks once or twice, and once it filled up I
could erase old backups with only one write and reuse it, right? (If I
do this, should I reformat to FAT so the controllers won't be confused?)

Thanks,
Brian




It seems to me (given the above remarks) that you could define a block
or file that contains your backup with suitable space for the year (as
already suggested). I think using the FAT for filename suggestion, at
the end of each year make the block containing your backup a read only
block. FAT does not have the permissions as does EXT3 but can be set
for read only operation. You can easily try this out using a test
block. Each year (since you were concerned about unused space) you
would then define a new block with the beginning of the new block as the
contents of the previous block unless of course you don't want to keep
all the old backup info.


Hope you find this suggestion helpful.

Thanks, Ted Hilts


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-11-2008, 09:37 PM
Shachar Or
 
Default rsync backup to ext3-formatted usb flash drive?

On Monday 11 August 2008 23:25, Brian Wells wrote:

> I'm looking at an alternative utility, svn-backup-dumps, in the same
> package. I could do ~800 full backups, or many more incremental backups
> with a full backup here and there, before running out of space. That
> would only write most blocks once or twice, and once it filled up I
> could erase old backups with only one write and reuse it, right? (If I
> do this, should I reformat to FAT so the controllers won't be confused?)

If you're worried about not confusing the controllers, you could create a vfat
fs normally in the device and then make a file in it, like
dd if=/dev/zero of=/media/usbdisk/ext3.fs bs=1M count=4000
and then make an ext3 filesystem in that file, like
mkfs.ext3 -L my_backup_fs /media/usbdisk/ext3.fs
You will be able to mount this file with the -o loop option and make your
backups there

But anyways... I'm not sure that those controllers even deal with data in
fs-level.
>
> Thanks,
> Brian

--
Shachar Or | שחר אור
http://ox.freeallweb.org/


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-11-2008, 10:33 PM
Rick Thomas
 
Default rsync backup to ext3-formatted usb flash drive?

On Aug 11, 2008, at 4:25 PM, Brian Wells wrote:


I'm looking at an alternative utility, svn-backup-dumps, in the same
package. I could do ~800 full backups, or many more incremental
backups

with a full backup here and there, before running out of space. That
would only write most blocks once or twice, and once it filled up I
could erase old backups with only one write and reuse it, right?
(If I
do this, should I reformat to FAT so the controllers won't be
confused?)


The inodes are going to be your problem in an ext3 filesystem, not
the data blocks. Keep that in mind. So you should be trying to
control the number of writes per inode. One way to get a better
handle on that is by using the "noatime" mount option. Or switching
to FAT, if your backup system supports FAT as a destination filesystem.


USB flash drives are not expensive -- cdw.com advertises a variety of
4GB drives for US$15-$20, and they aren't the cheapest by a good
shot. No matter what you finally wind up doing, I'd still plan to
replace the flash drive at least once a year. That works out to
about 5 US cents/day.


Rick


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-12-2008, 01:39 AM
Brian Wells
 
Default rsync backup to ext3-formatted usb flash drive?

Thanks to Shachar Or and Ted Hilts for the suggestions, I might try them
sometime (particularly the nested filesystem idea, that's very clever

My main reply, below, is to Rick.

On Mon, 2008-08-11 at 18:33 -0400, Rick Thomas wrote:
> On Aug 11, 2008, at 4:25 PM, Brian Wells wrote:
>
> > I'm looking at an alternative utility, svn-backup-dumps, in the same
> > package. I could do ~800 full backups, or many more incremental
> > backups
> > with a full backup here and there, before running out of space. That
> > would only write most blocks once or twice, and once it filled up I
> > could erase old backups with only one write and reuse it, right?
> > (If I
> > do this, should I reformat to FAT so the controllers won't be
> > confused?)
>
> The inodes are going to be your problem in an ext3 filesystem, not
> the data blocks. Keep that in mind. So you should be trying to
> control the number of writes per inode. One way to get a better
> handle on that is by using the "noatime" mount option. Or switching
> to FAT, if your backup system supports FAT as a destination filesystem.
>
> USB flash drives are not expensive -- cdw.com advertises a variety of
> 4GB drives for US$15-$20, and they aren't the cheapest by a good
> shot. No matter what you finally wind up doing, I'd still plan to
> replace the flash drive at least once a year. That works out to
> about 5 US cents/day.
>
> Rick

Looks like you caught some wishful thinking on my part. I guessed that
the inode blocks would still take a lot of writes, hence "most blocks"
instead of "all blocks"; but I don't know enough details about whether
inodes in ext3 are kept close together (i.e. in the same block) or not.

If they are not, then adding new files should create inodes in another
block, which shouldn't cause excessive writes to any single block. (The
access time change might, though.) Again, I don't know enough to be
sure, but I was hoping it might be so.

Interestingly, while I was researching on the web I found another issue:
the journal that ext3 keeps. Quoting http://en.wikipedia.org/wiki/Ext2

---Begin Quote---
ext2 is still recommended over journaling file systems on bootable USB
flash drives and other solid-state drives. ext2 performs fewer writes
than ext3 since it does not need to write to the journal. As the major
aging factor of a flash chip is the number of erase cycles, and as those
happen frequently on writes, this increases the life span of the
solid-state device. Another good practice for filesystems on flash
devices is the use of the noatime mount option, for the same reason.
---End Quote---

So it looks like there's another factor that might run the drive into
the ground. Wonderful, I think I see why most people keep FAT on them.

On the noatime issue, GNOME in etch mounts the drive automatically for
me when I plug it in; I'm not sure how to configure GNOME to use custom
mount options, short of unmounting and remounting manually every time I
stick the drive in. I think I'm also seeing one kind of reason that
many people hate automagical GUIs. Can someone point me to the docs?

Thanks,
Brian



--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 

Thread Tools




All times are GMT. The time now is 09:55 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org