Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Gentoo User (http://www.linux-archive.org/gentoo-user/)
-   -   backup to a cold-swap drive (http://www.linux-archive.org/gentoo-user/363012-backup-cold-swap-drive.html)

Iain Buchanan 04-28-2010 01:41 AM

backup to a cold-swap drive
 
Hi,

A winblows colleague said he uses a utility to backup his internal hard
drive to an external disk, such that if his internal disk fails he can
replace it with the external disk and continue straight away.

Since I go to weird locations with unreliable power and sometimes drop
my laptop I thought it should be simple to do the same in Linux. I have
an external disk the same size, but now what?

* I want to copy changes intelligently (ie. no dd, gparted, or
Ghost4Linux).
* I want to copy a specific device only (no usb keys, etc) to a
specific external device.
* Windows partitions can be ignored.
* It doesn't matter if the copy is not unmounted properly, eg. if
power is shut of without shutting down.
* The external disk must be able to be absent

Can md use one internal and one external disk in a RAID 1 setup, with
the external disk not always there? Any other suggestions?

thanks :)
--
Iain Buchanan <iaindb at netspace dot net dot au>

Better tried by twelve than carried by six.
-- Jeff Cooper

Florian Philipp 04-28-2010 03:31 PM

backup to a cold-swap drive
 
Am 28.04.2010 03:41, schrieb Iain Buchanan:
> Hi,
>
> A winblows colleague said he uses a utility to backup his internal hard
> drive to an external disk, such that if his internal disk fails he can
> replace it with the external disk and continue straight away.
>
> Since I go to weird locations with unreliable power and sometimes drop
> my laptop I thought it should be simple to do the same in Linux. I have
> an external disk the same size, but now what?
>
> * I want to copy changes intelligently (ie. no dd, gparted, or
> Ghost4Linux).
> * I want to copy a specific device only (no usb keys, etc) to a
> specific external device.
> * Windows partitions can be ignored.
> * It doesn't matter if the copy is not unmounted properly, eg. if
> power is shut of without shutting down.
> * The external disk must be able to be absent
>
> Can md use one internal and one external disk in a RAID 1 setup, with
> the external disk not always there? Any other suggestions?
>
> thanks :)

md would be extremely slow because it has to rebuild/resync the complete
array.

I suggest you manually recreate the partitioning scheme, install grub,
then mount it with some little script and call some backup tool to do
the actual copying.

If you can live with just one big partition as a backup (probably with
separate /boot), you should replace fstab and grub.conf on the backup
medium and blacklist them from the files which you want to back up.

Concerning the backup tool, I would use `rsync --delete` plus all
relevant switches for permissions, times, acls, etc. If you use another
tool, just make sure it doesn't put some metadata onto the backup medium
and that it can delete files which no longer exist on the original medium.

With regard to your requirement to just 'pull the cord' without
umounting it: Better mount it with '-o sync' to increase your chance
that everything works fine afterwards. But in reality, nothing can
really protect you from filesystem corruption in this situation. If you
can afford, better keep two backup media which you round-robin.

Hope this helps,
Florian Philipp

Frank Steinmetzger 04-28-2010 09:16 PM

backup to a cold-swap drive
 
Am Mittwoch, 28. April 2010 schrieb Iain Buchanan:
> Hi,
>
> A winblows colleague said he uses a utility to backup his internal hard
> drive to an external disk, such that if his internal disk fails he can
> replace it with the external disk and continue straight away.
>
> Since I go to weird locations with unreliable power and sometimes drop
> my laptop I thought it should be simple to do the same in Linux. I have
> an external disk the same size, but now what?
>
> * I want to copy changes intelligently (ie. no dd, gparted, or
> Ghost4Linux).
> * I want to copy a specific device only (no usb keys, etc) to a
> specific external device.
> * Windows partitions can be ignored.
> * It doesn't matter if the copy is not unmounted properly, eg. if
> power is shut of without shutting down.
> * The external disk must be able to be absent
>
> Can md use one internal and one external disk in a RAID 1 setup, with
> the external disk not always there? Any other suggestions?

After I upgraded my laptop with an internal HDD of 500 GB, I started using my
old external 500 GB drive as backup. Though of different dimensions and
makers, they both have the same number of sectors. So I dd'ed the entire disk
first, which gave me an exact mirror of the internal disk. I though this would
be faster, because I have lots of small files in some places.

But now I can update the backup by a simple call to rsync:

rsync -aX --delete / /dev/<backup root partition>/

-a (archive) copies permissions, ownerships and the likes
-X stops at file system boundaries, i.e. it will only backup the actual root
partition, without other mounted file systems such as /proc, /dev and /home.
--
Gruß | Greetings | Qapla'
Why did the tachyon cross the road?
Because it was on the other side.

Iain Buchanan 04-29-2010 12:38 AM

backup to a cold-swap drive
 
Hi & thanks,

On Wed, 2010-04-28 at 17:31 +0200, Florian Philipp wrote:

> > Can md use one internal and one external disk in a RAID 1 setup, with
> > the external disk not always there? Any other suggestions?
> >
> > thanks :)
>
> md would be extremely slow because it has to rebuild/resync the complete
> array.

so every time you unplug and re-plug the external disk, it will
essentially re-copy everything? Damn, there goes that fine idea!

> If you can live with just one big partition as a backup (probably with
> separate /boot), you should replace fstab and grub.conf on the backup
> medium and blacklist them from the files which you want to back up.

why wouldn't I backup fstab and grub.conf as well? If my internal disk
dies, I assume I'll swap them over, meaning grub and fstab will have to
be the same.

> Concerning the backup tool, I would use `rsync --delete` plus all
> relevant switches for permissions, times, acls, etc. If you use another
> tool, just make sure it doesn't put some metadata onto the backup medium
> and that it can delete files which no longer exist on the original medium.

I was thinking of rsync, but I didn't want to do it in an hourly cron
fashion, I was hoping for some "gamin" alteration-triggered idea.

> With regard to your requirement to just 'pull the cord' without
> umounting it:

I wasn't thinking of pulling the chord without unmounting, I was
thinking of the machine dying, hence leaving the disk in a non-shutdown
state.

thanks for the tips :) rsync will at least get me going quickly.
Yesterday I tried iotop to with dd - some slowness but otherwise quite
nice.

--
Iain Buchanan <iaindb at netspace dot net dot au>

Real computer scientists don't comment their code. The identifiers are
so long they can't afford the disk space.

Iain Buchanan 04-29-2010 12:41 AM

backup to a cold-swap drive
 
Hi,

On Wed, 2010-04-28 at 23:16 +0200, Frank Steinmetzger wrote:

> After I upgraded my laptop with an internal HDD of 500 GB, I started using my
> old external 500 GB drive as backup. Though of different dimensions and
> makers, they both have the same number of sectors. So I dd'ed the entire disk
> first, which gave me an exact mirror of the internal disk. I though this would
> be faster, because I have lots of small files in some places.

I tried that, but after dd finished, I was left with strange partitions
and id's that I couldn't mount. The two disks are both 160Gb with the
same sector size...

It might be easier to do the fdisk-ing by hand.

thanks,
--
Iain Buchanan <iaindb at netspace dot net dot au>

Necessity is the plea for every infringement of human freedom.
It is the argument of tyrants; it is the creed of slaves.
-- William Pitt, 1783

Iain Buchanan 04-29-2010 03:38 AM

backup to a cold-swap drive
 
On Wed, 2010-04-28 at 23:16 +0200, Frank Steinmetzger wrote:

> rsync -aX --delete / /dev/<backup root partition>/
>
> -a (archive) copies permissions, ownerships and the likes
> -X stops at file system boundaries, i.e. it will only backup the actual root
> partition, without other mounted file systems such as /proc, /dev and /home.

actually, lower case x is --one-file-system or "don't cross filesystem
boundaries". Upper case X is --xattrs or "preserve extended attributes"

:)
--
Iain Buchanan <iaindb at netspace dot net dot au>

When you don't know what to do, walk fast and look worried.

Alex Schuster 04-29-2010 02:44 PM

backup to a cold-swap drive
 
Iain Buchanan writes:

> A winblows colleague said he uses a utility to backup his internal hard
> drive to an external disk, such that if his internal disk fails he can
> replace it with the external disk and continue straight away.

I do the same, but with a 2nd internal drive. The drive is partitioned
similar, with some partitions being a bit larger so I can do incremental
backups, too.

I am using rdiff-backup, which makes use of rsync. The backup partition
has exactly the same contents as the source partition, except for an
additional 'rdiff-backup' directory that contains incremental backups of
files that were modified from backup to backup, gzipped.

Some other partitions are handled differently: /boot is just being dd'ed,
contents of /usr/src are tarred each, and /var/portage/packages/ is just
plain rsynced. Some unnecessary stuff like .ccache and /var/tmp/portage is
excluded.

All my partitions are LVM volumes, so before the backup starts, I make a
LVM snapshot of the partition. This way I can modify it while the backup
is still in progress.

I wrote a shell script to do this, so I do not have to issue a lot of
commands every time I want to do the backup. As there are now some others
using this script, adapted to their needs, I started to rewrite it in a
way that it reads a config file, and no modification of the script itself
is necessary. If anyone is interested, send me an email.


Some time ago my first drive started having bad blocks. Without LVM, I
could just have swapped the disks, but so I had to rename the backup
volume group to the original volume group from a live cd. And the system
was running from the new drive as it was before - only that I no longer
had a backup until the new drive arrived. This makes an uneasy feeling
with these 1.5 TB drives.

Wonko

Florian Philipp 04-30-2010 02:24 PM

backup to a cold-swap drive
 
Am 29.04.2010 02:38, schrieb Iain Buchanan:
> Hi & thanks,
>
> On Wed, 2010-04-28 at 17:31 +0200, Florian Philipp wrote:
[...]
>
>> If you can live with just one big partition as a backup (probably with
>> separate /boot), you should replace fstab and grub.conf on the backup
>> medium and blacklist them from the files which you want to back up.
>
> why wouldn't I backup fstab and grub.conf as well? If my internal disk
> dies, I assume I'll swap them over, meaning grub and fstab will have to
> be the same.

I think you misunderstood me or I didn't explain it correctly. I try it
again:

You probably have a lot of partitions on your internal disk; one for /,
/usr and /home, for example. My reasoning was that it is probably easier
to have just one big partition on your backup medium instead of many
small ones. If you would then swap your backup medium in, with exactly
the same fstab as in your original installation, your system would try
to mount partitions which are simply not there. Therefore you need a
different fstab and most likely also another grub.conf.

>
>> Concerning the backup tool, I would use `rsync --delete` plus all
>> relevant switches for permissions, times, acls, etc. If you use another
>> tool, just make sure it doesn't put some metadata onto the backup medium
>> and that it can delete files which no longer exist on the original medium.
>
> I was thinking of rsync, but I didn't want to do it in an hourly cron
> fashion, I was hoping for some "gamin" alteration-triggered idea.
>

Ah, I see what you mean. I've never worked with the file alteration
monitor (FAM) but once evaluated inotify for some administrative
purposes. AFAIK they are not scalable good enough to work on a system
wide basis. For example, I think the default limit of observable files
with inotify is 8192.

>> With regard to your requirement to just 'pull the cord' without
>> umounting it:
>
> I wasn't thinking of pulling the chord without unmounting, I was
> thinking of the machine dying, hence leaving the disk in a non-shutdown
> state.
>

Okay, I thought you meant the unreliable power at those "weird
locations" you were talking about. Such a black-out or brown-out is
basically the same as pulling the cord.

> thanks for the tips :) rsync will at least get me going quickly.
> Yesterday I tried iotop to with dd - some slowness but otherwise quite
> nice.
>

To reduce the performance impact, you can also use the ionice command.

Hope this helps,
Florian Philipp

Iain Buchanan 05-03-2010 12:13 PM

backup to a cold-swap drive
 
On Thu, 2010-04-29 at 16:44 +0200, Alex Schuster wrote:

[snip]
> All my partitions are LVM volumes, so before the backup starts, I make a
> LVM snapshot of the partition. This way I can modify it while the backup
> is still in progress.

hmm, never got into LVM. Sounds interesting though...

[snip]
> I wrote a shell script to do this, so I do not have to issue a lot of
> commands every time I want to do the backup.

I don't use too many commands, something like this
in /etc/cron.daily/custom-backup:

sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAx --exclude suspend_file
--delete-delay --delete-excluded --partial
--human-readable / /<unique-mount-of-external-drive> || echo external
backup failed!

> As there are now some others
> using this script, adapted to their needs, I started to rewrite it in a
> way that it reads a config file, and no modification of the script itself
> is necessary. If anyone is interested, send me an email.

interested! So is it on sourceforge yet ;)

thanks,
--
Iain Buchanan <iaindb at netspace dot net dot au>

BOFH Excuse #418:

Sysadmins busy fighting SPAM.

Iain Buchanan 05-03-2010 12:44 PM

backup to a cold-swap drive
 
On Fri, 2010-04-30 at 16:24 +0200, Florian Philipp wrote:
> Am 29.04.2010 02:38, schrieb Iain Buchanan:
> > Hi & thanks,
> >
> > On Wed, 2010-04-28 at 17:31 +0200, Florian Philipp wrote:
> [...]
> >
> >> If you can live with just one big partition as a backup (probably with
> >> separate /boot), you should replace fstab and grub.conf on the backup
> >> medium and blacklist them from the files which you want to back up.
> >
> > why wouldn't I backup fstab and grub.conf as well? If my internal disk
> > dies, I assume I'll swap them over, meaning grub and fstab will have to
> > be the same.
>
> I think you misunderstood me or I didn't explain it correctly. I try it
> again:

[snip]

ah, NOW I get it :)

[snip]

> Ah, I see what you mean. I've never worked with the file alteration
> monitor (FAM) but once evaluated inotify for some administrative
> purposes. AFAIK they are not scalable good enough to work on a system
> wide basis. For example, I think the default limit of observable files
> with inotify is 8192.

hm, there goes that idea! Is there any kernel based "watch" on all file
based I/O that I could queue up somehow? Just thinking aloud here. I
know "everything is a file", but no doubt I could watch all write
operations; filter out /dev and put the rest into a file; and then use
it like an rsync file-list...

> > thanks for the tips :) rsync will at least get me going quickly.
> > Yesterday I tried iotop to with dd - some slowness but otherwise quite
> > nice.
> >
>
> To reduce the performance impact, you can also use the ionice command.

whoops, that's what I meant. Even with ionice, there was some
noticeable delay when switching screens, opening programs, etc. More so
when my RAM had been swapped from the large amount of I/O (I assume). I
didn't try ionice with nice.

thanks,
--
Iain Buchanan <iaindb at netspace dot net dot au>

One family builds a wall, two families enjoy it.


All times are GMT. The time now is 02:06 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.