FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo User

 
 
LinkBack Thread Tools
 
Old 05-07-2010, 06:23 PM
Alex Schuster
 
Default snackup (was: backup to a cold-swap drive)

Iain Buchanan writes:

> On Thu, 2010-04-29 at 16:44 +0200, Alex Schuster wrote:

> > using this script, adapted to their needs, I started to rewrite it in
> > a way that it reads a config file, and no modification of the script
> > itself is necessary. If anyone is interested, send me an email.
>
> interested! So is it on sourceforge yet

Sorry, it took a little longer. I was ill, and then these scripts tend to
grow and grow until I am satisfied with them. It's still not perfect, but
I think at the moment it does what it should do.

Still not on sourceforge, but here: http://www.wonkology.org/utils/snackup
The name is silly, but I kinda like it now. A backup utilizing LVM
snapshots... snapshot backup... snackup!

It needs a config file that is looked up in some default locations, or can
be specified with the -c option. Use option -T to generate a template that
has information on the syntax and some examples. In short:

The config file defines targets you give as arguments to snackup. So
'snackup home' backs up your /home partition, 'snackup etc' creates a .tgz
file of your /etc. The config file is sourced, so you can put any bash
stuff you want into there, like some shell functions you want to have
executed. For example, I want my /var/log/portage/*.log files to be
compressed before I backup my /var partition.

A target is started by a colon, followed by name, type, source and
destination (and optionally more), separated by one or more tab
characters. Type can be 'cp', 'dd', 'tar', 'rsync' or 'rdiff'. I only
tested dd, tar and rdiff so far, though. The source directory may be
prefixed by the volume group and LVM of that partition in order to create
an lvm snapshot first. You may add a LUKS key if the partition is
encrypted. The source may contain wildcards. In that case, all matching
files are backed up to the destination. If the type has a * appended, all
matching files are backed up individually. Some examples:

: tar kde .kde* /backup/kde.%s

Target 'kde' will backup all your .kde* directories to /backup/kde.tar
(the %s is replaced by a suffix, usually tar). You need to be in your home
directory, otherwise the path will not be found. This is not necessary if
you use ~//.kde* instead.

: kernels tar* /usr/src//linux-* /backup/src/%f.%s -j - lsf

Target 'kernels' will backup your /usr/src/linux-* directories
individually - one tar file for each (note the * appendended to the tar
target type). The -z option will be passed to the tar command, and the
suffix is changed to tgz accordingly.
The // denotes that the backup should be done locally from the directory
left to the //. Otherwise, the full path would be included in the tar
file.

: tar etc /etc /backup/etc_%d.%s -j

Target 'etc' will tar your /etc directory, compressed with bzip2. The %d
will be replaced by the current date. You can change the date format by
re-defining a date() shell function.
You will get a message about tar removing the leading /, you can avoid it
by replacing /etc by ///etc, meaning that your local directroy will be /.

: boot dd /dev/sda5 /dev/sdb5 bs=32M

Target 'boot' will backup a boot partition with dd. Options like the
bs=32M are added directly to the dd command.

: home rsync system/home::. /backup/home/

Target 'home' will create a LVM snapshot of the /dev/system/home logical
volume, mount it and back it up with rdiff-backup. The '.' could also be a
/ or left off.

: var rsync system/var::/ /backup/var/ - ziplog

Similar, but the command ziplog will be executed before the backup,
compressing some log files. It must be defined in the config file.

The script has some options:

-c file location of config file
-C clear destination first
-d dummy mode, just show what would be done
-f force backup (the initial rdiff-backup may need this)
-h show this help
-l output to log file, too
-L log file (default:/home/wonko/log/snackup.log); may be a
directory (add a trailing slash) to create target-specific logs
-n l use nice level l (default:10, 0 to turn off)
-N l use ionice level l (default:3, 0 to turn off)
-o extra options that will added to the actual backup command
-s size size of LVM snapshot (default:2G)
-S char replace tab as delimiter for targets in config file
-T output config file template
-v verbose output; may be given multiple times
1: some extra output; 2: add -v to commands; 3: set -xv
-V output version information and exit

Most important are:
-c to specify the location of the config file
-d to output what would be done, you will see the final tar / rdiff-
backup / whatever command with all its options.
-T to see a template. Save it with snackup -T > ~/.snackup and then
modify your ~/.snackup file.
-o to add whichever options you like to the final command. These (and -j
and -z) override options already defined in the config file.

Too complicated? Overkill? Maybe. But I like it, and now I do not have to
write another script for every system I want to backup. And I like the LVm
snapshotting to be done automatically. It's not that hard, but it's
annoying to do it by hand, even more as all my volumes are encrypted.

These are my own targets:

: boot dd /dev/boot /dev/sdb5 - - -
: backup rdiff weird/backup::/etc/key::. /backup/weird/backup - - -
: local rdiff weird/local::/etc/key::. /backup/weird/local - - -
: home rdiff weird/home::/etc/key::. /backup/weird/home - - -
: med rdiff weird/med::/etc/key::. /backup/weird/med - - -
: mp3 rdiff weird/mp3::/etc/key::. /backup/weird/mp3 - - -
: mpeg rdiff weird/mpeg::/etc/key::. /backup/weird/mpeg - - -
: opt rdiff weird/opt::/etc/key::. /backup/weird/opt - - -
: root rdiff weird/root::/etc/key::. /backup/weird/root - - -
: usr rdiff weird/usr::/etc/key::. /backup/weird/usr - - -
: var rdiff weird/var::/etc/key::. /backup/weird/var - zipLog -
: etc tar ///etc /backup/conf/etc/etc_%d.%s -z - lsf
: kernels tar* /usr/src//linux-* /backup/weird/src/%f.%s -j - lsf
: portage rsync /var/portage/packages /backup/weird/portage - - -
: kde tar ~//.kde* /backup/wonko/home/kde-%d.%s -z - lsf
: dot tar ~//.??* /backup/extern/dot-%d.%s -z - lsf


At last, the disclaimer: Use it at your own risk! This is a script of some
800 lines of bash code some guy wrote, it messes around with LVM stuff,
creates and removes temporary directories and even has an option to wipe
the destination completely. I'm quite confident that no harm will be done,
and I do not suggest you to take a backup before daring to try this script,
but anyway.

Use the -d option first to test what actually would be going on, before you
do a real backup!

Hope this is of good use to someone. For me it is, I will be using it on
several systems, and now I only have to make little changes in the config
file for each system, but not at the script.

Suggestions and bug reports are welcome of course.

Wonko
 
Old 05-07-2010, 07:08 PM
 
Default snackup (was: backup to a cold-swap drive)

Alex Schuster <wonko@wonkology.org> wrote:

> Iain Buchanan writes:
>
> > On Thu, 2010-04-29 at 16:44 +0200, Alex Schuster wrote:
>
> > > using this script, adapted to their needs, I started to rewrite it in
> > > a way that it reads a config file, and no modification of the script
> > > itself is necessary. If anyone is interested, send me an email.
> >
> > interested! So is it on sourceforge yet
>
> Sorry, it took a little longer. I was ill, and then these scripts tend to
> grow and grow until I am satisfied with them. It's still not perfect, but
> I think at the moment it does what it should do.
>
> Still not on sourceforge, but here: http://www.wonkology.org/utils/snackup
> The name is silly, but I kinda like it now. A backup utilizing LVM
> snapshots... snapshot backup... snackup!
>
> It needs a config file that is looked up in some default locations, or can
> be specified with the -c option. Use option -T to generate a template that
> has information on the syntax and some examples. In short:
>
> The config file defines targets you give as arguments to snackup. So
> 'snackup home' backs up your /home partition, 'snackup etc' creates a .tgz
> file of your /etc. The config file is sourced, so you can put any bash
> stuff you want into there, like some shell functions you want to have
> executed. For example, I want my /var/log/portage/*.log files to be
> compressed before I backup my /var partition.
>
> A target is started by a colon, followed by name, type, source and
> destination (and optionally more), separated by one or more tab
> characters. Type can be 'cp', 'dd', 'tar', 'rsync' or 'rdiff'. I only
> tested dd, tar and rdiff so far, though. The source directory may be
> prefixed by the volume group and LVM of that partition in order to create
> an lvm snapshot first. You may add a LUKS key if the partition is
> encrypted. The source may contain wildcards. In that case, all matching
> files are backed up to the destination. If the type has a * appended, all
> matching files are backed up individually. Some examples:
>
> : tar kde .kde* /backup/kde.%s
>
> Target 'kde' will backup all your .kde* directories to /backup/kde.tar
> (the %s is replaced by a suffix, usually tar). You need to be in your home
> directory, otherwise the path will not be found. This is not necessary if
> you use ~//.kde* instead.
>
> : kernels tar* /usr/src//linux-* /backup/src/%f.%s -j - lsf
>
> Target 'kernels' will backup your /usr/src/linux-* directories
> individually - one tar file for each (note the * appendended to the tar
> target type). The -z option will be passed to the tar command, and the
> suffix is changed to tgz accordingly.
> The // denotes that the backup should be done locally from the directory
> left to the //. Otherwise, the full path would be included in the tar
> file.
>
> : tar etc /etc /backup/etc_%d.%s -j
>
> Target 'etc' will tar your /etc directory, compressed with bzip2. The %d
> will be replaced by the current date. You can change the date format by
> re-defining a date() shell function.
> You will get a message about tar removing the leading /, you can avoid it
> by replacing /etc by ///etc, meaning that your local directroy will be /.
>
> : boot dd /dev/sda5 /dev/sdb5 bs=32M
>
> Target 'boot' will backup a boot partition with dd. Options like the
> bs=32M are added directly to the dd command.
>
> : home rsync system/home::. /backup/home/
>
> Target 'home' will create a LVM snapshot of the /dev/system/home logical
> volume, mount it and back it up with rdiff-backup. The '.' could also be a
> / or left off.
>
> : var rsync system/var::/ /backup/var/ - ziplog
>
> Similar, but the command ziplog will be executed before the backup,
> compressing some log files. It must be defined in the config file.
>
> The script has some options:
>
> -c file location of config file
> -C clear destination first
> -d dummy mode, just show what would be done
> -f force backup (the initial rdiff-backup may need this)
> -h show this help
> -l output to log file, too
> -L log file (default:/home/wonko/log/snackup.log); may be a
> directory (add a trailing slash) to create target-specific logs
> -n l use nice level l (default:10, 0 to turn off)
> -N l use ionice level l (default:3, 0 to turn off)
> -o extra options that will added to the actual backup command
> -s size size of LVM snapshot (default:2G)
> -S char replace tab as delimiter for targets in config file
> -T output config file template
> -v verbose output; may be given multiple times
> 1: some extra output; 2: add -v to commands; 3: set -xv
> -V output version information and exit
>
> Most important are:
> -c to specify the location of the config file
> -d to output what would be done, you will see the final tar / rdiff-
> backup / whatever command with all its options.
> -T to see a template. Save it with snackup -T > ~/.snackup and then
> modify your ~/.snackup file.
> -o to add whichever options you like to the final command. These (and -j
> and -z) override options already defined in the config file.
>
> Too complicated? Overkill? Maybe. But I like it, and now I do not have to
> write another script for every system I want to backup. And I like the LVm
> snapshotting to be done automatically. It's not that hard, but it's
> annoying to do it by hand, even more as all my volumes are encrypted.
>
> These are my own targets:
>
> : boot dd /dev/boot /dev/sdb5 - - -
> : backup rdiff weird/backup::/etc/key::. /backup/weird/backup - - -
> : local rdiff weird/local::/etc/key::. /backup/weird/local - - -
> : home rdiff weird/home::/etc/key::. /backup/weird/home - - -
> : med rdiff weird/med::/etc/key::. /backup/weird/med - - -
> : mp3 rdiff weird/mp3::/etc/key::. /backup/weird/mp3 - - -
> : mpeg rdiff weird/mpeg::/etc/key::. /backup/weird/mpeg - - -
> : opt rdiff weird/opt::/etc/key::. /backup/weird/opt - - -
> : root rdiff weird/root::/etc/key::. /backup/weird/root - - -
> : usr rdiff weird/usr::/etc/key::. /backup/weird/usr - - -
> : var rdiff weird/var::/etc/key::. /backup/weird/var - zipLog -
> : etc tar ///etc /backup/conf/etc/etc_%d.%s -z - lsf
> : kernels tar* /usr/src//linux-* /backup/weird/src/%f.%s -j - lsf
> : portage rsync /var/portage/packages /backup/weird/portage - - -
> : kde tar ~//.kde* /backup/wonko/home/kde-%d.%s -z - lsf
> : dot tar ~//.??* /backup/extern/dot-%d.%s -z - lsf
>
>
> At last, the disclaimer: Use it at your own risk! This is a script of some
> 800 lines of bash code some guy wrote, it messes around with LVM stuff,
> creates and removes temporary directories and even has an option to wipe
> the destination completely. I'm quite confident that no harm will be done,
> and I do not suggest you to take a backup before daring to try this script,
> but anyway.
>
> Use the -d option first to test what actually would be going on, before you
> do a real backup!
>
> Hope this is of good use to someone. For me it is, I will be using it on
> several systems, and now I only have to make little changes in the config
> file for each system, but not at the script.
>
> Suggestions and bug reports are welcome of course.

I have a question -- where would lvm put a snapshot and how could I pass
some list of excludes to rdiff-backup. I have an lvm which is taking
all the PEs and a snapshot would take up lots of disk space -- or would
it. Would I need some free pes to put the snapshot?

Thanks.

--
Your life is like a penny. You're going to lose it. The question is:
How do
you spend it?

John Covici
covici@ccs.covici.com
 

Thread Tools




All times are GMT. The time now is 07:52 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org