Maximilian Bräutigam writes:

> You should backup all in / except
> /tmp/*
> /sys/*
> /proc/*
> /lost+found/*
> /dev/*

Attention here, you need at least the null and console entries in /dev, or
the system will not come up. I also have tty and tty1 in there, I think
those were neede for tuxonice.

Wonko

* Nganon <nganon+gentoo@gmail.com> wrote:

Hi,

> My first post on the list. I thought I would start with something that I
> started
> to think of as 'essential' after losing 90GB of data. Now I have two main
> questions in mind: what to and how to back up on gentoo most efficiently.

I'm using a little script like that:

#!/bin/bash
cd /var/backup/blackwidow || exit 1

DATE=`date +%F-%H-%M-%S`

rsync --exclude-from /scr/etc/backup-blackwidow.exclude
-avz blackwidow:/ /var/backup/blackwidow/ROOT
--backup
"--backup-dir=/var/backup/blackwidow/BACKUP-$DATE"
--delete
--delete-excluded
--delay-updates
--progress


(of course with a carefully maintained exclude file ;-p)
This doesnt make a real rotating backup, but stores all files that
get overwritten into their own directory (named by the current date).
And from time to time, I'm cleaning up the backup volume and look
for things that I still might need.

For things I'd like to keep an history (eg. /etc) I'm using git, and
pushing the repo to a remote server (denying non-fastfoward updates
there, so an theorectical highjacker cannot destroy my history)

For really long-term backups with bigger content (that might be
too big for git), you could also give venti+vac (from plan9port)
a try. I'm using it for storing larger media files (videos, etc)
in my MediaCloud platform.

> 1. Apart from users' home directories and the followings, what should be
> backed
> up on a gentoo machine?
> /etc/portage/
> /root
> /var/lib/portage
> ...?

Depends on how long the recovery may take. For example, if you can
afford recompiling world (or have another compatible image somewhere
else), you can exclude everything where packages are installed
(bindirs, libdirs, /usr/share, etc, etc) - assuming everything's
installed by portage.

> Though I can find enough space on the external drives, I don't trust them
> any more. See above..sigh..(No I recovered about one third of it with
> testdisk/photorec
> which names them as file000001 file00002.. and half them are zero sized..
> which
> quite justifies my agony)

Make multiple copies on different media (eg. different servers).


cu
--
----------------------------------------------------------------------
Enrico Weigelt, metux IT service -- http://www.metux.de/

phone: +49 36207 519931 email: weigelt@metux.de
mobile: +49 151 27565287 icq: 210169427 skype: nekrad666
----------------------------------------------------------------------
Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

2010/8/17 Maximilian Bräutigam <max-braeu@gmx.de>


You should backup all in / except

/tmp/*

/sys/*

/proc/*

/lost+found/*

/dev/*


Distfiles are saved outside the root *and*I can afford to rebuild world. My*main concern was losing (gentoo) config files, speaking of which, I*remembered to*back up /usr/src/linux/.config, and user folders.*



I have no solution how to bzip or gzip your backups or how to make a dvd

backup, but I use "app-backup/rsnapshot" which uses rsync but implements

an intelligent rotating system that is done daily, weekly, monthly,

yearly according to your config. Of course you should store the backup

on another physical hdd.


By the way, since a new hdd of one TB is pretty cheap, think about

running your gentoo in a software RAID. Guides:



http://www.gentoo.org/doc/en/gentoo-x86+raid+lvm2-quickinstall.xml

http://en.gentoo-wiki.com/wiki/Software_RAID_Install
Thanks for the advices. I am adding rsnapshot to my list.*

On 17 August 2010 22:34, Enrico Weigelt <weigelt@metux.de> wrote:


For things I'd like to keep an history (eg. /etc) I'm using git, and

pushing the repo to a remote server (denying non-fastfoward updates

there, so an theorectical highjacker cannot destroy my history)

Using git for /etc is a great idea.*Thanks.*

On Tue, 17 Aug 2010 14:14:27 +0200, Maximilian Bräutigam wrote:

> You should backup all in / except
> /tmp/*
> /sys/*
> /proc/*
> /lost+found/*
> /dev/*

That backs up a lot of stuff that isn't needed. As long as you have /etc
and /var/lib you can recreate the system. Depending on space vs. time,
you may prefer not to backup the gigabytes in /usr that can be recreated
by portage (although saving /usr/local is a good idea).

> By the way, since a new hdd of one TB is pretty cheap, think about
> running your gentoo in a software RAID. Guides:

RAID is not an alternative to backups, a corrupted filesystem on a RAID is
just as corrupted as if it were on a single disk, you just get extra
copies of the corruption.


--
Neil Bothwick

Despite the cost of living, have you noticed how it remains so popular?

On 18 August 2010 14:34, Neil Bothwick <neil@digimed.co.uk> wrote:


On Tue, 17 Aug 2010 14:14:27 +0200, Maximilian Bräutigam wrote:



> You should backup all in / except

> /tmp/*

> /sys/*

> /proc/*

> /lost+found/*

> /dev/*



That backs up a lot of stuff that isn't needed. As long as you have /etc

and /var/lib you can recreate the system. Depending on space vs. time,

you may prefer not to backup the gigabytes in /usr that can be recreated

by portage (although saving /usr/local is a good idea).



Thanks a lot for the valuable advice. I have a dozen of scripts in /usr/local/bin*that I forgot about.**


> By the way, since a new hdd of one TB is pretty cheap, think about

> running your gentoo in a software RAID. Guides:



RAID is not an alternative to backups, a corrupted filesystem on a RAID is

just as corrupted as if it were on a single disk, you just get extra

copies of the corruption.


*I did not know that. I was thinking of, in couple of months, buying a notebook*with two HDDs with RAID1 installed and using the usb drive as a backup*

destination. So if RAID got corruped, the backups, made since then, would be*useless? How would you resolve it?*



--

Neil Bothwick



Despite the cost of living, have you noticed how it remains so popular?

On Wed, 2010-08-18 at 14:09 +0300, Nganon wrote:
>
>
> On 17 August 2010 22:34, Enrico Weigelt <weigelt@metux.de> wrote:
> For things I'd like to keep an history (eg. /etc) I'm using
> git, and
> pushing the repo to a remote server (denying non-fastfoward
> updates
> there, so an theorectical highjacker cannot destroy my
> history)
>
>
> Using git for /etc is a great idea.
> Thanks.
>
Another option is:
* app-backup/dirvish
Latest version available: 1.2.1
Latest version installed: 1.2.1
Size of downloaded files: 47 kB
Homepage: http://www.dirvish.org/
Description: Dirvish is a fast, disk based, rotating network
backup system.
License: OSL-2.0


Works by first creating a copy (--init) and then hard-linking subsequent
versions of files/directories back to the original original if its
identical. If a file is changed/new, it is copied instead of linked so
actual space usage quickly stabilises even with a varying number of
versions. Backup over the network (this is how I have configured mine)
uses rsync over ssh with keys and is "pull" from a cron job on the
backup server or manual on demand (i.e., server initiated).

Version management is by a reasonably sophisticated date of version
scheme where by running "dirvish-expire" deletes out of date versions
(runs in a cron job). The smart part is that once the last hard link to
file is deleted, its gone, otherwise its kept in the remaining
versions

Restore is a simple matter of identifying the version you want and
copying it back - Ive restored individual files through to complete
systems after total disk failure.

Can do includes/excludes, whole systems or just directories such as /etc
and can be easily automated.

Doesnt use compression, but most backup regimes (every day for a weekly
rota + a Sunday kept for 6 months) stabilise at about 2x the original
(gross) copy size, no matter how many copies with average changes
between versions. Though large scale changes such as an "emerge -e
world" will take more as it will generate new copies of most files.

Downside is it will hammer the destination file system - reiserfs3 works
well, ext2/ext3 have been hopeless everytime I've tried - mass
corruption. The file system will need a large number of inodes (for
links) if there are an excessive number of files x versions - again
reiserfs3 scores well here.

Highly recommended!

BillK

On 08/18/2010 04:53 AM, Nganon wrote:
> I did not know that. I was thinking of, in couple of months, buying a
> notebook
> with two HDDs with RAID1 installed and using the usb drive as a backup
> destination. So if RAID got corruped, the backups, made since then,
> would be
> useless? How would you resolve it?

The ONLY thing RAID will save you from is hardware failure.

Copying data from one disk to another is not RAID. If you use dd to copy
from one corrupt filesystem to another, you have two corrupt filesystems.

If you want a robust filesystem, look into ZFS/BTRFS.

On 18 August 2010 14:59, William Kenworthy <billk@iinet.net.au> wrote:


On Wed, 2010-08-18 at 14:09 +0300, Nganon wrote:

>

>

> On 17 August 2010 22:34, Enrico Weigelt <weigelt@metux.de> wrote:

> * * * * For things I'd like to keep an history (eg. /etc) I'm using

> * * * * git, and

> * * * * pushing the repo to a remote server (denying non-fastfoward

> * * * * updates

> * * * * there, so an theorectical highjacker cannot destroy my

> * * * * history)

>

>

> Using git for /etc is a great idea.

> Thanks.

>

Another option is:

* *app-backup/dirvish

* * *Latest version available: 1.2.1

* * *Latest version installed: 1.2.1

* * *Size of downloaded files: 47 kB

* * *Homepage: * *http://www.dirvish.org/

* * *Description: Dirvish is a fast, disk based, rotating network

backup system.

* * *License: * * OSL-2.0





Works by first creating a copy (--init) and then hard-linking subsequent

versions of files/directories back to the original original if its

identical. *If a file is changed/new, it is copied instead of linked so

actual space usage quickly stabilises even with a varying number of

versions. *Backup over the network (this is how I have configured mine)

uses rsync over ssh with keys and is "pull" from a cron job on the

backup server or manual on demand (i.e., server initiated).



Version management is by a reasonably sophisticated date of version

scheme where by running "dirvish-expire" deletes out of date versions

(runs in a cron job). *The smart part is that once the last hard link to

file is deleted, its gone, otherwise its kept in the remaining

versions



Restore is a simple matter of identifying the version you want and

copying it back - Ive restored individual files through to complete

systems after total disk failure.



Can do includes/excludes, whole systems or just directories such as /etc

and can be easily automated.



Doesnt use compression, but most backup regimes (every day for a weekly

rota + a Sunday kept for 6 months) stabilise at about 2x the original

(gross) copy size, no matter how many copies with average changes

between versions. *Though large scale changes such as an "emerge -e

world" will take more as it will generate new copies of most files.



Downside is it will hammer the destination file system - reiserfs3 works

well, ext2/ext3 have been hopeless everytime I've tried - mass

corruption. *The file system will need a large number of inodes (for

links) if there are an excessive number of files x versions - again

reiserfs3 scores well here.



Highly recommended!



BillK







Thanks. It sound just it is made just for this. It even call itself 'time machine'.*Obviously compression is left out by using links but it sounds kind of*

overwhelming to me.*I don't have a reiserfs partition and cannot afford to*have one at the mo..**

On 18 August 2010 17:53, Bill Longman <bill.longman@gmail.com> wrote:


On 08/18/2010 04:53 AM, Nganon wrote:

> I did not know that. I was thinking of, in couple of months, buying a

> notebook

> with two HDDs with RAID1 installed and using the usb drive as a backup

> destination. So if RAID got corruped, the backups, made since then,

> would be

> useless? How would you resolve it?



The ONLY thing RAID will save you from is hardware failure.



Copying data from one disk to another is not RAID. If you use dd to copy

from one corrupt filesystem to another, you have two corrupt filesystems.


Clear now, thanks.*
If you want a robust filesystem, look into ZFS/BTRFS.




AFAIK ZFS is unmaintained and BTRFS is not stable, am I wrong?*