FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora User

 
 
LinkBack Thread Tools
 
Old 02-18-2011, 09:11 PM
suvayu ali
 
Default Qn about rsnapshot/rsync directory filter rules

Hi everyone,

I am trying to setup my backups using rsnapshot. I have a rather large
and frequently changing /home/<user>/ that I want to backup. I don't
want to backup the frequently changing directories (e.g. my downalods
directory or temporary work directory). In other words I wanted to
rsnapshot to backup _only_ the directories/files explicitly specified
and skip everythign else.

So I created a file ~/.rsync-filter with include rules like these,

include_/.config/**
include_/dir1/**
include_/.gitconfig
include_/.gitignore
include_/.gitk
include_/dir2/**

Now say I have other files and direcotries like,

~/.oldconfig/
~/dir2_i_dont_want/
~/dir3_i_dont_want/

Calling rsync with -F works but it also backs up the other unwanted
directory and files. How do I exclude them? The problem is I can't write
explicit exclude rules as that would need to be updated every time
something new is created (as I want rsnapshot to _not_ backup anything
by default).

I tried exclude_/* at the end of the filter rules, that made rsnapshot
skip the explicitly mentioned includes like,

include_/dir1/**
include_/dir2/**

The hidden files were backed up just fine though. Any ways of doing what
I want?

--
Suvayu

Open source is the future. It sets us free.
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 02-19-2011, 12:35 AM
Patrick O'Callaghan
 
Default Qn about rsnapshot/rsync directory filter rules

On Fri, 2011-02-18 at 14:11 -0800, suvayu ali wrote:
> Hi everyone,
>
> I am trying to setup my backups using rsnapshot. I have a rather large
> and frequently changing /home/<user>/ that I want to backup. I don't
> want to backup the frequently changing directories (e.g. my downalods
> directory or temporary work directory). In other words I wanted to
> rsnapshot to backup _only_ the directories/files explicitly specified
> and skip everythign else.
>
> So I created a file ~/.rsync-filter with include rules like these,
>
> include_/.config/**
> include_/dir1/**
> include_/.gitconfig
> include_/.gitignore
> include_/.gitk
> include_/dir2/**
>
> Now say I have other files and direcotries like,
>
> ~/.oldconfig/
> ~/dir2_i_dont_want/
> ~/dir3_i_dont_want/
>
> Calling rsync with -F works but it also backs up the other unwanted
> directory and files. How do I exclude them? The problem is I can't write
> explicit exclude rules as that would need to be updated every time
> something new is created (as I want rsnapshot to _not_ backup anything
> by default).
>
> I tried exclude_/* at the end of the filter rules, that made rsnapshot
> skip the explicitly mentioned includes like,
>
> include_/dir1/**
> include_/dir2/**
>
> The hidden files were backed up just fine though. Any ways of doing what
> I want?

I have the following rule in my /etc/rsnapshot.conf:

exclude_file /etc/rsnapshot-exclude

where /etc/rsnapshot-exclude contains the following:

.adobe/
.alexandria/
.amaya/
.bittorrent/
.cddb/
.claws-mail/tagsdb/
.cpan/
.dbus/
.gcjwebplugin/
.macromedia/Flash_Player/
.opera/cache4/
.thumbnails/
.waf-*/
.xine/skins/
.gvfs/
.cache/
.ccache/
[Cc]ache/
ccache/
.nautilus/
SVN/
.local/share/evolution
.claws-mail/imapcache
.thunderbird/**/ImapMail
.miro/icon-cache/
.wine/drive_c/windows/temp/
.VirtualBox/
*.o
*.flv
*.mpg
*.mov
*.torrent

There's probably some redundancy there (e.g. I forget why I have cache/
and [Cc]ache/) but it seems to work.

poc

--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 02-20-2011, 08:20 AM
Suvayu Ali
 
Default Qn about rsnapshot/rsync directory filter rules

Hi Patrick,

Thanks for your response.

On Fri, 18 Feb 2011 21:05:38 -0430
Patrick O'Callaghan <pocallaghan@gmail.com> wrote:

> On Fri, 2011-02-18 at 14:11 -0800, suvayu ali wrote:
>
> > In other words I wanted to rsnapshot to backup _only_ the
> > directories/files explicitly specified and skip everythign else.
> >
>
> I have the following rule in my /etc/rsnapshot.conf:
>
> exclude_file /etc/rsnapshot-exclude
>
> where /etc/rsnapshot-exclude contains the following:
>
> .adobe/
> .alexandria/
> .amaya/
> ...<snipped>

Yes I follow that, but my problem is this exclude list needs to be
updated everytime I create a new directory (maybe I try out some new
application) or file (which I don't really want to backup). I only want
to backup certian directories/files.

> *.o
> *.flv
> *.mpg
> *.mov
> *.torrent
>

This however helped me. With this I can consistently ignore certain file
types I don't want to backup ever e.g. downloaded video files or
object/binary files from compilations. I can't do the same for stray
text files I usually end up with when trying to experiment with coding
or looking at log files or temporarily generated list of things with
grep, temporarily downloaded pdf files (I can't ignore all pdfs as I
want to keep papers or notes I am interested in) and many others like
that.

I wish to understand the filter so that I can build the entire rule set
in layers. Something like this,

1. ignore all files
2. add backup_root/dir1
3. add backup_root/some_file

So the subsequent layers overrule (I think logically its an OR) the
previous rule set. This way the backup process will ignore files unless
other wise specified. Is that possible? Or am I asking for something
which is achieved more easily/differently?

>
> poc
>

Thanks for any thoughts.

--
Suvayu

Open source is the future. It sets us free.
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 02-20-2011, 11:49 PM
suvayu ali
 
Default Qn about rsnapshot/rsync directory filter rules

Hi Patrick,

On Sun, Feb 20, 2011 at 5:32 AM, Patrick O'Callaghan
<pocallaghan@gmail.com> wrote:
> On Sun, 2011-02-20 at 01:20 -0800, Suvayu Ali wrote:
>> I wish to understand the filter so that I can build the entire rule
>> set
>> in layers. Something like this,
>>
>> 1. ignore all files
>> 2. add backup_root/dir1
>> 3. add backup_root/some_file
>>
>> So the subsequent layers overrule (I think logically its an OR) the
>> previous rule set. This way the backup process will ignore files
>> unless
>> other wise specified. Is that possible? Or am I asking for something
>> which is achieved more easily/differently?
>
> Can't you just run rsnapshot on backup_root? I'm afraid I don't really
> understand what you're trying to do.
>

Sorry I wasn't clear. I am trying to setup my include/exclude ruleset
on the sending side instead of as a global setting as I intend to back
up multiple machines simultaneously. I wasn't getting the desired
results when I was using directory filters. I think it was because I
had misunderstood how rsync uses the filter rules for directory
specific filters. My problem was I had put the exclude rules _before_
the include rules, so rsync was skipping the directories before my
include rules could take effect.

> poc

Thanks for your response.

--
Suvayu

Open source is the future. It sets us free.
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 02-21-2011, 01:52 AM
Patrick O'Callaghan
 
Default Qn about rsnapshot/rsync directory filter rules

On Sun, 2011-02-20 at 16:49 -0800, suvayu ali wrote:
> Hi Patrick,
>
> On Sun, Feb 20, 2011 at 5:32 AM, Patrick O'Callaghan
> <pocallaghan@gmail.com> wrote:
> > On Sun, 2011-02-20 at 01:20 -0800, Suvayu Ali wrote:
> >> I wish to understand the filter so that I can build the entire rule
> >> set
> >> in layers. Something like this,
> >>
> >> 1. ignore all files
> >> 2. add backup_root/dir1
> >> 3. add backup_root/some_file
> >>
> >> So the subsequent layers overrule (I think logically its an OR) the
> >> previous rule set. This way the backup process will ignore files
> >> unless
> >> other wise specified. Is that possible? Or am I asking for something
> >> which is achieved more easily/differently?
> >
> > Can't you just run rsnapshot on backup_root? I'm afraid I don't really
> > understand what you're trying to do.
> >
>
> Sorry I wasn't clear. I am trying to setup my include/exclude ruleset
> on the sending side instead of as a global setting as I intend to back
> up multiple machines simultaneously. I wasn't getting the desired
> results when I was using directory filters. I think it was because I
> had misunderstood how rsync uses the filter rules for directory
> specific filters. My problem was I had put the exclude rules _before_
> the include rules, so rsync was skipping the directories before my
> include rules could take effect.

The include/exclude rules are just passed through to rsync. The man page
for rsync has a fairly extensive discussion of the syntax. I don't see a
direct statement that the rules are evaluated in strict order, but it
seems natural.

poc

--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 02-22-2011, 02:49 AM
Cameron Simpson
 
Default Qn about rsnapshot/rsync directory filter rules

On 20Feb2011 22:22, Patrick O'Callaghan <pocallaghan@gmail.com> wrote:
| The include/exclude rules are just passed through to rsync. The man page
| for rsync has a fairly extensive discussion of the syntax. I don't see a
| direct statement that the rules are evaluated in strict order, but it
| seems natural.

They're just like ACLs: first match applies. From the man page, undef
"FILTER RULES":

As the list of files/directories to transfer is built, rsync checks
each name to be transferred against the list of include/exclude pat-
terns in turn, and the first matching pattern is acted on

So to rsync just specific directories one's rules might read (however
expressed):

+ /dir1
+ /dir2
- /*

to do just dir1 and dir2.

Cheers,
--
Cameron Simpson <cs@zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

The ZZR-1100 remains the big, fast, versatile bike for the rider with a
regular pillion passenger or the desire to humiliate just about every
other road user who attempted to match its straight line acceleration. - REVS
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 02-22-2011, 03:06 AM
Patrick O'Callaghan
 
Default Qn about rsnapshot/rsync directory filter rules

On Tue, 2011-02-22 at 14:49 +1100, Cameron Simpson wrote:
> On 20Feb2011 22:22, Patrick O'Callaghan <pocallaghan@gmail.com> wrote:
> | The include/exclude rules are just passed through to rsync. The man page
> | for rsync has a fairly extensive discussion of the syntax. I don't see a
> | direct statement that the rules are evaluated in strict order, but it
> | seems natural.
>
> They're just like ACLs: first match applies. From the man page, undef
> "FILTER RULES":
>
> As the list of files/directories to transfer is built, rsync checks
> each name to be transferred against the list of include/exclude pat-
> terns in turn, and the first matching pattern is acted on

Good. It's what I supposed but hadn't seen in the man page.

poc

--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 02-22-2011, 04:33 AM
Suvayu Ali
 
Default Qn about rsnapshot/rsync directory filter rules

Hi Cameron,

On Tue, 22 Feb 2011 14:49:39 +1100
Cameron Simpson <cs@zip.com.au> wrote:

> They're just like ACLs: first match applies. From the man page, undef
> "FILTER RULES":
>
> As the list of files/directories to transfer is built, rsync
> checks each name to be transferred against the list of
> include/exclude pat- terns in turn, and the first matching pattern
> is acted on
>
> So to rsync just specific directories one's rules might read (however
> expressed):
>
> + /dir1
> + /dir2
> - /*
>
> to do just dir1 and dir2.
>
> Cheers,

Yes, I finally figured this out. For some reason the language in the
man page seemed unclear to me at the time. Now that I understand the
subtlety, I'm not sure there is a better way to phrase it. :-p Maybe it
needs more examples.

--
Suvayu

Open source is the future. It sets us free.
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 03-05-2011, 06:02 AM
Tim
 
Default Qn about rsnapshot/rsync directory filter rules

On Sun, 2011-02-20 at 01:20 -0800, Suvayu Ali wrote:
> my problem is this exclude list needs to be updated everytime I create
> a new directory (maybe I try out some new application) or file (which
> I don't really want to backup). I only want to backup certian
> directories/files.

You could structure your directory tree so that you keep everything you
don't want to back-up inside one parent directory, and/or everything
that you do want to get backed-up inside another parent directory.


--
[tim@localhost ~]$ uname -r
2.6.27.25-78.2.56.fc9.i686

Don't send private replies to my address, the mailbox is ignored. I
read messages from the public lists.



--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 

Thread Tools




All times are GMT. The time now is 04:14 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org