kernel notification of file system changes
 

On Wed, 2010-05-05 at 15:12 +0800, Bill Kenworthy wrote:
> Can the older dnotify do what you want? - it monitors files differently
> to inotify. There is also gammin/fam.

dnotify locks the files or directories you want to watch, so it would
prevent external media from being unmounted. dnotify also uses a file
descripter per watched item, which could get interesting for large
amounts of watches!

I'm not sure about FAM, but Gamin uses inotify or dnotify anyway (in
Linux). I think they're all designed in a similar way: you have to
register a whole bunch of files or directories to watch.

thanks,
--
Iain Buchanan <iaindb at netspace dot net dot au>

He that is giddy thinks the world turns round.
-- William Shakespeare, "The Taming of the Shrew"

kernel notification of file system changes
 

On Wed, 2010-05-05 at 18:35 +0100, Stroller wrote:
> On 5 May 2010, at 18:24, Florian Philipp wrote:
> >> ...
> >> man inotify(7):
> >> ... When a directory is monitored, inotify will return events for the
> >> directory itself, and for files inside the directory.
> >> ...
> >
> > To repeat my comment on Iain's original "backup to a cold-swap drive"
> > thread ...
>
> Sorry, I started ignoring that almost immediately it was posted.

ooohh, ouch! :)

> He
> rejected too quickly too many workable solutions to basically
> functional backup. Perhaps Iain is a perfectionist, but I did not wish
> to follow the thread.

Perhaps I am a bit of a perfectionist, but I think you misunderstood my
aim. I rejected 3 options straight away (dd, gparted, and Ghost4Linux)
because they're not designed for backing up a live filesystem in a
"change only" fashion ("intelligent" is the word I used).

Beyond that I didn't reject anything else that anyone mentioned.

> > ... Inotify has two drawbacks which make it hard or even impossible
> > to use for Iain's use case:
> >
> > a) It does not work recursively which means that you have to create a
> > new handle for each subdirectory. Of course, this only means more work
> > for the programmer but there is also the problem that
>
> Pardon me. I assumed that "files inside the directory" meant that foo
> would be be changed when foo/bar changed, thus monitoring grunt would
> reflect changes in grunt/foo/bar. I overlooked that a directory is not
> a file.

isn't it? I thought a directory was just a file containing names (or
inodes) of other files? Which would explain why monitoring grunt
wouldn't show changes in grunt/foo/bar, since the directory/file called
grunt remains the same (ie. contains the same list of inodes) even if
grunt/foo/bar changes.

Let me tell you what I actually want to do, which I may not have made
clear originally:

I want to backup root to an external drive (or that could be rephrased
as "I want to backup any mount to any other mount), such that:
1. My backup is an hour or so out of date (at most)
2. I don't need to copy the entire filesystem every time

To do that, I could either:
* Run rsync every hour over the entire filesystem (I'm doing this
now with ionice, takes about 10 minutes when there are no
changes)
* Use some file notification monitor to tell me which file was
just changed, and only rsync that file

The problems with rsync is that during the rsync process, the filesystem
is changing, so I will end up with a slightly inconsistent backup. If I
use some notification method that tells me a file has changed, I can
greatly reduce any inconsistency, and I reduce my hour down to seconds
or minutes, depending on how much changes at any one time.

I'm considering LVM for it's snapshot capability, but I'd still have to
rsync root. I would prefer a file notification method as well, so I can
just rsync the file that just changed.

So far all the file monitoring tools are based on individual files (even
the recursive ones), and you eventually reach a system limit.

Thanks, and willing to listen to any ideas from anyone (except
Stroller :p )

--
Iain Buchanan <iaindb at netspace dot net dot au>

Usually, when a lot of men get together, it's called a war.
-- Mel Brooks, "The Listener"

kernel notification of file system changes
 

On Thu, 2010-05-06 at 01:33 +0200, Volker Armin Hemmann wrote:

> oh god... fam... that crap caused me so much pain over the years. This bug
> ridden zombie is still around?

thanks for the heads-up - I guess I should leave FAM to plan B?
--
Iain Buchanan <iaindb at netspace dot net dot au>

I am a friend of the working man, and I would rather be his friend
than be one.
-- Clarence Darrow

kernel notification of file system changes
 

On Donnerstag 06 Mai 2010, Iain Buchanan wrote:
> On Thu, 2010-05-06 at 01:33 +0200, Volker Armin Hemmann wrote:
> > oh god... fam... that crap caused me so much pain over the years. This
> > bug ridden zombie is still around?
>
> thanks for the heads-up - I guess I should leave FAM to plan B?

Plan Z really. I can't even remember how many times I had to kill that pig to
unmount something - or because it suddenly decided to eat CPU cycles like a
fat stoner breakfast cereals.

kernel notification of file system changes
 

On Thu, 2010-05-06 at 08:54 +0930, Iain Buchanan wrote:

> but it's an angle to follow. I wonder how max_user_watches would handle
> being 100k or more... no doubt you just need some RAM?!
>
> thanks,

To answer my own questions, I'm now trying this:

# echo 100000 >/proc/sys/fs/inotify/max_user_watches

$ time sudo find / -xdev -type d | sudo inotifywatch -v -t 1 -e modify,attrib,move,create,delete,delete_self,unmou nt --fromfile -
Establishing watches...
Total of 71169 watches.
Finished establishing watches, now collecting statistics.
Will listen for events for 1 seconds.
total modify filename
6 6 /tmp/
2 2 /dev/

real 0m3.177s
user 0m0.768s
sys 0m1.378s

This sets up a watch on all directories under / that aren't part of
another filesystem, and then exits after one second. It's quite fast :)

The idea, off the top of my head, would be this:
1. inotifywatch as above but without the time restriction
2. wait for it to finishing "setting up"
3. rsync the whole directory structure to the backup
4. continuously do this loop:
1. get list of changes from inotifywatch
2. rsync those changes

Unfortunately inotifywatch only returns output on ctrl-c, which I don't
want to do or you loose anything changed between instances. This could
be changed to another signal, no doubt.

How does that sound for a continuous running backup? This is starting
to stray OT from Gentoo, but your thoughts are welcome :)
--
Iain Buchanan <iaindb at netspace dot net dot au>

One cannot make an omelette without breaking eggs -- but it is amazing
how many eggs one can break without making a decent omelette.
-- Professor Charles P. Issawi

kernel notification of file system changes
 

On 6 May 2010, at 00:51, Iain Buchanan wrote:

...

... Inotify has two drawbacks which make it hard or even impossible
to use for Iain's use case:

a) It does not work recursively which means that you have to
create a
new handle for each subdirectory. Of course, this only means more
work

for the programmer but there is also the problem that


Pardon me. I assumed that "files inside the directory" meant that foo
would be be changed when foo/bar changed, thus monitoring grunt would
reflect changes in grunt/foo/bar. I overlooked that a directory is
not

a file.


isn't it? I thought a directory was just a file containing names (or
inodes) of other files? Which would explain why monitoring grunt
wouldn't show changes in grunt/foo/bar, since the directory/file
called

grunt remains the same (ie. contains the same list of inodes) even if
grunt/foo/bar changes.


But if bar changes, the modification time of foo changes, which is a
change in grunt. Apparently not. :(



Let me tell you what I actually want to do, which I may not have made
clear originally:

I want to backup root to an external drive (or that could be rephrased
as "I want to backup any mount to any other mount), such that:
1. My backup is an hour or so out of date (at most)
2. I don't need to copy the entire filesystem every time

To do that, I could either:
* Run rsync every hour over the entire filesystem (I'm doing this
now with ionice, takes about 10 minutes when there are no
changes)
* Use some file notification monitor to tell me which file was
just changed, and only rsync that file

The problems with rsync is that during the rsync process, the
filesystem

is changing, so I will end up with a slightly inconsistent backup.


I really appreciate where you're coming from, as I have the same
concerns regarding the backing up of emails. I have systems with gigs
of email on them - belonging to myself and to others - and in the time
taken to rsync the lot, more mail is sure to have arrived. All one can
do is rsync regularly - then you're only an hour or so out of date.


I'm considering LVM for it's snapshot capability, but I'd still have
to

rsync root.


Indeed, snapshotting is ideal.


I would prefer a file notification method as well, so I can
just rsync the file that just changed.


I think you're unrealistic to expect your backup to be uptodate to
within minutes or seconds. I don't really see the benefit of that, to
be honest. As far as I'm concerned I don't really care if /bin, /sbin
&c are backed up or not, because I can reinstall Gentoo in a couple of
hours. Here /etc does not generally change on a daily basis - more
like a backup from last month would be fine! Your milage may vary, but
I really don't see a day-old backup of /etc as a problem. So we are
really only down to /home which needs backing up as frequently as
hourly.


I assume that you're aware that "live" backups require multiple
copies. This is the RAID-is-not-a-backup argument - if a file becomes
corrupt you don't want it overwriting your backup copy.


Having said that you're "unrealistic" and that "I don't see the
benefit", I do appreciate where you're coming from. It's the answer
that us perfectionists want - it's the answer that all Linux users
want. And thus, if there were a clean and elegant solution then it
would be a well-trodden path. I think that if there was a clean and
elegant solution to "the perfect backup" everyone would know what it
was, and everyone would be using it, and you wouldn't even have to
ask. This is why I stopped reading your thread early.


Having said all that, if the stumbling block is snapshotting of your /
partition, you might take a look at btrfs - it looks like it allows
you to do this. brrfs is probably good enough now for everyday use, as
long as you have a backup, and I think you could reasonably easily
backup your snapshots on some regular basis to some other "safe"
filesystem, such as ext[234].


I find it slightly interesting that none of the ionotify solutions do
what you want. The point of such tools is to "do something when a file
is written or changed", so they would seem an ideal fit to solve the
"perfect backup problem" - in that context it would seem trivial to
write a kernel module / userspace app to write a copy of each changed
file to /path/to/backup/$UID and record some information about the
original filename, $UID and the time of change. I imagine that some
types of changes to files - such as bittorrent downloads, in which a
binary file may change like A___E -> A_C_E -> ABCDE - are more
difficult to handle than others, so this will be left to the reader as
a practical programming exercise.


Stroller.

kernel notification of file system changes
 

On 05/06/2010 07:22 AM, Stroller wrote:





I find it slightly interesting that none of the ionotify solutions do what you want. The point of such tools is to "do something when a file is written or changed", so they would seem an ideal fit to solve the "perfect backup problem" - in that
context it would seem trivial to write a kernel module / userspace app to write a copy of each changed file to /path/to/backup/$UID and record some information about the original filename, $UID and the time of change. I imagine that some types of
changes to files - such as bittorrent downloads, in which a binary file may change like A___E -> A_C_E -> ABCDE - are more difficult to handle than others, so this will be left to the reader as a practical programming exercise.> Stroller.


Matt Dillon created a new file system for his DragonflyBSD project to solve
the backup problem (and others):

http://en.wikipedia.org/wiki/HAMMER