I still haven't been able to pinpoint exactly where bacula hangs up when
LD_PRELOAD is set to use spd_readdir, but I have a suspect. bacula-fd
gets directory entries with readdir_r, which is a function that is not
reimplemented in spd_readdir.

So when bacula calls opendir it gets the shadow version, which calls the
original open, read, and closedir functions. It then returns its
private dir_s structure. The (unshadowed) readdir_r then tries to work
with dir_s.

It looks as if I (or one of you gurus?) need to implement a wrapper for
readdir_r. A quick looks suggests there may be a couple of subtleties
(the spd_readdir struct dir_s is allocated, and so thread safe, but it's
dir entry is not; and readdir_r is expecting some "real" system data
structures back and users may have problems with fake ones).

Ross

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

I've attached a modified version of Ted's spd_readdir.c that adds
support for readdir_r and readdir64_r. It appears to be working
(readdir64_r is the only new routine getting exercised), but should be
taken as a rough cut. I also added a Makefile and a test program.

It also looks as if this is giving me a huge speed improvement (at least
x4) of my backups of my ext3 partitions. I'll try to report after a
full and incremental backup complete, which will be a couple of days.

Originally I tried taking the threading code from the system
implementations of the original readdir_r. When that didn't work (since
it was designed to be part of a libc build) I switched to pthreads. I
don't know if recursive locking is essential; I activated it at one
point while trying to get things to work.

For big directories this code could use quite a lot of memory. It
allows an optional max size, beyond which it reverts to the original
system calls. I wonder if instead taking large directories in chunks
would preserve much of the speedup while putting a bound on memory use.

Ross Boylan

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

My previous attachment had only a link for the main file; the current
one should have the real thing.

For the full backup, using the preload library changed the backup time
from over 35 hours to 22 hours for a full backup. The full backup got
much slower as it progressed; my guess is something other than the
preload library (perhaps the snapshotting itself, bacula, or postgresql)
accounts for that.

The percentage change for incremental backups, which involve relatively
more time scanning, is larger: from 3 hours to under .5 hours.

There's no obvious speedup for the jobs involving Reiser filesystems.

All in all, a big win. Thanks to everyone for your help, and especially
to Ted for the original code.

Ross Boylan

On Sun, 2008-06-08 at 21:26 -0700, Ross Boylan wrote:
> I've attached a modified version of Ted's spd_readdir.c that adds
> support for readdir_r and readdir64_r. It appears to be working
> (readdir64_r is the only new routine getting exercised), but should be
> taken as a rough cut. I also added a Makefile and a test program.
>
> It also looks as if this is giving me a huge speed improvement (at least
> x4) of my backups of my ext3 partitions. I'll try to report after a
> full and incremental backup complete, which will be a couple of days.
>
> Originally I tried taking the threading code from the system
> implementations of the original readdir_r. When that didn't work (since
> it was designed to be part of a libc build) I switched to pthreads. I
> don't know if recursive locking is essential; I activated it at one
> point while trying to get things to work.
>
> For big directories this code could use quite a lot of memory. It
> allows an optional max size, beyond which it reverts to the original
> system calls. I wonder if instead taking large directories in chunks
> would preserve much of the speedup while putting a bound on memory use.
>
> Ross Boylan
>
_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users