FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > EXT3 Users

 
 
LinkBack Thread Tools
 
Old 06-08-2008, 05:30 AM
Ross Boylan
 
Default spd_readdir.c and readdir_r

I still haven't been able to pinpoint exactly where bacula hangs up when
LD_PRELOAD is set to use spd_readdir, but I have a suspect. bacula-fd
gets directory entries with readdir_r, which is a function that is not
reimplemented in spd_readdir.

So when bacula calls opendir it gets the shadow version, which calls the
original open, read, and closedir functions. It then returns its
private dir_s structure. The (unshadowed) readdir_r then tries to work
with dir_s.

It looks as if I (or one of you gurus?) need to implement a wrapper for
readdir_r. A quick looks suggests there may be a couple of subtleties
(the spd_readdir struct dir_s is allocated, and so thread safe, but it's
dir entry is not; and readdir_r is expecting some "real" system data
structures back and users may have problems with fake ones).

Ross

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 06-09-2008, 04:26 AM
Ross Boylan
 
Default spd_readdir.c and readdir_r

I've attached a modified version of Ted's spd_readdir.c that adds
support for readdir_r and readdir64_r. It appears to be working
(readdir64_r is the only new routine getting exercised), but should be
taken as a rough cut. I also added a Makefile and a test program.

It also looks as if this is giving me a huge speed improvement (at least
x4) of my backups of my ext3 partitions. I'll try to report after a
full and incremental backup complete, which will be a couple of days.

Originally I tried taking the threading code from the system
implementations of the original readdir_r. When that didn't work (since
it was designed to be part of a libc build) I switched to pthreads. I
don't know if recursive locking is essential; I activated it at one
point while trying to get things to work.

For big directories this code could use quite a lot of memory. It
allows an optional max size, beyond which it reverts to the original
system calls. I wonder if instead taking large directories in chunks
would preserve much of the speedup while putting a bound on memory use.

Ross Boylan

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 06-16-2008, 03:46 AM
Ross Boylan
 
Default spd_readdir.c and readdir_r

My previous attachment had only a link for the main file; the current
one should have the real thing.

For the full backup, using the preload library changed the backup time
from over 35 hours to 22 hours for a full backup. The full backup got
much slower as it progressed; my guess is something other than the
preload library (perhaps the snapshotting itself, bacula, or postgresql)
accounts for that.

The percentage change for incremental backups, which involve relatively
more time scanning, is larger: from 3 hours to under .5 hours.

There's no obvious speedup for the jobs involving Reiser filesystems.

All in all, a big win. Thanks to everyone for your help, and especially
to Ted for the original code.

Ross Boylan

On Sun, 2008-06-08 at 21:26 -0700, Ross Boylan wrote:
> I've attached a modified version of Ted's spd_readdir.c that adds
> support for readdir_r and readdir64_r. It appears to be working
> (readdir64_r is the only new routine getting exercised), but should be
> taken as a rough cut. I also added a Makefile and a test program.
>
> It also looks as if this is giving me a huge speed improvement (at least
> x4) of my backups of my ext3 partitions. I'll try to report after a
> full and incremental backup complete, which will be a couple of days.
>
> Originally I tried taking the threading code from the system
> implementations of the original readdir_r. When that didn't work (since
> it was designed to be part of a libc build) I switched to pthreads. I
> don't know if recursive locking is essential; I activated it at one
> point while trying to get things to work.
>
> For big directories this code could use quite a lot of memory. It
> allows an optional max size, beyond which it reverts to the original
> system calls. I wonder if instead taking large directories in chunks
> would preserve much of the speedup while putting a bound on memory use.
>
> Ross Boylan
>
_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 09-03-2008, 01:45 PM
Theodore Tso
 
Default spd_readdir.c and readdir_r

Hey Ross,

Sorry for not responding early; I was travelling a lot over the
summer, and I never got around to responding to your e-mail.

Many thanks for adding support for readdir_r and readdir64_r! As it
turns out, I was doing some updates to spd_readdir.c to support
fdopendir (which rm uses). Also, it looks like you based your changes
off of an older version of spd_readdir.c that didn't support the
dirfd() call. I probably will try to package this up into its own
package, since I suspect it would be useful to a larger set of people.

In any case here's the merged version I have. Please let me know if
this works for you, and if you have any other suggested improvements!

- Ted

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 

Thread Tools




All times are GMT. The time now is 03:21 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org