FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > EXT3 Users

 
 
LinkBack Thread Tools
 
Old 06-10-2008, 10:24 PM
Santi Saez
 
Default 2GB memory limit running fsck on a +6TB device

Theodore Tso escribió:

It's always a good idea when running e2fsck (aka fsck.ext3) directly
and/or on a terminal/console to include as command-line options "-C
0". This will display a progress bar, so you can gauge how it is
doing. (0 through 70% is pass 1, which requires scanning the inode
table and following all of the indirect blocks.)



Thanks for the tip! :-)

'/var/cache/e2fsck' is in the _same_ disk, perhaps mounting via iSCSI,
NFS, etc.. this directory will improve, we will work with this in other
test.


I have enabled progress bar sending SIGUSR1 signal to the process, and
it's still on 2% ;-(


"scratch_files" directory size is now 251M, it has grown 81MB in the
last 7 hours:


# ls -lh /var/cache/e2fsck/
total 251M
-rw------- 1 root root 112M 2008-06-11 00:09
7701b70e-f776-417b-bf31-3693dba56f86-dirinfo-VkmFXP
-rw------- 1 root root 139M 2008-06-11 00:09
7701b70e-f776-417b-bf31-3693dba56f86-icount-YO08bu


strace's output is the same, and also memory usage is the same.

I will let the process more time.. but I think it will take too much
time to complete, at least to finish the pass 1, perhaps more than 50
hours? According that now is only on 2% of the process + take 12 hours
to complete, and pass 1 is from 0% through 70%.. is there any other
solution to solve this?


ext4 will solve this problem? I have not tested ext4 already, but I have
read that it will improve fast filesytem checking...


Regards,

--
Santi Saez

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 06-10-2008, 11:01 PM
Theodore Tso
 
Default 2GB memory limit running fsck on a +6TB device

On Wed, Jun 11, 2008 at 12:24:27AM +0200, Santi Saez wrote:
>
> '/var/cache/e2fsck' is in the _same_ disk, perhaps mounting via iSCSI, NFS,
> etc.. this directory will improve, we will work with this in other test.
>
> I have enabled progress bar sending SIGUSR1 signal to the process, and it's
> still on 2% ;-(
>
> "scratch_files" directory size is now 251M, it has grown 81MB in the last 7
> hours:

hmm..... can you send me the output of dumpe2fs /dev/sdXX? You can
run that command while e2fsck is running, since it's read-only. I'm
curious exactly how big the filesystem is, and how many directories
are in the first part of the filesystem.

How big is the filesystem(s) that you are backing up via BackupPC, in
terms of size (megabytes) and files (number of inodes)? And how many
days of incremental backups are you keeping? Also, how often do files
change? Can you give a rough estimate of how many files get modified
per backup cycle?

Thanks,

- Ted

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 06-10-2008, 11:48 PM
Santi Saez
 
Default 2GB memory limit running fsck on a +6TB device

Theodore Tso escribió:

hmm..... can you send me the output of dumpe2fs /dev/sdXX? You can
run that command while e2fsck is running, since it's read-only. I'm
curious exactly how big the filesystem is, and how many directories
are in the first part of the filesystem.

Upsss... dumpe2fs takes about 3 minutes to complete and generates about
133MB output file:


dumpe2fs 1.40.8 (13-Mar-2008)
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: 7701b70e-f776-417b-bf31-3693dba56f86
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal dir_index filetype sparse_super
large_file

Default mount options: (none)
Filesystem state: clean with errors
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 792576000
Block count: 1585146848
Reserved block count: 0
Free blocks: 913341561
Free inodes: 678201512
First block: 0
Block size: 4096
Fragment size: 4096
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 16384
Inode blocks per group: 512
Filesystem created: Mon Nov 13 10:12:49 2006
Last mount time: Mon Jun 9 19:37:12 2008
Last write time: Tue Jun 10 12:18:25 2008
Mount count: 37
Maximum mount count: -1
Last checked: Mon Nov 13 10:12:49 2006
Check interval: 0 (<none>)
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
Default directory hash: tea
Directory Hash Seed: afabe3f6-4405-44f4-934b-76c23945db7b
Journal backup: inode blocks
Journal size: 32M

Some example output from group 0 to 5 is available at:

http://pastebin.com/f5341d121


How big is the filesystem(s) that you are backing up via BackupPC, in
terms of size (megabytes) and files (number of inodes)? And how many
days of incremental backups are you keeping? Also, how often do files
change? Can you give a rough estimate of how many files get modified
per backup cycle?



Where are backing up several servers, near about 15 in this case, with
60-80GB data size to backup in each server and +2-3 millon inodes, with
15 day incrementals. I think near about 2-3% of the files changes each
day, but I will ask for more info to the backup administrator.


I have found and old doc with some build info for this server, the
partition was formated with:


# mkfs.ext3 -b 4096 -j -m 0 -O dir_index /dev/sda4
# tune2fs -c 0 -i 0 /dev/sda4
# mount -o data=writeback,noatime,nodiratime,commit=60 /dev/sda4 /backup

I'm going to fetch more info about BackupPC and backup cycles, thanks Ted!!

Regards,

--
Santi Saez

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 06-11-2008, 02:18 AM
Theodore Tso
 
Default 2GB memory limit running fsck on a +6TB device

On Wed, Jun 11, 2008 at 01:48:35AM +0200, Santi Saez wrote:
> Theodore Tso escribió:
>> hmm..... can you send me the output of dumpe2fs /dev/sdXX? You can
>> run that command while e2fsck is running, since it's read-only. I'm
>> curious exactly how big the filesystem is, and how many directories
>> are in the first part of the filesystem.
>>
> Upsss... dumpe2fs takes about 3 minutes to complete and generates about
> 133MB output file:

True, but it compresses well. :-) And the aside from the first part
of the dumpe2fs, the part that I was most interested could have been
summarized by simply doing a "grep directories dumpe2fs.out".

But simply looking at your dumpe2fs, and take an average from the
first 6 block groups which you included in the pastebin, I can
extrapolate and guess that you have about 63 million directories, out
of approximately 114 million total inodes (so about 51 million regular
files, nearly all of which have hard link counts > 1). Unfortunately,
BackupPC blows out of the water all of our memory reduction
hueristics. I estimate you need something like 2.6GB to 3GB of memory
just for these data structures alone. (Not to mention 94 MB for each
inode bitmap, and 188 MB for each block bitmap.) The good news is
that 4GB of memory should do you --- just. (I'd probably put in a bit
more physical memory just to be on the safe side, or enable swap
before running e2fsck). The bad news is you really, REALLY need a
64-bit kernel on your system.

Because /var/cache/e2fsck is on the same disk spindle as the
filesystem you are checking, you're probably getting killed on seeks.
Moving /var/cache/e2fsck to another disk partition will help (or
better yet, battery backed memory device), but the best thing you can
do is get a 64-bit kernel and not need to use the auxiliary storage in
the first place.

As far as what to advice to give you, why are you running e2fsck? Was
this an advisory thing caused by the mount count and/or length of time
between filesystem checks? Or do you have real reason to believe the
filesystem may be corrupt?

- Ted

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 06-11-2008, 08:14 AM
 
Default 2GB memory limit running fsck on a +6TB device

On Tue, 10 Jun 2008 22:18:00 -0400, Theodore Tso <tytso@mit.edu> wrote:

> True, but it compresses well. :-) And the aside from the first part
> of the dumpe2fs, the part that I was most interested could have been
> summarized by simply doing a "grep directories dumpe2fs.out".



"grep directories" is available at:

http://santi.usansolo.net/tmp/dumpe2fs_directories.txt.gz (317K)

Full "dumpe2fs" output compressed is 34M and available at:

http://santi.usansolo.net/tmp/dumpe2fs.txt.gz



> But simply looking at your dumpe2fs, and take an average from the
> first 6 block groups which you included in the pastebin, I can
> extrapolate and guess that you have about 63 million directories, out
> of approximately 114 million total inodes (so about 51 million regular
> files, nearly all of which have hard link counts > 1).

# grep directories dumpe2fs.txt | awk '{sum += $7} END {print sum}'
78283294



> BackupPC blows out of the water all of our memory reduction
> hueristics. I estimate you need something like 2.6GB to 3GB of memory
> just for these data structures alone. (Not to mention 94 MB for each
> inode bitmap, and 188 MB for each block bitmap.) The good news is
> that 4GB of memory should do you --- just. (I'd probably put in a bit
> more physical memory just to be on the safe side, or enable swap
> before running e2fsck). The bad news is you really, REALLY need a
> 64-bit kernel on your system.

Unfortunately, I have killed the process, in 21 hours only 2.5% of the fsck
is completed ;-(

'scratch_files' directory has grown to 311M

================================================== =================
# time fsck -y /dev/sda4
fsck 1.40.8 (13-Mar-2008)
e2fsck 1.40.8 (13-Mar-2008)
Adding dirhash hint to filesystem.

/dev/sda4 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes

/dev/sda4: e2fsck canceled.

/dev/sda4: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sda4: ********** WARNING: Filesystem still has errors **********

real 1303m19.306s
user 1079m58.898s
sys 217m10.130s
================================================== =================

> Because /var/cache/e2fsck is on the same disk spindle as the
> filesystem you are checking, you're probably getting killed on seeks.
> Moving /var/cache/e2fsck to another disk partition will help (or
> better yet, battery backed memory device), but the best thing you can
> do is get a 64-bit kernel and not need to use the auxiliary storage in
> the first place.

I'm trying a fast test with "mount tmpfs /var/cache/e2fsck -t tmpfs -o
size=2048M", but appears that will take a long time to complete too.. so
the next test will be with a 64-bit LiveCD



> As far as what to advice to give you, why are you running e2fsck? Was
> this an advisory thing caused by the mount count and/or length of time
> between filesystem checks? Or do you have real reason to believe the
> filesystem may be corrupt?

No, it's not related with mount count and/or length of time between
filesystem checks. When booting we get this error/warning:

EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
EXT3 FS on sda4, internal journal
EXT3-fs: mounted filesystem with writeback data mode.

And "tune2fs" returns that ext3 is in "clean with errors" state.. so, we
think that completing a full fsck process is a good idea; what means in
this case "clean with errors" state, running a fsck is not needed?

Thanks again for all the help and advices!!

--
Santi Saez

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 06-11-2008, 11:51 AM
 
Default 2GB memory limit running fsck on a +6TB device

On Wed, 11 Jun 2008 10:14:45 +0200, <santi@usansolo.net> wrote:

>> Because /var/cache/e2fsck is on the same disk spindle as the
>> filesystem you are checking, you're probably getting killed on seeks.
>> Moving /var/cache/e2fsck to another disk partition will help (or
>> better yet, battery backed memory device), but the best thing you can
>> do is get a 64-bit kernel and not need to use the auxiliary storage in
>> the first place.
>
> I'm trying a fast test with "mount tmpfs /var/cache/e2fsck -t tmpfs -o
> size=2048M", but appears that will take a long time to complete too.. so
> the next test will be with a 64-bit LiveCD

Note that putting '/var/cache/e2fsck' in a memory filesystem is aprox. 3
times faster ;-)

Making some fast test with e2fsck v1.40.10 appears that is a bit faster
than v1.40.8, last version improves this feature? Anyway, finally I had to
cancel the process..

# ./e2fsck -nfvttC0 /dev/sda4
e2fsck 1.40.10 (21-May-2008)
Pass 1: Checking inodes, blocks, and sizes
/dev/sda4: e2fsck canceled.


/dev/sda4: ********** WARNING: Filesystem still has errors **********

Memory used: 260k/581088k (183k/78k)

Regards,

--
Santi Saez

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 06-11-2008, 02:59 PM
Andreas Dilger
 
Default 2GB memory limit running fsck on a +6TB device

On Jun 11, 2008 13:51 +0200, santi@usansolo.net wrote:
> On Wed, 11 Jun 2008 10:14:45 +0200, <santi@usansolo.net> wrote:
>
> >> Because /var/cache/e2fsck is on the same disk spindle as the
> >> filesystem you are checking, you're probably getting killed on seeks.
> >> Moving /var/cache/e2fsck to another disk partition will help (or
> >> better yet, battery backed memory device), but the best thing you can
> >> do is get a 64-bit kernel and not need to use the auxiliary storage in
> >> the first place.
> >
> > I'm trying a fast test with "mount tmpfs /var/cache/e2fsck -t tmpfs -o
> > size=2048M", but appears that will take a long time to complete too.. so
> > the next test will be with a 64-bit LiveCD
>
> Note that putting '/var/cache/e2fsck' in a memory filesystem is aprox. 3
> times faster ;-)

...but, isn't the problem that you don't have enough RAM? Using tdb+ramfs
isn't going to be faster than using the RAM directly.

I suspect that the only way you are going to check this filesystem efficiently
is to boot a 64-bit kernel (even just from a rescue disk), set up some swap
just in case, and run e2fsck from there.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 06-11-2008, 04:49 PM
Bryan Kadzban
 
Default 2GB memory limit running fsck on a +6TB device

On Wed, Jun 11, 2008 at 08:59:08AM -0600, Andreas Dilger wrote:
> On Jun 11, 2008 13:51 +0200, santi@usansolo.net wrote:
> > On Wed, 11 Jun 2008 10:14:45 +0200, <santi@usansolo.net> wrote:
> >
> > >> Moving /var/cache/e2fsck to another disk partition will help (or
> > >> better yet, battery backed memory device), but the best thing you
> > >> can do is get a 64-bit kernel and not need to use the auxiliary
> > >> storage in the first place.
> > >
> > > I'm trying a fast test with "mount tmpfs /var/cache/e2fsck -t tmpfs
> > > -o size=2048M", but appears that will take a long time to complete
> > > too.. so the next test will be with a 64-bit LiveCD
> >
> > Note that putting '/var/cache/e2fsck' in a memory filesystem is aprox.
> > 3 times faster ;-)
>
> ...but, isn't the problem that you don't have enough RAM? Using
> tdb+ramfs isn't going to be faster than using the RAM directly.

It won't be faster, no, but it will be faster than tdb-on-disk, and much
faster than tdb on the same disk as the one that's being checked.

And it *might* allow e2fsck to allocate all the virtual memory that it
needs, depending on how the tmpfs driver works. If tmpfs uses the same
VA space as e2fsck and the rest of the kernel, then it probably won't
help. But if tmpfs can use a different pool somehow (whether that's
because the kernel uses a different set of pagetables, or whatever),
then it might.

> I suspect that the only way you are going to check this filesystem
> efficiently is to boot a 64-bit kernel (even just from a rescue disk),
> set up some swap just in case, and run e2fsck from there.

And try to run a 64-bit e2fsck binary, too. The virtual address space
usage estimate that someone (Ted?) came up with earlier in this thread
was close to 4G, which means that even with a 64-bit kernel, a 32-bit
e2fsck binary might still run out of virtual address space. (It will
need to map lots of disk, plus any real RAM usage, plus itself and any
libraries. That last bit *might* push it over 4G, depending on how
accurate the estimate of 4G turns out to be.)

The easiest way to do this is probably run the e2fsck from the LiveCD
itself; don't try to run the 32-bit version that the system has
installed. That version *might* work, but it'll be tight; a 64-bit
version that can use 40-odd bits in its virtual addresses (44? 48? I
think it depends on the exact CPU model -- and the kernel, of course)
will have a *lot* more headroom.

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 06-12-2008, 05:24 AM
Theodore Tso
 
Default 2GB memory limit running fsck on a +6TB device

On Wed, Jun 11, 2008 at 08:59:08AM -0600, Andreas Dilger wrote:
> > Note that putting '/var/cache/e2fsck' in a memory filesystem is aprox. 3
> > times faster ;-)
>
> ...but, isn't the problem that you don't have enough RAM? Using tdb+ramfs
> isn't going to be faster than using the RAM directly.

Tmpfs is swap backed, if swap has been configured. So it can help.

Another possibility is to use a statically linked e2fsck, since the
shared libraries chew up a lot of VM address space. But in this
particular case, it probably wouldn't be enough.

I think the best thing to do is this case to use a 64-bit kernel and a
64-bit compiled e2fsck binary.

- Ted

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 

Thread Tools




All times are GMT. The time now is 08:10 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org