FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > EXT3 Users

 
 
LinkBack Thread Tools
 
Old 05-29-2008, 09:37 AM
Jelle de Jong
 
Default needs help, root inode gone after usb bus reset on sata disks

Theodore Tso wrote:

On Wed, May 28, 2008 at 04:44:21PM +0200, Jelle de Jong wrote:

dumpe2fs -o superblock=32768 /dev/sdXX


I asked you to do the above, but you did this instead:


dumpe2fs -ob 32768 /dev/sda1 > dumpe2fs-32768-info-sda1.txt 2>&1




My humble excuse, i had to place the disk in a server and this server
had an older version of the dumpe2fs tool that did not supported the
superblock option. I upgraded the server and rerun all the test for you.


dumpe2fs /dev/sda1 > dumpe2fs-info-sda1.txt 2>&1
dumpe2fs -o superblock=32768 /dev/sda1 >
dumpe2fs-superblock-32768-info-sda1.txt 2>&1
dumpe2fs -o superblock=98304 /dev/sda1 >
dumpe2fs-superblock-98304-info-sda1.txt 2>&1

e2fsck -n /dev/sda1 > e2fsck-n-info-sda1.txt 2>&1

http://www.powercraft.nl/temp/dumpe2fs-info-sda1.txt.gz
http://www.powercraft.nl/temp/dumpe2fs-superblock-32768-info-sda1.txt.gz
http://www.powercraft.nl/temp/dumpe2fs-superblock-98304-info-sda1.txt.gz
http://www.powercraft.nl/temp/e2fsck-n-info-sda1.txt.gz

I hope this is the correct information, that can tell you want command
is best to run to restore the filesystem with the data.



Resulting in this:

dumpe2fs: No such file or directory while trying to open 32768

So I can't tell if the backup superblock was corrupted, but this is
definitely one for the record books. Looking at primary superblock,
we see the following:

dumpe2fs 1.40-WIP (14-Nov-2006)
Filesystem volume name: <none>
Last mounted on: ^^<BA><8B>
Filesystem UUID: 2e27ae79-fc96-43f5-9758-15ed74dd55fb
Filesystem magic number: 0xEF53
Filesystem revision #: 0 (original)
Filesystem features: (none)
Default mount options: MNTOPT_15 MNTOPT_16 MNTOPT_18 MNTOPT_20 MNTOPT_21 MNTOPT_22 MNTOPT_24 MNTOPT_26

The above, especially the Filesystem features, and default mount
options, are garbage. But it looks like the rest of the superblock,
including the magic number, the block counts, etc., look sane --- at
least in sane enough that it passed e2fsck's sanity checking.

This is unlike *any* corruption I've seen before; usually there will
be a single bit flip, or the entire disk sector is corrupted, but it's
extremely rare to see this kind of selective corruption.

It's even wierder that this apparently happened on more than one hard
drive. In any case, I would ditch that USB<->SATA converter as fast
as possible, because there is something seriously wrong. The other
possibility is that you're running with buggy kernel, but no one else
has ever reported anything like this, and for two disks to be
corrupted the same way means it's unlikely to be caused by a random
wild pointer or some such. So if I really had to guess I'd go with
the USB converter, but that's not for certain.

In terms of how to fix it, I'd would have to see the results of


dumpe2fs -o superblock=32768 /dev/sdXX

and


dumpe2fs -o superblock=98304 /dev/sdXX

Hopefully one of the superblocks look OK. We could also try manually
repairing the superblock with debugfs, in the worse case, but it'll be
easier if we can fix things via the backup superblock.

- Ted


I always seem to get the impossible out of Linux tools, but most times
this is during quality tests... however this was on "normal usage". I
hope it has noting to do with the latest release changes or with corrupt
binaries on my client system.


Thank you Ted,

Kind regards,

Jelle


_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 05-29-2008, 12:58 PM
Theodore Tso
 
Default needs help, root inode gone after usb bus reset on sata disks

On Thu, May 29, 2008 at 11:37:25AM +0200, Jelle de Jong wrote:
>
> My humble excuse, i had to place the disk in a server and this server had
> an older version of the dumpe2fs tool that did not supported the superblock
> option. I upgraded the server and rerun all the test for you.
>
> dumpe2fs -o superblock=32768 /dev/sda1 >
> dumpe2fs-superblock-32768-info-sda1.txt 2>&1
> dumpe2fs -o superblock=98304 /dev/sda1 >
> dumpe2fs-superblock-98304-info-sda1.txt 2>&1

Unfortunately, it looks like the backup superblocks were also
corrupted, and in the same way. Did you *ever* run e2fsck in
read/write mode (i.e., without the -n option) on this filesystem after
when you think it had gotten corrupted?

So what I will suggest at this point is that you do the following:

debugfs -w /dev/sda1
debugfs: features dir_index filetype sparse_super
debugfs: quit

The run "e2fsck -nf /dev/sda1" and make the output looks relatively
clean. You should *not* see any messages about needing to relocate
inode tables.

If so, you can then run "e2fsck -f /dev/sda1 to fully recover the
filesysten.

> I always seem to get the impossible out of Linux tools, but most times this
> is during quality tests... however this was on "normal usage". I hope it
> has noting to do with the latest release changes or with corrupt binaries
> on my client system.

Well, absolutely no one else reporting this problem or anything like
it...

- Ted

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 05-29-2008, 02:44 PM
Jelle de Jong
 
Default needs help, root inode gone after usb bus reset on sata disks

Theodore Tso wrote:

On Thu, May 29, 2008 at 11:37:25AM +0200, Jelle de Jong wrote:
My humble excuse, i had to place the disk in a server and this server had
an older version of the dumpe2fs tool that did not supported the superblock
option. I upgraded the server and rerun all the test for you.


dumpe2fs -o superblock=32768 /dev/sda1 >
dumpe2fs-superblock-32768-info-sda1.txt 2>&1
dumpe2fs -o superblock=98304 /dev/sda1 >
dumpe2fs-superblock-98304-info-sda1.txt 2>&1


Unfortunately, it looks like the backup superblocks were also
corrupted, and in the same way. Did you *ever* run e2fsck in
read/write mode (i.e., without the -n option) on this filesystem after
when you think it had gotten corrupted?


yes i made one mistake the first time i run fsck /dev/sdX i answered yes
to the fist question (autoreponse), then i saw the second question and
saw that it was the same issue as with the previous disk and cancel the
fsck and reported my issue to the list.




So what I will suggest at this point is that you do the following:

debugfs -w /dev/sda1
debugfs: features dir_index filetype sparse_super
debugfs: quit

The run "e2fsck -nf /dev/sda1" and make the output looks relatively
clean. You should *not* see any messages about needing to relocate
inode tables.

If so, you can then run "e2fsck -f /dev/sda1 to fully recover the
filesysten.

I always seem to get the impossible out of Linux tools, but most times this
is during quality tests... however this was on "normal usage". I hope it
has noting to do with the latest release changes or with corrupt binaries
on my client system.


Well, absolutely no one else reporting this problem or anything like
it...


Ok, this did not when so great...

e2fsck -nf /dev/sda1 > e2fsck-nf-info-sda1.txt 2>&1
e2fsck -fy /dev/sda1 > e2fsck-fy-info-sda1.txt 2>&1

http://www.powercraft.nl/temp/e2fsck-nf-info-sda1.txt.gz
http://www.powercraft.nl/temp/e2fsck-fy-info-sda1.txt.gz (log is of an
second resumed run)


root@ashley:/media/sda1# ls -hal
total 8.0K
drwxr-xr-x 3 root root 4.0K 2008-05-29 15:28 .
drwxr-xr-x 5 root root 200 2008-05-29 16:10 ..
drwx------ 2 root root 4.0K 2008-05-29 15:28 lost+found
root@ashley:/media/sda1# cd lost+found/
root@ashley:/media/sda1/lost+found# ls -hal
total 8.0K
drwx------ 2 root root 4.0K 2008-05-29 15:28 .
drwxr-xr-x 3 root root 4.0K 2008-05-29 15:28 ..
root@ashley:/media/sda1/lost+found# df -hal
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 688G 8.0K 653G 1% /media/sda1

nothing there anymore :-S :-S

I am going to restore the backup image back to sda1 it is 750GB so this
takes a while (14 hours)


Any ideas what went wrong?




_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 05-29-2008, 08:01 PM
Theodore Tso
 
Default needs help, root inode gone after usb bus reset on sata disks

On Thu, May 29, 2008 at 04:44:08PM +0200, Jelle de Jong wrote:
> Ok, this did not when so great...
>
> e2fsck -nf /dev/sda1 > e2fsck-nf-info-sda1.txt 2>&1
> e2fsck -fy /dev/sda1 > e2fsck-fy-info-sda1.txt 2>&1

OK, this makes no sense whatsoever. In the first pass, it complained
about the root inode being corrupted; that's fine, that was probably
from the initial hardware corruption.

The second time you ran e2fsck, it didn't complain about anything
until pass #5, so apparently the root inode was OK the second time
around. But when you run e2fsck with -n, it opens the device
read-only, so theres no way the filesystem could have changed.

If you didn't run any other commands or do anything else between the
two runs of e2fsck, then you have some serious hardware problem where
the disk is not returning the same data between the first and second
e2fsck run. And if you have wierd hardware problems, either with the
hard drive or how the hard drive is connected to the OS, there's
really nothing e2fsck can do to help you.....

- Ted

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 05-29-2008, 08:15 PM
Jelle de Jong
 
Default needs help, root inode gone after usb bus reset on sata disks

Theodore Tso wrote:

On Thu, May 29, 2008 at 04:44:08PM +0200, Jelle de Jong wrote:

Ok, this did not when so great...

e2fsck -nf /dev/sda1 > e2fsck-nf-info-sda1.txt 2>&1
e2fsck -fy /dev/sda1 > e2fsck-fy-info-sda1.txt 2>&1


OK, this makes no sense whatsoever. In the first pass, it complained
about the root inode being corrupted; that's fine, that was probably
from the initial hardware corruption.

The second time you ran e2fsck, it didn't complain about anything
until pass #5, so apparently the root inode was OK the second time
around. But when you run e2fsck with -n, it opens the device
read-only, so theres no way the filesystem could have changed.

If you didn't run any other commands or do anything else between the
two runs of e2fsck, then you have some serious hardware problem where
the disk is not returning the same data between the first and second
e2fsck run. And if you have wierd hardware problems, either with the
hard drive or how the hard drive is connected to the OS, there's
really nothing e2fsck can do to help you.....



no no, i think i did not give enough info sorry :-p

I did the following:

debugfs -w /dev/sda1
debugfs: features dir_index filetype sparse_super
debugfs: quit

then i run

e2fsck -nf /dev/sda1

to see if it still wanted to relocate inodes. This was not the case
anymore, however it still wanted to relocate the root inode...


I then run:

e2fsck -f /dev/sda1

and manual answer yes to the question until i had to enter a lot of "y"
(see logs) and killed the program with ctrl-c


then i run the following commands:

e2fsck -nf /dev/sda1 > e2fsck-nf-info-sda1.txt 2>&1
e2fsck -fy /dev/sda1 > e2fsck-fy-info-sda1.txt 2>&1

I am now restoring the backup so i we can try again....

Hope things make more sens now.


Peace,

Jelle

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 05-29-2008, 09:20 PM
Theodore Tso
 
Default needs help, root inode gone after usb bus reset on sata disks

On Thu, May 29, 2008 at 10:15:08PM +0200, Jelle de Jong wrote:
> I did the following:
>
> debugfs -w /dev/sda1
> debugfs: features dir_index filetype sparse_super
> debugfs: quit
>
> then i run
>
> e2fsck -nf /dev/sda1
>
> to see if it still wanted to relocate inodes. This was not the case
> anymore, however it still wanted to relocate the root inode...
>
> I then run:
>
> e2fsck -f /dev/sda1
>
> and manual answer yes to the question until i had to enter a lot of "y"
> (see logs) and killed the program with ctrl-c

what answers did you answer yes to? I don't have a log of your
"e2fsck -f /dev/sda1" run, and so I can't tell what happened. The
e2fsck -fy run you gave me was large, but information-free, since it
just had pass #5 messages regarding adjusting accounting information.

If it was just deleting the root inode (because it was corrupted), and
creating a new root inode, that doesn't explain why all of the inodes
disappeared, unless the inode table had somehow gotten completely
zero'ed out

At this point, what I would probably suggest is that you run

e2image -r /dev/hda1 - | bzip2 > hda1.e2i.bz2

... and put it someplace where I can download it and take a look at
what the heck happened to your filesystem.

By the way, please look at the "script" command ("man script"); it is
very handy for capturing a record of what an interactive session with
a program like e2fsck.

- Ted

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 06-06-2008, 06:24 PM
Jelle de Jong
 
Default needs help, root inode gone after usb bus reset on sata disks

Theodore Tso wrote:

On Thu, May 29, 2008 at 10:15:08PM +0200, Jelle de Jong wrote:

I did the following:

debugfs -w /dev/sda1
debugfs: features dir_index filetype sparse_super
debugfs: quit

then i run

e2fsck -nf /dev/sda1

to see if it still wanted to relocate inodes. This was not the case
anymore, however it still wanted to relocate the root inode...


I then run:

e2fsck -f /dev/sda1

and manual answer yes to the question until i had to enter a lot of "y"
(see logs) and killed the program with ctrl-c


what answers did you answer yes to? I don't have a log of your
"e2fsck -f /dev/sda1" run, and so I can't tell what happened. The
e2fsck -fy run you gave me was large, but information-free, since it
just had pass #5 messages regarding adjusting accounting information.

If it was just deleting the root inode (because it was corrupted), and
creating a new root inode, that doesn't explain why all of the inodes
disappeared, unless the inode table had somehow gotten completely
zero'ed out

At this point, what I would probably suggest is that you run


e2image -r /dev/hda1 - | bzip2 > hda1.e2i.bz2

... and put it someplace where I can download it and take a look at
what the heck happened to your filesystem.

By the way, please look at the "script" command ("man script"); it is
very handy for capturing a record of what an interactive session with
a program like e2fsck.



Thanks for all the info Ted,

http://www.powercraft.nl/temp/e2image-r-sda1-v0.1.1.e2i.bz2

I did some experimenting and to see if I can find some data on the disk
by doing the below command on an unaltered backup:


e2fsck -fy /dev/sda1 > e2fsck-fy-info-sda1-v0.1.1j.txt 2>&1

However no files where found, so maybe something when wrong with the dd
backup.


I don't now if there is a way to see if there is actual data on the disk.

So for now i am giving up on recovery the data, maybe you can get a glue
of what the hack happened to the file system and learn something new...


The only thing i would like to now is how to backup and restore the
filesystem. (for example i am going to setup a raid setup but this kind
of file system crashes are not covered with a raid setup)


Thanks in advance,

Jelle

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 

Thread Tools




All times are GMT. The time now is 06:52 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org