FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > EXT3 Users

 
 
LinkBack Thread Tools
 
Old 05-27-2008, 08:56 AM
Jelle de Jong
 
Default needs help, root inode gone after usb bus reset on sata disks

Hello everybody,

I am new to this list, so welcome everybody.

Last 2 week I had two harddisk crashes with my ext2 file system.

This is what sort of happed with both of the disk:

I pluged in my USB to SATA converter in my harddisk that has an ext2
filesystem. I mounted the partition, went to a directory that had a DVD
image. I mounted the dvd image in the same directory and started
watching the movie. After 40 minutes the movie stops.

After some investigation I saw the ISO image was not mounted anymore.
And dmesg was showing 3 USB bus reset log entries.

I rebooted the computer and tried to mount the usb ext2 disk partition
again. But it failed.

dmesg and a fsck showed messages about the root inode being gone!

fsck.ext2 -p /dev/sdd1 did not work manual run is needed.

On the first 500GB disk I did an fsck.ext2 -y /dev/sdd1 did did not
fixed my disk it had still errors, I lost 35% of my data, but the
partition was mountable again, and the files where in the lost+found
directory.

I don't want this to happen with the second 750GB harddisk, I would like
all my data back.

fsck.ext3 -n /dev/sdd1 > fsck-crash-info.txt 2>&1
http://filebin.ca/mczmks/fsck-crash-info.zip

What should I do? What commands do you want me to run to provide more
info? How can i restore my root inode?

Thanks in advance,

Jelle de Jong

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 05-27-2008, 11:01 AM
"Christian Kujau"
 
Default needs help, root inode gone after usb bus reset on sata disks

On Tue, May 27, 2008 10:56, Jelle de Jong wrote:
> image. I mounted the dvd image in the same directory and started watching
> the movie. After 40 minutes the movie stops.

Maybe someone on the list can tell you how the movie ends, if you tell us
the title :-)

> dmesg and a fsck showed messages about the root inode being gone!
> fsck.ext2 -p /dev/sdd1 did not work manual run is needed.
> On the first 500GB disk I did an fsck.ext2 -y /dev/sdd1 did did not

Do you still have the logs from these fsck runs? Might be interesting what
the exact errors were...

> fsck.ext3 -n /dev/sdd1 > fsck-crash-info.txt 2>&1
> http://filebin.ca/mczmks/fsck-crash-info.zip

Hm, filebin.ca times out, any chance to put this fsck-crash-info.txt
somewhere else? (no need to zip, gzip will do fine...)

> What should I do? What commands do you want me to run to provide more
> info? How can i restore my root inode?

500GB, 750GB...and no backups? Ouch :-

Christian.
--
make bzImage, not war

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 05-27-2008, 11:25 AM
Jelle de Jong
 
Default needs help, root inode gone after usb bus reset on sata disks

Christian Kujau wrote:

On Tue, May 27, 2008 10:56, Jelle de Jong wrote:

image. I mounted the dvd image in the same directory and started watching
the movie. After 40 minutes the movie stops.


Maybe someone on the list can tell you how the movie ends, if you tell us
the title :-)


I also had the dvd on disk, it was Memento. But I am not so lucky with
all the other data on the disk.


dmesg and a fsck showed messages about the root inode being gone!
fsck.ext2 -p /dev/sdd1 did not work manual run is needed.
On the first 500GB disk I did an fsck.ext2 -y /dev/sdd1 did did not


Do you still have the logs from these fsck runs? Might be interesting what
the exact errors were...


The logs will be almost exactly the same as with the second disk (see
the gziped file)


fsck.ext3 -n /dev/sdd1 > fsck-crash-info.txt 2>&1
http://filebin.ca/mczmks/fsck-crash-info.zip


Hm, filebin.ca times out, any chance to put this fsck-crash-info.txt
somewhere else? (no need to zip, gzip will do fine...)


http://www.powercraft.nl/temp/fsck-crash-info.txt.gz


What should I do? What commands do you want me to run to provide more
info? How can i restore my root inode?


500GB, 750GB...and no backups? Ouch :-


Indeed, I really hope we can solve it.

If I need to buy an additional 750G disk to make a backup dd image
please tell me, it will make a hole in my wallet but if it is necessary...



Christian.


Thanks in advance,

Jelle


_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 05-27-2008, 12:23 PM
"Christian Kujau"
 
Default needs help, root inode gone after usb bus reset on sata disks

On Tue, May 27, 2008 13:25, Jelle de Jong wrote:
> http://www.powercraft.nl/temp/fsck-crash-info.txt.gz

So, you've ran fsck but did not try to repair yet, right? If so and you do
happen to have a spare 750 GB *now*, try to dd(1) your data to this spare
disk: then you can fsck your filesystems as many times as you want to.

Are there any (USB-)device related errors in the syslogs? We don't wanna
run fsck when the underlying device is unstable. dd(1) would be a good way
to find out.

Also, is LVM or RAID or sth. like this involved? Searching the net for
these errors brought up quite a few hits related to LVM b0rkage...

However, let's hope some ext3 hacker will comment on the logs...

C.
--
make bzImage, not war

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 05-27-2008, 12:47 PM
Theodore Tso
 
Default needs help, root inode gone after usb bus reset on sata disks

On Tue, May 27, 2008 at 10:56:32AM +0200, Jelle de Jong wrote:
>
> I pluged in my USB to SATA converter in my harddisk that has an ext2
> filesystem. I mounted the partition, went to a directory that had a DVD
> image. I mounted the dvd image in the same directory and started
> watching the movie. After 40 minutes the movie stops.

Were you doing anything else on the computer; where there any write
operations taking place? If you were just reading from the
filesystem, the fact that your filesystem was that badly damaged makes
me deeply suspicious about your USB to SATA converter.

> fsck.ext2 -p /dev/sdd1 did not work manual run is needed.
>
> On the first 500GB disk I did an fsck.ext2 -y /dev/sdd1 did did not
> fixed my disk it had still errors, I lost 35% of my data, but the
> partition was mountable again, and the files where in the lost+found
> directory.

It looks like garbage was written into your block group descriptors,
but since the superblock looked OK, e2fsck -y tried its best, but in
this case it may have done more harm than good. (In general, if you
see e2fsck asking permission to relocate an inode table; there's
something very wrong, and you probably want to say 'n' and do an image
level backup of the filesystem before proceeding.)

> I don't want this to happen with the second 750GB harddisk, I would like
> all my data back.

Well, there's no guarantee the same corruption will have taken place
on your other hard drive. Running e2fsck -n on that second hard drive
and letting an expert examine it would be a good first step, *before*
blindly running e2fsck -y.

In the next version of e2fsprogs (in development in the git
repository), e2fsck will have the ability to create an "undo" log
which will make e2fsck -y safer, but personally I've always liked to
individually hit return to say 'yes' to each >question.

> fsck.ext3 -n /dev/sdd1 > fsck-crash-info.txt 2>&1
> http://filebin.ca/mczmks/fsck-crash-info.zip
>
> What should I do? What commands do you want me to run to provide more
> info? How can i restore my root inode?

So this is from your 500GB disk, as I understand it, right? I'd
really need to see the results of "e2fsck -n" *before* you ran "e2fsck
-y" but seeing what I see there, taking an image-level backup before
you had begun would have been really good idea.

I'm not sure there's anythign you'll be able to do about restoring
your root inode. But if it was just the root inode that was
destroyed, that's actually not a big deal; you'll just have files in
lost+found, and you can usually piece together the root directory
fairly easily.

The bigger problem is the other parts of the filesystem that were
corrupted, due to what was apparently a hardware failure. I'm
actually really not a fan of USB as an interconnect for disks, because
the cables can be flakey; it's not that hard for them to come lose,
which may have been what caused your USB<->SATA converter to flake
out, but it apparently did so in a very spectacular fashion.

When I have time I'll have to add a better automated hueristic to
e2fsck try to do this automatically (although even when I make e2fsck
-y smarter, there *still* will be cases where a human with experience
and intelligence and common sense will do better than a program), but
for now, if you see a message about wanting to relocate an inode
table, you'll want to look at the output of "dumpe2fs /dev/sdXX",
"dumpe2fs -o superblock=32768 /dev/sdXX", and "dumpe2fs -o
superblock=98304 /dev/sdXX" (these numbers are assuming a 4k
blocksize, which is the common default). If the location of the inode
table blocks makes more sense when dumpe2fs is told to look at the
backup superblock at 32768, it may be that e2fsck -b 32768 /dev/sdXX
will do a better job of recovering the filesystem.

- Ted

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 05-27-2008, 12:52 PM
Jelle de Jong
 
Default needs help, root inode gone after usb bus reset on sata disks

Christian Kujau wrote:

On Tue, May 27, 2008 13:25, Jelle de Jong wrote:

http://www.powercraft.nl/temp/fsck-crash-info.txt.gz


So, you've ran fsck but did not try to repair yet, right? If so and you do
happen to have a spare 750 GB *now*, try to dd(1) your data to this spare
disk: then you can fsck your filesystems as many times as you want to.


I have not run fsck to repair with the previous disk this when wrong, so
I will buy a spare 750 GB disk today, and make a dd image from the
one disk to the other.



Are there any (USB-)device related errors in the syslogs? We don't wanna
run fsck when the underlying device is unstable. dd(1) would be a good way
to find out.


I have seen the usb bus reset errors before but I cant put my finger on
the broken part. I have now been copying data from one usb disk to an
other usb disk without any usb error messages and this is a 4 hour transfer.



Also, is LVM or RAID or sth. like this involved? Searching the net for
these errors brought up quite a few hits related to LVM b0rkage...


No RAID setup, but i am planning a software raid setup soon.


However, let's hope some ext3 hacker will comment on the logs...


yes lets hope so...


C.


_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 05-27-2008, 01:09 PM
Jelle de Jong
 
Default needs help, root inode gone after usb bus reset on sata disks

Theodore Tso wrote:

On Tue, May 27, 2008 at 10:56:32AM +0200, Jelle de Jong wrote:

I pluged in my USB to SATA converter in my harddisk that has an ext2
filesystem. I mounted the partition, went to a directory that had a DVD
image. I mounted the dvd image in the same directory and started
watching the movie. After 40 minutes the movie stops.


Were you doing anything else on the computer; where there any write
operations taking place? If you were just reading from the
filesystem, the fact that your filesystem was that badly damaged makes
me deeply suspicious about your USB to SATA converter.


There was nothing else going on then watching a DVD form the disk. It
may have been an usb issue, but I am using the same sata usb converter
for several hours now without any problem. But an usb converter / power
failure should not be able to create so much damage when just reading
files...





fsck.ext2 -p /dev/sdd1 did not work manual run is needed.

On the first 500GB disk I did an fsck.ext2 -y /dev/sdd1 did did not
fixed my disk it had still errors, I lost 35% of my data, but the
partition was mountable again, and the files where in the lost+found
directory.


It looks like garbage was written into your block group descriptors,
but since the superblock looked OK, e2fsck -y tried its best, but in
this case it may have done more harm than good. (In general, if you
see e2fsck asking permission to relocate an inode table; there's
something very wrong, and you probably want to say 'n' and do an image
level backup of the filesystem before proceeding.)


I don't want this to happen with the second 750GB harddisk, I would like
all my data back.


Well, there's no guarantee the same corruption will have taken place
on your other hard drive. Running e2fsck -n on that second hard drive
and letting an expert examine it would be a good first step, *before*
blindly running e2fsck -y.

>

In the next version of e2fsprogs (in development in the git
repository), e2fsck will have the ability to create an "undo" log
which will make e2fsck -y safer, but personally I've always liked to
individually hit return to say 'yes' to each >question.


fsck.ext3 -n /dev/sdd1 > fsck-crash-info.txt 2>&1
http://filebin.ca/mczmks/fsck-crash-info.zip

What should I do? What commands do you want me to run to provide more
info? How can i restore my root inode?


So this is from your 500GB disk, as I understand it, right? I'd
really need to see the results of "e2fsck -n" *before* you ran "e2fsck
-y" but seeing what I see there, taking an image-level backup before
you had begun would have been really good idea.


The log is of the second hard drive. I don't have a log of the first
hard drive, but it had very very similar outputs. Going to create an
image and hope an expert can tell me how to try fixing the file system.



I'm not sure there's anythign you'll be able to do about restoring
your root inode. But if it was just the root inode that was
destroyed, that's actually not a big deal; you'll just have files in
lost+found, and you can usually piece together the root directory
fairly easily.

The bigger problem is the other parts of the filesystem that were
corrupted, due to what was apparently a hardware failure. I'm
actually really not a fan of USB as an interconnect for disks, because
the cables can be flakey; it's not that hard for them to come lose,
which may have been what caused your USB<->SATA converter to flake
out, but it apparently did so in a very spectacular fashion.


The reason i used usb connections is power saving, just plug in the hard
drive you need. I think I will have an closer look at a placing my
harddrives in my server finding some way to hot-swap hot-powerplug
drivers enable and disable the power to harddrivers.



When I have time I'll have to add a better automated hueristic to
e2fsck try to do this automatically (although even when I make e2fsck
-y smarter, there *still* will be cases where a human with experience
and intelligence and common sense will do better than a program), but
for now, if you see a message about wanting to relocate an inode
table, you'll want to look at the output of "dumpe2fs /dev/sdXX",
"dumpe2fs -o superblock=32768 /dev/sdXX", and "dumpe2fs -o
superblock=98304 /dev/sdXX" (these numbers are assuming a 4k
blocksize, which is the common default). If the location of the inode
table blocks makes more sense when dumpe2fs is told to look at the
backup superblock at 32768, it may be that e2fsck -b 32768 /dev/sdXX
will do a better job of recovering the filesystem.


dumpe2fs /dev/sdXX
dumpe2fs -o superblock=32768 /dev/sdXX
dumpe2fs -o superblock=98304 /dev/sdXX
e2fsck -b 32768 /dev/sdXX

Sound like a lot of experimentation, so I am going to make a backup first.

I do not have an journaling system on my disk, would it have been a lot
saver to have journaling on usb disk? and what about an auto sync option
flag for usb disks?


Thank you for the information Ted,

Jelle

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 05-28-2008, 02:44 PM
Jelle de Jong
 
Default needs help, root inode gone after usb bus reset on sata disks

Jelle de Jong wrote:

Theodore Tso wrote:

On Tue, May 27, 2008 at 10:56:32AM +0200, Jelle de Jong wrote:

I pluged in my USB to SATA converter in my harddisk that has an ext2
filesystem. I mounted the partition, went to a directory that had a DVD
image. I mounted the dvd image in the same directory and started
watching the movie. After 40 minutes the movie stops.


Were you doing anything else on the computer; where there any write
operations taking place? If you were just reading from the
filesystem, the fact that your filesystem was that badly damaged makes
me deeply suspicious about your USB to SATA converter.


There was nothing else going on then watching a DVD form the disk. It
may have been an usb issue, but I am using the same sata usb converter
for several hours now without any problem. But an usb converter / power
failure should not be able to create so much damage when just reading
files...





fsck.ext2 -p /dev/sdd1 did not work manual run is needed.

On the first 500GB disk I did an fsck.ext2 -y /dev/sdd1 did did not
fixed my disk it had still errors, I lost 35% of my data, but the
partition was mountable again, and the files where in the lost+found
directory.


It looks like garbage was written into your block group descriptors,
but since the superblock looked OK, e2fsck -y tried its best, but in
this case it may have done more harm than good. (In general, if you
see e2fsck asking permission to relocate an inode table; there's
something very wrong, and you probably want to say 'n' and do an image
level backup of the filesystem before proceeding.)


I don't want this to happen with the second 750GB harddisk, I would like
all my data back.


Well, there's no guarantee the same corruption will have taken place
on your other hard drive. Running e2fsck -n on that second hard drive
and letting an expert examine it would be a good first step, *before*
blindly running e2fsck -y.

>

In the next version of e2fsprogs (in development in the git
repository), e2fsck will have the ability to create an "undo" log
which will make e2fsck -y safer, but personally I've always liked to
individually hit return to say 'yes' to each >question.


fsck.ext3 -n /dev/sdd1 > fsck-crash-info.txt 2>&1
http://filebin.ca/mczmks/fsck-crash-info.zip

What should I do? What commands do you want me to run to provide more
info? How can i restore my root inode?


So this is from your 500GB disk, as I understand it, right? I'd
really need to see the results of "e2fsck -n" *before* you ran "e2fsck
-y" but seeing what I see there, taking an image-level backup before
you had begun would have been really good idea.


The log is of the second hard drive. I don't have a log of the first
hard drive, but it had very very similar outputs. Going to create an
image and hope an expert can tell me how to try fixing the file system.



I'm not sure there's anythign you'll be able to do about restoring
your root inode. But if it was just the root inode that was
destroyed, that's actually not a big deal; you'll just have files in
lost+found, and you can usually piece together the root directory
fairly easily.

The bigger problem is the other parts of the filesystem that were
corrupted, due to what was apparently a hardware failure. I'm
actually really not a fan of USB as an interconnect for disks, because
the cables can be flakey; it's not that hard for them to come lose,
which may have been what caused your USB<->SATA converter to flake
out, but it apparently did so in a very spectacular fashion.


The reason i used usb connections is power saving, just plug in the hard
drive you need. I think I will have an closer look at a placing my
harddrives in my server finding some way to hot-swap hot-powerplug
drivers enable and disable the power to harddrivers.



When I have time I'll have to add a better automated hueristic to
e2fsck try to do this automatically (although even when I make e2fsck
-y smarter, there *still* will be cases where a human with experience
and intelligence and common sense will do better than a program), but
for now, if you see a message about wanting to relocate an inode
table, you'll want to look at the output of "dumpe2fs /dev/sdXX",
"dumpe2fs -o superblock=32768 /dev/sdXX", and "dumpe2fs -o
superblock=98304 /dev/sdXX" (these numbers are assuming a 4k
blocksize, which is the common default). If the location of the inode
table blocks makes more sense when dumpe2fs is told to look at the
backup superblock at 32768, it may be that e2fsck -b 32768 /dev/sdXX
will do a better job of recovering the filesystem.


dumpe2fs /dev/sdXX
dumpe2fs -o superblock=32768 /dev/sdXX
dumpe2fs -o superblock=98304 /dev/sdXX
e2fsck -b 32768 /dev/sdXX

Sound like a lot of experimentation, so I am going to make a backup first.

I do not have an journaling system on my disk, would it have been a lot
saver to have journaling on usb disk? and what about an auto sync option
flag for usb disks?


So, it took 14 hours to pump over the 750G to an other disk, but i hope
it went ok. I executed the below command created the logs and hope
somebody can tell me what to do next?


dd if=/dev/sdb of=/dev/sda > dd-run-info.txt 2>&1
dumpe2fs /dev/sda1 > dumpe2fs-info-sda1.txt 2>&1
dumpe2fs -ob 32768 /dev/sda1 > dumpe2fs-32768-info-sda1.txt 2>&1
dumpe2fs -ob 98304 /dev/sda1 > dumpe2fs-98304-info-sda1.txt 2>&1
e2fsck -b 32768 /dev/sda1 (need terminal for interactive repairs)

http://www.powercraft.nl/temp/fsck-crash-info.txt.gz
http://www.powercraft.nl/temp/dumpe2fs-info-sda1.txt.gz
http://www.powercraft.nl/temp/dumpe2fs-32768-info-sda1.txt.gz
http://www.powercraft.nl/temp/dumpe2fs-98304-info-sda1.txt.gz

Thanks in advance,

Jelle

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 05-28-2008, 11:24 PM
Theodore Tso
 
Default needs help, root inode gone after usb bus reset on sata disks

On Wed, May 28, 2008 at 04:44:21PM +0200, Jelle de Jong wrote:
>> dumpe2fs -o superblock=32768 /dev/sdXX

I asked you to do the above, but you did this instead:

> dumpe2fs -ob 32768 /dev/sda1 > dumpe2fs-32768-info-sda1.txt 2>&1

Resulting in this:

dumpe2fs: No such file or directory while trying to open 32768

So I can't tell if the backup superblock was corrupted, but this is
definitely one for the record books. Looking at primary superblock,
we see the following:

dumpe2fs 1.40-WIP (14-Nov-2006)
Filesystem volume name: <none>
Last mounted on: ^^<BA><8B>
Filesystem UUID: 2e27ae79-fc96-43f5-9758-15ed74dd55fb
Filesystem magic number: 0xEF53
Filesystem revision #: 0 (original)
Filesystem features: (none)
Default mount options: MNTOPT_15 MNTOPT_16 MNTOPT_18 MNTOPT_20 MNTOPT_21 MNTOPT_22 MNTOPT_24 MNTOPT_26

The above, especially the Filesystem features, and default mount
options, are garbage. But it looks like the rest of the superblock,
including the magic number, the block counts, etc., look sane --- at
least in sane enough that it passed e2fsck's sanity checking.

This is unlike *any* corruption I've seen before; usually there will
be a single bit flip, or the entire disk sector is corrupted, but it's
extremely rare to see this kind of selective corruption.

It's even wierder that this apparently happened on more than one hard
drive. In any case, I would ditch that USB<->SATA converter as fast
as possible, because there is something seriously wrong. The other
possibility is that you're running with buggy kernel, but no one else
has ever reported anything like this, and for two disks to be
corrupted the same way means it's unlikely to be caused by a random
wild pointer or some such. So if I really had to guess I'd go with
the USB converter, but that's not for certain.

In terms of how to fix it, I'd would have to see the results of

dumpe2fs -o superblock=32768 /dev/sdXX

and

dumpe2fs -o superblock=98304 /dev/sdXX

Hopefully one of the superblocks look OK. We could also try manually
repairing the superblock with debugfs, in the worse case, but it'll be
easier if we can fix things via the backup superblock.

- Ted

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 05-28-2008, 11:50 PM
Theodore Tso
 
Default needs help, root inode gone after usb bus reset on sata disks

Oh, so I forgot to mention the main thing which caused e2fsck to
report that various inode tables needed to be moved. The filesystem
feature field was zero'ed out, and looking at the dumpe2fs output,
it's clear the sparse_super feature should have been enabled.

The backup superblocks should have that feature set; if so, running
e2fsck telling it to pay attention to one of the backup superblocks
should address the problem.

Again, the really funny thing was how the superblock got corrupted in
such a funny and specific way. It's almost as if it was corrupted by
Murphy (as in Murphy's Law) himself.

- Ted

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 

Thread Tools




All times are GMT. The time now is 04:51 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org