ext3 filesystem is not recognized
 

Hello all,

I have a few ext3 file systems that are not being recognized.

Here is the setup: MD software RAID 5 on 4 disks (md0), a LVM logical
volume (/dev/volume_group/logical_volume) comprised of one physical
device (/dev/md0), a encryption layer provided by the cryptoloop driver
(losetup -e aes /dev/loop0 /dev/volume_group/logical_volume), then a
EXT3 file system (mkfs.ext3 /dev/loop0).

Recently the RAID device kicked out one of the disks during a large file
transfer. After re-adding the disk to the array (smartctl didn't report
anything wrong with it, I am not sure why this happened), authenticating
against the cryptographic layer, then trying to mount the drive, I get
the following error:

[root@storage redhat]# mount -t ext3 /dev/loop1 /terrorbyte/1/
mount: wrong fs type, bad option, bad superblock on /dev/loop1,

The message in /var/log/message is:
VFS: Can't find ext3 filesystem on dev loop1.

I then tried to e2fsck the /dev/loop1 partition with all of the
different blocks that were reported from:
mke2fs -n /dev/loop1
with no luck still.

I am unsure of where the problem actually is, and how to go about
debugging it. Any suggestions would be appreciated.

Sincerely,
Dennison Williams
--
************************************************** ***************
* To communicate with me securely, please email me and I will *
* send you my public key. We can then verify each others *
* fingerprints in person, or over the phone. *
* *
* I am open and willing to talk about setting up PGP, the *
* security problems inherent with PGP, and alternatives to PGP *
* for secure electronic communication. *
************************************************** ***************

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

ext3 filesystem is not recognized
 

Given what* you've described, then only drive that it would make sense to pull out would be the one that was dropped and then re-inserted.


On Jan 4, 2008 10:20 PM, Dennison Williams <
evoltech@2inches.com> wrote:

> Did you try and re-insert the kicked-out drive as if it was clean, or did
> you try to re-sync it to the existing filesystem. *If the former, then
> that's a HUGE mistake because the data on the drive is no longer in sync

> with what is on the other drives. (unless the entire filesystem was made
> read-only when (or before) the drive was dropped out.)

I re-inserted it with:
mdadm /dev/md0 --add /dev/sde
At which point it seemed to resync with the raid device (ie. the output

of /proc/mdstat showed that it was incrementally syncing)

> Check the SMART logs for each of the drives to see if they've had any
> problems.

there are messages like this:

/dev/sdc, failed to read SMART Attribute Data
...but this wasn't one of the disks that was removed from the raid deviceIf there are complaints about SDC, then I'd be inclined to do a long test of it

in smart. it's possible that the real problem started here.

A badblock read test (or just a dd if=/dev/sdc of=/dev/null) would also test the I/O path between the drive and the CPU. If there are complaints about that drive, then .. at this point, you should consider it suspicious.

*
>
> Try pullling the (candidate) compromized drive out of the array and see if
> the (degraded) filesystem works OK and has good data. *If it does, then I'd
> guess that the pulled drive had bad data written to it somehow --- re-add it

> (as if it was hot-swapped in), and hope it doesn't happen again.
> Try that with each of the *drives, in turn until you find the badly written
> drive. *If one of the drives has badly written data, the system really can't

> tell, for sure, which one is wrong.

I want to make sure I understand you here. *Say my raid device is

comprised of for devices /dev/md0 = /dev/sd[abcd], are you sugesting
that for each drive I do somthing like this:

mdadm /dev/md0 --fail /dev/sda --remove /dev/sdaDon't bother. If the drive got resynced, then pulling it won't do any good unless software RAID gets silently confused by random data on one plex,



then try to mount up the FS as usual to see if it is there? *Wouldn't

this point be moot if the device already re-assembled itself?
Yes. it would be moot.


>
> [[ unless the array was read-only when the drive was dropped, then you will
> only have any hope of good data with the dropped drive pulled ]]

It wasn't read-only, but nothing was writing to it.


Thanks for your time and prompt response.
Sincerely,
Dennison Williams

Unless noatime was set, then the drive was being written to (if only atime data).* if all that got scrambled was atime data you should still have been able to mount the drive.


--
Stephen Samuel http://www.bcgreen.com
778-861-7641
_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

ext3 filesystem is not recognized
 

Stephen Samuel wrote:
> Given what you've described, then only drive that it would make sense to
> pull out would be the one that was dropped and then re-inserted.

I did this with the following set of commands:
mdadm -S /dev/md1
mdadm -A /dev/md1 /dev/sdf /dev/sdg /dev/sdh
mdadm --run /dev/md1
lvchange -a y /dev/volume_group/logical_volume
losetup -e aes /dev/loop1 /dev/volume_group/logical_volume
mount -t ext3 -o ro /dev/loop1 /mnt/logical_volume

and got the same errors; "mount: wrong fs type, bad option, bad
superblock on /dev/loop1"

>>> Check the SMART logs for each of the drives to see if they've had any
>>> problems.
>> there are messages like this:
>> /dev/sdc, failed to read SMART Attribute Data
>> ...but this wasn't one of the disks that was removed from the raid device
>
> If there are complaints about SDC, then I'd be inclined to do a long test of
> it
> in smart. it's possible that the real problem started here.
>
> A badblock read test (or just a dd if=/dev/sdc of=/dev/null) would also test
> the I/O path between the drive and the CPU. If there are complaints about
> that drive, then .. at this point, you should consider it suspicious.

Ran "dd if=/dev/sdc of=/dev/null" while monitoring /var/log/messages,
with no messages. Must have been a fluke. I will try doing a extended
run of smartctl.

>> Try pullling the (candidate) compromized drive out of the array and see if
>> the (degraded) filesystem works OK and has good data. If it does, then
> I'd
>> guess that the pulled drive had bad data written to it somehow --- re-add
> it
>> (as if it was hot-swapped in), and hope it doesn't happen again.
>> Try that with each of the drives, in turn until you find the badly
> written
>> drive. If one of the drives has badly written data, the system really
> can't
>> tell, for sure, which one is wrong.

I ended up doing this with each drive as above and still the FS wasn't
recognized.

One thing that confuses me though is that the data seems to be partially
valid. When the array device is assembled and running the logical
volume is recognized, and furthermore losetup accepts the correct
password. The only thing that doesn't seem to be in working order is
the ext3 filesystem.

In the linux encryption howto
(http://encryptionhowto.sourceforge.net/Encryption-HOWTO-6.html, section
6.1), there is a entry describing possible problems if the kernel was
compiled without CONFIG_BLK_LOOP_DEV_USE_REL_BLOCK. I can't find this
option anywhere in the config for my kernel (2.6.18-1.2798.fc6xen). At
this point I am thinking that the problem is at the cryptoloop or ext3
level, but I am not sure what else I can do to check. Any more ideas?

Sincerely,
Dennison Williams

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users