Linux Archive

Linux Archive (
-   EXT3 Users (
-   -   Overwritten beginning of ext3 filesystem. Recovery? (

Florian Weber 12-27-2010 03:37 PM

Overwritten beginning of ext3 filesystem. Recovery?
Hello list

I accidentally trashed the first ~10-20GB of a 1TB ext3 filesystem with
a heedless RAID1 rebuild (excruciating detail below). I'm now looking for
options to get as much as possible of the remaining data back.

I've been searching the web for over a day now but all my results are
either not

what I need (MBR, partition table and superblock are OK) or too lowlevel
(revocering many thousands of nameless and structureless mails/jpgs/docs just
doesn't cut it here, IMVHO).
My main problem is not that I accidentally deleted files, but that
basically my

/ directory just went "poof" and left the rest sitting around.

Since the damaged filesystem was clean before my accident, I'm figuring I just
might get most of the data back: even much of the directory structure should
still be there if I only knew how to get at it.

I'd be most grateful for any tips, tools, or even documentation to aid in
writing my own tool.

Thans in advance for your time,
Florian Weber

PS: that _was_ my backup :-( Thanks for not mentioning it.



Starting point:
I've been running the following setup on my machine:
* Two same-size harddisks, currently 1TB, one big partition each -->
sda[1], sdb[1]

* Linux software RAID1 consisting of these partitions --> md0
* A single ext3 filesystem, default parameters, reserved blocks lowered to 1%
* All system and data inside this single partition, ca. 350-400GB
* (Much too) infrequent backups ... yes, yes, I know, I know ...

After many years, I wanted to move from Gentoo to KUbuntu. No big deal:
* Shutdown PC, pull disk sdb from the RAID
* Install Ubuntu on sda as if working on a blank disk (setup as above, with
one of the RAID1 disks physically missing during the install)
* Boot the new system from sda, still in degraded mode
* Treating sdb like a standalone ext3 disk: mount, copy configs and /home,
* Get the system into working order (config files reconciled, all applications
* Determine that the "old" stuff is not needed anymore
* Put sdb back into the RAID1 and rebuild

What went wrong:
Before the initial shutdown, I did not change the partition type on sdb from
0xFD to 0x83 to prevent RAID autodetection. Booting with sdb
reattached (to get
at my personal data) would therefore (correctly) have resulted in a
RAID rebuild --> very bad.

So I figured: I'll attach the disk, boot with "raid=noautodetect" in the
kernel commandline, and I'll be fine. But: unlike my previous setup,
Ubuntu has

a silent bootloader and I missed my chance to enter the commandline.

And the RAID instantly started rebuilding itself onto my backup disk :-O

I quickly realised what was happening and cleanly shut down my
system (incurring some additional damage from the running rebuild, but
the worst

was already done). Total running time was about 3 minutes, in parallel to the
system booting up and shutting down.

What I have now:
* A working, new Ubuntu installation on a degraded RAID1 array, without
personal data. I'm currently typing on this system.

* A harddisk (sdb) that previously contained a working system with a total of
350-400GB data, but was subject to a RAID1 rebuild for <3-4 minutes
at <=100MB/sec. The disc is not connected at the moment.

* The MBR on sdb is the new one. That's OK.
* The partition table on sdb is the new one. It looks identical to the
old one.
* The ext3 superblock on sdb1 is the new one. It's basically the same
as the old
one. I compared it against one of the (old) backup superblocks at the
end of the

* I have a dd image of partition sdb1
* I can mount the image of sdb1 and do an ls. I see data from the new
system. Much content is missing, obviously, since it was not synced
over yet

* I can "fsck -n" the image of sdb1. Many errors of course ("inode contains
invalid block", "too many illegal blocks", "i_size wrong", "i_Blocks wrong"),
since much stuff was not synced over yet
* At some point, "fsck -n" stops with "illegal indirect block"
* I have not yet tried to "fsck -y". That would be my next step.

* I have 1TB of free space available and can organise more

I do realise this is not for the faint of heart, but I'm done with my
fainting for this instance ;-)

Still with hope,
Florian Weber

Buergernetz Pfaffenhofen Webmail -

Ext3-users mailing list

Florian Weber 02-06-2011 04:23 PM

Overwritten beginning of ext3 filesystem. Recovery?
Hello list

For the benefit of those searching the archives, here's how I got out of the
mess described in my first mail on27dec2010.

0a. Be extra careful and use your brain before even touching the keyboard. Go
to extreme measures to prevent typos.
0b. Keep backups: make a dd copy of your disk/partition and put the original
hardware into a safe. Do not work on the master image. Make it read-only,
create _another_ working copy and use _that_ for recovery. You will make
mistakes and you do not want to touch the hardware.
0c. Be prepared to learn a lot of things about filesystems you never wanted to
0c. Whatevery you do, check the units your tools are using. Each time. They
might be filesystem blocks, disk blocks, inodes, bytes, kB, kiB, ....
0d. Have a look at the thread "recovery recommendations" started by "m.p." on

1. I overwrote the stuff I definitely knew to be faulty with zeroes, i.e. the
first 10GB. I erred on the safe side, rather keeping bogus stuff than deleting
good data.
2. This action killed my partition table. I had to restore it manually but was
prepared for that.
3. I ran e2fsck on the broken partition. After the first run, it prompted me to
be run again, which I did.
4. The filesystem now contained lost+found/ as it's only toplevel directory.
Below that were many files and directories
5. I sorted those files/dirs and gave them meaningful names instead of block
numbers. About 70% were recognizable from their content, among it *all* my
personal data! The rest was unrecognizable binary and ASCII fragments which I
discarded, there's a high likelyhood that most of it was actually deleted in
the old filesystem, and it couldn't be restored anyway.
6. I compared against very old backups which showed no data loss.
7. I'm still doing lots of random samples to check for damaged files and loss
of newer files, but found none so far.

I got all my data back, but that was pure luck.
To account not only for hardware but also for software and human failures, I
have bought a USB harddisk which I use for weekly backups. I'm still
evaluating which backup tools best fit my needs.

Hope this helps someone.

With best regards,
Florian Weber

Ext3-users mailing list

All times are GMT. The time now is 09:49 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.