emergency call for help: raid5 fallen apart
 

I also sent this one to the linux-raid-ml, I am in trouble:

Sorry for maybe FAQing, I am in emergency mode:

customer server, RAID5 + hotspare, 4 drives ...

gentoo Linux version 2.6.25-gentoo-r7

mdadm 2.6.4-r1 here

-

one of the 4 drives showed massive errors in dmesg, /dev/sdc
SMART-errors etc.
bought new drive and wanted to swap today.

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4]
md1 : active raid1 sdb1[1] sda1[0]
104320 blocks [2/2] [UU]

md3 : active raid5 sdb3[1] sda3[0]
19550976 blocks level 5, 64k chunk, algorithm 2 [3/2] [UU_]

md4 : inactive sdb4[1](S) sdd4[3](S) sdc4[2](S) sda4[0](S)
583641088 blocks



-


I did:

mdadm /dev/md3 --fail /dev/sdc3

went OK

mdadm /dev/md4 --remove /dev/sdc3

OK as well, raid md3 rebuilt

-

With md4 I was too aggressive maybe:

mdadm /dev/md4 --fail /dev/sdc4 --remove /dev/sdc4

this rendered md4 unusable, even after a reboot it can't be reassambled.

This is bad, to say the least.

md4 : inactive sdb4[1](S) sdd4[3](S) sdc4[2](S) sda4[0](S)
583641088 blocks


What to try?

This is a crucial server and I feel a lot of pressure.
Rebuilding that raid would mean a lot of restore-work etc.
So I would really appreciate a goo advice here.

THANKS!

Stefan

emergency call for help: raid5 fallen apart
 

I also sent this one to the linux-raid-ml, I am in trouble:


Am 24.02.2010 15:54, schrieb Stefan G. Weichinger:

> What to try?
>
> This is a crucial server and I feel a lot of pressure.
> Rebuilding that raid would mean a lot of restore-work etc.
> So I would really appreciate a goo advice here.

Followup:

--examine shows different statii for the four partitions:


server-gentoo ~ # mdadm --examine /dev/sda4
/dev/sda4:
Magic : a92b4efc
Version : 00.90.00
UUID : d4b0e9c1:067357ce:2569337e:e9af8bed
Creation Time : Tue Aug 5 14:14:16 2008
Raid Level : raid5
Used Dev Size : 145910272 (139.15 GiB 149.41 GB)
Array Size : 291820544 (278.30 GiB 298.82 GB)
Raid Devices : 3
Total Devices : 4
Preferred Minor : 4

Update Time : Wed Feb 24 15:33:37 2010
State : active
Active Devices : 2
Working Devices : 3
Failed Devices : 1
Spare Devices : 1
Checksum : 3039381e - correct
Events : 0.13

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 0 8 4 0 active sync /dev/sda4

0 0 8 4 0 active sync /dev/sda4
1 1 8 20 1 active sync /dev/sdb4
2 2 0 0 2 faulty removed
3 3 8 52 3 spare /dev/sdd4


server-gentoo ~ # mdadm --examine /dev/sdb4
/dev/sdb4:
Magic : a92b4efc
Version : 00.90.00
UUID : d4b0e9c1:067357ce:2569337e:e9af8bed
Creation Time : Tue Aug 5 14:14:16 2008
Raid Level : raid5
Used Dev Size : 145910272 (139.15 GiB 149.41 GB)
Array Size : 291820544 (278.30 GiB 298.82 GB)
Raid Devices : 3
Total Devices : 4
Preferred Minor : 4

Update Time : Wed Feb 24 15:37:05 2010
State : clean
Active Devices : 1
Working Devices : 2
Failed Devices : 1
Spare Devices : 1
Checksum : 3039393f - correct
Events : 0.32

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 1 8 20 1 active sync /dev/sdb4

0 0 0 0 0 removed
1 1 8 20 1 active sync /dev/sdb4
2 2 0 0 2 faulty removed
3 3 8 52 3 spare /dev/sdd4


server-gentoo ~ # mdadm --examine /dev/sdc4
/dev/sdc4:
Magic : a92b4efc
Version : 00.90.00
UUID : d4b0e9c1:067357ce:2569337e:e9af8bed
Creation Time : Tue Aug 5 14:14:16 2008
Raid Level : raid5
Used Dev Size : 145910272 (139.15 GiB 149.41 GB)
Array Size : 291820544 (278.30 GiB 298.82 GB)
Raid Devices : 3
Total Devices : 4
Preferred Minor : 4

Update Time : Wed Feb 24 15:33:28 2010
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Checksum : 30393836 - correct
Events : 0.10

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 2 8 36 2 active sync /dev/sdc4

0 0 8 4 0 active sync /dev/sda4
1 1 8 20 1 active sync /dev/sdb4
2 2 8 36 2 active sync /dev/sdc4
3 3 8 52 3 spare /dev/sdd4


server-gentoo ~ # mdadm --examine /dev/sdd4
/dev/sdd4:
Magic : a92b4efc
Version : 00.90.00
UUID : d4b0e9c1:067357ce:2569337e:e9af8bed
Creation Time : Tue Aug 5 14:14:16 2008
Raid Level : raid5
Used Dev Size : 145910272 (139.15 GiB 149.41 GB)
Array Size : 291820544 (278.30 GiB 298.82 GB)
Raid Devices : 3
Total Devices : 4
Preferred Minor : 4

Update Time : Wed Feb 24 15:37:05 2010
State : clean
Active Devices : 1
Working Devices : 2
Failed Devices : 1
Spare Devices : 1
Checksum : 3039395d - correct
Events : 0.32

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 3 8 52 3 spare /dev/sdd4

0 0 0 0 0 removed
1 1 8 20 1 active sync /dev/sdb4
2 2 0 0 2 faulty removed
3 3 8 52 3 spare /dev/sdd4



---- Does this info help?
Thanks, Stefan

emergency call for help: raid5 fallen apart
 

Am 24.02.2010 16:11, schrieb Stefan G. Weichinger:
>
> I also sent this one to the linux-raid-ml, I am in trouble:

--

>> This is a crucial server and I feel a lot of pressure.
>> Rebuilding that raid would mean a lot of restore-work etc.
>> So I would really appreciate a goo advice here.

thanks for any off-list replies ... situation calmed so far.
Stefan