FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 08-17-2011, 10:40 PM
Mathieu Baudier
 
Default RAID5 suddenly broken

Hello,

I have a RAID5 array on my CentOS 5.6 x86_64 workstation which
"suddenly" failed to work (actually after the system could not resume
from a suspend).

I had recently issues after moving the workstation to another office,
where one of the disks got accidently unplugged. But the RAID was
working and it had reconstructed (as far as I can tell) the data.
After I replugged the disk, it was working normally over the last two
days (enough to get back to a proper state I guess/hoped)

This RAID is used as an LVM volume group for all my important data,
among them the root of the operating system(s).
It based on four partitions on four separate disks (the third
partition of each disk, 3 active, one spare)

When booting, I get an error message similar to:

raid5 failed: No md superblock detected on /dev/md0.

and the LVM volume group does not come up.

I then booted using the CentOS 5.6 LiveCD and tried to run a few mdadm
command (see just below).
It seems that there are some data still lying around, but I'm not very
experienced with RAID and I thought that I would ask for advice before
trying commands which may impact the data such as assemble, etc.

Thanks in advance for any help!

Mathieu

[root@livecd ~]# mdadm --misc -E /dev/md0
mdadm: No md superblock detected on /dev/md0.
[root@livecd ~]# mdadm --misc -Q /dev/md0
/dev/md0: is an md device which is not active
/dev/md0: No md super block found, not an md component.
[root@livecd ~]# mdadm --misc -D /dev/md0
mdadm: md device /dev/md0 does not appear to be active.
[root@livecd ~]# mdadm -E /dev/sd*3
/dev/sda3:
Magic : a92b4efc
Version : 0.90.00
UUID : 7533411a:f066a145:1e89d48e:1a8374a3
Creation Time : Tue Dec 1 12:01:05 2009
Raid Level : raid5
Used Dev Size : 204796416 (195.31 GiB 209.71 GB)
Array Size : 409592832 (390.62 GiB 419.42 GB)
Raid Devices : 3
Total Devices : 2
Preferred Minor : 0

Update Time : Wed Aug 17 14:47:36 2011
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 1
Spare Devices : 0
Checksum : ed6d5dcd - correct
Events : 38857

Layout : left-symmetric
Chunk Size : 256K

Number Major Minor RaidDevice State
this 0 8 3 0 active sync /dev/sda3

0 0 8 3 0 active sync /dev/sda3
1 1 0 0 1 faulty removed
2 2 8 51 2 active sync /dev/sdd3
/dev/sdb3:
Magic : a92b4efc
Version : 0.90.00
UUID : 7533411a:f066a145:1e89d48e:1a8374a3
Creation Time : Tue Dec 1 12:01:05 2009
Raid Level : raid5
Used Dev Size : 204796416 (195.31 GiB 209.71 GB)
Array Size : 409592832 (390.62 GiB 419.42 GB)
Raid Devices : 3
Total Devices : 4
Preferred Minor : 0

Update Time : Wed Aug 10 11:52:15 2011
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Checksum : ed63a948 - correct
Events : 9022

Layout : left-symmetric
Chunk Size : 256K

Number Major Minor RaidDevice State
this 1 8 19 1 active sync /dev/sdb3

0 0 8 3 0 active sync /dev/sda3
1 1 8 19 1 active sync /dev/sdb3
2 2 8 51 2 active sync /dev/sdd3
3 3 8 35 3 spare /dev/sdc3
/dev/sdc3:
Magic : a92b4efc
Version : 0.90.00
UUID : 7533411a:f066a145:1e89d48e:1a8374a3
Creation Time : Tue Dec 1 12:01:05 2009
Raid Level : raid5
Used Dev Size : 204796416 (195.31 GiB 209.71 GB)
Array Size : 409592832 (390.62 GiB 419.42 GB)
Raid Devices : 3
Total Devices : 3
Preferred Minor : 0

Update Time : Tue Aug 16 17:48:59 2011
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Checksum : ed6bf2b0 - correct
Events : 10670

Layout : left-symmetric
Chunk Size : 256K

Number Major Minor RaidDevice State
this 1 8 35 1 active sync /dev/sdc3

0 0 8 3 0 active sync /dev/sda3
1 1 8 35 1 active sync /dev/sdc3
2 2 8 51 2 active sync /dev/sdd3
/dev/sdd3:
Magic : a92b4efc
Version : 0.90.00
UUID : 7533411a:f066a145:1e89d48e:1a8374a3
Creation Time : Tue Dec 1 12:01:05 2009
Raid Level : raid5
Used Dev Size : 204796416 (195.31 GiB 209.71 GB)
Array Size : 409592832 (390.62 GiB 419.42 GB)
Raid Devices : 3
Total Devices : 2
Preferred Minor : 0

Update Time : Wed Aug 17 14:47:36 2011
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 1
Spare Devices : 0
Checksum : ed6d5e01 - correct
Events : 38857

Layout : left-symmetric
Chunk Size : 256K

Number Major Minor RaidDevice State
this 2 8 51 2 active sync /dev/sdd3

0 0 8 3 0 active sync /dev/sda3
1 1 0 0 1 faulty removed
2 2 8 51 2 active sync /dev/sdd3
[root@livecd ~]#
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 08-18-2011, 08:36 AM
Mathieu Baudier
 
Default RAID5 suddenly broken

> [root@livecd ~]# mdadm --misc -E /dev/md0
> mdadm: No md superblock detected on /dev/md0.
> [root@livecd ~]# mdadm --misc -Q /dev/md0
> /dev/md0: is an md device which is not active
> /dev/md0: No md super block found, not an md component.
> [root@livecd ~]# mdadm --misc -D /dev/md0
> mdadm: md device /dev/md0 does not appear to be active.

I could fix the issue.

Since the information on internet is a bit messy and scary here is a
summary of the problem and of the solution, for future reference:

## PROBLEM
Due to some failure related to suspend the RAID5 array became inconsistent.
The symptom was that the superblock (which allow auto-configuration of
the RAID array) was not recognized.

But the underlying member partitions were still recognized as per the
following command:
> [root@livecd ~]# mdadm -E /dev/sd*3
which provided details about their states

## SOLUTION
(detailed output of the commands at the end of the post)

# Assemble the array with force option
mdadm -v --assemble --force /dev/md0 /dev/sd{a,b,c,d}3

# NOTE: assembling without force option was not enough
[root@livecd ~]# mdadm -v --assemble /dev/md0 /dev/sd{a,b,c,d}3
...
mdadm: /dev/md0 assembled from 2 drives - not enough to start the
array while not clean - consider --force.

# Check state
mdadm -D /dev/md0
...
Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 0 0 1 removed
2 8 51 2 active sync /dev/sdd3

# Add missing partitions to the array
mdadm /dev/md0 -a /dev/sdb3
mdadm /dev/md0 -a /dev/sdc3

# Check that it is now OK
mdadm -D /dev/md0
...
Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 8 19 1 spare rebuilding /dev/sdb3
2 8 51 2 active sync /dev/sdd3

3 8 35 - spare /dev/sdc3

# and watch it rebuild:
...
Rebuild Status : 3% complete

Many thanks to the CentOS LiveCD team, they saved the day.

Cheers,

Mathieu

## DETAILED OUTPUT

[root@livecd ~]# mdadm -v --assemble --force /dev/md0 /dev/sd{a,b,c,d}3
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda3 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdb3 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdc3 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdd3 is identified as a member of /dev/md0, slot 2.
mdadm: added /dev/sdc3 to /dev/md0 as 1
mdadm: added /dev/sdd3 to /dev/md0 as 2
mdadm: added /dev/sda3 to /dev/md0 as 0
mdadm: /dev/md0 has been started with 2 drives (out of 3).

[root@livecd ~]# mdadm -D /dev/md0
/dev/md0:
Version : 0.90
Creation Time : Tue Dec 1 12:01:05 2009
Raid Level : raid5
Array Size : 409592832 (390.62 GiB 419.42 GB)
Used Dev Size : 204796416 (195.31 GiB 209.71 GB)
Raid Devices : 3
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Wed Aug 17 14:47:36 2011
State : clean, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 256K

UUID : 7533411a:f066a145:1e89d48e:1a8374a3
Events : 0.38857

Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 0 0 1 removed
2 8 51 2 active sync /dev/sdd3


[root@livecd ~]# mdadm /dev/md0 -a /dev/sdb3
mdadm: re-added /dev/sdb3
[root@livecd ~]# mdadm /dev/md0 -a /dev/sdc3
mdadm: added /dev/sdc3
[root@livecd ~]# mdadm -D /dev/md0
/dev/md0:
Version : 0.90
Creation Time : Tue Dec 1 12:01:05 2009
Raid Level : raid5
Array Size : 409592832 (390.62 GiB 419.42 GB)
Used Dev Size : 204796416 (195.31 GiB 209.71 GB)
Raid Devices : 3
Total Devices : 4
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Wed Aug 17 14:47:36 2011
State : clean, degraded, recovering
Active Devices : 2
Working Devices : 4
Failed Devices : 0
Spare Devices : 2

Layout : left-symmetric
Chunk Size : 256K

Rebuild Status : 0% complete

UUID : 7533411a:f066a145:1e89d48e:1a8374a3
Events : 0.38857

Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 8 19 1 spare rebuilding /dev/sdb3
2 8 51 2 active sync /dev/sdd3

3 8 35 - spare /dev/sdc3
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 07:33 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org