A while ago I found myself in the unfortunate situation of having a degraded RAID5 array failing to boot after a blackout. So I hooked a monitor to it to see what was going on and found that it got stuck at boot time with identifying one of the disks. After a long timeout, it finally booted into Ubuntu from the attached Flash drive where it reported the degraded RAID array and dropped back to diagnostic mode.
Finding out what broke
I was pleased to find that only one of the disks was out and the superblock was in good shape, and that the rest of the system was OK:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
admin@machine:~$ sudo mdadm --detail /dev/md0 [sudo] password for admin: /dev/md0: Version : 1.2 Creation Time : Wed May 30 19:26:25 2012 Raid Level : raid5 Array Size : 5860538880 (5589.05 GiB 6001.19 GB) Used Dev Size : 1953512960 (1863.02 GiB 2000.40 GB) Raid Devices : 4 Total Devices : 3 Persistence : Superblock is persistent Update Time : Thu Dec 27 13:10:16 2012 State : clean, degraded Active Devices : 3 Working Devices : 3 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Name : machine:0 (local to host machine) UUID : d53b6e38:cc0ca3ae:7d3a3e69:cba8bfe1 Events : 156429 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 17 1 active sync /dev/sdb1 2 8 33 2 active sync /dev/sdc1 3 0 0 3 removed admin@machine:~$
It took me a while to interpret the fact that the total number of devices was smaller than the number of Raid devices, as usually one needs to explicitly remove a failed disk from the array. Such "automatic removal" could also complicate things if there were more disks in the system, but in my case I could readily see that
/dev/sdd was the faulty one and that indeed it was automatically dropped from the array due to the total failure to discover it at boot time.
Doing a backup, getting a new disk
Needless to say, before I did anything else I did a full backup of the array data - this took a few days but it is definitely the way to go. In the meantime, I purchased a new disk that matched those in the system, then once the backup finished added it to the machine replacing the failed drive.
Partitioning the new disk
With the new disk in place, I wanted to initialize it with the same partition table as the others in the array - this seemed like a convenient way to move on with all other drives having same sized partitions. So I tried copying the partition table from
/dev/sda to the new disk:
sfdisk -d /dev/sda | sfdisk /dev/sdd
This reported back:
WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util sfdisk doesn't support GPT. Use GNU Parted.
Indeed, since the RAID array is above the 2TB limit for MBR partition tables it uses GPT. Luckily, I learned of
sgdisk and did a quick
apt-get install sdisk to install it on the machine. With this tool, I could now rephrase the previous command as:
1 2 3 4 5
admin@machine:~$ sudo sgdisk -R=/dev/sdd /dev/sda [sudo] password for admin: The operation has completed successfully. admin@machine:~$ sudo sgdisk -G /dev/sdd The operation has completed successfully.
The latter command is necessary to randomize the GUIDs copied from the existing partitions, apparently, this is something that RAID GPT disks require.
Adding the new disk into the array and waiting it out
The rest of the "rescue operation" was according to the books. I added the new disk to the array:
mdadm --manage /dev/md0 -a /dev/sdd1
This started a sync with the other disks, copying data where necessary. Doing a watch to monitor progress reported:
1 2 3 4 5 6 7 8
admin@machine:~$ watch cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid5 sdd1 sdc1 sda1 sdb1 5860538880 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_] [>....................] recovery = 1.5% (29949252/1953512960) finish=500.9min speed=63994K/sec unused devices: <none>
Once the sync was over, everything looked OK:
1 2 3 4 5 6
admin@machine:~$ cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid5 sdd1 sdc1 sda1 sdb1 5860538880 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU] unused devices: <none>
Now, if I could only find out why the
 after sdd1 appears instead of
... But either way, the array seems OK and the server now starts normally. Hope this helps someone with the same problem.