Howto rebuilding a RAID array after a disk fails

Author: Vivek Gite Last updated: December 27, 2007 9 comments

Q. I have two disk-mirrored array, suppose if one of my disk in mirrored RAID array fails, then I will replace that disk with new one (I have hot swapping SCSI drives). Now question is how I rebuild a RAID array after a disk fails.

ADVERTISEMENTS

A. A redundant array of inexpensive disks, (redundant array of independent disks) is a system, which uses multiple hard drives to share or replicate data among the drives. You can use both IDE and SCSI disk for mirroring.

If you are not using hot swapping drives then you need to shutdown server. Once hard disk has been replaced to system, you need to use used raidhotadd to add disks from RAID-1, -4 and -5 arrays, while they are active.

Assuming that new SCSI disk is /dev/sdb, type the following command:# raidhotadd /dev/mdX /dev/sdbReplace /dev/mdX with actual raid device name (e.g. /dev/md0). While array is under construction, you can see progress and other information by typing following command:# cat /proc/mdstat

Category	List of Unix and Linux commands
File Management	cat
Firewall	Alpine Awall • CentOS 8 • OpenSUSE • RHEL 8 • Ubuntu 16.04 • Ubuntu 18.04 • Ubuntu 20.04
Network Utilities	dig • host • ip • nmap
OpenVPN	CentOS 7 • CentOS 8 • Debian 10 • Debian 8/9 • Ubuntu 18.04 • Ubuntu 20.04
Package Manager	apk • apt
Processes Management	bg • chroot • cron • disown • fg • jobs • killall • kill • pidof • pstree • pwdx • time
Searching	grep • whereis • which
User Information	groups • id • lastcomm • last • lid/libuser-lid • logname • members • users • whoami • who • w
WireGuard VPN	CentOS 8 • Debian 10 • Firewall • Ubuntu 20.04

ADVERTISEMENTS

9 comments… add one

franklin Aug 4, 2008 @ 5:25
i used raid5 with 3 hard disk.each partition has 5gb,now totally how much space i have in this situation?
Reply Link
franklin Aug 4, 2008 @ 5:58
Anyone tells please what is the hot swap drive and use also where used in raid?
Reply Link
franklin Aug 4, 2008 @ 8:54
Do you kno abowt relationshoip btn kernel,shell,and
userinterface also please tell to me boss
Reply Link
Nikolay Mar 8, 2011 @ 20:32
for RAID5 subtract 1 disk.
If you have 3 disks * 10 GB each, RAID5 would be 20 GB
If you have 4 disk s* 10 GB each, RAID5 would be 30 GB
If you have 8 disks * 10 GB each, RAID5 would be 70 GB
…
for RAID6 subtract 1 disk more (2 total).
If you have 4 disks * 10 GB each, RAID5 would be 20 GB
If you have 8 disks * 10 GB each, RAID5 would be 60 GB
…
Reply Link
Paco Colomer Dec 23, 2013 @ 19:38
Hi,
Great tips, however there is something that I might be missing.
Firstly ‘raidhotadd’ command is neither in centos 6.4 nor in its yum repo.
And secondly, as per example above:
# raidhotadd /dev/mdX /dev/sdb
Where ‘X’ is de number of metadevice, but a metadevice should map a disk partition, not a full disk. isn’t it?
Could anyone clarify?
Reply Link
- TB Dec 23, 2013 @ 21:07
  Say RAID-1 array /dev/md0 is made of two disks /dev/sda+/dev/sdb and /dev/sda failed. In this case you need to remove /dev/sda1 as follows:
  Mark failed /dev/sda1
```
mdadm --manage /dev/md0 --fail /dev/sda1
```
  Remove /dev/sda1 from md0
```
mdadm --manage /dev/md0 --remove /dev/sdb1
```
  Watch status at /proc/mdstat
```
watch cat /proc/mdstat
```
  Finally shutdown the server
```
shutdown -h 0
```
  Remove and replace the /dev/sda disk. Start the box. Next, create same paritions on /dev/sda as active /dev/sdb. I use the following at the terminal
```
sfdisk -d /dev/sdb | sfdisk /dev/sda
```
  Use fdisk to verify partitions on both disks:
```
fdisk -l
```
  Add /dev/sda1 to /dev/md0:
```
mdadm --manage /dev/md0 --add /dev/sda1
```
  The above will start the synchronization aka raid-1 rebuild:
```
watch cat /proc/mdstat
```
  Reply Link
Paco Colomer Dec 23, 2013 @ 21:23
Thanks TB for your response.
Correct, that’s the traditional way… but I’m wondering if is possible to plug the new disk and manage to do all steps at once: clone partitions, add partitions to meta-devices and sync all.
That’s the reason I got the link to this article, but I’m a bit upset that ‘raidhotadd’ is not found in CentOS. Moreover the example above is talking about cloning the entire disk but condidering that you only have a meta-device.
In my case I’m just now mirroring 3 meta-devices: md0, md1 and md2 from the same disk (sda) to a new disk (sdb), so how could be done using ‘raidhotadd’?
Regards
Reply Link
- TB Dec 23, 2013 @ 21:44
  raidhotadd (raidsetfaulty and raidhotremove) was old tool and it was part of raidtools. It is now replaced by mdadm. So you will not find raidhotadd on any modern distro including CentOS and co. In short you can replicate the raidhotadd command with –add option
```
 mdadm /dev/md0 --add /dev/sda1
```
  If you have /dev/md0, md1, md2 and so on:
```
# mark faulty #
mdadm --manage /dev/md0 --fail /dev/sda1
mdadm --manage /dev/md1 --fail /dev/sda2
mdadm --manage /dev/md2 --fail /dev/sda3

# remove it #
mdadm --manage /dev/md0 --remove /dev/sda1
mdadm --manage /dev/md1 --remove /dev/sda2
mdadm --manage /dev/md2 --remove /dev/sda3

# Shutdown, replace the disk, run sfdisk to mirror partitions and type #
mdadm --manage /dev/md0 --add /dev/sda1
mdadm --manage /dev/md1 --add /dev/sda2
mdadm --manage /dev/md2 --add /dev/sda3
```
  Sadly, there is no shortcut. If you find one, let us know.
  Reply Link
Paco Colomer Dec 23, 2013 @ 21:51
Ack’ed! I was trying to mimic by software what hardware raid does; I mean, you plug the disk and forget about going toward all steps you mentioned.
That would be great, because the goal is to clone the disk, any tools may take care about all steps within the appropriated order ;) I don’t see the complexity whether both disk are exactly the same size, just copy bit to bit from beginning to end.
If don’t exist yet… this would be a great tool for Sysadmins. When I need to replace a server disk, from time to time, I use to forget about all steps… and I’m always wondering why this is not as easy as plug the disk and forget about the remaining steps lol
Thanks anyway.
Reply Link

Howto rebuilding a RAID array after a disk fails

See also: