HowTo: Speed Up Linux Software Raid Building And Re-syncing

by Vivek Gite · 12 comments

It is no secret that I am a pretty big fan of excellent Linux Software RAID. Creating, assembling and rebuilding small array is fine. But, things started to get nasty when you try to rebuild or resync large size array. You may get frustrated when you see it is going to take 22 hours to rebuild the array. You can always increase RAID resync performance using the following technique.

Recently, I build a small NAS server running Linux for one my client with 5 x 2TB disks in RAID 6 configuration for all in one backup server for Mac OS X and Windows XP/Vista clients computers. Then whey I cat /proc/mdstat it reported that md0 is created and resync is in progress. The resync speed was around 4000K/sec and resync will complete in approximately in 22 hours. I wanted to finish this early.

/proc/sys/dev/raid/{speed_limit_max,speed_limit_min}

The /proc/sys/dev/raid/speed_limit_min is config file that reflects the current "goal" rebuild speed for times when non-rebuild activity is current on an array. The speed is in Kibibytes per second, and is a per-device rate, not a per-array rate . The default is 1000.

The /proc/sys/dev/raid/speed_limit_max is config file that reflects the current "goal" rebuild speed for times when no non-rebuild activity is current on an array. The default is 100,000.

To see current limits, enter:
# sysctl dev.raid.speed_limit_min # sysctl dev.raid.speed_limit_max

To increase speed, enter:

 
echo value > /proc/sys/dev/raid/speed_limit_min

 
sysctl -w dev.raid.speed_limit_min=value

In this example, set it to 50000 K/Sec, enter:
# echo 50000 > /proc/sys/dev/raid/speed_limit_min
OR
# sysctl -w dev.raid.speed_limit_min=50000
If you want to override the defaults you could add these two lines to /etc/sysctl.conf:

dev.raid.speed_limit_min = 50000
dev.raid.speed_limit_max = 200000

Bitmap Option

Bitmaps optimize rebuild time after a crash, or after removing and re-adding a device. Turn it on by typing the following command:
# mdadm --grow --bitmap=internal /dev/md0
Once array rebuild or fully synced, disable bitmaps:
# mdadm --grow --bitmap=none /dev/md0

Result

My speed went from 4k to 51k:
cat /proc/mdstat
Sample outputs:


Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md5 : active raid1 sde2[2](S) sdd2[3](S) sdc2[4](S) sdb2[1] sda2[0]
      530048 blocks [2/2] [UU]

md0 : active raid6 sde3[4] sdd3[3] sdc3[2] sdb3[1] sda3[0]
      5855836800 blocks level 6, 64k chunk, algorithm 2 [5/5] [UUUUU]
      [============>........]  resync = 61.7% (1205475036/1951945600) finish=242.9min speed=51204K/sec

References:

man page md and mdadm

Featured Articles:

4000+ howtos and counting! Want to read more Linux / UNIX howtos, tips and tricks? We request you to sign up for the Daily email newsletter or Weekly newsletter and get intimated about our new howtos / faqs as soon as it is released.

{ 12 comments… read them below or add one }

1 zosky June 16, 2010

i love your blog, but now i hate you … i added a new drive last week and watched mdadm reshape for ~80hrs … guess it’ll be easier next time. thanks for the tip :P

2 Vivek Gite June 16, 2010: Heh.. no problem. I’ve updated the post with bitmap option which will increase speed further.

Reply

3 ManvsTech June 16, 2010

That’s an awesome tip. From my rough calculations I assume it finished in about 1hr 45min. May not be a good thing if you bill your clients per hour :)

4 Jord June 17, 2010

Hi there,

this for sure will help a lot when I’ll install our budget backup SAN :-). Do you by any chance also now whether such limits exists for drbd-syncing?

Cheers!

5 Vivek Gite June 17, 2010

Yes, drbd also suffer from the same issue and can be fixed by editing /etc/drbd.conf. Look out for rate directive:

 rate 40M;

Another note LVM really slows down performance with drbd; so make sure you avoid it. See this help page [drbd.org].

6 Thank You June 18, 2010: I always faced this speed problem and I thought this is due to software issue. With this tip I reduced my build time from 9.2 hours to less than 1 hour.

Reply

7 Cr0t June 18, 2010

One of my machines has 5x 2TB in a RAID5. I got even more speed out of my volume after tweaking the drives and the md0 volume itself.

—BEGIN—

#!/bin/bash

blockdev --setra 16384 /dev/sd[abcdefg]

echo 1024 > /sys/block/sda/queue/read_ahead_kb
echo 1024 > /sys/block/sdb/queue/read_ahead_kb
echo 1024 > /sys/block/sdc/queue/read_ahead_kb
echo 1024 > /sys/block/sdd/queue/read_ahead_kb
echo 1024 > /sys/block/sde/queue/read_ahead_kb

echo 256 > /sys/block/sda/queue/nr_requests
echo 256 > /sys/block/sdb/queue/nr_requests
echo 256 > /sys/block/sdc/queue/nr_requests
echo 256 > /sys/block/sdd/queue/nr_requests
echo 256 > /sys/block/sde/queue/nr_requests

# Set read-ahead.
echo "Setting read-ahead to 64 MiB for /dev/md0"
blockdev --setra 65536 /dev/md0

# Set stripe-cache_size for RAID5.
echo "Setting stripe_cache_size to 16 MiB for /dev/md0"
echo 16384 > /sys/block/md0/md/stripe_cache_size
echo 8192 > /sys/block/md0/md/stripe_cache_active 

# Disable NCQ on all disks.
echo "Disabling NCQ on all disks..."
echo 1 > /sys/block/sda/device/queue_depth
echo 1 > /sys/block/sdb/device/queue_depth
echo 1 > /sys/block/sdc/device/queue_depth
echo 1 > /sys/block/sdd/device/queue_depth
echo 1 > /sys/block/sde/device/queue_depth

—END—

8 Vivek Gite June 18, 2010

Thanks for sharing your script.

9 Jord July 16, 2010: One more question on the software raid possibilities:
with a hardware raid, when one disk dies, you can still boot the machine. I had achieved the same for RAID-1 with info found on the net, but how does one goes to achieve the same for a 6 disk RAID-5 setup? I can’t find that info…. Any suggestions or links?

One possible solution I came up with (but I’m not too keen on using it) would be to install the system to an USB drive/stick (of which one can easily make backups, or even use 2 usb devices in RAID-1) and keep the 6 drives as pure data-disks (no OS) in RAID 5.

Thanks in advance

Reply

10 Jord July 29, 2010

The easiest solution I found was to combine raid 1 (for booting the system) & raid 5 (for data) on the same six disks. Agreed, I now have a 6 disk raid 1, but it is only 2GB. The rest is raid 5. Hopes this helps someone facing the same problems sometimes.

Cheers, Jord

11 Andy Lee Robinson August 9, 2010

A suggestion, boot from a pendrive.
I just leave a pendrive in a usb port and boot from that as there isn’t any space for more drives.
Had some problems installing Fedora 12 some months ago using the onboard fakeraid and locking up, so reinstalled as standalone drives and mdadm.
I wanted to make a kickass colocated web/mysql server and I’m still soaking it, but I’m worried that resyncing may cause performance issues at critical times, as I have seen while developing.
(it’s now part of a 3-node multiple master mysql cluster+slave experiment, with one node at the ISP linked to here at home over 30mbps / ssh tunnels) :-)

My setup:
RAID10, persistent superblock (host i7/920 + 12Gb RAM)
|=========md0========|====md1====| /dev/sda 1Tb
|=========md0========|====md1====| /dev/sdb 1Tb
|=========md0========|====md1====| /dev/sdc 1Tb
|=========md0========|====md1====| /dev/sdd 1Tb
|=========md0========| /dev/sde 750Gb
|=========md0========| /dev/sdf 750Gb
|======| /dev/sdg 2×250Gb (HW raid1)
|=| /dev/sdh 1Gb pendrive /boot

12 Rory August 20, 2010

So can you add the intent bitmap during a RAID 5 reshape? That would really help me out…

Definitely should have looked at this before. Great blog post, man.

HowTo: Speed Up Linux Software Raid Building And Re-syncing

/proc/sys/dev/raid/{speed_limit_max,speed_limit_min}

Bitmap Option

Result

References:

Featured Articles:

Search

Sponsored links

Related Posts