Comments on: HowTo: Speed Up Linux Software Raid Building And Re-syncing

By: M1TH

M1TH — Mon, 21 Nov 2011 17:02:42 +0000

Also, thanks to Cr0t!! sped up my raid build from about 28-30K/sec to 40-45K/sec, topping out over 50K occasionally. recognize what you are doing issuing those commands, which is tuning the hard drive parameters that deal with pretty low-level communication options of the device. options tied to the drivers, firmware and kernel. so, use them with care, and don’t be surprised if extreme circumstances cause a lock-up or other failure case causing you to have to start your raid rebuild over. point being, use with caution and in experimental environment as mentioned. still, this did not cause any issues with my setup even doing it during the build, and in most cases on properly working hardware it should not. but be warned!

anywho, the original problem for me turned out to be one bad hard drive causing the array to be built at <1K/sec. sad thing is that nothing about the system or even smartctl really indicated errors. it was really diagnosed by noticing the smartctl 'remap' value skyrocketing on one device where it was not on any of the others, and physically looking at the box showed 10x more LED activity on the problematic disk as it tried every read and write vs the other 19 in the array. so, don't rely on your os to inform you of hard drive issues and you can't rely on SMART at face value either. i think any experienced sys-admin would validate that statement.

happy tuning everyone =) thanks for the info.

By: rtcbad

rtcbad — Mon, 24 Oct 2011 22:37:10 +0000

Thanks to this article and the comments on tweaking the drive parameters from Cr0t, rebuild performance on RAID1 went from 32K to over 110K during system idle. The combination of increasing the dev.raid.speed_limit_min to 50000 and disabling NCQ on both drives gave a phenomenal increase in performance. After seeing how much disabling NCQ helped, I further increased dev.raid.speed_limit_min to 110000 to further speed up the rebuild. The rebuild went from an initial estimate of 6 hours, to only taking about 1.5 hours after these tweaks.

By: David Clymer

David Clymer — Wed, 10 Aug 2011 12:32:33 +0000

You could split the disks into two partitions, a small 100M partition, and a data partition which uses the rest of the disk. Set up two raid volumes. The small partitions would be raid1, and the rest raid5/6 whatever. install everything on the raid6 volume and mount the raid1 as /boot. Setup grub for each disk:

root (hd0,0)
setup (hd0)
root (hd1,0)
setup (hd1)
…etc.

In this way, any drive could be removed and you could still boot the machine.

By: cd

cd — Wed, 08 Jun 2011 17:20:48 +0000

just for sharing : replacing a failed drive , 2TB caviar black – WD2002FAEX, 3 good drives + 1 new on a production server (still gets 100Mbs trafic out )

[>....................] recovery = 2.8% (56272128/1942250240) finish=888.0min speed=35394K/sec

The drives ar having a 92% busy
DSK | sda | busy 92%

—-total-cpu-usage—- -dsk/total- -net/total- —paging– —system–
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
6 6 83 3 0 2| 107M 51M| 0 0 | 5.6B 44B|8900 14k
1 10 72 14 0 3| 363M 150M|8421k 23M| 0 0 |9330 11k
7 12 63 14 1 4| 416M 150M|8009k 25M| 0 0 |9804 12k

won’t push the speed limit to keep the server responsive.

ps: careful running “magic” scripts with “magic” settings. got my server (colocated) frozen while changing some readhead values “on the fly”. Costed me 30$ extra intervention to the colo guys to reboot it (it was on weekend :)) .

Can be fun tweaking but really there’s no silver bullet that comes without some risks. For oldies : there were times when computers came with a Turbo button. If you’d press it the cpu frequency would get higher. Of course 50% of the time Duke Nukem would freeze if you press the turbo button while playing:)

By: indulis

indulis — Sun, 15 May 2011 06:35:48 +0000

Sounds like the RAID setup can do 100 MB/sec, but you have changed the /proc/sys/dev/raid/speed_limit_max setting only, this is used if there is no other activity on the disks at all. SO the rise to 100MB/sec is wen no other I/O activity happens. Try changing /proc/sys/dev/raid/speed_limit_min

echo “50000″ > /proc/sys/dev/raid/speed_limit_min

then see what happens.

By: jmuellers

jmuellers — Thu, 28 Apr 2011 09:33:40 +0000

edit:
Sometimes the speed is rising from 20000k to 100000k for a few seconds or minutes and then falling back to 20. I have no explanation for that behaviour.
Except for the resync the system is idle.
md0_raid6 is using about 5% while speed ist 20000k, 25% while speed ist 100000k
md0_resync is using a few percent at 20000k, 20% at 100000k.
Just in this moment the speed went up to 100000k for only 10 seconds… no clue.

By: jmuellers

jmuellers — Thu, 28 Apr 2011 08:58:37 +0000

Setting the speed limits didn’t help in my case.
I have raid6 with 6xwd20ears, with 0xfb partitions (4kb physical 4kb logical, aligned).
Resync speed after setting up the array is around 20000kb/s, I assume it will get even slower towards the end of the process.

Maybe it would be faster with AAM set to 255 (now set to 128), unfortunatly I cannot set it through my raid controller (only works if the drive is plugged in on the mainboard sata controller).

I’m also not able to set the bitmap, but i think it would not affect the initial resync after having just built the array, am I right?

I would also be interested in further information about setting read ahead etc.

By: upsite

upsite — Mon, 28 Feb 2011 01:04:21 +0000

Hi Cr0t,

on which hardware specs reliy your setra and read_ahead_kb values?
i got WD20EARS with 64mb memory.
so is it save/usefull to increase your values?

Best practice values ? limits?
thanx ;)

By: dalobo

dalobo — Mon, 21 Feb 2011 02:32:28 +0000

When I run sudo mdadm –grow –bitmap=internal /dev/md1

I get:

mdadm: failed to set internal bitmap.

The RAID is rebuilding when I ran this. Do I need to do this before I rebuild? If so, how??

md1 : active raid5 sde[4] sdf[2] sdg[3] sdd[1] sdb[0]
5860543488 blocks super 0.91 level 5, 4k chunk, algorithm 2 [5/5] [UUUUU]
[=>...................] reshape = 8.2% (161204840/1953514496) finish=1231.1min speed=24262K/sec

No change in speed.

By: Rory

Rory — Fri, 20 Aug 2010 04:29:19 +0000

So can you add the intent bitmap during a RAID 5 reshape? That would really help me out…

Definitely should have looked at this before. Great blog post, man.

By: Andy Lee Robinson

Andy Lee Robinson — Mon, 09 Aug 2010 06:00:58 +0000

A suggestion, boot from a pendrive.
I just leave a pendrive in a usb port and boot from that as there isn’t any space for more drives.
Had some problems installing Fedora 12 some months ago using the onboard fakeraid and locking up, so reinstalled as standalone drives and mdadm.
I wanted to make a kickass colocated web/mysql server and I’m still soaking it, but I’m worried that resyncing may cause performance issues at critical times, as I have seen while developing.
(it’s now part of a 3-node multiple master mysql cluster+slave experiment, with one node at the ISP linked to here at home over 30mbps / ssh tunnels) :-)

My setup:
RAID10, persistent superblock (host i7/920 + 12Gb RAM)
|=========md0========|====md1====| /dev/sda 1Tb
|=========md0========|====md1====| /dev/sdb 1Tb
|=========md0========|====md1====| /dev/sdc 1Tb
|=========md0========|====md1====| /dev/sdd 1Tb
|=========md0========| /dev/sde 750Gb
|=========md0========| /dev/sdf 750Gb
|======| /dev/sdg 2x250Gb (HW raid1)
|=| /dev/sdh 1Gb pendrive /boot

By: Jord

Jord — Thu, 29 Jul 2010 13:07:18 +0000

The easiest solution I found was to combine raid 1 (for booting the system) & raid 5 (for data) on the same six disks. Agreed, I now have a 6 disk raid 1, but it is only 2GB. The rest is raid 5. Hopes this helps someone facing the same problems sometimes.

Cheers, Jord

By: Jord

Jord — Fri, 16 Jul 2010 11:20:46 +0000

One more question on the software raid possibilities:
with a hardware raid, when one disk dies, you can still boot the machine. I had achieved the same for RAID-1 with info found on the net, but how does one goes to achieve the same for a 6 disk RAID-5 setup? I can’t find that info…. Any suggestions or links?

One possible solution I came up with (but I’m not too keen on using it) would be to install the system to an USB drive/stick (of which one can easily make backups, or even use 2 usb devices in RAID-1) and keep the 6 drives as pure data-disks (no OS) in RAID 5.

Thanks in advance

By: Vivek Gite

Vivek Gite — Fri, 18 Jun 2010 20:23:53 +0000

Thanks for sharing your script.

By: Cr0t

Cr0t — Fri, 18 Jun 2010 20:11:48 +0000

One of my machines has 5x 2TB in a RAID5. I got even more speed out of my volume after tweaking the drives and the md0 volume itself.

—BEGIN—

#!/bin/bash
blockdev --setra 16384 /dev/sd[abcdefg]
echo 1024 > /sys/block/sda/queue/read_ahead_kb
echo 1024 > /sys/block/sdb/queue/read_ahead_kb
echo 1024 > /sys/block/sdc/queue/read_ahead_kb
echo 1024 > /sys/block/sdd/queue/read_ahead_kb
echo 1024 > /sys/block/sde/queue/read_ahead_kb
echo 256 > /sys/block/sda/queue/nr_requests
echo 256 > /sys/block/sdb/queue/nr_requests
echo 256 > /sys/block/sdc/queue/nr_requests
echo 256 > /sys/block/sdd/queue/nr_requests
echo 256 > /sys/block/sde/queue/nr_requests
# Set read-ahead.
echo "Setting read-ahead to 64 MiB for /dev/md0"
blockdev --setra 65536 /dev/md0
# Set stripe-cache_size for RAID5.
echo "Setting stripe_cache_size to 16 MiB for /dev/md0"
echo 16384 > /sys/block/md0/md/stripe_cache_size
echo 8192 > /sys/block/md0/md/stripe_cache_active
# Disable NCQ on all disks.
echo "Disabling NCQ on all disks..."
echo 1 > /sys/block/sda/device/queue_depth
echo 1 > /sys/block/sdb/device/queue_depth
echo 1 > /sys/block/sdc/device/queue_depth
echo 1 > /sys/block/sdd/device/queue_depth
echo 1 > /sys/block/sde/device/queue_depth

—END—

By: Thank You

Thank You — Fri, 18 Jun 2010 14:57:45 +0000

I always faced this speed problem and I thought this is due to software issue. With this tip I reduced my build time from 9.2 hours to less than 1 hour.

By: Vivek Gite

Vivek Gite — Thu, 17 Jun 2010 18:04:39 +0000

Yes, drbd also suffer from the same issue and can be fixed by editing /etc/drbd.conf. Look out for rate directive:

 rate 40M;

Another note LVM really slows down performance with drbd; so make sure you avoid it. See this help page [drbd.org].

By: Jord

Jord — Thu, 17 Jun 2010 17:53:08 +0000

Hi there,

this for sure will help a lot when I’ll install our budget backup SAN :-). Do you by any chance also now whether such limits exists for drbd-syncing?

Cheers!

By: ManvsTech

ManvsTech — Wed, 16 Jun 2010 15:56:36 +0000

That’s an awesome tip. From my rough calculations I assume it finished in about 1hr 45min. May not be a good thing if you bill your clients per hour :)

By: Vivek Gite

Vivek Gite — Wed, 16 Jun 2010 10:33:32 +0000

Heh.. no problem. I’ve updated the post with bitmap option which will increase speed further.