Software Vs Hardware RAID

by on July 7, 2009 · 33 comments· LAST UPDATED July 7, 2009

in , ,

A redundant array of inexpensive disks (RAID) allows high levels of storage reliability. RAID is not a backup solution. It is used to improve disk I/O (performance) and reliability of your server or workstation. A RAID can be deployed using both software and hardware. But the real question is whether you should use a hardware RAID solution or a software RAID solution.

In this post I will document my experience with both software and hardware RAID.

Comparison: Hardware vs Software RAID

FeatureSoftware RAIDHardware RAID
Cost:
Software RAID is part of OS, so no need to spend extract money.
LowHigh
Complexity:
The software RAID works on partition level and it can sometime increase complexity if you mix different partitions and hardware RAID.
Medium to highLow
Write back caching (BBU):
The software RAID cannot add a battery. Hardware RAID can run in write-back mode if it has a BBU installed. With BBU pending writes are not lost on a power failure.
NoYes
Performance:
With the software based RAID0 and RAID1 performance is negligible. However, performance goes down when you use parity-based arrays and/or several arrays at the same time. The performance of a software-based array is dependent on the server CPU performance and current load.
Depend upon usageHigh
Overheads (CPU, RAM etc):
The software RAID must use server's CPU and RAM for RAID software. The more hard drives means more CPU cycle will go to software RAID instead of your Apache / Postfix or MySQL server.
Depend upon usageNo
Disk hot swapping:
It means replacing hard disk without shutting down the server. Many RAID controller supports disk hot swapping.
NoYes
Hot spare support:
A hard disk is physically installed in the array which stays inactive until an active drive fails, when the system automatically replaces the failed drive with the spare, rebuilding the array with the spare hard disk included.
YesYes
/boot partition:
It is hard to make fail over with software RAID if /boot fails while booting the server. This can result into unexpected errors and data loss. However, LILO and FreeBSD loader can get around this problem too.
NoYes
Open source factor:
*BSD / OpenSolaris and Linux RAID software drivers are open source. It means more people can fix problems as compare to a closed source hardware firmware. You can move, mix and match different sizes with open source software RAID.
YesNo
Vendor lock in (open formats): See above.NoYes
Higher write throughput:
Hardware RAID with BBU may offers higher write throughput.
NoYes
Faster rebuilds:
Hardware RAID with BBU may offers faster rebuilds as compare to software based solution.
NoYes
Can act as a backup solution?:
Both software and hardware RAID cannot protect you against human errors or system failures or viruses. Daily scheduled and off site backups of your system are highly recommended. Use tools such as rsync, rsnapshot, tar, dump, restore and others to make daily backups.
NoNo
Recommend usage:
+Low cost solution
+Better for RAID0 or RAID1
+Single server / workstation
+Perfect for home and small business users.
+No vendor lock-ins
+Do you run a mission critical cluster or setup?
+Heavy database driven dynamic site
+Do you want the highest performance possible?

Other Factors

Powerful Modern CPU

The performance of a software-based array is dependent on the server CPU performance and load. With today's faster CPUs, software RAID outperforms hardware RAID.

Can RAID Array Fail?

Yes. The entire RAID array can fail taking down all your data (yes hardware RAID card do dies). Use tapes and other servers that can hold copies of the data, but don't allow much interaction with it. Move your data offsite. Another option is to use two or three RAID cards. Combine them together to protect your data. This make sure you gets back your data when one of your RAID card dies out.

Hardware vs Software Recovery

My personal experience - recovering from software RAID is easy. However, sometime finding out exact hardware RAID requirements can be a nightmare. A good backup can save from RAID hardware incompatibility problems. Software RAID allows you to mix different drive and sizes. You can not do something like this with hardware RAID cards. With software RAID you can swap the drives to a different server and read the data. There is no vendor lock in with software RAID solution.

You Can Not Go Wrong With Hardware RAID

There is an old saying in IT - no one ever got fired for picking RAID controllers.

Use Both Hardware and Software RAID

Sometime you need to use both hardware and software RAID to get the best of both worlds. For e.g. set up 4 mirror pairs, 2 on each hardware RAID controller, and use software RAID0 to put it all together. This will give the best performance for database server. Here is another example from one of our DR site server (this box mirrors our 30+ production server files and database):

  1. Server chassis with redundant power supplies
  2. Intel or AMD Dual Core CPU x 2
  3. 16GB ECC RAM
  4. 24 hot swappable drive bays
  5. 2 x RAID PCIe / PCIx RAID hardware controller
  6. 4 x Intel 1000 PCIx Lan cards (bond them together)
  7. 24 x 1TB SATA hard disk
  8. OS - Pick - Linux / FreeBSD / OpenSolaris
  9. Filesystem - Pick - ZFS / UFS / Ext3 (we use RAID-Z)
  10. Backup software - rsync, rsnapshots and MySQL in slave mode.

Now you can configure RAID0 stripe across the three RAID6 arrays (8 x 24 disks) using both hardware and software solution together. This massive storage system is perfect for online live backups.

Conclusion

So which one is better software raid or hardware raid?

Short answer - None.

Long answer - It depends upon your setup and requirements. I strongly recommend running both with benchmarking software to find out your disk I/O. Test both solutions by removing hard disk i.e. fail a few drives. Try running system while drives are failed. Note down system load and errors (if any). Reboot the system. See if you can boot. Can you see your data again? Are you comfortable using tools provided with both solutions? See what works for you.

Finally, while choosing a storage always consider speed, reliability, and cost - pick any two.

A Final Note About My Personal Choice

I have been successfully using Linux and FreeBSD software RAID for several years for backing up my own data. I prefer to use software RAID to save money and to avoid vendor lock ins. All my personal data backup using the following hardware:

  • 1.5 TB USB hard disk - rsync and rsnapshot is used to make backup of all my servers and digitial camera.
  • 80GB x 3 hard disk software RAID using FreeNAS. Again, rsync is used to make all backups. I'm planing to replace UFS with RAID-Z under FreeBSD 8.

RAID Alternatives

Local disks on MogileFS storage nodes can be in a RAID, or not. It's cheaper not to, as RAID doesn't buy you any safety that MogileFS doesn't already provide. This is quite popular among the web 2.0 companies where lots of photos, images and files are uploaded by their users.

RAID-Z ZFS Storage is a data/parity scheme like RAID-5, but it uses dynamic stripe width. Every block is its own RAID-Z stripe, regardless of blocksize. This means that every RAID-Z write is a full-stripe write. It doesn't require any special hardware.

References:

What do you think? Please add your experience in the comments below.

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 33 comments… read them below or add one }

1 Solaris July 7, 2009 at 7:31 pm

Simple test to find out the approximate linear write speed of your hard disk/RAID arrays/etc

dd if=/dev/zero/ of=/root/1GBfile bs=1M count=1024

For example I got around 160 MB/s on 2 Hard disks in RAID1 ext3. Using XFS the values
are slightly higher.

The command above can give you a hint if you don’t have tine/don’t want to run bonnie and
similar programs.

Today I created a 32 GB secondary (pri=0) swap file and the average write speed was
~ 80 MB/s. This means that my RAID controller has some errors. Vivek can you post
an article about how to debug hardware RAID problems like bad write/reads ?

Note: The speed returned is not the average read/write speed since usually files tend
to fragment and very important the information that is being read in normal operation mode is not continuous, but the aberration is lower than 35%.

Note2: The above is for Linux (tested under Debian and Ubuntu Server).

Reply

2 kim jong il July 7, 2009 at 9:08 pm

i really dont get it.
I CANT buy a server with LESS THAN 4 CPUs.
One of them HAS to be able to run the Raid-5 parity calculations, right?
or is the MD/LVM driver not that smart yet?

Reply

3 nixCraft July 8, 2009 at 4:33 am

hardware RAID problems like bad write/reads

You can use smartool and cli tools provided by your RAID device. Watch out our FAQ section. I will add another FAQ regarding this issue.

@kim: No it is not that smart yet. It will not do anything special.

HTH

Reply

4 Solaris July 8, 2009 at 7:11 am

smartmontools looks like what I need, I am looking forward for your article.

Thanks

Reply

5 Cosmin July 8, 2009 at 10:55 am

have been using software raid1 on CentOs 4.5 but find it hard to manage, every kernel upgrade can break the array and drivers are not too good also this not tested on CentOs 5.
lately we switched to hardware raid but there are also somme issues between raid1 array on 15K and jbod on 10K.
so not found yet the solution that would be good for us.

Reply

6 t13nr4 July 8, 2009 at 11:33 am

thanks for the info!!

Reply

7 mdm July 8, 2009 at 12:05 pm

Nice.

finally found what I have been looking for sometime..

Reply

8 Cristiano July 8, 2009 at 1:02 pm

I agree with the author. I do too use software RAID for a couple of years and been really satisfied with it.

FYI: Cosmin, I suggest u to use mdadm for your array solution and mount the partition at startup. If you need any help, ill be glad to assist u… Just send me an e-mail.

Reply

9 Raja July 8, 2009 at 1:07 pm

Awesome write up man. Just a question, dont you think those 80GB hard drives eat up power?

@Solaris thanks for that nice quasi-hack on testing drive throughput :)

Reply

10 Raja July 8, 2009 at 1:09 pm

Sorry for the extra spam, but i think a subscribe to comments plugin will allow people to turn this into a semi discussion.

Reply

11 nixCraft July 8, 2009 at 1:21 pm

@Raja,

I only use those 80GB to backup data once a day.

There is rss comment feed for each page (just click on rss button @ firefox awesome bar). Nevertheless, I will add a subscribe to comments plugin as well.

Reply

12 Amr El-Sharnoby July 8, 2009 at 1:35 pm

I’d highly recommend that you consider using LVM striping and mirroring under Linux, that works like RAID-0 and RAID-1 allowing you to do all the magic of LVM ….

I’m using it for highly load servers with relatively new Xeon and Opteron CPUs and I can find its performance outperforming Modern Hardware RAID controllers with much more stability …

I’ve faced too much trouble with hardware raid controllers with those servers, the controllers caused too much instability and many failures , I’ve changed them, upgraded to newer models, updated firmware but all didn’t solve the problem,
so I’ve moved to LVM solution, as I’ve more than 30% idle CPU time even under high loads , while that LVM overhead is roughly 3 to 5% …
All problems has been solved, and I’m quite happy with tht stabilty and performance …

I’m using RHEL 5.3 on those servers ..

Reply

13 valqk July 8, 2009 at 5:27 pm

I personally use Software raids (md linux and gmirror in fbsd). That’s because I can’t afford expensive models of 3ware and similar and also because you MUST have at LEAST 1 spare exactly the same one as the one in machine. I’ve bought 2 3ware 8006lp2. They work very good but unfortunately the speed test fails because they are SATAI not SATAII. The good thing is that they DO have HOSTSWAP and it WORKS!
(they are LOW PROFILE and dont have BBU). Funny thing is that I’ve read that there IS A WAY to do HOSTSWAP WITH SATA disk (as far as I can remember – detach the disk call hdparm to spin it off and then plug out, plug in the new one and call hdparm to spin it up again… never tested this!)
anyways. I’m for using both technologies but only if you have spare controller which is exactly the same as the one running, otherwise you’ll get in troubles some day. And this day will be the day that you can’t wait 5 hours to get a new hw raid card… :-D
@Amr El-Sharnoby here is why you should not use LVM2 mirroring.

Reply

14 Harshad July 9, 2009 at 12:33 pm

I personally feel if RAID is used along with LVM gives you much more reliable way of storing data without wasting any disk space.

Reply

15 Amr El-Sharnoby July 11, 2009 at 4:37 pm

Thanks, valqk ,

But for striping, LVM should be chosen, !

Reply

16 Charanjit Singh July 21, 2009 at 4:09 pm

Awesome friend whenever I visit at your site i always find some thing new tips and tricks. You are doing very good job this will encourage people for choosing open source as career. Thanks.

Regards
Charanjit Singh
System Admin, RHCT

Reply

17 Jon July 27, 2009 at 3:28 am

I’ve been using Software RAID for personal use for a while now and it’s worked great. I have a software RAID 5 and RAID 1 under LVM. This is a great solution as I can mix different RAID arrays together to make the most efficient use out of my drives. For example, I have two 250gb drives in a RAID 1 and three 500gb drives in a RAID 5, with LVM I can make these two arrays function as one large volume. If anybody is interested here is a tutorial on How to Install Fedora 11 and Set up a Software RAID 0, 1, 5, 6 or 10 Array with LVM Great comparison, enjoyed the article, for home use software RAID is the best because of lower cost and increased flexibility.

Reply

18 SSEGAWA DRRICK September 4, 2009 at 12:44 pm

THANKS FOR THAT GOOD INFORMATION AND KEEP ON. DERRICK SSEGAWA KAMPALA UGANDA.

Reply

19 Galvao January 27, 2011 at 3:54 am

Hi,

My server Windows crash with RAID1 software. the technician said “the two discs stopped working” Is this possible? “we attach these drives to this server or to another server, the server’s BIOS does not recognize them”

I find it amazing that they can not recover anything.

I hope your comment

Thanks!

Reply

20 Kris March 12, 2011 at 7:32 am

@Galvao
This is recoverable. Each disk should contain exactly the same info (provided the software doing your RAID1 didn’t fail before this hardware failure, in which case one disk would be time stated to that point) unless they both completely failed (hard drive motor no longer functions and/or every head on the hard drive crashed and scratched the disk platters). This would be extremely unlikely for both to be failed at the exact same time. Unless there was a serious power surge that had gone through your entire system and reached both hard drives. EXTREMELY unlikely.

In any case, there are still bits that exist on the hard drive and for a high price there are data recovery centers that can place the platters in another hard drive encasement and read what is available. Provided the heads didn’t scratch the platters to severely, any data outside of scratched areas should be recoverable. Depends what the data is worth to you

Cheers

Reply

21 Andrew April 1, 2011 at 7:12 am

@ galvo, sound suspicious to me that both would crash at the same time. Are there other WORKING drives connected to the same controller ?? Just a thought.. Well i am from the old school of thought of thought that use only H/W raid. But i am starting to change my point of view on that and am going to change my current setup. ,, I am beginning to think my controllers are bottle necking my performance while i have (2) xeon 3.6 sitting doing nothing all day. I was just looking for an articale to reaffim my thoughts. When i set up my first S/Ware raid (12 fiber disc on my home PC) it was pretty fast. The only drawback i noticed was that the drives wined alot and sounded like machine gun fire or popcorn, even when just sitting there. Through the H/Ware raid they didn’t do that. The dont clicity klack on my server.. I got 2 seperate controllers in my PC and hardly make make any noise in there iether (h/w raid 0, 2chan x 3disc) I dont know. it will probably take me a week to figure out. Will probably go 0 – 0 and stripe or span through OS

Reply

22 Rixkster April 27, 2011 at 5:15 pm

You said:
“…So which one is better software raid or hardware raid?
Short answer – None…”

Wow, Hardware is the only important answer, short, or long.

Dude, Dedicated hardware Raid is infinitely better than Software Raid.
You might as well say that your “1″ analog Modem/phone was fantastic until someone picked up the phone line -oops ?
In other words, if you lose your OS, you have a good possibilty of losing all your Raid -you just lost ALL your Data. get it?
I lost my OS completely, my Windows software raid got pffft’d, But luckily, I had my important data on a Hardware Raid 10 controller card.
After I swapped out for a new MB and re-installed OS, my hardware raid controller came up with flying colors -NO data loss.
-not to mention, that
1./ Software Raid is extremely “limited” on its Raid variations, and capabilities depending on the “robustness” of your OS, and also how deep is your wallet,
You wanna talk Windows software Raid, where all your eggs are in 1 Basket? , wasting CPU/Memory cycles ?, then just forget it -LMAO.
And, even then, you will defintely pay for “good” software Raid -in other words, you’re much better off getting a dedicated Hardware raid (HBA) controller.
BTW, Windows Sotware Raid, such as Windows 8 Sever, still comes in only 2 colors, (Raid 1, and Raid 5) -and you’re gonna pay for it $.
For that extra money -just buy the safety of a high quality dedicated hardware-raid controiller card /with the Cables….
:)
2./ Hardware Raid is blazingly faster than any Software/BIOS Raid garbage too -hands down.
3./ Don’t be silly, if your DATA is that important.

seriously,

Software Raid ? -get a grip will ya.

gzzz,

Reply

23 JohnSmith March 5, 2012 at 8:02 am

“Wow, Hardware is the only important answer, short, or long.”

Now I’m the one LAMO. You concern yourself about wasting CPU cycles & state H/W RAID only _real_ option, yet you use Windows. Don’t be silly, if your DATA is that important. Are you a MCSE? My little n00blet, how about you read this before making such absurd statements:

Backblaze Blog: Petabytes on a budget: How to build cheap cloud storage

67 TB, RAID6, not a dedicated HBA in sight. Seriously -get a grip will ya.

ps/ MCSE =
Must
Consult
Someone
Else

Reply

24 kjaid March 11, 2012 at 8:21 am

“At Backblaze we have developed software that de-duplicates and chops data into blocks; encrypts and transfers it for backup; reassembles, decrypts, re-duplicates, and packages the data for recovery; and monitors and manages the entire cloud storage system. This process is proprietary technology that we have developed over the years.”

what does the backblaze blog proof here?
in my mind, only if you deploy such a complicated setup as the guys at backblaze has done, you can probobly sit back and relax about your software based raid …

sorry johnsmith, but just because software raid is possible, it doesn’t replace a professional solution

Reply

25 keith August 31, 2011 at 11:10 am

I do not believe anyone here would seriously consider using a WINDOWS software Raid!
:-D

Reply

26 pablo February 19, 2014 at 10:16 pm

I’m afraid they do. I was refusing to believe it but I think what they posted amounts to proof. xD

Reply

27 Bill September 2, 2011 at 1:47 pm

Hi,

I was thinking of creating the following RAID set on one of our powerpc xserves running 10.5 (non server). Curious if anyone has feedback on the configuration. I am just learning about RAID setups, and to this point I have been simply using RAID 1 on all our data to avoid drive failure issues. However we run a design firm and having multiple separate volumes can be annoying. So after reading a tiny bit I was considering the following configuration:

One RAID 0 stripped set (created using the raid feature in OSX disk utility).
Inside this Raid set I would add RAID 1 drives (like lacie 2big or WD Mybook II) that are mirrored, thinking 5 or 6. Not sure if there are any benefits to the number of drives one way or the other.

So it would be a combination of software and hardware. I like this idea because i could easily swap failed drives but also increase access speed. My one concern is the RAID 0 portion becoming corrupt and not being able to retrieve the data. Will I loose it all? It will be a lot of data to keep backed up.

I would love to hear any and all feedback! thanks.

Bill

Reply

28 mekh February 17, 2012 at 4:30 pm

very useful article
Thank you

Reply

29 Michael August 17, 2012 at 12:58 pm

You are wrong in several aspects. Never mix hardware raid and zfs, for instance.

Reply

30 David A. Lethe October 12, 2012 at 2:09 am

This article is absolute crap and grossly inaccurate. I’ve got 20+ years designing RAID hardware, firmware, software, diagnostics, software RAID stacks … Here are just a few things that are wrong:
* Writeback caching. A BBU is not necessary depending on the implementation of the RAID write hole. Example, Solaris does NOT need a BBU, nothing is lost in event of an incomplete rewrite. Read up on ZFS file system to learn why
* Performance – GROSSLY wrong. Microsoft, linux, and all the *nix do read load balancing. If each disk can read 100MB/sec sustained then anybody who runs performance monitor and dynamic RAID1 can easily see aggregate read I/O in the 160+ MB/sec.
* Overhead? It takes MILLISECONDS to get the data, it takes NANOSECONDS to calculate parity. Do the math.
* No software RAID hot swapping? Ever see a rackmount JBOD enclosure that WASN’T hot swap? The manufacturers wouldn’t go to the expense if they didn’t work.
* software RAID such as in solaris can do hot snapshots. Want a full backup as of 2:17 PM? Type in the snapshot command, give it a file name, and then creates a backup as of that instant in parallel while you are doing I/O.
* Cache? You can add SSDs in same RAID pool. You USE system RAM for additional cache. That is a plus. No limit of 256MB of cache, if it benefits you, then you could have GB of cache.

Reply

31 dangerous May 26, 2013 at 8:07 pm

how can i see the hardware & Software raid with ssh ?
i want know in myserver which hdd has software raid ? and witch hdd has hardware raid ? do you have any ssh command for do it ?
tnx

Reply

32 wumpus June 20, 2013 at 3:19 pm

I to am amused by the idea of keeping important data on windows. Really shows how what level of reliability you are talking about.

Also hardware RAID gives the added benefit that once it fries you have to find *exactly* compatible hardware to run your disks *at all*. Obviously, you should keep a few tested (and tested on at least duplicated live arrays) cards around, but there is always the case where the spare is as dead as the former live hardware. At that point you can only hope that you can buy the correct board and flash it to the correct bios level and hope it does what it says on the tin. Don’t expect to get going any time soon, while software RAID can run on *anything*.

Real pros avoid hardware RAID like the plague until they absolutely need its performance. Even then it is a disaster waiting to happen.

Reply

33 pablo February 19, 2014 at 10:24 pm

I would like to mention this interesting read: http://www.miracleas.com/BAARF/BAARF2.html

Reply

Leave a Comment

Tagged as: , , , , , , , , ,

Previous post:

Next post: