A redundant array of inexpensive disks (RAID) allows high levels of storage reliability. RAID is not a backup solution. It is used to improve disk I/O (performance) and reliability of your server or workstation. A RAID can be deployed using both software and hardware. But the real question is whether you should use a hardware RAID solution or a software RAID solution.
In this post I will document my experience with both software and hardware RAID.
Comparison: Hardware vs Software RAID
| Feature | Software RAID | Hardware RAID |
|---|---|---|
| Cost: Software RAID is part of OS, so no need to spend extract money. |
Low | High |
| Complexity: The software RAID works on partition level and it can sometime increase complexity if you mix different partitions and hardware RAID. |
Medium to high | Low |
| Write back caching (BBU): The software RAID cannot add a battery. Hardware RAID can run in write-back mode if it has a BBU installed. With BBU pending writes are not lost on a power failure. |
No | Yes |
| Performance: With the software based RAID0 and RAID1 performance is negligible. However, performance goes down when you use parity-based arrays and/or several arrays at the same time. The performance of a software-based array is dependent on the server CPU performance and current load. |
Depend upon usage | High |
| Overheads (CPU, RAM etc): The software RAID must use server's CPU and RAM for RAID software. The more hard drives means more CPU cycle will go to software RAID instead of your Apache / Postfix or MySQL server. |
Depend upon usage | No |
| Disk hot swapping: It means replacing hard disk without shutting down the server. Many RAID controller supports disk hot swapping. |
No | Yes |
| Hot spare support: A hard disk is physically installed in the array which stays inactive until an active drive fails, when the system automatically replaces the failed drive with the spare, rebuilding the array with the spare hard disk included. |
Yes | Yes |
| /boot partition: It is hard to make fail over with software RAID if /boot fails while booting the server. This can result into unexpected errors and data loss. However, LILO and FreeBSD loader can get around this problem too. |
No | Yes |
| Open source factor: *BSD / OpenSolaris and Linux RAID software drivers are open source. It means more people can fix problems as compare to a closed source hardware firmware. You can move, mix and match different sizes with open source software RAID. |
Yes | No |
| Vendor lock in (open formats): See above. | No | Yes |
| Higher write throughput: Hardware RAID with BBU may offers higher write throughput. |
No | Yes |
| Faster rebuilds: Hardware RAID with BBU may offers faster rebuilds as compare to software based solution. |
No | Yes |
| Can act as a backup solution?: Both software and hardware RAID cannot protect you against human errors or system failures or viruses. Daily scheduled and off site backups of your system are highly recommended. Use tools such as rsync, rsnapshot, tar, dump, restore and others to make daily backups. |
No | No |
| Recommend usage: |
+Low cost solution +Better for RAID0 or RAID1 +Single server / workstation +Perfect for home and small business users. +No vendor lock-ins |
+Do you run a mission critical cluster or setup? +Heavy database driven dynamic site +Do you want the highest performance possible? |
Other Factors
Powerful Modern CPU
The performance of a software-based array is dependent on the server CPU performance and load. With today's faster CPUs, software RAID outperforms hardware RAID.
Can RAID Array Fail?
Yes. The entire RAID array can fail taking down all your data (yes hardware RAID card do dies). Use tapes and other servers that can hold copies of the data, but don't allow much interaction with it. Move your data offsite. Another option is to use two or three RAID cards. Combine them together to protect your data. This make sure you gets back your data when one of your RAID card dies out.
Hardware vs Software Recovery
My personal experience - recovering from software RAID is easy. However, sometime finding out exact hardware RAID requirements can be a nightmare. A good backup can save from RAID hardware incompatibility problems. Software RAID allows you to mix different drive and sizes. You can not do something like this with hardware RAID cards. With software RAID you can swap the drives to a different server and read the data. There is no vendor lock in with software RAID solution.
You Can Not Go Wrong With Hardware RAID
There is an old saying in IT - no one ever got fired for picking RAID controllers.
Use Both Hardware and Software RAID
Sometime you need to use both hardware and software RAID to get the best of both worlds. For e.g. set up 4 mirror pairs, 2 on each hardware RAID controller, and use software RAID0 to put it all together. This will give the best performance for database server. Here is another example from one of our DR site server (this box mirrors our 30+ production server files and database):
- Server chassis with redundant power supplies
- Intel or AMD Dual Core CPU x 2
- 16GB ECC RAM
- 24 hot swappable drive bays
- 2 x RAID PCIe / PCIx RAID hardware controller
- 4 x Intel 1000 PCIx Lan cards (bond them together)
- 24 x 1TB SATA hard disk
- OS - Pick - Linux / FreeBSD / OpenSolaris
- Filesystem - Pick - ZFS / UFS / Ext3 (we use RAID-Z)
- Backup software - rsync, rsnapshots and MySQL in slave mode.
Now you can configure RAID0 stripe across the three RAID6 arrays (8 x 24 disks) using both hardware and software solution together. This massive storage system is perfect for online live backups.
Conclusion
So which one is better software raid or hardware raid?
Short answer - None.
Long answer - It depends upon your setup and requirements. I strongly recommend running both with benchmarking software to find out your disk I/O. Test both solutions by removing hard disk i.e. fail a few drives. Try running system while drives are failed. Note down system load and errors (if any). Reboot the system. See if you can boot. Can you see your data again? Are you comfortable using tools provided with both solutions? See what works for you.
Finally, while choosing a storage always consider speed, reliability, and cost - pick any two.
A Final Note About My Personal Choice
I have been successfully using Linux and FreeBSD software RAID for several years for backing up my own data. I prefer to use software RAID to save money and to avoid vendor lock ins. All my personal data backup using the following hardware:
- 1.5 TB USB hard disk - rsync and rsnapshot is used to make backup of all my servers and digitial camera.
- 80GB x 3 hard disk software RAID using FreeNAS. Again, rsync is used to make all backups. I'm planing to replace UFS with RAID-Z under FreeBSD 8.
RAID Alternatives
Local disks on MogileFS storage nodes can be in a RAID, or not. It's cheaper not to, as RAID doesn't buy you any safety that MogileFS doesn't already provide. This is quite popular among the web 2.0 companies where lots of photos, images and files are uploaded by their users.
RAID-Z ZFS Storage is a data/parity scheme like RAID-5, but it uses dynamic stripe width. Every block is its own RAID-Z stripe, regardless of blocksize. This means that every RAID-Z write is a full-stripe write. It doesn't require any special hardware.
References:
- RAID 5 vs RAID 10 - Recommended RAID For Safety and Performance
- RAID article from the wikipedia, the free encyclopedia.
- Linux: Why software RAID? by Jeff Garzik.
- How To: Linux Software RAID
- How To: FreeBSD Software RAID, from the official FreeBSD handbook.
- ZFS (the z file system) from Sun
.
What do you think? Please add your experience in the comments below.
Featured Articles:
- 20 Linux System Monitoring Tools Every SysAdmin Should Know
- 20 Linux Server Hardening Security Tips
- 10 Greatest Open Source Software Of 2009
- My 10 UNIX Command Line Mistakes
- Top 5 Email Client For Linux, Mac OS X, and Windows Users
- Top 20 OpenSSH Server Best Security Practices
- Top 10 Open Source Web-Based Project Management Software
- Top 5 Linux Video Editor Software
- Email this to a friend
- Download PDF version
- Printable version
- Comment RSS feed
- Last Updated: Jul/7/2009
{ 18 comments… read them below or add one }
Simple test to find out the approximate linear write speed of your hard disk/RAID arrays/etc
dd if=/dev/zero/ of=/root/1GBfile bs=1M count=1024
For example I got around 160 MB/s on 2 Hard disks in RAID1 ext3. Using XFS the values
are slightly higher.
The command above can give you a hint if you don’t have tine/don’t want to run bonnie and
similar programs.
Today I created a 32 GB secondary (pri=0) swap file and the average write speed was
~ 80 MB/s. This means that my RAID controller has some errors. Vivek can you post
an article about how to debug hardware RAID problems like bad write/reads ?
Note: The speed returned is not the average read/write speed since usually files tend
to fragment and very important the information that is being read in normal operation mode is not continuous, but the aberration is lower than 35%.
Note2: The above is for Linux (tested under Debian and Ubuntu Server).
i really dont get it.
I CANT buy a server with LESS THAN 4 CPUs.
One of them HAS to be able to run the Raid-5 parity calculations, right?
or is the MD/LVM driver not that smart yet?
You can use smartool and cli tools provided by your RAID device. Watch out our FAQ section. I will add another FAQ regarding this issue.
@kim: No it is not that smart yet. It will not do anything special.
HTH
smartmontools looks like what I need, I am looking forward for your article.
Thanks
have been using software raid1 on CentOs 4.5 but find it hard to manage, every kernel upgrade can break the array and drivers are not too good also this not tested on CentOs 5.
lately we switched to hardware raid but there are also somme issues between raid1 array on 15K and jbod on 10K.
so not found yet the solution that would be good for us.
thanks for the info!!
Nice.
finally found what I have been looking for sometime..
I agree with the author. I do too use software RAID for a couple of years and been really satisfied with it.
FYI: Cosmin, I suggest u to use mdadm for your array solution and mount the partition at startup. If you need any help, ill be glad to assist u… Just send me an e-mail.
Awesome write up man. Just a question, dont you think those 80GB hard drives eat up power?
@Solaris thanks for that nice quasi-hack on testing drive throughput :)
Sorry for the extra spam, but i think a subscribe to comments plugin will allow people to turn this into a semi discussion.
@Raja,
I only use those 80GB to backup data once a day.
There is rss comment feed for each page (just click on rss button @ firefox awesome bar). Nevertheless, I will add a subscribe to comments plugin as well.
I’d highly recommend that you consider using LVM striping and mirroring under Linux, that works like RAID-0 and RAID-1 allowing you to do all the magic of LVM ….
I’m using it for highly load servers with relatively new Xeon and Opteron CPUs and I can find its performance outperforming Modern Hardware RAID controllers with much more stability …
I’ve faced too much trouble with hardware raid controllers with those servers, the controllers caused too much instability and many failures , I’ve changed them, upgraded to newer models, updated firmware but all didn’t solve the problem,
so I’ve moved to LVM solution, as I’ve more than 30% idle CPU time even under high loads , while that LVM overhead is roughly 3 to 5% …
All problems has been solved, and I’m quite happy with tht stabilty and performance …
I’m using RHEL 5.3 on those servers ..
I personally use Software raids (md linux and gmirror in fbsd). That’s because I can’t afford expensive models of 3ware and similar and also because you MUST have at LEAST 1 spare exactly the same one as the one in machine. I’ve bought 2 3ware 8006lp2. They work very good but unfortunately the speed test fails because they are SATAI not SATAII. The good thing is that they DO have HOSTSWAP and it WORKS!
(they are LOW PROFILE and dont have BBU). Funny thing is that I’ve read that there IS A WAY to do HOSTSWAP WITH SATA disk (as far as I can remember – detach the disk call hdparm to spin it off and then plug out, plug in the new one and call hdparm to spin it up again… never tested this!)
anyways. I’m for using both technologies but only if you have spare controller which is exactly the same as the one running, otherwise you’ll get in troubles some day. And this day will be the day that you can’t wait 5 hours to get a new hw raid card… :-D
@Amr El-Sharnoby here is why you should not use LVM2 mirroring.
I personally feel if RAID is used along with LVM gives you much more reliable way of storing data without wasting any disk space.
Thanks, valqk ,
But for striping, LVM should be chosen, !
Awesome friend whenever I visit at your site i always find some thing new tips and tricks. You are doing very good job this will encourage people for choosing open source as career. Thanks.
Regards
Charanjit Singh
System Admin, RHCT
I’ve been using Software RAID for personal use for a while now and it’s worked great. I have a software RAID 5 and RAID 1 under LVM. This is a great solution as I can mix different RAID arrays together to make the most efficient use out of my drives. For example, I have two 250gb drives in a RAID 1 and three 500gb drives in a RAID 5, with LVM I can make these two arrays function as one large volume. If anybody is interested here is a tutorial on How to Install Fedora 11 and Set up a Software RAID 0, 1, 5, 6 or 10 Array with LVM Great comparison, enjoyed the article, for home use software RAID is the best because of lower cost and increased flexibility.
THANKS FOR THAT GOOD INFORMATION AND KEEP ON. DERRICK SSEGAWA KAMPALA UGANDA.