How To Avoid Sudden Outburst Of Backup Shell Script / Program Disk I/O
A sudden outburst of violent disk I/O activity can bring down mail or web server. Usually, a web / mysql or mail server serving millions and millions pages per months are prone to this kind of problem. Backup activity can increase current system load from usual 5 to 20 (or more).
To avoid this kind of sudden outburst problem, run your script with scheduling class and priority. Linux comes with various utilities to manage this kind of madness.
CFQ scheduler
You need Linux kernels 2.6.13+ with the CFQ IO scheduler. CFQ (Completely Fair Queuing) is an I/O scheduler for the Linux kernel, which is default in 2.6.18+ kernel. RHEL 4/ 5 and SuSE Linux has all scheduler built into kernel so no need to rebuild your kernel. To find out your scheduler name, enter:
# for d in /sys/block/sd[a-z]/queue/scheduler; do echo "$d => $(cat $d)" ; done
Sample output for each disk:
/sys/block/sda/queue/scheduler => noop anticipatory deadline [cfq] /sys/block/sdb/queue/scheduler => noop anticipatory deadline [cfq] /sys/block/sdc/queue/scheduler => noop anticipatory deadline [cfq]
CFQ is default and recommended for good performance.
Old good nice program
You can run a program with modified scheduling priority using nice command (19 = least favorable):
# nice -n19 /path/to/backup.sh
Sample cronjob:
@midnight /bin/nice -n19 /path/to/backup.sh
iosnice utility
ionice command provide more control as compare to nice command. This program sets the io scheduling class and priority for a program or script. It supports following three scheduling classes (quoting from the man page):
- Idle : A program running with idle io priority will only get disk time when no other program has asked for disk io for a defined grace period. The impact of idle io processes on normal system activity should be zero. This scheduling class does not take a priority argument.
- Best effort : This is the default scheduling class for any process that hasn’t asked for a specific io priority. Programs inherit the CPU nice setting for io priorities. This class takes a priority argument from 0-7, with lower number being higher priority. Programs running at the same best effort priority are served in a round-robin fashion. This is usually recommended for most application.
- Real time : The RT scheduling class is given first access to the disk, regardless of what else is going on in the system. Thus the RT class needs to be used with some care, as it can starve other processes. As with the best effort class, 8 priority levels are defined denoting how big a time slice a given process will receive on each scheduling window. This is should be avoided for all heavily loaded system.
How do I use ionice command?
Linux refers the scheduling class using following number system and priorities:
| Scheduling class | Number | Possible priority |
| real time | 1 | 8 priority levels are defined denoting how big a time slice a given process will receive on each scheduling window |
| best-effort | 2 | 0-7, with lower number being higher priority |
| idle | 3 | Nil ( does not take a priority argument) |
To display the class and priority of the running process, enter:
# ionice -p {PID}
# ionice -p 1
Sample output:
none: prio 0
Dump full web server disk / mysql backup using best effort scheduling (2) and 7 priority:
# /usr/bin/ionice -c2 -n7 /root/scripts/nas.backup.full
Open another terminal and watch disk I/O network stats using atop or top or your favorite monitoring tool:
# atop
Sample cronjob:
@weekly /usr/bin/ionice -c2 -n7 /root/scripts/nas.backup.full >/dev/null 2>&1
You can set process with PID 1004 as an idle io process, enter:
# ionice -c3 -p 1004
Runs rsync.sh script as a best-effort program with highest priority, enter:
# ionice -c2 -n0 /path/to/rsync.sh
Finally, you can combine both nice and ionice together:
# nice -n 19 ionice -c2 -n7 /path/to/shell.script
Related: chrt command to set / manipulate real time attributes of a Linux process and taskset command to retrieve or set a processes's CPU affinity.
Other suggestion to improve disk I/O
- Use hardware RAID controller
- Use fast SCSI / SA-SCSI disk 15k speed disk
- Use slave / passive server to backup MySQL
Further readings:
- man page ionice
You may also be interested in other helpful articles:
- Linux Manipulating real-time attributes of a process
- Performance Tuning for Linux swap partition
- Comparison: Linux Disk Scheduler
- Programming: Understanding Linux Completely Fair Scheduler
- Display IP Address Allocation Table According to Subnet Mask
Discussion on This Article:
Leave a Reply
We encourage your comments, and suggestions. But please stay on topic, be polite, and avoid spam. Thank you very much for stopping by our site!
Tags: bandwidth management, bandwidth throttle software, bandwidth throttling, best effort, cfq, disk bandwidth throttling, grace period, hard disk, ionice command, limit bandwidth, linux bandwidth throttling, linux kernel, linux kernels, outburst, per second, priorities, priority programs, queue, sudden outburst, throttle bandwidth, throttle network bandwidth
Utilizing a CDP block-level backup solution will reduce your backup load tremendously.
Adam
Why in heck didn’t I know about ionice? Many of my apps every day involve major disk IO which slows down everybody else. This is a life saver!
ionice is letting me run backup scripts and the yum updater, etc with the websites flying along instead of timing out, this is fantastic, thanks for pointing it out.