How To Avoid Sudden Outburst Of Backup Shell Script or Program Disk I/O on Linux

A sudden outburst of violent disk I/O activity can bring down your email or web server. Usually, a web, mysql, or mail server serving millions and millions pages (requests) per months are prone to this kind of problem. Backup activity can increase current system load too. To avoid this kind of sudden outburst problem, run your script with scheduling class and priority. Linux comes with various utilities to manage this kind of madness.

ADVERTISEMENTS

CFQ scheduler

You need Linux kernels 2.6.13+ with the CFQ IO scheduler. CFQ (Completely Fair Queuing) is an I/O scheduler for the Linux kernel, which is default in 2.6.18+ kernel. RHEL 4/ 5 and SuSE Linux has all scheduler built into kernel so no need to rebuild your kernel. To find out your scheduler name, enter:
# for d in /sys/block/sd[a-z]/queue/scheduler; do echo "$d => $(cat $d)" ; done
Sample output for each disk:

/sys/block/sda/queue/scheduler => noop anticipatory deadline [cfq]
/sys/block/sdb/queue/scheduler => noop anticipatory deadline [cfq]
/sys/block/sdc/queue/scheduler => noop anticipatory deadline [cfq] 

CFQ is default and recommended for good performance.

Old good nice program

You can run a program with modified scheduling priority using nice command (19 = least favorable):
# nice -n19 /path/to/backup.sh
Sample cronjob:
@midnight /bin/nice -n19 /path/to/backup.sh

Say hello to ionice utility

The ionice command provide better control as compare to nice command for the I/O scheduling class and priority of a program or script. It supports following three scheduling classes (quoting from the man page):

  • Idle : A program running with idle io priority will only get disk time when no other program has asked for disk io for a defined grace period. The impact of idle io processes on normal system activity should be zero. This scheduling class does not take a priority argument.
  • Best effort : This is the default scheduling class for any process that hasn’t asked for a specific io priority. Programs inherit the CPU nice setting for io priorities. This class takes a priority argument from 0-7, with lower number being higher priority. Programs running at the same best effort priority are served in a round-robin fashion. This is usually recommended for most application.
  • Real time : The RT scheduling class is given first access to the disk, regardless of what else is going on in the system. Thus the RT class needs to be used with some care, as it can starve other processes. As with the best effort class, 8 priority levels are defined denoting how big a time slice a given process will receive on each scheduling window. This is should be avoided for all heavily loaded system.

Syntax

The syntax is:

ionice options  PID
ionice options -p PID
ionice -c1 -n0  PID

How do I use the ionice command on Linux?

Linux refers the scheduling class using following number system and priorities:

Scheduling class Number Possible priority
real time 1 8 priority levels are defined denoting how big a time slice a given process will receive on each scheduling window
best-effort 2 0-7, with lower number being higher priority
idle 3 Nil ( does not take a priority argument)

Examples

To display the class and priority of the running process, enter:
# ionice -p {PID}
# ionice -p 1

Sample output:

none: prio 0

Dump full web server disk / mysql or pgsql database backup using best effort scheduling (2) and 7 priority:
# /usr/bin/ionice -c2 -n7 /root/scripts/nas.backup.full
Open another terminal and watch disk I/O network stats using atop/tip or top or your favorite monitoring tool:
# atop
Sample cronjob:
@weekly /usr/bin/ionice -c2 -n7 /root/scripts/nas.backup.full >/dev/null 2>&1
You can set process with PID 1004 as an idle io process, enter:
# ionice -c3 -p 1004
Runs rsync.sh script as a best-effort program with highest priority, enter:
# ionice -c2 -n0 /path/to/rsync.sh
Type the following command to run ‘zsh’ as a best-effort program with highest priority.
# ionice -c 2 -n 0 zsh
Finally, you can combine both nice and ionice together:
# nice -n 19 ionice -c2 -n7 /path/to/shell.script
Related: chrt command to set / manipulate real time attributes of a Linux process and taskset command to retrieve or set a processes’s CPU affinity.
To see help on options type:
$ ionice --help
Sample outputs:

Sets or gets the IO scheduling class and priority of processes.
 
Usage:
 ionice [options] -p <pid>...
 ionice [options] -P <pgid>...
 ionice [options] -u <uid>...
 ionice [options] <command>
 
Options:
 -c, --class <class>    name or number of scheduling class,
                          0: none, 1: realtime, 2: best-effort, 3: idle
 -n, --classdata <num>  priority (0..7) in the specified scheduling class,
                          only for the realtime and best-effort classes
 -p, --pid <pid>...     act on these already running processes
 -P, --pgid <pgrp>...   act on already running processes in these groups
 -t, --ignore           ignore failures
 -u, --uid <uid>...     act on already running processes owned by these users
 
 -h, --help     display this help and exit
 -V, --version  output version information and exit

Other suggestion to improve disk I/O

  1. Use hardware RAID controller.
  2. Use fast SCSI / SA-SCSI / SAS 15k speed disk.
  3. Use fast SSD based storage (costly option).
  4. Use slave / passive server to backup MySQL
Recommended readings:
  • man page – ionice(1)
🐧 Get the latest tutorials on SysAdmin, Linux/Unix, Open Source & DevOps topics via:
CategoryList of Unix and Linux commands
File Managementcat
FirewallAlpine Awall CentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network Utilitiesdig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot cron disown fg jobs killall kill pidof pstree pwdx time
Searchinggrep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNAlpine CentOS 8 Debian 10 Firewall Ubuntu 20.04

ADVERTISEMENTS
14 comments… add one
  • Adam Ward Jun 3, 2008 @ 19:40

    Utilizing a CDP block-level backup solution will reduce your backup load tremendously.

    Adam

  • Gregor Jun 4, 2008 @ 15:59

    Why in heck didn’t I know about ionice? Many of my apps every day involve major disk IO which slows down everybody else. This is a life saver!

  • Steve Jul 7, 2008 @ 21:46

    ionice is letting me run backup scripts and the yum updater, etc with the websites flying along instead of timing out, this is fantastic, thanks for pointing it out.

  • Amr El-Sharnoby May 23, 2009 @ 13:15

    Thanks,
    Please fix the spelling mistake; iosnice should be ionice

  • Martin Aug 5, 2009 @ 18:56

    OT but have to mention that here shortly:

    Often I google/yahoo for a specific problem and in the end I’m landing here, could even start looking this site.

    Great work, man! Go on like that.

    Greetings from Europe!
    Martin

  • AskApache Aug 22, 2009 @ 16:40

    Been googling for an hour, this is the finest ionice page I’ve found yet! This is exciting, saves me from having to do an unneccessary upgrade.

  • Rob Nov 2, 2009 @ 21:14

    Tnx!
    Allways bugging me when running cp script or another script eating up the performance.. put it in my webserver… exactly what i was looking for…and yeah also for me the same, was searching for an hour to find till ended up here and found what i want.. Tnx

  • Holger Mar 31, 2011 @ 19:24

    Thanks for your hint. I wasn’t aware of the existence of ionice. Using ionice plus nice for rsync backup jobs tremendously lowers the load of your system. This is great when backing up your Linux workstation to an external connected USB hdd an still want to play a game or browse the net.

    CFQ itself is not always the answer for bad responsiveness of x64 Linux hosts.

    BTW: A hardware raid controller does not always help in the first place. Our IBM servers which run a KVM-based virtualisation solution bog down if you copy the HDD image of one VM to another VMs HDD image (= cloning the VM). I think ionice will help here a lot.

  • David Jul 12, 2011 @ 10:50

    This is stupid. A heavy IO activity shouldn’t make the system unresponsive. This behavior should be automatic. That what schedulers are for.

  • Andres Jul 21, 2012 @ 7:43

    Thanks for that info.

    Shouldn’t you add “use SSD storage” under your suggestions to improve I/O?. I cannot understand why it’s not already on the article.

  • Julio Jul 23, 2012 @ 18:40

    And what about using nice to modify priority to a Cron Job that consists on a “/bin/php -f /home/user/script.php” ?
    Is it possible? I tried creating a bash script that executes php, but it’s not executed.

  • Paul Thomson Sep 30, 2013 @ 9:55

    Perfect for s3cmd sync not hoggin’ resources.

    Keep up the good work, stay safe!
    Paul 🙂

  • Mike Johnson Dec 16, 2015 @ 21:13

    Don’t forget cgroups!

  • Cr0t Dec 18, 2015 @ 2:40

    I forgot what it is called, but the kernel has a new query feature or something like that, which makes that option obsolete. – Sorry, I am unable to re-call the feature.

Leave a Reply

Your email address will not be published. Required fields are marked *

Use HTML <pre>...</pre>, <code>...</code> and <kbd>...</kbd> for code samples.