Perform backups for the Linux operating system

This question asked again and again by a new Linux sys admins:

How do I perform backups for my Linux operating system?

So I am putting up all necessary information you ever need to know about backup. The main aim is to provide you necessary software, links and commands to get started as soon as possible.

Backup is essential

First a backup is essential. You need a good backup strategy to:

Minimize time from disaster such as server failure or human error (file deleted) or acts of God
To avoid downtime
Save money and time
And ultimately to save your job ;)

A backup must provide

Restoration of a single/individual files
Restoration of file systems

What to backup?

User files and dynamic data [databases] (stored in /home or specially configured partitions or /var etc).
Application software (stored in /usr)
OS files
Application configuration files (stored in /etc, /usr/local/etc or /home/user/.dotfiles)

Different types of backups

Full backups: Each file and directory is written to backup media
Incremental backups (Full + Incremental backup): This backups are used in conjunction with full backup. These backups will be incremental if each original piece of backed up information is stored only once , and then successive backups only contain the information that changed since the previous one. It use file's modification time to determine which file need to backup.

So when you restore incremental backup:

First restore the last full backup
Next every subsequent incremental backup you need to restore

Preferred Backup Media

Tape (old and trusted method)
Network (ftp, nas, rsync etc)
Disk (hard disk, optical disk etc)

Test backups

Please note that whichever backup media you choose, you need to test your backup. Perform tests to make sure that data can be read from media.

Backup Recommendation

My years of experience show that if you follow following formulas you are most likely to get back your data in worst scenario:
(a) Rotate backup media
(b) Use multiple backup media for same data such as ftp and tape
(c) Keep old copies of backups offsite

In short create good disaster recovery plan.

General procedure to restore a Linux/UNIX box

There is not golden rule or procedure but I follow these two methods:

Method # 1: Reinstall everything, restore everything, and secure everything

Use this method (bare metal recovery) if your server is cracked or hacked or hard drive is totally out of order:

Format everything
Reinstall os
Configure data partitions (if any)
Install drivers
Restore data from backup media
Configure security

Method # 2: Use of recovery CD/DVD rom

Use this method if your box is not hacked and system cannot boot or MBR damaged or accidental file deletion etc:

Boot into rescue mode.
Debug (or troubleshoot) the problem
Verify that disk partitions stable enough (use fsck) to put backup data
Install drivers
Restore data from backup media
Configure security

Linux (and other UNIX oses) backup tools

Luckily Linux/UNIX provides good set of tools for backup. We have almost covered each and every tool mentioned below. Just follow the link to get more information about each command and its usage:

Backing to tape using tar to tape, tar over ssh, cpio, and dump command. tar and friends are good for small backups. For large scale backup or backup that demands large CPU and I/O, use other solution (see below).
Backing to another server using ftp, ftp to NAS, access NAS server using NFS protocol, access NAS from samba/windows, rsync UNIX/Linux server or windows servers
Backing to recordable media such as CDR or DVD

It is also recommended that you use RAID or LVM (see consistent backup with LVM) or combination of both to increase reliability of data.

A note about MySQL or Oracle database backup

Backing up database server such as MySQL or Oracle needs more planning. Generally you can apply a table write lock and use mysql database dump utility to backup database. You can also use LVM volume to save database data.

A note about large scale backup

As I said earlier tar is good if you need to backup small amount of data that does not demands high CPU or I/O. Following are recommended tools for backup that demands high CPU or I/O rate:

(a) amanda - AMANDA, the Advanced Maryland Automatic Network Disk Archiver, is a backup system (open source software) that allows the administrator to set up a single master backup server to back up multiple hosts over network to tape drives/changers or disks or optical media.

(b) Third party commercial proprietary solutions:
Top three excellent commercial solutions:

If you are looking to perform the tasks of protecting large-scale computer systems use above solutions and following two books will give you good idea:

Implementing Backup and Recovery: The Readiness Guide for the Enterprise - This gives you an introduction to backup and recovery concepts in the enterprise and covers disaster recovery, and other solutions such as Veritas Netbackup.
Backup & Recovery (UNIX) - This unique book shows you how to ensure data protection without resorting to expensive commercial solutions