Perform backups for the Linux operating system

Posted on in Categories Backup, Data recovery, Sys admin, Tips last updated October 25, 2006

This question asked again and again by a new Linux sys admins:

How do I perform backups for my Linux operating system?

So I am putting up all necessary information you ever need to know about backup. The main aim is to provide you necessary software, links and commands to get started as soon as possible.

Backup is essential

First a backup is essential. You need a good backup strategy to:

  • Minimize time from disaster such as server failure or human error (file deleted) or acts of God
  • To avoid downtime
  • Save money and time
  • And ultimately to save your job 😉

A backup must provide

  • Restoration of a single/individual files
  • Restoration of file systems

What to backup?

  • User files and dynamic data [databases] (stored in /home or specially configured partitions or /var etc).
  • Application software (stored in /usr)
  • OS files
  • Application configuration files (stored in /etc, /usr/local/etc or /home/user/.dotfiles)

Different types of backups

  • Full backups: Each file and directory is written to backup media
  • Incremental backups (Full + Incremental backup): This backups are used in conjunction with full backup. These backups will be incremental if each original piece of backed up information is stored only once , and then successive backups only contain the information that changed since the previous one. It use file’s modification time to determine which file need to backup.

So when you restore incremental backup:

  1. First restore the last full backup
  2. Next every subsequent incremental backup you need to restore

Preferred Backup Media

  1. Tape (old and trusted method)
  2. Network (ftp, nas, rsync etc)
  3. Disk (hard disk, optical disk etc)

Test backups

Please note that whichever backup media you choose, you need to test your backup. Perform tests to make sure that data can be read from media.

Backup Recommendation

My years of experience show that if you follow following formulas you are most likely to get back your data in worst scenario:
(a) Rotate backup media
(b) Use multiple backup media for same data such as ftp and tape
(c) Keep old copies of backups offsite

In short create good disaster recovery plan.

General procedure to restore a Linux/UNIX box

There is not golden rule or procedure but I follow these two methods:

Method # 1: Reinstall everything, restore everything, and secure everything

Use this method (bare metal recovery) if your server is cracked or hacked or hard drive is totally out of order:

  1. Format everything
  2. Reinstall os
  3. Configure data partitions (if any)
  4. Install drivers
  5. Restore data from backup media
  6. Configure security

Method # 2: Use of recovery CD/DVD rom

Use this method if your box is not hacked and system cannot boot or MBR damaged or accidental file deletion etc:

  1. Boot into rescue mode.
  2. Debug (or troubleshoot) the problem
  3. Verify that disk partitions stable enough (use fsck) to put backup data
  4. Install drivers
  5. Restore data from backup media
  6. Configure security

Linux (and other UNIX oses) backup tools

Luckily Linux/UNIX provides good set of tools for backup. We have almost covered each and every tool mentioned below. Just follow the link to get more information about each command and its usage:

It is also recommended that you use RAID or LVM (see consistent backup with LVM) or combination of both to increase reliability of data.

A note about MySQL or Oracle database backup

Backing up database server such as MySQL or Oracle needs more planning. Generally you can apply a table write lock and use mysql database dump utility to backup database. You can also use LVM volume to save database data.

A note about large scale backup

As I said earlier tar is good if you need to backup small amount of data that does not demands high CPU or I/O. Following are recommended tools for backup that demands high CPU or I/O rate:

(a) amandaAMANDA, the Advanced Maryland Automatic Network Disk Archiver, is a backup system (open source software) that allows the administrator to set up a single master backup server to back up multiple hosts over network to tape drives/changers or disks or optical media.

(b) Third party commercial proprietary solutions:
Top three excellent commercial solutions:

If you are looking to perform the tasks of protecting large-scale computer systems use above solutions and following two books will give you good idea:

Recommended further readings

I hope this small how to provide enough information to anyone to kick start your backup operation. Tell me if I am missing something or if you have a better backup solution or strategy, please comment back.

How to MySQL backup and data recovery with mysql-zrm

Posted on in Categories Backup, Data recovery, Howto, MySQL last updated September 28, 2006

This guide describes how to back up and recover your MySQL databases with mysql-zrm on a Debian Sarge system. mysql-zrm is short for Zmanda Recovery Manager for MySQL, it is a new tool that lets you create full logical or raw backups of your databases (regardless of your storage engine and MySQL configuration), generate reports about the backups, verify the integrity of the backups, and recover your databases. It can also send email notifcations about the backup status, and you can implement multiple backup policies (based on your applications and based on time (e.g. daily, weekly, etc.)).

Linux Get List of Installed Software for Reinstallation / Restore All the Software Programs

Posted on in Categories Backup, Linux, Shell scripting, Tips last updated August 22, 2006

Hardware and software failures are part of life. And that is why you need to have a backup plan. I have already written about backing up files and MySQL databases. There is no need to backup all installed binaries and software programs. The following tip will not just save your time, but both Debian and RHEL based distro can be updated instantly.

Linux and UNIX Backing up key information or files

Posted on in Categories Backup, Data recovery, Linux last updated August 15, 2006

Martin Brown shows us how to backup key files using tar and perl / shell scripts. It is a good article for all new sys admins (this goes well with my own shell scripts collection).

Most UNIX administrators have processes in place to back up the data and information on their UNIX machines, but what about the configuration files and other elements that provide the configuration data your machines need to operate? This article provides detailed information on techniques for achieving an effective and efficient backup system for these key files

The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes.

Read more at developerworks

Protect harddisk-based backup from accidental removal

Posted on in Categories Backup, FreeBSD, Linux, Security last updated July 28, 2006

If you store a backup on hard disk, the risk of rm -rf does exists.

Use chattr command changes the file attributes on a Linux second extended file system so that you will stay away from accidentally removing your backup files with rm -rf.

chattr command write protect the backup directory so that no one can delete it (including root). This is also known as making file immutable. For example your backup directory is /backup, type the following command to write protect directory /backup (only root can set or remove these flags):

# chattr -R +i /backup

To remove immutable flag, type the following command:

# chattr -R -i /backup

If you are using FreeBSD use chflags command to setup immutable flag:

# chflags -R schg /backup

To remove immutable flag, type the following command:

# chflags -R noschg /backup

These commands will keep you from accidentally removing your backup files with rm -rf 😀

Connecting Linux or UNIX system to Network attached storage device

Posted on in Categories Backup, CentOS, Data recovery, Debian Linux, File system, FreeBSD, Gentoo Linux, Howto, Linux, Troubleshooting, Ubuntu Linux, UNIX last updated May 20, 2006

Network attached storage (NAS) allows using TCP/IP network to backup files. This enables multiple servers in IDC to share the same storage for backup at once, which minimizes overhead by centrally managing hard disks. NAS is scalable, high performance network solution. The main advantage is more hard disk storage space added to a network that already utilizes servers without shutting them down for maintenance and upgrades.

Please note that NAS are not just common in IDC or offices but you can use it for file sharing and backup at home. You can purchase 200+GB NAS for less than $200 these days. Personally, I am using Maxtor ShareStorage 200GB Network Attached Storage at home. This is a step-by-step guide on connecting Linux or UNIX systems to SAN for backup or sharing files.

The protocol used with NAS is a file-based protocol such as NFS or Microsoft’s Common Internet File System (CIFS). Both of them allow storing backups using UNIX and Linux servers or Windows 2003 server.

However many new Linux or UNIX sys admin find it difficult to use NAS backup. Here are quick handy tips most newbie will find useful.

(A) Use IP address of NAS. If you do not have properly configured SAMBA server it is difficult to resolve hostnames. IP address will save your time.

(B) If you are using IPTABLES or PF firewall then make sure the following UDP/TCP ports are open between your firewall and the NAS Backup Server:

  1. TCP 21 (ftp)
  2. TCP 20 (ftp-data)
  3. TCP/UDP 137 (NETBIOS Name Service aka netbios-ns)
  4. TCP/UDP 138 (NETBIOS Datagram Service aka netbios-dgm)
  5. TCP/UDP 139 (NETBIOS session service aka netbios-ssn )
  6. TCP/UDP 445 (Microsoft Naked CIFS aka microsoft-ds )

Sample network diagram

Following is sample network diagram for our setup:

+-------------+               +-------------+
|             |               |             |
|   N A S     |<=============>|   Linux/    |
|             |               |   UNIX      |
IP:202.54.20.111              IP:202.54.1.13

Iptables configuration

FTP outgoing client request using iptables (assuming that your server IP is 202.54.1.13 and NAS IP is 202.54.20.111). Append following iptables rules to your script:

iptables -A OUTPUT -p tcp -s 202.54.1.13 –sport 1024:65535 -d 202.54.20.111 –dport 21 -m state –state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp -s 202.54.20.111 –sport 21 -d 202.54.1.13 –dport 1024:65535 -m state –state ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp -s 202.54.1.13 –sport 1024:65535 -d 202.54.20.111 –dport 1024:65535 -m state –state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp -s 202.54.20.111 –sport 1024:65535 -d 202.54.1.13 –dport 1024:65535 -m state –state ESTABLISHED -j ACCEPT

NETBIOS/CIFS outgoing client request

Please add following rules to your iptables script:

iptables -A OUTPUT -p udp -s 202.54.1.13 –sport 137 -d 0/0 –dport 137 -j ACCEPT
iptables -A OUTPUT -p udp -s 202.54.1.13 –sport 138 -d 0/0 –dport 138 -j ACCEPT
iptables -A OUTPUT -p tcp -s 202.54.1.13 –sport 1024:65535 -d 202.54.20.111 –dport 139 -m state –state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p udp -s 202.54.20.111 –sport 137 -d 202.54.1.13 –dport 137 -j ACCEPT
iptables -A INPUT -p udp -s 202.54.20.111 –sport 138 -d 202.54.1.13 –dport 138 -j ACCEPT
iptables -A INPUT -p tcp -s 202.54.20.111 –sport 139 -d 202.54.1.13 –dport 1024:65535 -m state –state ESTABLISHED -j ACCEPT

Please note that when configuring a firewall, the high order ports (1024-65535) are often used for outgoing connections and therefore should be permitted through the firewall. It is prudent to block incoming packets on the high order ports except for established connections. This is what you are doing in above FTP and CIFS client request.

How do I access NAS server using FTP?

You need to use Internet file transfer program (FTP) that comes with UNIX/Linux or windows. Most service provider will provide you:

  • NAS Server IP (e.g. 202.54.20.111 / nas.myserviceprovider.com)
  • NAS FTP Username (e.g. nixcraft)
  • NAS FTP Password (e.g. mySecret)

Let us assume you have file called mysqldump.tar.gz. You can put this file to NAS backup server using following ftp command:

$ ftp nas.myserviceprovider.com

OR

$ ftp 202.54.20.111

Output:

Username: nixcraft
Password: mySecret
ftp> bin
200 Type set to I.
ftp> prom
Interactive mode off.
ftp> put mysqldump.tar.gz
ftp> quit

How do I access NAS server using SAMBA client?

Make sure you have samba client installed. Use apt-get or up2date command to install SAMBA client.

a) Create a directory

# mkdir /backup

b) Mount remote NAS share (NOTE: you must type following command on a single line)

# mount -t smbfs -o username=nixcraft,password=mySecret //202.54.20.111/sharename /backup

OR

# smbmount -o username=nixcraft,password=mySecret //202.54.20.111/sharename /backup

You can skip password option for security reason (samba will prompt you for password).

c) Copy files using cp command:

# cp sitebackup.tar.gz /backup

d) You can use /backup directory to dump backup using mysql script or backup shell script.

A note for FreeBSD user

If you would like to access NAS server from FreeBSD use following command (NOTE: you must type following command on a single line):

# mkdir /backup
# mount_smbfs -I 202.54.20.111 [email protected]/sharename /backup

Output:

Password:

Related previous articles

Updated for accuracy.

How do I sync data between two Load balanced Windows 2003 servers?

Posted on in Categories Backup, Networking, Windows, Windows server last updated March 30, 2006

For UNIX/Linux server you can use rsync command to keep data sync’d between two servers. rsync also works under Windows NT/2003 server but it will take lots of time to setup.

For Microsoft Windows 2003 server use robocopy tool, to sync files between two windows server. You can download robocopy.exe and other files from Windows 2003 Resource Kit Tools set.

Once tool is downloaded, install the Resource Kit. By default robocopy located at c:\Program Files\Windows Resource Kits\Tools\robocopy.exe. Now let us assume that you want to sync d:\Inetpub\wwwroot\ directory between two Windows server. First create hidden share on www2 (secondary server)

  1. Login to www2
  2. Select d:\Inetpub\wwwroot\
  3. Right click > Properties > Sharing tab > Share this folder > Add $ at the end of share name and save the changes.

How do I use robocopy command?

=> Login to primary server (www1)
=> Click on stat > run > type cmd
=> At command prompt type command:
c:>"c:\program files\windows resource kits\tools\robocopy.exe d:\inetpub\wwwroot www2\\inetpub\wwwroot\"ALTERNATIVELY, use IP address of www2 server :
c:>"c:\program files\windows resource kits\tools\robocopy.exe d:\inetpub\wwwroot 202.54.10.200\\inetpub\wwwroot\"robocopy has many more options and lots of other information available in Robocopy.DOC file.

How do I sync data between two Load balanced Linux / UNIX servers?

Posted on in Categories Backup, FreeBSD, Linux, RedHat/Fedora Linux, Solaris, Tips, UNIX last updated March 29, 2006

Load balancing is way to provide to handle the load of the large number of visitors particular website or network service receives. It helps provide redundancy to your website. There are several ways to accomplish load balancing:
* Use of Linux Virtual Server
* Use of layer 4 routers
* Round robin DNS with squid cache
* Proprietary clustering solution from vendor such as Microsoft or HP/IBM, Cisco, Nortel etc

However, one frequently asked question is how to keep your webpages (HTML/PHP/PERL scripts) synchronized with each server. For example if you create a new web page called viewnews.php on one www2 server, how does new page get copied over to the second server www1?

You can use rsync – a network file distribution/synchronization utility on Unixish (Linux, FreeBSD, Solaris etc) systems. It does not simply send new files; it updates all files by sending only changed files. This saves time.

Install the rsync

Debian Linux user type the following command:# apt-get install rsyncFedora Linux user, user type the following command:# yum install rsyncRed Hat Linux user, user type the following command:# up2date rsyncFreeBSD user, user type the following command:# pkg_add -r -v rsyncALTERNATIVELY, use FreeBSD ports collection:# cd /usr/ports/net/rsync
# make; make install; make clean

How do I use rsync command?

You do not need to run rsync as a service or daemon. For example, if you would like to sync’d between www1 and www2 servers, type the following command on www1 server:rsync -avrR --links --rsh=/usr/bin/ssh 202.54.1.11:/var/html/ /var/htmlWhere,

  • -avrR : archive mode (a), verbose (v), recurse into directories (r), use relative path names (R)
  • –links : copy symlinks as symlinks
  • –rsh=/usr/bin/ssh : Use to specify the remote shell ssh to use (secure copy).
  • 202.54.1.11:/var/html/ : WWW2 server IP address and path to synchronize to www1 server
  • /var/html : WWW1 server path

A sample shell script for same job

#!/bin/bash
MASTER="master-server-ip"
DIR="/var/www/change-me"
LDIR="/local/dir"
SSH="/usr/bin/ssh"
rsync -avrR --links --rsh=$SSH $MASTER:$DIR $LDIR

See also:

  • If you are going to use rsync command from cron job/shell script, generate ssh keys for password less login over ssh.
  • See official rsync site for more complex examples.