Perform backups for the Linux operating system

Posted on in Categories Backup, Data recovery, Sys admin, Tips last updated October 25, 2006

This question asked again and again by a new Linux sys admins:

How do I perform backups for my Linux operating system?

So I am putting up all necessary information you ever need to know about backup. The main aim is to provide you necessary software, links and commands to get started as soon as possible.

Backup is essential

First a backup is essential. You need a good backup strategy to:

  • Minimize time from disaster such as server failure or human error (file deleted) or acts of God
  • To avoid downtime
  • Save money and time
  • And ultimately to save your job 😉

A backup must provide

  • Restoration of a single/individual files
  • Restoration of file systems

What to backup?

  • User files and dynamic data [databases] (stored in /home or specially configured partitions or /var etc).
  • Application software (stored in /usr)
  • OS files
  • Application configuration files (stored in /etc, /usr/local/etc or /home/user/.dotfiles)

Different types of backups

  • Full backups: Each file and directory is written to backup media
  • Incremental backups (Full + Incremental backup): This backups are used in conjunction with full backup. These backups will be incremental if each original piece of backed up information is stored only once , and then successive backups only contain the information that changed since the previous one. It use file’s modification time to determine which file need to backup.

So when you restore incremental backup:

  1. First restore the last full backup
  2. Next every subsequent incremental backup you need to restore

Preferred Backup Media

  1. Tape (old and trusted method)
  2. Network (ftp, nas, rsync etc)
  3. Disk (hard disk, optical disk etc)

Test backups

Please note that whichever backup media you choose, you need to test your backup. Perform tests to make sure that data can be read from media.

Backup Recommendation

My years of experience show that if you follow following formulas you are most likely to get back your data in worst scenario:
(a) Rotate backup media
(b) Use multiple backup media for same data such as ftp and tape
(c) Keep old copies of backups offsite

In short create good disaster recovery plan.

General procedure to restore a Linux/UNIX box

There is not golden rule or procedure but I follow these two methods:

Method # 1: Reinstall everything, restore everything, and secure everything

Use this method (bare metal recovery) if your server is cracked or hacked or hard drive is totally out of order:

  1. Format everything
  2. Reinstall os
  3. Configure data partitions (if any)
  4. Install drivers
  5. Restore data from backup media
  6. Configure security

Method # 2: Use of recovery CD/DVD rom

Use this method if your box is not hacked and system cannot boot or MBR damaged or accidental file deletion etc:

  1. Boot into rescue mode.
  2. Debug (or troubleshoot) the problem
  3. Verify that disk partitions stable enough (use fsck) to put backup data
  4. Install drivers
  5. Restore data from backup media
  6. Configure security

Linux (and other UNIX oses) backup tools

Luckily Linux/UNIX provides good set of tools for backup. We have almost covered each and every tool mentioned below. Just follow the link to get more information about each command and its usage:

It is also recommended that you use RAID or LVM (see consistent backup with LVM) or combination of both to increase reliability of data.

A note about MySQL or Oracle database backup

Backing up database server such as MySQL or Oracle needs more planning. Generally you can apply a table write lock and use mysql database dump utility to backup database. You can also use LVM volume to save database data.

A note about large scale backup

As I said earlier tar is good if you need to backup small amount of data that does not demands high CPU or I/O. Following are recommended tools for backup that demands high CPU or I/O rate:

(a) amandaAMANDA, the Advanced Maryland Automatic Network Disk Archiver, is a backup system (open source software) that allows the administrator to set up a single master backup server to back up multiple hosts over network to tape drives/changers or disks or optical media.

(b) Third party commercial proprietary solutions:
Top three excellent commercial solutions:

If you are looking to perform the tasks of protecting large-scale computer systems use above solutions and following two books will give you good idea:

Recommended further readings

I hope this small how to provide enough information to anyone to kick start your backup operation. Tell me if I am missing something or if you have a better backup solution or strategy, please comment back.

How To: Extract an RPM Package Files Without Installing It

Posted on in Categories CentOS, Data recovery, Howto, Linux, Linux distribution, package management, RedHat/Fedora Linux, Sys admin, Tips last updated October 18, 2006

As most of you may know to how extract a tarballs and/or a zip files. Someone, recently PM me with a question:

Dear nixCraft,

How do I extract an RPM package without installing it on my Fedora Linux or CentOS or RHEL (Red Hat Enterprise Linux) Suse Linux?

Sincerely,

CentOS user.

How to MySQL backup and data recovery with mysql-zrm

Posted on in Categories Backup, Data recovery, Howto, MySQL last updated September 28, 2006

This guide describes how to back up and recover your MySQL databases with mysql-zrm on a Debian Sarge system. mysql-zrm is short for Zmanda Recovery Manager for MySQL, it is a new tool that lets you create full logical or raw backups of your databases (regardless of your storage engine and MySQL configuration), generate reports about the backups, verify the integrity of the backups, and recover your databases. It can also send email notifcations about the backup status, and you can implement multiple backup policies (based on your applications and based on time (e.g. daily, weekly, etc.)).

Knoppix Linux rescue: Four Tough Lessons of System Recovery

Posted on in Categories Data recovery, Linux last updated September 1, 2006

I have already written about how to save (recover) data from crashed disks with dd and ddrescue command, and repairing ReiserFS file system with reiserfsck

KIVILCIM Hindistan recently migrated to a new laptop and promptly destroyed his partition table. Flailing around with false laziness taught him the true value of true laziness. Here’s a real-life tale of Knoppix rescue and a cautionary tale about troubleshooting. I hope someone will find this experience and howto useful in similar conditions.

FTA, “Last week, I received a brand new laptop with 1.5Gb RAM, a 100GB SATA HD, and a 15.4-inch wide screen, brightview display. It has basically all the technical gizmos that can spoil a new employee.

The computer came to me installed with Windows XP Pro. My game plan was to transfer my files via a USB disk to the NTFS partition and then transfer my second partition which is Debian Sarge (so-called) Unstable, and keep up with my regular business.

I turned red. I had just wiped the partition that contained my backup data and the installation files of my laptop. Fortunately, my boot partition was still there, so I just had to collect my backup data (some 60GB) from different computers and copy them again, which looked like half a day or so of work.”

Read more at linuxdevcenter

Howto: Recover lost files after you accidentally wipe your hard drive

Posted on in Categories Data recovery, Linux last updated August 28, 2006

Shawn Hermans show us how to How to recover lost files after you accidentally wipe your hard drive.

FTA, “Recently I wanted to make sure I had enough space to back up my home digital videos and pictures, so I purchased a new hard drive to add to my home Linux server. I moved all the files I wanted to save onto a single hard drive and repartitioned the old hard drive so I could upgrade to a newer version of Linux. After going through the process of reinstalling the operating system, I mounted the backup hard drive and discovered that it was empty. I had some how mixed up the hard drive I used to back up all the data with a hard drive that I wanted to wipe. Because I had done such a poor job of retaining backups on external media, I did not have any backups of my pictures and videos.”

Finally, I discovered TestDisk and PhotoRec and was able to use the latter to recover my lost files.

Read more at Linux.com

Linux and UNIX Backing up key information or files

Posted on in Categories Backup, Data recovery, Linux last updated August 15, 2006

Martin Brown shows us how to backup key files using tar and perl / shell scripts. It is a good article for all new sys admins (this goes well with my own shell scripts collection).

Most UNIX administrators have processes in place to back up the data and information on their UNIX machines, but what about the configuration files and other elements that provide the configuration data your machines need to operate? This article provides detailed information on techniques for achieving an effective and efficient backup system for these key files

The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes.

Read more at developerworks

HowTo: Backup MySQL Databases, Web server Files to a FTP Server Automatically

Posted on in Categories Data recovery, Linux, MySQL, Shell scripting last updated August 10, 2006

This is a simple backup solution for people who run their own web server and MySQL database server on a dedicated or VPS server. Most dedicated hosting provider provides backup service using NAS or FTP servers. These service providers will hook you to their redundant centralized storage array over private VLAN. Since, I manage couple of boxes, here is my own automated solution. If you just want a shell script, go here (you just need to provided appropriate input and it will generate FTP backup script for you on fly, you can also grab my php script generator code).

Repairing ReiserFS file system with reiserfsck

Posted on in Categories CentOS, Data recovery, Debian Linux, File system, Linux, Troubleshooting last updated July 13, 2006
Repairing ReiserFS file system with reiserfsck

The idea and commands in this article submitted by Jacques Wagener via email. In his own words, “After nuking my partition by accident (and through my stupidity) I was really disappointed in myself, especially in losing my bookmarks and rss-feeds“. The following article is based upon our email communication. I am just putting them as an article.

We have already written about ext2/ext3 file repair using fsck and other utilities. Linux comes with different filesystems and different repair utilities. To repair a ReiserFS filesystem you need to run reiserfsck command, which is a checking tool for the ReiserFS filesystem (just like fsck command for ext2/ext3 file system).

Reiserfsck searches for a Reiserfs filesystem on a device, replays any necessary transactions, and either checks or repairs the file system. ReiserFS saves data or log in a special file for pending disk updates and later on it commit updates to disk resulting into very good filesystem consistency.

Step # 1: Install reiserfsck

You need to install reiserfsprogs package, which includes reiserfsck user level tools for ReiserFS filesystems.
# apt-get install reiserfsprogs

Or if you are using RedHat/Fedor Core Linux:
# yum install reiserfsprogs

Step # 2: Backup partition/disk

Take system down to runlevel 1.
# init 1

Unmount /dev/sda1 (if it is still mounted as read/write only):
# umount /dev/sda1; umount /dev/sda2

Before using any one of the following command you are strongly advised to make a backup copy of the whole partition using dd or ddrescue (recommended) command.
# ddrescue /dev/sda /dev/sdb

OR use dd command:
# dd if=/dev/sda of=/dev/sdb conv=noerror

Step # 3: Check filesystem consistency

Above command makes a backup of your drive. Next check filesystem consistency which will reports problem to you with the following command (assuming that /dev/sda1 is your partition):
# reiserfsck --check /dev/sda1

If you get an error Bad root block 0If you get following message:
Running with –rebuild-tree is required

Then you need to run following command rebuilds the entire filesystem tree using leaf nodes found on the device (this is nothing but indication of corrupted :
# reiserfsck --scan-whole-partition --rebuild-tree /dev/sda1

Where,

  • –scan-whole-partition: This option causes –rebuild-tree to scan the whole partition but not only the used space on the partition. You should always use this option with –rebuild-tree.
  • –check: Checks filesystem consistency and reports, but does not repair any corruption that it finds.
  • –rebuild-tree: This option rebuilds the entire filesystem tree using leaf nodes found on the device. Once you passed this option, do not stop or interrupt building operating.

Recovering corrupted superblock

Each file system has a superblock, which contains information about file system such as:

  • File system type
  • Size
  • Status
  • Information about other metadata structures

If this information lost, you are in trouble (data loss) so Linux maintains multiple redundant copies of the superblock in every file system.

During check (reiserfsck –check /dev/sda1) if you get an error superblock was missing, use following command to fix superblock:
# reiserfsck --rebuild-sb /dev/sda1

Where,

  • –rebuild-sb: This option recovers the superblock on a Reiserfs partition. Normally you only need this option if mount reports “read_super_block: can’t find a reiserfs file system”.

Caution: Do not run above command twice on same drive. You will damage your partition (data).

Final note

Next logical step is mount your partition /dev/sda1 and check for your data:
# mkdir -p /mnt/data
# mount /dev/sda1 /mnt/data
# cd /mnt/data
# ls
# ls lost+found/ -l

lost+found is a special directory where recovered files are kept by Linux/reiserfsck. You can examine these files and restore the data.

Better backup entire partition using tar or ssh session:
# tar cvf /dev/nst0 /mnt/data

OR use scp to dump data to remote system:
# scp -r /mnt/data [email protected]:/backup

See also:

Connecting Linux or UNIX system to Network attached storage device

Posted on in Categories Backup, CentOS, Data recovery, Debian Linux, File system, FreeBSD, Gentoo Linux, Howto, Linux, Troubleshooting, Ubuntu Linux, UNIX last updated May 20, 2006

Network attached storage (NAS) allows using TCP/IP network to backup files. This enables multiple servers in IDC to share the same storage for backup at once, which minimizes overhead by centrally managing hard disks. NAS is scalable, high performance network solution. The main advantage is more hard disk storage space added to a network that already utilizes servers without shutting them down for maintenance and upgrades.

Please note that NAS are not just common in IDC or offices but you can use it for file sharing and backup at home. You can purchase 200+GB NAS for less than $200 these days. Personally, I am using Maxtor ShareStorage 200GB Network Attached Storage at home. This is a step-by-step guide on connecting Linux or UNIX systems to SAN for backup or sharing files.

The protocol used with NAS is a file-based protocol such as NFS or Microsoft’s Common Internet File System (CIFS). Both of them allow storing backups using UNIX and Linux servers or Windows 2003 server.

However many new Linux or UNIX sys admin find it difficult to use NAS backup. Here are quick handy tips most newbie will find useful.

(A) Use IP address of NAS. If you do not have properly configured SAMBA server it is difficult to resolve hostnames. IP address will save your time.

(B) If you are using IPTABLES or PF firewall then make sure the following UDP/TCP ports are open between your firewall and the NAS Backup Server:

  1. TCP 21 (ftp)
  2. TCP 20 (ftp-data)
  3. TCP/UDP 137 (NETBIOS Name Service aka netbios-ns)
  4. TCP/UDP 138 (NETBIOS Datagram Service aka netbios-dgm)
  5. TCP/UDP 139 (NETBIOS session service aka netbios-ssn )
  6. TCP/UDP 445 (Microsoft Naked CIFS aka microsoft-ds )

Sample network diagram

Following is sample network diagram for our setup:

+-------------+               +-------------+
|             |               |             |
|   N A S     |<=============>|   Linux/    |
|             |               |   UNIX      |
IP:202.54.20.111              IP:202.54.1.13

Iptables configuration

FTP outgoing client request using iptables (assuming that your server IP is 202.54.1.13 and NAS IP is 202.54.20.111). Append following iptables rules to your script:

iptables -A OUTPUT -p tcp -s 202.54.1.13 –sport 1024:65535 -d 202.54.20.111 –dport 21 -m state –state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp -s 202.54.20.111 –sport 21 -d 202.54.1.13 –dport 1024:65535 -m state –state ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp -s 202.54.1.13 –sport 1024:65535 -d 202.54.20.111 –dport 1024:65535 -m state –state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp -s 202.54.20.111 –sport 1024:65535 -d 202.54.1.13 –dport 1024:65535 -m state –state ESTABLISHED -j ACCEPT

NETBIOS/CIFS outgoing client request

Please add following rules to your iptables script:

iptables -A OUTPUT -p udp -s 202.54.1.13 –sport 137 -d 0/0 –dport 137 -j ACCEPT
iptables -A OUTPUT -p udp -s 202.54.1.13 –sport 138 -d 0/0 –dport 138 -j ACCEPT
iptables -A OUTPUT -p tcp -s 202.54.1.13 –sport 1024:65535 -d 202.54.20.111 –dport 139 -m state –state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p udp -s 202.54.20.111 –sport 137 -d 202.54.1.13 –dport 137 -j ACCEPT
iptables -A INPUT -p udp -s 202.54.20.111 –sport 138 -d 202.54.1.13 –dport 138 -j ACCEPT
iptables -A INPUT -p tcp -s 202.54.20.111 –sport 139 -d 202.54.1.13 –dport 1024:65535 -m state –state ESTABLISHED -j ACCEPT

Please note that when configuring a firewall, the high order ports (1024-65535) are often used for outgoing connections and therefore should be permitted through the firewall. It is prudent to block incoming packets on the high order ports except for established connections. This is what you are doing in above FTP and CIFS client request.

How do I access NAS server using FTP?

You need to use Internet file transfer program (FTP) that comes with UNIX/Linux or windows. Most service provider will provide you:

  • NAS Server IP (e.g. 202.54.20.111 / nas.myserviceprovider.com)
  • NAS FTP Username (e.g. nixcraft)
  • NAS FTP Password (e.g. mySecret)

Let us assume you have file called mysqldump.tar.gz. You can put this file to NAS backup server using following ftp command:

$ ftp nas.myserviceprovider.com

OR

$ ftp 202.54.20.111

Output:

Username: nixcraft
Password: mySecret
ftp> bin
200 Type set to I.
ftp> prom
Interactive mode off.
ftp> put mysqldump.tar.gz
ftp> quit

How do I access NAS server using SAMBA client?

Make sure you have samba client installed. Use apt-get or up2date command to install SAMBA client.

a) Create a directory

# mkdir /backup

b) Mount remote NAS share (NOTE: you must type following command on a single line)

# mount -t smbfs -o username=nixcraft,password=mySecret //202.54.20.111/sharename /backup

OR

# smbmount -o username=nixcraft,password=mySecret //202.54.20.111/sharename /backup

You can skip password option for security reason (samba will prompt you for password).

c) Copy files using cp command:

# cp sitebackup.tar.gz /backup

d) You can use /backup directory to dump backup using mysql script or backup shell script.

A note for FreeBSD user

If you would like to access NAS server from FreeBSD use following command (NOTE: you must type following command on a single line):

# mkdir /backup
# mount_smbfs -I 202.54.20.111 [email protected]/sharename /backup

Output:

Password:

Related previous articles

Updated for accuracy.