≡ Menu

Backup

Mirror directory with mirrordir command

rsyn is the best tool for mirroring servers and data. However, there is also mirrordir tool. As name suggest it can mirror directories and do lots of other things. It offers lots of options for die hard shell scripting fans. It can copy, mirror directory trees via a minimal set of changes, locally or over FTP, or over a secure tcp connection.

From the man page:

mirrordir copies files that are different between the directories control and mirror to the directory mirror. Files whose modification times or sizes differ are copied. File permissions, ownerships, modification times, access times (only if --access-times is used), sticky bits, and device types are duplicated. Symlinks are duplicated without any translation. Symlink modification and access times (of the symlink itself, not the file it points to) are not preserved. Hard linked files are merely copied.

mirrordir command supports strong stream cipher encryption and Diffie-Hellman key exchanges with several possible key sizes.

Install mirrordir

Use apt-get or ports system command:
$ sudo apt-get install mirrordir

mirrordir examples

Mirror ~/scripts to ~/scripts.backup:
$ mkdir ~/scripts.backup
$ mirrordir -v ~/scripts ~/scripts.backup

If you rerun mirrordir again, only the updated files are copied.

You can mirroring FTP sites:
$ mirrordir -v ftp://ftp.nixcraft.in/pub /home/backup/nixcraft.in/ftp/pub

mirrordir offers tons of option, refer mirrordir man page for more examples and options:
$ man mirrordir

Linux mobile phone manager – Wammu

Wammu is mobile phone manager running on Linux, Windows and possibly other platforms, where Gammu and wxPython works. The communication is made by Gammu library. With this software you can:
=> Edit / delete contacts, todo, calendar etc
=> Send files
=> Text (sms) composer for multi part smses
=> Search phone
=> Backup etc

Phones supported by Gammu:

* Nokia DCT3 (3210, 3310, 3330, 3390, 3410, 5110, 5110i, 5130, 5190, 5210, 5510, 6110, 6130, 6150, 6190, 8210, 8250, 8290, 8850, 8855, 8890 6210, 6250, 7110, 9110) and compatible

* Nokia DCT4 (3510, 3510i, 3530, 5100, 6100, 6310, 6310i, 6510, 6610, 7210, 8310, 8910) and compatible

* Siemens M20, MC35, SL45 and compatible

* Alcatel One Touch 501, 701, 715, 535, 735

* Sony Ericsson phones

* AT compatible phones

* Symbian phones up to Symbian 9.0 are supported through gnapplet

Install wammu - Phone manager

Debian / Ubuntu Linux user, enter:
$ sudo apt-get install wammu

Download Wammu

Visit official project home page here

Please note that wammu also works under Windows XP.

Internet SCSI (iSCSI) is a network protocol s that allows you to use of the SCSI protocol over TCP/IP networks. It is good alternative to Fibre Channel-based SANs. You can easily manage, mount and format iSCSI Volume under Linux. It allows access to SAN storage over Ethernet.

Open-iSCSI Project

Open-iSCSI project is a high-performance, transport independent, multi-platform implementation of iSCSI. Open-iSCSI is partitioned into user and kernel parts.

Instructions are tested on:
[a] RHEL 5
[b] CentOS 5
[c] Fedora 7
[d] Debian / Ubuntu Linux

Install Required Package

iscsi-initiator-utils RPM package - The iscsi package provides the server daemon for the iSCSI protocol, as well as the utility programs used to manage it. iSCSI is a protocol for distributed disk access using SCSI commands sent over Internet Protocol networks. This package is available under Redhat Enterprise Linux / CentOS / Fedora Linux and can be installed using yum command:
# yum install iscsi-initiator-utils

A note about Debian / Ubuntu Linux

If you are using Debian / Ubuntu Linux install open-iscsi package, enter:
$ sudo apt-get install open-iscsi

iSCSI Configuration

There are three steps needed to set up a system to use iSCSI storage:

  1. iSCSI startup using the init script or manual startup. You need to edit and configure iSCSI via /etc/iscsi/iscsid.conf file
  2. Discover targets.
  3. Automate target logins for future system reboots.
  4. You also need to obtain iSCSI username, password and storage server IP address (target host)

Step # 1: Configure iSCSI

Open /etc/iscsi/iscsid.conf with vi text editor:
# vi /etc/iscsi/iscsid.conf
Setup username and password:
node.session.auth.username = My_ISCSI_USR_NAME
node.session.auth.password = MyPassword
discovery.sendtargets.auth.username = My_ISCSI_USR_NAME
discovery.sendtargets.auth.password = MyPassword

Where,

  • node.session.* is used to set a CHAP username and password for initiator authentication by the target(s).
  • discovery.sendtargets.* is used to set a discovery session CHAP username and password for the initiator authentication by the target(s)

You may also need to tweak and set other options. Refer to man page for more information. Now start the iscsi service:
# /etc/init.d/iscsi start

Step # 2: Discover targets

Now use iscsiadm command, which is a command-line tool allowing discovery and login to iSCSI targets, as well as access and management of the open-iscsi database. If your storage server IP address is 192.168.1.5, enter:
# iscsiadm -m discovery -t sendtargets -p 192.168.1.5
# /etc/init.d/iscsi restart

Now there should be a block device under /dev directory. To obtain new device name, type:
# fdisk -l
or
# tail -f /var/log/messages
Output:

Oct 10 12:42:20 ora9is2 kernel:   Vendor: EQLOGIC   Model: 100E-00           Rev: 3.2
Oct 10 12:42:20 ora9is2 kernel:   Type:   Direct-Access                      ANSI SCSI revision: 05
Oct 10 12:42:20 ora9is2 kernel: SCSI device sdd: 41963520 512-byte hdwr sectors (21485 MB)
Oct 10 12:42:20 ora9is2 kernel: sdd: Write Protect is off
Oct 10 12:42:20 ora9is2 kernel: SCSI device sdd: drive cache: write through
Oct 10 12:42:20 ora9is2 kernel: SCSI device sdd: 41963520 512-byte hdwr sectors (21485 MB)
Oct 10 12:42:20 ora9is2 kernel: sdd: Write Protect is off
Oct 10 12:42:20 ora9is2 kernel: SCSI device sdd: drive cache: write through
Oct 10 12:42:20 ora9is2 kernel:  sdd: unknown partition table
Oct 10 12:42:20 ora9is2 kernel: sd 3:0:0:0: Attached scsi disk sdd
Oct 10 12:42:20 ora9is2 kernel: sd 3:0:0:0: Attached scsi generic sg3 type 0
Oct 10 12:42:20 ora9is2 kernel: rtc: lost some interrupts at 2048Hz.
Oct 10 12:42:20 ora9is2 iscsid: connection0:0 is operational now

/dev/sdd is my new block device.

Step # 3: Format and Mount iSCSI Volume

You can now partition and create a filesystem on the target using usual fdisk and mkfs.ext3 commands:
# fdisk /dev/sdd
# mke2fs -j -m 0 -O dir_index /dev/sdd1

OR
# mkfs.ext3 /dev/sdd1

Tip: If your volume is large size like 1TB, run mkfs.ext3 in background using nohup:
# nohup mkfs.ext3 /dev/sdd1 &

Mount new partition:
# mkdir /mnt/iscsi
# mount /dev/sdd1 /mnt/iscsi

Step #4: Mount iSCSI drive automatically at boot time

First make sure iscsi service turned on at boot time:
# chkconfig iscsi on
Open /etc/fstab file and append config directive:
/dev/sdd1 /mnt/iscsi ext3 _netdev 0 0
Save and close the file.

Further readings:

Most time you have a limited space on the remote SFTP/ SSH backup server. Here is the script that periodically cleanup old backup files from the server i.e it will remove old directories.

Requirements

Script will automatically calculate date from today's date. By default it will keep only last 7 days backup on server. You can easily increase / decrease this limit. In order to run script you must meet the following criteria:

  • Remote SSH server with rm command execution permission
  • SSH Keys for password less login (see how to setup RSA / DSA keys for password less login)
  • Accurate date and time on local system (see how to synchronize clock using ntpdate ntp client)
  • Remote backup directory must be in dd-mm-yyyy or mm-dd-yyyy format. For example daily mysql backup should be stored in /mysql/mm-dd-yyyy format.

Sample Script Usage

Run the script as follows:
./rot.backup.sh 7 /mysql "rm -rf"
Where,

  • 7 : Remove last 7 days files
  • /mysql : Base directory to clean up. If todays date is 9/Oct/2007, it will remove last 7 days directory /mysql/02-10-2007, /mysql/01-10-2007, .... /mysql/26-09-2007, /mysql/25-09-2007. It means script will only keep last 7 days backup on remote sftp / ssh server.
  • rm -rf : Command to run on directory structure

Sample Shell Script

Install following script:

#!/bin/bash
if [ "$#" == "0" ];then
  echo "$0 upper-limit path {command}"
  exit 1
fi
### SSH Server setup ###
SSH_USER="vivek"
SSH_SERVER="nas.nixcraft.in"
START=7
DIR_FORMAT="%d-%m-%Y" # DD-MM-YYYY format
#DIR_FORMAT="%m-%d-%Y" #MM-DD-YYYY format
## do not edit below ##
LIMIT=$( expr $START + $1 )
## default CMD ##
CMD="ls"
SSH_PATH="."
[ "$3" != "" ] && CMD="$3" || :
[ "$2" != "" ] && SSH_PATH="$2" || :
DAYS=$(for d in $(seq $START $LIMIT);do date --date="$d days ago" +"${DIR_FORMAT}"; done)
for d in $DAYS
do
  ssh ${SSH_USER}@${SSH_SERVER} ${CMD} ${SSH_PATH}/$d
done

Run above script via cron tab (cronjob):
@daily /path/to/rot.ssh.script 7 "/html" "rm -rf"
@daily /path/to/rot.ssh.script 7 "/mysql" "rm -rf"

Using UNIX pipe concept one can dump database to another server securely using ssh protocol. All you need remote execution rights for the 'dd' command, over SSH. This allows you to run database dumps across an encrypted channel.

Dump Postgres Database using ssh

Use pg_dump command command:
pg_dump -U USERNAME YOUR-DATABASE-NAME | ssh user@remote.server.com "dd of=/pgsql/$(date +'%d-%m-%y')"

Dump MySQL Database using ssh

Type the following command:
mysqldump -u USERnAME -p'PASSWORD' YOUR-DATABASE-NAME | ssh user@remote.server.com "dd of=/mysql/$(date +'%d-%m-%y')"

netcat utility (nc command) considered as TCP/IP swiss army knife. It reads and writes data across network connections, using TCP or UDP protocol. It is designed to be a reliable "back-end" tool that can be used directly or easily driven by other programs and scripts. At the same time, it is a feature-rich network debugging and exploration tool, since it can create almost any kind of connection you would need and has several interesting built-in capabilities.

I also install the netcat package for administering a network and you'd like to use its debugging and network exploration capabilities.

One my favorite usage is to migrating data between two server hard drives using netcat over a network. It is very easy to copy complete drive image from one server to another.

You can also use ssh for the same purpose, but encryption adds its own overheads. This is tried and trusted method (hat tip to karl) .

Make sure you have backup of all important data.

Install netcat

It is possible that nc may not be installed by default under Redhat / CentOS / Debian Linux.

Insall nc under Redhat / CentOS / Fedora Linux

Use yum command as follows:
# yum install nc
Output:

Loading "installonlyn" plugin
Loading "rhnplugin" plugin
Setting up Install Process
Setting up repositories
rhel-x86_64-server-vt-5   100% |=========================| 1.2 kB    00:00
rhel-x86_64-server-5      100% |=========================| 1.2 kB    00:00
Reading repository metadata in from local files
Parsing package install arguments
Resolving Dependencies
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for nc to pack into transaction set.
nc-1.84-10.fc6.x86_64.rpm 100% |=========================| 6.9 kB    00:00
---> Package nc.x86_64 0:1.84-10.fc6 set to be updated
--> Running transaction check
Dependencies Resolved
=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
 nc                      x86_64     1.84-10.fc6      rhel-x86_64-server-5   56 k
Transaction Summary
=============================================================================
Install      1 Package(s)
Update       0 Package(s)
Remove       0 Package(s)
Total download size: 56 k
Is this ok [y/N]: y
Downloading Packages:
(1/1): nc-1.84-10.fc6.x86 100% |=========================|  56 kB    00:00
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing: nc                           ######################### [1/1]
Installed: nc.x86_64 0:1.84-10.fc6
Complete!

Debian / Ubuntu Linux netcat installation

Simply use apt-get command:
$ sudo apt-get install netcat

WARNING! These examples may result into data loss, ensure there are good backups before doing this, as using command wrong way can be dangerous.

How do I use netcat to copy hard disk image?

Our sample setup

-----------------------
HostA // 192.168.1.1
------------------------
           sda
        NETWORK
           sdb
------------------------
HostB // 192.168.1.2
-------------------------

Your task is copy HostA /dev/sda to HostB's /dev/sdb using netcat command. First login as root user

Command to type on hostB (receiving end ~ write image mode)

You need to open port on hostB using netcat, enter :
# netcat -p 2222 -l |bzip2 -d | dd of=/dev/sdb
Where,

  • -p 2222 : Specifies the source port nc should use, subject to privilege restrictions and availability. Make sure port 2222 is not used by another process.
  • -l : Used to specify that nc should listen for an incoming connection rather than initiate a connection to a remote host.
  • bzip2 -d : Compresses image using the Burrows-Wheeler block sorting text compression algorithm, and Huffman coding. This will speed up network transfer ( -d : force decompression mode)
  • dd of=/dev/sda : /dev/sda is your hard disk. You can also specify partition such as /dev/sda1

Command to type on hostA (send data over a network ~ read image mode)

Now all you have to do is start copying image. Again login as root and enter:
# bzip2 -c /dev/sda | netcat hostA 2222
OR use IP address:
# bzip2 -c /dev/sda | netcat 192.168.1.1 2222

This process takes its own time.

A note about latest netcat version 1.84-10 and above

If you are using latest nc / netcat version above syntax will generate an error. It is an error to use -l option in conjunction with the -p, -s, or -z options. Additionally, any timeouts specified with the -w option are ignored. So use nc command as follows.

On hostA, enter:
# nc -l 2222 > /dev/sdb
On hostB, enter:
# nc hostA 2222< /dev/sda
OR
# nc 192.168.1.1 2222< /dev/sda

Using a second machine (hostB), connect to the listening nc process at 2222 (hostA), feeding it the file (/dev/sda)which is to be transferred. You can use bzip2 as follows.
On hostA, enter:
# nc -l 2222 | bzip2 -d > /dev/sdb
On hostB, enter:
# bzip2 -c /dev/sda | nc 192.168.1.1 2222

Further readings

How do I improve performance?

As suggested by anonymous user:

You should definitely use bs=16M or something like that. Otherwise, the copy will take forever. Copying a 300 GB hard drive over a 1 Gbps cross-over cable took about 1 1/2 hours or so using bs=16M Without this option, the same thing would have taken about 7 hours.

In short use command as follows:
# netcat -p 2222 -l |bzip2 -d | dd of=/dev/sdb bs=16M

Updated for accuracy.

You can easily mount remote server file system or your own home directory using special sshfs and fuse tools.

FUSE - Filesystem in Userspace

FUSE is a Linux kernel module also available for FreeBSD, OpenSolaris and Mac OS X that allows non-privileged users to create their own file systems without the need to write any kernel code. This is achieved by running the file system code in user space, while the FUSE module only provides a "bridge" to the actual kernel interfaces. FUSE was officially merged into the mainstream Linux kernel tree in kernel version 2.6.14.

You need to use SSHFS to access to a remote filesystem through SSH or even you can use Gmail account to store files.

Following instructions are tested on CentOS, Fedora Core and RHEL 4/5 only. But instructions should work with any other Linux distro without a problem.

Step # 1: Download and Install FUSE

Visit fuse home page and download latest source code tar ball. Use wget command to download fuse package:
# wget http://superb-west.dl.sourceforge.net/sourceforge/fuse/fuse-2.6.5.tar.gz
Untar source code:
# tar -zxvf fuse-2.6.5.tar.gz
Compile and Install fuse:
# cd fuse-2.6.5
# ./configure
# make
# make install

Step # 2: Configure Fuse shared libraries loading

You need to configure dynamic linker run time bindings using ldconfig command so that sshfs command can load shared libraries such as libfuse.so.2:
# vi /etc/ld.so.conf.d/fuse.conf
Append following path:
/usr/local/lib
Run ldconfig:
# ldconfig

Step # 3: Install sshfs

Now fuse is loaded and ready to use. Now you need sshfs to access and mount file system using ssh. Visit sshfs home page and download latest source code tar ball. Use wget command to download fuse package:
# wget http://easynews.dl.sourceforge.net/sourceforge/fuse/sshfs-fuse-1.7.tar.gz
Untar source code:
# tar -zxvf sshfs-fuse-1.7.tar.gz
Compile and Install fuse:
# cd sshfs-fuse-1.7
# ./configure
# make
# make install

Mounting your remote filesystem

Now you have working setup, all you need to do is mount a filesystem under Linux. First create a mount point:
# mkdir /mnt/remote
Now mount a remote server filesystem using sshfs command:
# sshfs vivek@rock.nixcraft.in: /mnt/remote
Where,

  • sshfs : SSHFS is a command name
  • vivek@rock.nixcraft.in: - vivek is ssh username and rock.nixcraft.in is my remote ssh server.
  • /mnt/remote : a local mount point

When promoted supply vivek (ssh user) password. Make sure you replace username and hostname as per your requirements.

Now you can access your filesystem securely using Internet or your LAN/WAN:
# cd /mnt/remote
# ls
# cp -a /ftpdata . &

To unmount file system just type:
# fusermount -u /mnt/remote
or
# umount /mnt/remote

Further readings: