≡ Menu

remote backup

Linux: Local / Remote Backup For Large Files

You can easily backup large files using Linux backup software combination - tar, split and md5sum program. From the article:

I have a directory containing some files for a virtual computer. The files that hold the data for the virtual computer's hard discs are not only big, they are also "sparse" files, which means that they use only enough disc space for the data that was actually written to the file. For example, the virtual computer may have a 30GB drive of which 2GB has been used. Even though it uses only 2GB on my hard disc, a program that reads the file may see it as a 30GB file. This type of file can be tricky to back up because, when you copy it, you can end up with a 30GB file, or it might simply fail to copy, depending on the type of file system used on the backup storage.

=> Backing up Large Files

SSH: Rotate backup shell script to remove directories (old backup files)

Most time you have a limited space on the remote SFTP/ SSH backup server. Here is the script that periodically cleanup old backup files from the server i.e it will remove old directories.


Script will automatically calculate date from today's date. By default it will keep only last 7 days backup on server. You can easily increase / decrease this limit. In order to run script you must meet the following criteria:

  • Remote SSH server with rm command execution permission
  • SSH Keys for password less login (see how to setup RSA / DSA keys for password less login)
  • Accurate date and time on local system (see how to synchronize clock using ntpdate ntp client)
  • Remote backup directory must be in dd-mm-yyyy or mm-dd-yyyy format. For example daily mysql backup should be stored in /mysql/mm-dd-yyyy format.

Sample Script Usage

Run the script as follows:
./rot.backup.sh 7 /mysql "rm -rf"

  • 7 : Remove last 7 days files
  • /mysql : Base directory to clean up. If todays date is 9/Oct/2007, it will remove last 7 days directory /mysql/02-10-2007, /mysql/01-10-2007, .... /mysql/26-09-2007, /mysql/25-09-2007. It means script will only keep last 7 days backup on remote sftp / ssh server.
  • rm -rf : Command to run on directory structure

Sample Shell Script

Install following script:

if [ "$#" == "0" ];then
  echo "$0 upper-limit path {command}"
  exit 1
### SSH Server setup ###
DIR_FORMAT="%d-%m-%Y" # DD-MM-YYYY format
#DIR_FORMAT="%m-%d-%Y" #MM-DD-YYYY format
## do not edit below ##
LIMIT=$( expr $START + $1 )
## default CMD ##
[ "$3" != "" ] && CMD="$3" || :
[ "$2" != "" ] && SSH_PATH="$2" || :
DAYS=$(for d in $(seq $START $LIMIT);do date --date="$d days ago" +"${DIR_FORMAT}"; done)
for d in $DAYS
  ssh ${SSH_USER}@${SSH_SERVER} ${CMD} ${SSH_PATH}/$d

Run above script via cron tab (cronjob):
@daily /path/to/rot.ssh.script 7 "/html" "rm -rf"
@daily /path/to/rot.ssh.script 7 "/mysql" "rm -rf"