Rsync Preserve / Copy Hard Links ( Backup Rsnapshot Directory Tree )

See all Debian/Ubuntu Linux related FAQ
I‘m using rsnapshot filesystem snapshot utility to make incremental snapshots of local and remote filesystems for 10 production servers running on RHEL 5.x system. The rsnapshot commands makes extensive use of hard links, so if the file doesn’t change, the next snapshot is simply a hard link to the exact same file. How do I use the rsync command to copy my the entire snapshot directory /raid6/rsnapshot/ (around 4TB) to a remote server for mirroring purpose?

The rsync command can preserve hard links and make the exact copy of /raid6/rsnapshot/ directory to a remote server using the following syntax. This is useful for making offsite backups or copy existing backups to a usb hard disk. Let us see how to use rsync to preserve and copy hard Links, softlinks and other data.
Tutorial details
Difficulty level Easy
Root privileges No
Requirements Linux or Unix terminal
Category Linux shell scripting
Prerequisites rsync command
OS compatibility BSD Linux macOS Unix WSL
Est. reading time 3 minutes
Advertisement

Rsync Preserve / Copy Hard Links For Rsnapshot Backup Directory

The syntax is as follows to keep track of hard links when using rsync:

rsync -az -H --delete --numeric-ids /path/to/source server2:/path/to/dest/
## OR ##
rsync -az -H --delete --numeric-ids /path/to/source 192.168.1.5:/path/to/dest/
 
## How about adding progress bar? ##
rsync -az -H --delete --progress --numeric-ids /path/to/source 192.168.1.5:/path/to/dest/
 
## Add a rsync log file ##
rsync --log-file="/var/log/my-rsync-script.log" -az -H --delete --progress --numeric-ids /path/to/source 192.168.1.5:/path/to/dest/

Where,

  1. -a : Archive mode (i.e. recurse into directories, and preserve symlinks, file permissions, file modification times, file group, file owner, device files & special files)
  2. -z : Compress file data during the transfer
  3. -H : Preserve hard links (i.e. copy hard links as hard links)
  4. --delete : Delete extraneous files from the receiving side (ones that aren't on the sending side), but only for the directories that are being synchronized i.e. keep exact replica of your /raid6/rsnapshot directory.
  5. --numeric-ids : Transfer numeric group and user IDs rather than using user and group names and mapping them at both ends.
  6. --progress : Show progress during transfer.
  7. --log-file="/var/log/my-rsync-script.log" : Log what rsync command is doing to the /var/log/my-rsync-script.log file.

Let us see some examples for Linux, macOS, FreeBSD and Unix-like systems.

Rsync copy hard links

In short type the following command as root user to copy /raid6/rsanpshot to another backup server named backupserver2:
# rsync -az -H --delete --numeric-ids /raid6/rsanpshot backupserver2:/backups/
You can retrieve backups from backupserver2 as follows:
# rsync -az -H --delete --numeric-ids backupserver1:/raid6/rsanpshot /backups/
Smaller size directories can be dumped to usb 2.0/3.0 or eSata external hard drives using the same syntax. First, mount usb hard drive:
# mount /dev/sdXY /mnt/usbdisk
Next, use the rsync command as follows:
# rsync -az -H --delete --numeric-ids /raid6/rsanpshot /mnt/usbdisk/
rsync copy hard link on linux unix

Preserving and coping hard links with rsync command

We can set rsync speed limit from eating all bandwidth with –bwlimit option:
# rsync -v -a --bwlimit=5000 -H --delete --numeric-ids rsnashot-prod user@192.168.2.19:/rsnapshot
The --bwlimit=KBPS option allows you to specify the maximum transfer rate. The RATE value can be suffixed with a string to indicate a size multiplier, and may be a fractional value (e.g. "–bwlimit=2.5m").

Shell script example for Rsync to preserve and copy hard links

Create a new shell script named dump-nixcraft-wks01-to-nas-server.sh:

#!/bin/bash
# Name: dump-nixcraft-wks01-to-nas-server.sh
# Purpose: 1. Dump dirs to a nas server while preserving hard and softlinks. 
#          2. Make sure you run this script via a cron job
#          3  Set up ssh pub keys log in to avoid password prompt
# Author: Vivek Gite, under GPL v2.x+
# ------------------------------------------------------------------------------
# Dest server info 
dest_box="user_name@server_ip_here"
dest_dir="/path/to/nixcraft-wks01-dumps/"
# backup dir lists 
src_dirs="/home/vivek /etc /root /var/spool/cron"
for d in $src_dirs
do
	rsync --exclude-from="$HOME/bin/dump-nixcraft-wks01-to-nas-server.sh.exclude" \
	      -azv -H --delete --numeric-ids \
	      --log-file="/var/log/rsync.dump-nixcraft-wks01-to-nas-server.sh.log" \
	      "$d" ${dest_box}:${dest_dir}
done

Also create a file named dump-nixcraft-wks01-to-nas-server.sh.exclude and add patterns to exclude from dumps/backups. For example:

# exclude from backups #
vivek/*.png
vivek/*.jpg
vivek/.mozilla/*
vivek/.config/google-chrome/*
vivek/snap/*
vivek/.cache*
vivek/.local/share/Trash/*

Next, set up a cron job as follows using the crontab command:

@hourly /path/dump-nixcraft-wks01-to-nas-server.sh

Conclusion

You learned how to copy and preserve hard Links when using rsync command on Linux, macOS, FreeBSD or Unix-like systems. For further info check rsync command manual using the man command or help command:
$ man rsync
$ rsync --help
$ rsync --help | grep -Ew -- '(-|--)(v|a|z|H|delete |numeric-ids|log-file.*FILE)'

Sample help session for the rsync command:

 -v, --verbose               increase verbosity
 -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
 -H, --hard-links            preserve hard links
     --del                   an alias for --delete-during
     --delete                delete extraneous files from destination dirs
     --delete-before         receiver deletes before transfer, not during
     --delete-during         receiver deletes during the transfer
     --delete-delay          find deletions during, delete after
     --delete-after          receiver deletes after transfer, not during
     --delete-excluded       also delete excluded files from destination dirs
     --delete-missing-args   delete missing source args from destination
     --numeric-ids           don't map uid/gid values by user/group name
 -z, --compress              compress file data during the transfer
     --log-file=FILE         log what we're doing to the specified FILE
     --log-file-format=FMT   log updates using the specified FMT

🥺 Was this helpful? Please add a comment to show your appreciation or feedback.

nixCrat Tux Pixel Penguin
Hi! 🤠
I'm Vivek Gite, and I write about Linux, macOS, Unix, IT, programming, infosec, and open source. Subscribe to my RSS feed or email newsletter for updates.

14 comments… add one
  • SomeOrdinarySysadmin Nov 26, 2023 @ 19:38

    I am about to migrate my backup server to new hardware and wondered which rsync options I needed to use to transfer my exististing backups. As I also use rsnapshot, this article exactly matches my use case. Thank you very much!

Leave a Reply

Your email address will not be published. Required fields are marked *

Use HTML <pre>...</pre> for code samples. Your comment will appear only after approval by the site admin.