I know how to use rsync for Unix systems which synchronizes files and directories from one location to another while minimizing data transfer. However, I would like to synchronizing files between two directories, either on one computer, or between a computer and another server. How do I maintain the same version of files on multiple servers?
Unison is a file-synchronization tool for Unix and Windows.
It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other. This is useful for:
- Backups.
- Web server cluster.
- Sync home and office files.
Install unison
Type the following command under RHEL / CentOS Linux (make sure you turn on EPEL repo):
# yum install unison
Type the following command under Debian / Ubuntu Linux:
# apt-get update && apt-get install unison
Type the following command under FreeBSD:
# cd /usr/ports/net/unison/ && make install clean
How Do I Use unison?
In this example, sync /tmp/test1 to /tmp/test2 as follows:
# mkdir /tmp/test{1,2}
# cd /tmp/test1
# touch file{1,2,3}
# ls -l
Sample outputs:
total 12 -rw-r--r-- 1 root root 0 Aug 16 12:09 file1 -rw-r--r-- 1 root root 0 Aug 16 12:09 file2 -rw-r--r-- 1 root root 0 Aug 16 12:09 file3
Now, try to sync it:
# unison /tmp/test1 /tmp/test2
Sample outputs:
Contacting server... Connected [//vivek-desktop//tmp/test1 -> //vivek-desktop//tmp/test2] Looking for changes Warning: No archive files were found for these roots, whose canonical names are: /tmp/test1 /tmp/test2 This can happen either because this is the first time you have synchronized these roots, or because you have upgraded Unison to a new version with a different archive format. Update detection may take a while on this run if the replicas are large. Unison will assume that the 'last synchronized state' of both replicas was completely empty. This means that any files that are different will be reported as conflicts, and any files that exist only on one replica will be judged as new and propagated to the other replica. If the two replicas are identical, then no changes will be reported. If you see this message repeatedly, it may be because one of your machines is getting its address from DHCP, which is causing its host name to change between synchronizations. See the documentation for the UNISONLOCALHOSTNAME environment variable for advice on how to correct this. Donations to the Unison project are gratefully accepted: http://www.cis.upenn.edu/~bcpierce/unison Press return to continue.[] Reconciling changes test1 test2 file ----> file1 [f] file ----> file2 [f] file ----> file3 [f] Proceed with propagating updates? [] y Propagating updates UNISON 2.27.57 started propagating changes at 12:11:18 on 16 Aug 2010 [BGN] Copying file1 from /tmp/test1 to /tmp/test2 [END] Copying file1 [BGN] Copying file2 from /tmp/test1 to /tmp/test2 [END] Copying file2 [BGN] Copying file3 from /tmp/test1 to /tmp/test2 [END] Copying file3 UNISON 2.27.57 finished propagating changes at 12:11:18 on 16 Aug 2010 Saving synchronizer state Synchronization complete (3 items transferred, 0 skipped, 0 failures)
The -batch (batch mode) option ask no questions at all, enter:
# rm /tmp/test1/file3
# echo 'foo' >> /tmp/test2/file2
# unison -batch /tmp/test1 /tmp/test2
Sample outputs:
Contacting server... Connected [//vivek-desktop//tmp/test1 -> //vivek-desktop//tmp/test2] Looking for changes Reconciling changes file3 test1 : deleted test2 : unchanged file modified on 2010-08-16 at 12:09:31 size 0 rw-r--r-- Propagating updates UNISON 2.27.57 started propagating changes at 12:15:11 on 16 Aug 2010 [BGN] Updating file file2 from /tmp/test2 to /tmp/test1 [END] Updating file file2 [BGN] Deleting file3 from /tmp/test2 [END] Deleting file3 UNISON 2.27.57 finished propagating changes at 12:15:11 on 16 Aug 2010 Saving synchronizer state Synchronization complete (2 items transferred, 0 skipped, 0 failures)
Remote Server Synchronizer
First, make sure you use the same version on local and remote server. Use the following command to test that the local unison client can start and connect to the remote server:
# unison -testServer /tmp/test1 ssh://server1.cyberciti.com//tmp/test1
Sample outputs:
Contacting server... Connected [//vivek-desktop//tmp/test1 -> //server1.cyberciti.com//tmp/test1]
To sync, enter:
# unison -batch /tmp/test1 ssh://server1.cyberciti.com//tmp/test1
Delete or add a new file from or to server1.cyberciti.com//tmp/test1 directory and try again:
# unison -batch /tmp/test1 ssh://server1.cyberciti.com//tmp/test1
Sample Shell Scripts
Create a sample shell script as follows (sync.dirs.sh):
#!/bin/bash # set paths / dirs _paths="/var/www/html/ \ /etc/ \ /home/vivek/ \ /projects/scripts/*.pl" # binary file name _unison=/usr/bin/unison # server names # sync server1.cyberciti.com with rest of the server in cluster _rserver="server2.cyberciti.com server3.cyberciti.com" # sync it for r in ${_rserver} do for p in ${_paths} do ${_unison} -batch "${p}" "ssh://${r}/${p}" done done
Save and close the file. Setup a cronjob as follows:
*/30 * * * * /path/to/sync.dirs.sh &>/tmp/sync.dirs.sh.log
Make sure you setup ssh keys or use keychain to avoid the password prompt.
How Do I Call unison On Demand?
You need to use the incrond (inotify cron daemon) is a daemon which monitors filesystem events (such as add a new file, delete a file and so on) and executes commands or shell scripts. It’s use is generally similar to cron. In this example, call sync.dirs.sh whenever files uploaded or deleted from /var/www/html (see inotify FAQ for more info):
/var/www/html IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /path/to/sync.dirs.sh
References:
🐧 9 comments so far... add one ↓
Category | List of Unix and Linux commands |
---|---|
File Management | cat |
Firewall | Alpine Awall • CentOS 8 • OpenSUSE • RHEL 8 • Ubuntu 16.04 • Ubuntu 18.04 • Ubuntu 20.04 |
Network Utilities | dig • host • ip • nmap |
OpenVPN | CentOS 7 • CentOS 8 • Debian 10 • Debian 8/9 • Ubuntu 18.04 • Ubuntu 20.04 |
Package Manager | apk • apt |
Processes Management | bg • chroot • cron • disown • fg • jobs • killall • kill • pidof • pstree • pwdx • time |
Searching | grep • whereis • which |
User Information | groups • id • lastcomm • last • lid/libuser-lid • logname • members • users • whoami • who • w |
WireGuard VPN | Alpine • CentOS 8 • Debian 10 • Firewall • Ubuntu 20.04 |
Hi,
I’ve looked at unison some time ago as a corporate solution to replace dropbox, it’s good stuff but not good enough to replace such a killer-app as dropbox.
Do you have any knowledge about an unison configuration to replace dropbox on a enterprise level? (share between groups, share with clients with an url, versioning,..)
Amazing blog/site/forum, congratulations.
Thanks for writing this ! It was very helpful for me.
Mohamed
Hello,
Thanks for a great post.
I have a question regarding your use of incron.
It seems that incron doesn’t detect changes in subdirectories.
does your use-case don’t deal with subdirectories? and if it does, what do you do?
Thanks,
Yoni.
Thanks Vivek,
I was the stuff i was looking for.
regards.
Hi Vivek,
after a small evaluation found that it is not fitting to my requirement.
I have two server which received approx 100 files in a minute sizing from few kbs to 10Mb. now the problem is, if it sync when a file is partly arrived on any of server, it will copy only that part. on next sync it will scape that same file.
Can you pls suggest what are the option available for this scenario.
Regards,
Rahul Janghel.
I have a question about incrond and unison combination (oer even rsync or whatever).
When I have incrond daemon looking for read/write/create jobs in folder and it’ll start unison – which will then start syncing – are then some inbuilt “lock” systems in incrond to avoid it starting a massive amount of colliding synch jobs.
When unison moves files from remote server into watched folder wouldn’t incornd fire up additional unisons doing the same thing for each one?
Or is there something witty already put into your shell script there that I didn’t catch?
I would like to use unison, but it doesn’t actually do batch mode, still popping up long error messages in dialog boxes.
take for example:
mkdir a
touch a/fa1
touch a/fa2
mkdir b
unison -batch a b
This then pops up *two windows* on the screen! Apparently ignoring the batch option. One of the windows is a long long error messages that promises to return every time I use unison because we serve addresses on the machine via dhcp. The other window is some GUI interface which apparently has no purpose (as we ran in batch)
Thus unison is unusable in scripts, unless you want to close 50 windows manually, unusable with dhcp, and is high maintenance with new users who don’t understand all the messages.
Too bad -batch doesn’t actually do batch processing, that would make it a completely different program.
The same results of two windows popping up on the screen also happens with the example above with:
unison -batch -silent a b
Silent mode is far from silent, spitting out a hundred line message and popping windows up on gnome. Though after all the noise and promises of making more noise, it does unify the directories …
Primero quera agradecer por crear este maravilloso post y segundo pedir ayuda :-( … tengo una red domestica con 4 computadoras , con sistema operativo windows 7, estoy usando UNISON-GTK gui compatible con win7, intento usar ssh para sincronizar mis computadoras pero no se que poner aquí :
Conection Setup
Synchronization Kind : Using SSH
configutarion
Host : ??????????
User : ???????????
, en cmd msconfig/all me sale la siguiente descripcion :
nombre de host …… :PC10
Direccion IPv4 …….. :192.168.1.110
ahora como debería de proseguir avanzando , si me pudieran ayudar les quedaría enormemente agradecido.