HowTo Use rsync For Transferring Files Under Linux or UNIX

by on December 5, 2006 · 60 comments· LAST UPDATED August 8, 2012

in , ,

How do you install and use rsync to synchronize files and directories from one location (or one server) to another location? - A common question asked by new sys admin.

rsync is a free software computer program for Unix and Linux like systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction.

So what is unique about the rsync command?

It can perform differential uploads and downloads (synchronization) of files across the network, transferring only data that has changed. The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection.

How do I install rsync?

Use any one of the following commands to install rsync. If you are using Debian or Ubuntu Linux, type the following command:
# apt-get install rsync
OR
$ sudo apt-get install rsync
If you are using Red Hat Enterprise Linux (RHEL) / CentOS 4.x or older version, type the following command:
# up2date rsync
RHEL / CentOS 5.x or newer (or Fedora Linux) user type the following command:
# yum install rsync

Always use rsync over ssh

Since rsync does not provide any security while transferring data it is recommended that you use rsync over ssh session. This allows a secure remote connection. Now let us see some examples of rsync command.

Comman rsync command options

  • --delete : delete files that don't exist on sender (system)
  • -v : Verbose (try -vv for more detailed information)
  • -e "ssh options" : specify the ssh as remote shell
  • -a : archive mode
  • -r : recurse into directories
  • -z : compress file data

Task : Copy file from a local computer to a remote server

Copy file from /www/backup.tar.gz to a remote server called openbsd.nixcraft.in
$ rsync -v -e ssh /www/backup.tar.gz jerry@openbsd.nixcraft.in:~
Output:

Password:
sent 19099 bytes  received 36 bytes  1093.43 bytes/sec
total size is 19014  speedup is 0.99

Please note that symbol ~ indicate the users home directory (/home/jerry).

Task : Copy file from a remote server to a local computer

Copy file /home/jerry/webroot.txt from a remote server openbsd.nixcraft.in to a local computer's /tmp directory:
$ rsync -v -e ssh jerry@openbsd.nixcraft.in:~/webroot.txt /tmp

Task: Synchronize a local directory with a remote directory

$ rsync -r -a -v -e "ssh -l jerry" --delete /local/webroot openbsd.nixcraft.in:/webroot

Task: Synchronize a remote directory with a local directory

$ rsync -r -a -v -e "ssh -l jerry" --delete openbsd.nixcraft.in:/webroot/ /local/webroot

Task: Synchronize a local directory with a remote rsync server or vise-versa

$ rsync -r -a -v --delete rsync://rsync.nixcraft.in/cvs /home/cvs
OR
$ rsync -r -a -v --delete /home/cvs rsync://rsync.nixcraft.in/cvs

Task: Mirror a directory between my "old" and "new" web server/ftp

You can mirror a directory between my "old" (my.old.server.com) and "new" web server with the command (assuming that ssh keys are set for password less authentication)
$ rsync -zavrR --delete --links --rsh="ssh -l vivek" my.old.server.com:/home/lighttpd /home/lighttpd

Read related previous articles

Other options - rdiff and rdiff-backup

The rdiff command uses the rsync algorithm. A utility called rdiff-backup has been created which is capable of maintaining a backup mirror of a file or directory over the network, on another server. rdiff-backup stores incremental rdiff deltas with the backup, with which it is possible to recreate any backup point. Next time I will write about these utilities.

rsync for Windows Server/XP/7/8

Please note if you are using MS-Windows, try any one of the program:

  1. DeltaCopy
  2. NasBackup
Further readings

=> Read rsync man page
=> Official rsync documentation

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 60 comments… read them below or add one }

1 n00b October 8, 2007 at 8:27 pm

I have RSYNC running to copy files from a mapped linux server (through a windows PC running cwrsync) to a remote linux server. Everything works fine, but the permissions and ownership changes. I added this line:

–chmod=u+rwx,g+rx,o+x

and that seems to work fine in ensuring I have chmod 0755 set for my folders and files. However, it keeps reverting my users:groups from myuser:nobody to 400:401.

What is the code that I add to rsync to have it:
a) Retain the existing ownerships, or
b) Force myuser:nobody for all files and folders?

Reply

2 n00b October 8, 2007 at 8:34 pm

Oops. Second question. I’d like for all directories to retain their original ownership and permissions (rather than forcing the chmod). The reason is because I’d like for most directories to be 0755, but need 0777 on some specific ones for a templating system and tiny_mc plug-in to work. How can i retain the original ownership and permissions? rather than having to go through and modify them one-by-one after the Rsync, or giving a blanket 0777 for everything? Regardless, I don’t know how to specify folder-specific privileges in rsync and would prefer to just have them remain the same as before the rsync.

Reply

3 nixCraft October 8, 2007 at 9:49 pm

<owever, it keeps reverting my users:groups from myuser:nobody to 400:401.

You need to add myuser:nobody to your local server.

<I don’t know how to specify folder-specific privileges in rsync and would prefer to just have them remain the same as before the rsync.

Make sure files are transferred in “archive” mode, which ensures that symbolic links, devices, attributes, permissions, ownerships, etc. are preserved in the transfer.
rsync -avr src/path /dest/path2

Reply

4 babu May 8, 2008 at 5:40 pm

HiFriends,

Can you give a help about. I have some unix specific setup.

We have the following directories on p1cpe, and p2cpe:
/opt/Apps/StaticContent
/opt/Apps/html_templates
/opt/Apps/bea/user_projects/domains/NDC-Dev/lib
/opt/Apps/webapps
/opt/Apps/src

I need them to be available, and in sync to both systems. The originals should all be one p1cpe, so if you have to copy or do anything, those should be considered the source.

The idea is, if a developer makes changes to files on p1cpe:/opt/Apps/StaticContent, I want it also available on p2cpe, without them having to modify it twice.

All the work should be done as the user “webdude”. You should have sudo rights to webdude on p1cpe and p2cpe.

Couls you please help me on that.

Thanks,
Babu

Reply

5 pardeep July 29, 2008 at 7:29 am

I want to rsync 800Gb data over 4Mbps link. there are many folders which more than 100Gb, so it is keep failing rsync. i am using even –bwlimit also.
i need some script which look into folders and rsync one by one folder instead of one 100Gb folder. does it make sense?

any kind of script shell or php?

e.g. $cd mainfolder–
$ls
folder1
folder2
folder3
folder4
folder5

thanks

Reply

6 wYke January 16, 2009 at 4:22 pm

hi, i hope someone could help me. I need to synchronize files between two servers like every few minues, but i need to do it automatedly using crontab. My problem is I need to use SSH with a different port, so i need to use the ssh n the rsync command line. If i put the command line into cronjob, it will prompt for password rite? then it will erroneous. Please help how to solve, as in it will rsync successfully in cronjob.

Reply

7 linehrr October 11, 2012 at 7:16 am

use `echo ‘your pw’ | rsync -e ssh ….` to automatically deliver the password to your rsync.
Have a try

Reply

8 nixCraft January 16, 2009 at 5:09 pm

You need to use password less login using ssh keys.

Reply

9 wYke January 16, 2009 at 5:12 pm

hmm.. i found the answer here

thanks alot

Reply

10 Mehdi January 26, 2009 at 4:35 pm

Hello;

I want to mirror our web server into a backup server. How can I use the rsync? I see –delete in the command line above and I dont know why? I am also afraid to run the rsync from the WRONG server putting useless files into our web server, so where do you run rsync from and what files do you copy to make a good mirror/backup server?
This is for in case the web server goes down we can bring this one up(back up) and use it until the other one is fixed.
I have setup cert on both servers so ssh-ing is no problem.

Thank You Very Much!

Reply

11 nixCraft January 26, 2009 at 8:19 pm

@ Mehdi,

I suggest you use rsnapshot tool, which uses rsync as backup tool. rsnapshot makes hourly, daily, weekly and monthly backup. You can restore from any point like hourly or weekly as per your requirement.

HTH

Reply

12 Mehdi January 27, 2009 at 9:58 am

Thank You Very Much Vivek for quick response!
I will try that.

Regards;
mehdi

Reply

13 David Johnston April 15, 2009 at 5:09 am

Isn’t the “r” redundant in the examples as it is implied by the “a” option? A look at the man page says that “a” is short for “rlptgoD”.

Reply

14 poswer June 2, 2009 at 6:30 am

how to do automatic backup using rsync.

Reply

15 Miro June 23, 2009 at 4:46 pm

How to sync specific files (in this case *.dat) from multiple directories where directory name is different?
For example, I’d like to copy/sync all server1:/dir1/1*/*.dat files to corresponding server2:/dir2/2*/*.dat files (where 1* directory names are 101,102.. 199 and 2* are corresponding 201,202 .. 299 directories).

Something like (calling rsync from server2):
rsync -zrv server1:/dir1/1*/*.dat /dir2/

Any suggestions?

Reply

16 nixCraft June 23, 2009 at 4:49 pm

Try:
rsync -zrv --include="*.dat" server1:/dir1/1 /dir2/

Reply

17 Ken Nil July 30, 2009 at 7:34 am

I’m trying to set up a classroom, and on my server I have a folder with virtual machine images and config files that I need to distribute to 12 classroom computers. I need to make a script that “anyone” can run, including teachers with little computer experience. The teacher has access to the server, and the plan is that he/she only needs to run one script on the server to update all the classroom computers. Is rsync the right tool?

Reply

18 nixCraft July 30, 2009 at 8:13 am

Yes, it is right tool to push (from server) or pull files (from clients).

Reply

19 John Niken August 31, 2009 at 1:49 pm

Hi Vivek. I need to make a script to send files between local linux computers. Is the easiest way is using rsync? Or what is your recommend?

Regards.

Reply

20 nixCraft August 31, 2009 at 2:11 pm

Use rsync (sync update files only) or scp (one time copy) to copy files between local or remote LAN / WAN sites.

Reply

21 Nishith September 23, 2009 at 9:05 am

Hello.

I am having 2 Server. 1 server contains Unix OS & another Server contains Linux. I want to merge some data directories from linux PC to Unix PC.

both servers contains same named directories & files. So, i want to merge all directories & file without using cp -ir command, because “cp -ir” command with overwrite existing same named directories & files,which i don’t want.

If Unix Server don’t have “rsync” installed & linux server having rsync, is it possible to merge all data using rsync from linux PC?

Reply

22 pd September 23, 2009 at 11:47 pm

try cp -ur command
-u, –update copy only when the SOURCE file is newer than the destination file or when the destination file is missing

make sure backup both directories before run any commands.

i hope, this will help you.

Reply

23 Michael January 30, 2010 at 2:45 am

Yes I want to become a download mirror for several large companies who require me to use rsync all I need to do is copy files from their server to a directory in mine I need some help here please

Reply

24 viji February 6, 2010 at 11:49 am

i am using suse linux server and windows xp as client.I am having a network connection here.I need to transfer the database back file from the linux server to my windows xp client.please give some steps/command line arguments to transfer this.

Reply

25 Vivek February 23, 2010 at 6:40 pm

What about rsync to different port other than 22 ?

Reply

26 Gregory March 6, 2010 at 7:25 pm

@Vivek

rsync -e ‘ssh -p 40000′

Reply

27 Jozef April 10, 2010 at 4:05 pm

We staringrting to switch from solaris to RedHat. I created RedHat intel machine for webserver. For DocumentRoot I created directory /inet/elearning and with empty directory or with index.html (wich I transfer from sparc machine apache started and working but when I rsync all content from sparc I’m getting syntax error at start saying ‘DocumentRoot must be directory’. I see problem with directory format between solaris and redhat. I use rsync syntax:
/usr/bin/rsync -avzp –delete -e /usr/bin/ssh root@sparc-machine:inet/elearning /inet
Is any chance do reformat directory during transfer them or after? On solaris creation block is 512, on redhat probably 4096
Any clue help may help
Thank you

Reply

28 Suresh.G June 29, 2010 at 11:39 am

I have some folder in the source machine in the format m_d_Y, now from the remote machine i need to pull only last 30 days folders using rsync. Both the source and destination are linux boxes. Any help is appreciated

Reply

29 Arie July 8, 2010 at 6:38 pm

Hello,

Is there an option to synchronize/backup/copy files from one server to another every day at a certain time? Or better yet, does rsync provide on the fly replication, meaning as sson as something changes on one server, the changes are replicated to a second server?

Thank you,

Reply

30 nixCraft July 8, 2010 at 6:59 pm

No rsync is not real time server file system mirroring tool. It is perfect for making backup and other purposes. You’ve two options here for failback mirroring use DRBD (distributed storage system for the Linux which is like RAID 1, except that it runs over a network between two servers) or use nfs / gfs2 / ofs2 with centralized storage for cluster.

HTH

Reply

31 Tone Williams August 22, 2010 at 6:06 pm

This was fantastic! I have a couple of questions:

1.) Can it show the percentage completed for each file on-screen?
2.) Is there a way to dump the results into a text file instead of on-screen?
3.) What happens if your SSH connection is accidentally terminated in mid-process? Does the process continue without you?

Thanks.

Reply

32 akpk September 3, 2010 at 6:33 am

@Tone

1) use -v attribute along with rsync command.
rsync -varpgo /usr/local/nagios/contacts.cfg stephen@172.0.0.1:~

2) We can redirect the output to another file adding this at the end of any command
> file.txt
>> file1.txt
EG: date > file.txt

Reply

33 Tone Williams September 3, 2010 at 10:34 am

@akpk and vivek
Thank you very much! This post along with your response on how to store results to a text file has been most useful!

Reply

34 Hartmut Laue September 17, 2010 at 9:13 am

Hello,

I am using rsync under ubuntu on my netbook to duplicate the contents of my /home to an SD-Card, by

rsync -av –delete /home/x/ /media/SD-Card/home.x.backup

Running this command, the program seems to do what it should. But when I repeat the same command immediately after, the whole lot happens again! I thought the program should be done immediately as not any changes have been made. What is wrong?

(I don’t know if it plays a role how the hard disks are formatted. The source is ext4 while the SD-Card is FAT32 (or VFAT). Could it be an explanation that FAT can’t save the information on ownership (rights) so that on starting rsync a second time the program views all files as different and does the same job as before again?)

Reply

35 shiva December 20, 2010 at 6:37 am

hello,
I have one clearification on working of rsyn
how data will transfor wheather bytes leval or block leval or file leval.

Reply

36 Jay Boyd February 24, 2011 at 5:34 pm

Very helpful post. Thanks.

Reply

37 Dinesh Jadhav March 1, 2011 at 7:39 am

Lets say we wanted to copy all of the files under our local directory, “/usbdisk/” and place a copy on the machine “somemachine” under the remote directory “/backups/”. The name of the user on the remote machine is “dinesh”. Notice we added the “-z” argument so all data will be compressed over the network. We could use the following line:

rsync -avz /usbdisk/ dinesh@somemachine:/backups/

Reply

38 Meghraj May 13, 2011 at 1:21 pm

I found this article very useful especially for beginners. I will try to contribute as more as possible, so it would be helpful for others…

Reply

39 Avin June 8, 2011 at 4:12 am

Very Nice Article…

Can You please give more details about –delete option ?
When Shall I use this option ?

Reply

40 Igo Konrads July 7, 2011 at 11:12 pm

This is exactly what I am after. Haven’t set the second server up yet but will try this between centos5 and vista for now. I hope this is gonna be good enough to get rid of the useless Plesk backup tool. Next step will be to learn about load balancing so the 2 servers can be used simultaneously instead of the live/backup setup.

Reply

41 Eric Gillette July 9, 2011 at 1:44 am

This is a fantastic explanation of rSync and all it’s idiosyncrasies.

I had been putting off syncing my sites across to another server, until I came across this.

Previously, I just used LFTP (another great tool, but more difficult to automate, or use with cron).

Thanks a bunch!

Reply

42 Tapan Bhanot July 11, 2011 at 7:58 am

Hello,

How to use rsync with a username and password that is supplied by the remote server ?

I have a tar file which I need to upload to a remote server. The remote server has provided me with IP address, a username and a password. How do I use rsync with the username and password to upload the file on remote server ??

Thanks.

Reply

43 Deepa February 8, 2012 at 7:35 pm

Hi,
I am doing a rsync between two servers. I have put the script in crontab file so that it runs automatically everyday at night. But the problem is that rsync stops once it encounters a large file. But, surprisingly, If I execute the same script from the CLI, it works perfectly right. Only, when it happens in the background via rsync, it gets stuck. Please advise what could be the issue. Do settings need to be changed somewhere.

Thanks in advance.
Deepa

Reply

44 dhan March 26, 2012 at 6:47 pm

i think you need -P option for resuming of file / folder … ;)

Reply

45 nyiko July 11, 2012 at 12:08 pm

hi
i have downloaded a file in one account and want to copy the file to the other account in the same machine,i have tried to copy using scp and cp command but without success,it keeps prompting for password.please help

Reply

46 Rowan Reid August 8, 2012 at 7:53 am

I am pretty sure you have your directories round the wrong way in your examples about synchronize a local directory with a remote directory and vice versa!

Reply

47 Sapna Joshi August 17, 2012 at 10:45 am

Hi Vivek,

I am totally fresher in Unix, and I want to cut the log files from server1 and paste them to server2. I am very much confused on the command part.

Below are my understanding , Please suggest.
1.) Create public key through ” ssh-keygen -t rsa”
2.) Copy the public key to Server2 –> If yes, then where exactly I have to copy the key?
3.) write command in Server1 “scp -C $file Path$ $public_key_username@Server2.com:/path$
–> I want to cut the file from server1, what is the exact command for that?
–> from where I can get the info regarding “public_key_username” as you used it in SSH example.
4.) You must have SSH running on Server2 to make this work.
5.) write cron job to make it automatic.

Require guidance on same.
Thanks in advance.

Regards,
Sapna Joshi

Reply

48 vik September 26, 2012 at 8:32 am

Hi,
Can anyone assure me that the rsync –delete & –delete-after parameter deletes files
only on the DESTINATION (target) & that NO file deletion occurs on the SOURCE?

Thanks,
vik

Reply

49 Rashid Iqbal September 30, 2012 at 2:35 pm

I am copying a 2gb single file over 2MB link,
If copy is fail and I want to start the copy again from where it left then what is the option I need.

Reply

50 BrianBlaze October 24, 2012 at 4:04 pm

Thank you thank you thank you :D

Reply

51 rum November 8, 2012 at 3:33 am

Bump

keep this alive.

Reply

52 Siddharth Rana December 9, 2012 at 9:43 am

which protocol version rsync support ?

Reply

53 Rahul Janghel April 25, 2013 at 3:15 pm

Hi Vivek, Friends,

Is there any way to exclude any directory from sync at either end ?
How it is better than unison ?

Regards,
Rahul Janghel.

Reply

54 pete July 26, 2013 at 2:56 am

thanks. that was helpful.

Reply

55 sachin August 1, 2013 at 1:49 am

I am trying to use r sync to sync up files between 4 servers . The issue is all the servers would be master servers and files can come on any of the server at anytime . Can i use rsync in this case ? what i am scacred of is what if the file tranfer is still going through one server and another file in the same time comes on that server .

Reply

56 Daniel September 18, 2013 at 3:58 pm

Thanks so much for this intro. I’ve been meaning to start using rsync for ages, this got me straight in.

Reply

57 vinoth September 24, 2013 at 6:49 am

i have rsync errror in my downloading system.the network ip address is to be changed.
so how to configure new static ip address in rsync

Reply

58 joe January 8, 2014 at 4:17 am

I’m not sure if I’m simply misreading the “Task” titles, but it seems you’ve got the “Synchronize a local directory with a remote directory” and “Synchronize a remote directory with a local directory” mixed up.

To say “Synchronize X with Y” implies X is the destination and Y is the source; X is the thing being acted upon, Y is what is being used to act. However, this is definitely NOT what is shown in the example command. Rsync’s first directory argument is the source, the second argument is the destination.

Reply

59 Ory CHHEAN May 7, 2014 at 8:18 am

Dear Everyone,

I need help.

I would like to do a schedule backup using cron job in Ubuntu.

If I use “rsync -ave ssh root@remote_ip:/var/mail/ /tmp/backup_location/”, it will need to enter remote host password.

So how can I use above command accompanied by remote host password.

Thank you for your help.

Reply

60 Mojtaba May 22, 2014 at 2:33 pm

Hi,

I think you must change these with each other:

Task: Synchronize a local directory with a remote directory

$ rsync -r -a -v -e “ssh -l jerry” –delete /local/webroot openbsd.nixcraft.in:/webroot

Task: Synchronize a remote directory with a local directory

$ rsync -r -a -v -e “ssh -l jerry” –delete openbsd.nixcraft.in:/webroot/ /local/webroot

the format is rsync source dest.

Reply

Leave a Comment

Tagged as: , , , , , , , , , , ,

Previous post:

Next post: