Debian / Ubuntu: Install Duplicity For Encrypted Backup In Cloud

Posted on in Categories , , , last updated October 4, 2013

My cloud based backup service provider has ssh access to the backup system. How do I install and configure duplicity software for remote backups in incremental and encrypted format on Debian or Ubuntu Linux based desktop / laptop / server?

Duplicity is a piece of software which provides easy encrypted versioned remote backup of files requiring little of the remote server. It does this using GnuPG, tar, and rdiff. To transmit data it can use ssh/scp, local file access, rsync, ftp, and Amazon S3.

You also need to setup a passphrase. A passphrase is a sequence of words or other text used to control access to a computer system, program or data. A passphrase is similar to a password in usage, but is generally longer for added security. Passphrases are often used to control both access to, and operation of, cryptographic programs and systems. Passphrases are particularly applicable to systems that use the passphrase as an encryption key. I strongly recommend that you setup a passphrase for your ssh-keys.

Install duplicity on Debian / Ubuntu Linux

Open the terminal and type the following command (Red hat and friends see our previous faq “Install duplicity on RHEL/CentOS based systems” for more information):
$ sudo apt-get install duplicity
# apt-get install duplicity
Sample outputs:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  librsync1 python-crypto python-gnupginterface python-paramiko
Suggested packages:
  python-boto ncftp python-pexpect python-cloudfiles python-gdata tahoe-lafs
  python-crypto-dbg python-crypto-doc
The following NEW packages will be installed:
  duplicity librsync1 python-crypto python-gnupginterface python-paramiko
0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,735 kB of archives.
After this operation, 10.4 MB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 stable/main librsync1 amd64 0.9.7-9 [72.1 kB]
Get:2 stable/main python-gnupginterface all 0.3.2-9.1 [21.0 kB]
Get:3 stable/main duplicity amd64 0.6.18-3 [309 kB]
Get:4 stable/main python-crypto amd64 2.6-4 [524 kB]
Get:5 stable/main python-paramiko all [809 kB]
Fetched 1,735 kB in 6s (262 kB/s)                                              
Selecting previously unselected package librsync1:amd64.
(Reading database ... 36532 files and directories currently installed.)
Unpacking librsync1:amd64 (from .../librsync1_0.9.7-9_amd64.deb) ...
Selecting previously unselected package python-gnupginterface.
Unpacking python-gnupginterface (from .../python-gnupginterface_0.3.2-9.1_all.deb) ...
Selecting previously unselected package duplicity.
Unpacking duplicity (from .../duplicity_0.6.18-3_amd64.deb) ...
Selecting previously unselected package python-crypto.
Unpacking python-crypto (from .../python-crypto_2.6-4_amd64.deb) ...
Selecting previously unselected package python-paramiko.
Unpacking python-paramiko (from .../python-paramiko_1.7.7.1-3.1_all.deb) ...
Processing triggers for man-db ...
Setting up librsync1:amd64 (0.9.7-9) ...
Setting up python-gnupginterface (0.3.2-9.1) ...
Setting up duplicity (0.6.18-3) ...
Setting up python-crypto (2.6-4) ...
Setting up python-paramiko ( ...
Processing triggers for python-support ...

How do I create SSH keys?

To run automated backups, you must set password-less SSH connections using an SSH keys. Use ssh-keygen command to create ssh-keys:
ssh-keygen -t rsa

Skip a passphrase (not recommend)

If you trust your local system/server/latop/desktop do not enter a passphrase. Just hit enter twice and set an empty passphrase. See the following step-by-step guide for detailed information on setting up ssh keys:

  1. Howto Linux / UNIX setup SSH with DSA public key authentication (password less login)
  2. Howto use multiple SSH keys for password less login?

Setup a passphrase for ssh keys (recommend)

If you are paranoid, set a passphrase when prompted and install keychain to setup password less login:
$ sudo apt-get install keychain
Edit your ~/.bash_profile, enter:
vi $HOME/.bash_profile
Append the following keychain config directives:

## Note --clear option is a security feature ##
/usr/bin/keychain --clear  $HOME/.ssh/id_rsa
source $HOME/.keychain/$HOSTNAME-sh

Save and close the file. See our faq: “keychain: Set Up Secure Passwordless SSH Access For Backup Scripts” for more information.

How do I create a GPG keys?

Install GNU privacy guard – a free PGP replacement:
# apt-get install gnupg
$ sudo apt-get install gnupg
Type the following command to create a GPG key to encrypt backups:
# gpg --gen-key
Sample outputs:

gpg (GnuPG) 1.4.12; Copyright (C) 2012 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
gpg: keyring `/root/.gnupg/secring.gpg' created
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <[email protected]>"
Real name: Home Nas Server
Email address: [email protected]
Comment: Home Nas Server Backup
You selected this USER-ID:
    "Home Nas Server (Home Nas Server Backup) <[email protected]>"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.

To list your gpg keys, run:
# gpg --list-keys
Sample outputs:

pub   4096R/4AABBCC7 2013-10-04
uid                  Home Nas Server (Home Nas Server Backup) <[email protected]>
sub   4096R/12345678 2013-10-04

You need to note down the public key 4AABBCC7.

How do I backup files?

The following example use scp to backup /home/nixcraft/ to ~/backups on the system:
duplicity /home/nixcraft/ scp:[email protected]/backups

Exclude the files from backup

The following command will backup the / (whole root file system) but excludes /tmp, /proc, /nas, /jails, and /mnt directories from backup:
duplicity --exclude /tmp/ --exclude /proc/ --exclude /nas/ \
--exclude /jails/ --exclude /mnt/ / scp:[email protected]/backups

Include the files in backup

The following command will backup only the /home/, /root/, /etc/, and /var/spool/cron/ directories under root (/) file system using the --include option:
duplicity --include /home/ --include /root --include /etc/ \
--include /var/spool/cron/ --exclude '**' / scp:[email protected]/backups

Encrypted backup commands

When backing up, encrypt to the given public key, instead of using symmetric (traditional) encryption. You need to use the --encrypt-key option as follows. To find out your gpg key-id use the following command:
gpg --list-keys

Full backup duplicity command

To make full encrypted backup, enter:
duplicity full --encrypt-key="4AABBCC7" /home/nixcraft/ scp:[email protected]/backups

Incremental backup duplicity command

To make incremental encrypted backup, enter:
duplicity incr --encrypt-key="4AABBCC7" /home/nixcraft/ scp:[email protected]/backups

List the files stored in the archive

To see the files currently backed up in the archive, enter:
duplicity list-current-files --encrypt-key="4AABBCC7" scp:[email protected]/backups

Verify backup duplicity command

You can verify backup with the following command:
duplicity verify --encrypt-key="4AABBCC7" scp:[email protected]/backups /home/nixcraft

Rotate backup duplicity command

You can delete all backup sets older than the given time. Now suppose you want to retain 60 days of backup and remove files older than 60 days, enter:
duplicity remove-older-than 60D --encrypt-key="4AABBCC7" --force scp:[email protected]/backups

Pass the remove-all-but-n-full count to delete all backups sets that are older than the count:th last full backup. In other words, keep the last count full backups and associated incremental ets). The count must be larger than zero. A value of 1 means that only the single most recent backup chain will be kept. Note that the option --force will be needed to delete the files rather than just list them.
duplicity remove-all-but-n-full 10 --encrypt-key="4AABBCC7" --force scp:[email protected]/backups

Restore (recover) last backup duplicity command

Now suppose you accidentally delete /home/nixcraft/ and want to restore it the way it was at the time of last backup, enter:
mkdir /home/nixcraft/
duplicity --encrypt-key="4AABBCC7" scp:[email protected]/backups /home/nixcraft/

Restore (recover) specific file duplicity command

If you wanted to restore just the file “Documents/resume.doc” in /home/nixcraft/ as it was seven days ago into /home/nixcraft/Documents, type:
duplicity -t 7D --file-to-restore="Documents/resume.doc" --encrypt-key="4AABBCC7" scp:[email protected]/backups /home/nixcraft/Documents

Cleaning up backups

You can delete the extraneous duplicity files. Non-duplicity files, or files in complete data sets will not be deleted. This should only be necessary after a duplicity session fails or is aborted prematurely. Note that the --force option will be needed to delete the files rather than just list them:
duplicity cleanup --force --encrypt-key="4AABBCC7" scp:[email protected]/backups

Putting it all together

A sample shell script:

# A simple backup script wrapper for duplicity. 
# Author - nixCraft <> under GPL v2+
# -----------------------------------------------------
## Define VARS ##
_target='scp:[email protected]/backups'
## Unlock ssh/scp/sftp ##
source $HOME/.keychain/$HOSTNAME-sh
## Cleanup ##
$_duplicity --force --encrypt-key="${_gpg_key}" ${_target}
## Rotate old backups than 60 days ##
$_duplicity remove-older-than 60D --encrypt-key="${_gpg_key}" ${_target}
## Backup our home sweet home i.e. /home/nixcraft/ ##
## Note: full backup if older than 60 day else do incremental backup ###
$_duplicity ${_src} --encrypt-key="${_gpg_key}" --full-if-older-than 60D ${_target}

Setup cron job as follows:
@daily /path/to/your/awesome/

Recommend readings:
  • Man pages: ssh(1),scp(1),duplicity(1),cron(8),bash(1)
  • duplicity home page

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin and a trainer for the Linux operating system/Unix shell scripting. He has worked with global clients and in various industries, including IT, education, defense and space research, and the nonprofit sector. Follow him on Twitter, Facebook, Google+.

7 comment

  1. i have a problem whit this
    duplicity /home/bintang/ scp:[email protected]/backups
    BackendException: ssh connection to [email protected]:22 failed: timed out

    how can fix this?

    1. I got the same error from Google. After investigating, I tried just 1 file to backup, which it did successfully. Then I reduced the amount of my data to backup and it works, so I would double-check your total backup size. Start small and check what your total data limit is for yahoo.

  2. Thanks again for a great tutorial, it was great that you ended with that useful BASH script.

    I read that you can specify “full” and “incr” to get duplicity to perform a full and an incremental backup respecitvely. However, one can not specify this, and it appears that the backup still takes place. What behaviour will this have? Will it remove the previous backup and create a fresh new one? Perform a full backup on top of the old one such that you now have 2 full restore points, or perform an incremental backup every time if there is already a full backup?

    Thanks again!

  3. I just noticed, are you missing the “cleanup” command in the script at:

    ## Cleanup ##
    $_duplicity –force –encrypt-key=”${_gpg_key}” ${_target}

    Should it not be:

    ## Cleanup ##
    $_duplicity cleanup –force –encrypt-key=”${_gpg_key}” ${_target}

Leave a Comment