Debian / Ubuntu: Install Duplicity For Encrypted Backup In Cloud

by on October 4, 2013 · 3 comments· LAST UPDATED October 4, 2013

in , , ,

My cloud based backup service provider has ssh access to the backup system. How do I install and configure duplicity software for remote backups in incremental and encrypted format on Debian or Ubuntu Linux based desktop / laptop / server?

Tutorial details
DifficultyAdvanced (rss)
Root privilegesYes
RequirementsPython+SSH+GPG
Linux
Estimated completion time20m
Duplicity is a piece of software which provides easy encrypted versioned remote backup of files requiring little of the remote server. It does this using GnuPG, tar, and rdiff. To transmit data it can use ssh/scp, local file access, rsync, ftp, and Amazon S3.

You also need to setup a passphrase. A passphrase is a sequence of words or other text used to control access to a computer system, program or data. A passphrase is similar to a password in usage, but is generally longer for added security. Passphrases are often used to control both access to, and operation of, cryptographic programs and systems. Passphrases are particularly applicable to systems that use the passphrase as an encryption key. I strongly recommend that you setup a passphrase for your ssh-keys.

Install duplicity on Debian / Ubuntu Linux

Open the terminal and type the following command (Red hat and friends see our previous faq "Install duplicity on RHEL/CentOS based systems" for more information):
$ sudo apt-get install duplicity
OR
# apt-get install duplicity
Sample outputs:

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  librsync1 python-crypto python-gnupginterface python-paramiko
Suggested packages:
  python-boto ncftp python-pexpect python-cloudfiles python-gdata tahoe-lafs
  python-crypto-dbg python-crypto-doc
The following NEW packages will be installed:
  duplicity librsync1 python-crypto python-gnupginterface python-paramiko
0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,735 kB of archives.
After this operation, 10.4 MB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://mirrors.kernel.org/debian/ stable/main librsync1 amd64 0.9.7-9 [72.1 kB]
Get:2 http://mirrors.kernel.org/debian/ stable/main python-gnupginterface all 0.3.2-9.1 [21.0 kB]
Get:3 http://mirrors.kernel.org/debian/ stable/main duplicity amd64 0.6.18-3 [309 kB]
Get:4 http://mirrors.kernel.org/debian/ stable/main python-crypto amd64 2.6-4 [524 kB]
Get:5 http://mirrors.kernel.org/debian/ stable/main python-paramiko all 1.7.7.1-3.1 [809 kB]
Fetched 1,735 kB in 6s (262 kB/s)
Selecting previously unselected package librsync1:amd64.
(Reading database ... 36532 files and directories currently installed.)
Unpacking librsync1:amd64 (from .../librsync1_0.9.7-9_amd64.deb) ...
Selecting previously unselected package python-gnupginterface.
Unpacking python-gnupginterface (from .../python-gnupginterface_0.3.2-9.1_all.deb) ...
Selecting previously unselected package duplicity.
Unpacking duplicity (from .../duplicity_0.6.18-3_amd64.deb) ...
Selecting previously unselected package python-crypto.
Unpacking python-crypto (from .../python-crypto_2.6-4_amd64.deb) ...
Selecting previously unselected package python-paramiko.
Unpacking python-paramiko (from .../python-paramiko_1.7.7.1-3.1_all.deb) ...
Processing triggers for man-db ...
Setting up librsync1:amd64 (0.9.7-9) ...
Setting up python-gnupginterface (0.3.2-9.1) ...
Setting up duplicity (0.6.18-3) ...
Setting up python-crypto (2.6-4) ...
Setting up python-paramiko (1.7.7.1-3.1) ...
Processing triggers for python-support ...

How do I create SSH keys?

To run automated backups, you must set password-less SSH connections using an SSH keys. Use ssh-keygen command to create ssh-keys:
ssh-keygen -t rsa

Skip a passphrase (not recommend)

If you trust your local system/server/latop/desktop do not enter a passphrase. Just hit enter twice and set an empty passphrase. See the following step-by-step guide for detailed information on setting up ssh keys:

  1. Howto Linux / UNIX setup SSH with DSA public key authentication (password less login)
  2. Howto use multiple SSH keys for password less login?

Setup a passphrase for ssh keys (recommend)

If you are paranoid, set a passphrase when prompted and install keychain to setup password less login:
$ sudo apt-get install keychain
Edit your ~/.bash_profile, enter:
vi $HOME/.bash_profile
Append the following keychain config directives:

 
## Note --clear option is a security feature ##
/usr/bin/keychain --clear  $HOME/.ssh/id_rsa
source $HOME/.keychain/$HOSTNAME-sh

Save and close the file. See our faq: "keychain: Set Up Secure Passwordless SSH Access For Backup Scripts" for more information.

How do I create a GPG keys?

Install GNU privacy guard - a free PGP replacement:
# apt-get install gnupg
OR
$ sudo apt-get install gnupg
Type the following command to create a GPG key to encrypt backups:
# gpg --gen-key
Sample outputs:

gpg (GnuPG) 1.4.12; Copyright (C) 2012 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
 
gpg: keyring `/root/.gnupg/secring.gpg' created
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y
 
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"
 
Real name: Home Nas Server
Email address: root@nas01
Comment: Home Nas Server Backup
You selected this USER-ID:
    "Home Nas Server (Home Nas Server Backup) <root@nas01>"
 
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.
 

To list your gpg keys, run:
# gpg --list-keys
Sample outputs:

/root/.gnupg/pubring.gpg
------------------------
pub   4096R/4AABBCC7 2013-10-04
uid                  Home Nas Server (Home Nas Server Backup) <root@nas01>
sub   4096R/12345678 2013-10-04

You need to note down the public key 4AABBCC7.

How do I backup files?

The following example use scp to backup /home/nixcraft/ to ~/backups on the cloud.example.com system:
duplicity /home/nixcraft/ scp://user@cloud.example.com/backups

Exclude the files from backup

The following command will backup the / (whole root file system) but excludes /tmp, /proc, /nas, /jails, and /mnt directories from backup:
duplicity --exclude /tmp/ --exclude /proc/ --exclude /nas/ \
--exclude /jails/ --exclude /mnt/ / scp://user@cloud.example.com/backups

Include the files in backup

The following command will backup only the /home/, /root/, /etc/, and /var/spool/cron/ directories under root (/) file system using the --include option:
duplicity --include /home/ --include /root --include /etc/ \
--include /var/spool/cron/ --exclude '**' / scp://user@cloud.example.com/backups

Encrypted backup commands

When backing up, encrypt to the given public key, instead of using symmetric (traditional) encryption. You need to use the --encrypt-key option as follows. To find out your gpg key-id use the following command:
gpg --list-keys

Full backup duplicity command

To make full encrypted backup, enter:
duplicity full --encrypt-key="4AABBCC7" /home/nixcraft/ scp://userNameHere@cloud.example.com/backups

Incremental backup duplicity command

To make incremental encrypted backup, enter:
duplicity incr --encrypt-key="4AABBCC7" /home/nixcraft/ scp://userNameHere@cloud.example.com/backups

List the files stored in the archive

To see the files currently backed up in the archive, enter:
duplicity list-current-files --encrypt-key="4AABBCC7" scp://userNameHere@cloud.example.com/backups

Verify backup duplicity command

You can verify backup with the following command:
duplicity verify --encrypt-key="4AABBCC7" scp://userNameHere@cloud.example.com/backups /home/nixcraft

Rotate backup duplicity command

You can delete all backup sets older than the given time. Now suppose you want to retain 60 days of backup and remove files older than 60 days, enter:
duplicity remove-older-than 60D --encrypt-key="4AABBCC7" --force scp://userNameHere@cloud.example.com/backups

Pass the remove-all-but-n-full count to delete all backups sets that are older than the count:th last full backup. In other words, keep the last count full backups and associated incremental ets). The count must be larger than zero. A value of 1 means that only the single most recent backup chain will be kept. Note that the option --force will be needed to delete the files rather than just list them.
duplicity remove-all-but-n-full 10 --encrypt-key="4AABBCC7" --force scp://userNameHere@cloud.example.com/backups

Restore (recover) last backup duplicity command

Now suppose you accidentally delete /home/nixcraft/ and want to restore it the way it was at the time of last backup, enter:
mkdir /home/nixcraft/
duplicity --encrypt-key="4AABBCC7" scp://userNameHere@cloud.example.com/backups /home/nixcraft/

Restore (recover) specific file duplicity command

If you wanted to restore just the file "Documents/resume.doc" in /home/nixcraft/ as it was seven days ago into /home/nixcraft/Documents, type:
duplicity -t 7D --file-to-restore="Documents/resume.doc" --encrypt-key="4AABBCC7" scp://userNameHere@cloud.example.com/backups /home/nixcraft/Documents

Cleaning up backups

You can delete the extraneous duplicity files. Non-duplicity files, or files in complete data sets will not be deleted. This should only be necessary after a duplicity session fails or is aborted prematurely. Note that the --force option will be needed to delete the files rather than just list them:
duplicity cleanup --force --encrypt-key="4AABBCC7" scp://userNameHere@cloud.example.com/backups

Putting it all together

A sample shell script:

#!/bin/bash
# A simple backup script wrapper for duplicity. 
# Author - nixCraft <www.cyberciti.biz> under GPL v2+
# -----------------------------------------------------
 
## Define VARS ##
_gpg_key='4AABBCC7'
_target='scp://userNameHere@cloud.example.com/backups'
_duplicity='/usr/bin/duplicity'
_src='/home/nixcraft/'
 
## Unlock ssh/scp/sftp ##
source $HOME/.keychain/$HOSTNAME-sh
 
## Cleanup ##
$_duplicity --force --encrypt-key="${_gpg_key}" ${_target}
 
## Rotate old backups than 60 days ##
$_duplicity remove-older-than 60D --encrypt-key="${_gpg_key}" ${_target}
 
## Backup our home sweet home i.e. /home/nixcraft/ ##
## Note: full backup if older than 60 day else do incremental backup ###
$_duplicity ${_src} --encrypt-key="${_gpg_key}" --full-if-older-than 60D ${_target}

Setup cron job as follows:
@daily /path/to/your/awesome/backup.script.sh

Recommend readings:
TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 3 comments… read them below or add one }

1 cocolocko October 17, 2013 at 12:40 pm

Thanks u “very much”, for your good tutorials and you work that you share!
Greetings from Germany

Cocolocko

Reply

2 el loco December 28, 2013 at 7:34 am

i have a problem whit this
duplicity /home/bintang/ scp://bintang_c0ol@yahoo.co.id/backups
BackendException: ssh connection to bintang_c0ol@yahoo.co.id:22 failed: timed out

how can fix this?

Reply

3 Cocolocko December 29, 2013 at 2:44 am

Is yahoo defently accepting ssh/scp for your connection?

Reply

Leave a Comment

Tagged as: , , , ,

Previous Faq:

Next Faq: