tchen
New Member
Duply is a thin command line wrapper around duplicity, handling configuration, keys and pre-post scripts. You can still pass in duplicity options directly, but this just makes things so much nicer.
This guide assumes
Installation
Start by installing the required packages
sudo apt-get install –y duplicity duply ncftp
Duplicity, and duply as a result, can have targets that use rsync, ftp, s3, scp, webdav. I’ll use FTP as that’s most easily available.
Encryption and Signature Keys
Duply is pretty flexible with how you setup encryption. For our purposes, we have the following requirements:
Let’s start by creating a master key
gpg --gen-key
Accept the default key type, bits, and expiry. Fill in the name, email as you see fit. Use a passphrase and note it down somewhere very safe. At the end, it’ll create a key and automatically add it to your keyring. For example,
gpg: key 3B70D27E marked as ultimately trusted
public and secret key created and signed.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
pub 2048R/3B70D27E 2013-10-31
Key fingerprint = 476A 1D6E 2C3E B4AC 0D94 5EA2 1A3D F71A 3B70 D27E
uid Bob Marley <[email protected]>
sub 2048R/317C2259 2013-10-31
Note the ID next to the pub line. 3B70D27E is the key ID we’ll be using in this example. We’ll need to export this public key
gpg ––armor ––export 3B70D27E > master-archive-enc.pub
We’ll also export the private key and store it in a very secure place – preferably offline on a CD/DVD. Delete it from the ring too just to be sure. You really don’t need this private key except in extreme situations so feel free to dust off those old ICBM launch code plastic cases and place it in there.
gpg ––armor ––export-secret-keys 3B70D27E > master-archive-enc.key
gpg ––delete-secret-keys 3B70D27E
On the servers you wish to backup, import your master public key and trust it
gpg --import master-archive-enc.pub
gpg --edit-key 3B70D27E
> Trust
> 5 = I trust ultimately
> quit
Then generate the encryption key for the server
gpg --gen-key
We don’t need to export these. Leave both public and private keys in the GPG keyring.
Profiles
Create a profile using the command
/usr/bin/duply myapp create
This will create a profile folder in /etc/duply/myapp with a skeleton conf file. There’s a lot to take in, so we’ll outline a simpler configuration file. Let’s start with declaring our encryption keys.
# MINIMAL DUPLY CONFIGURATION
# file: /etc/duply/myapp/conf
GPG_KEY='25CAF99E,3B70D27E'
GPG_PW='serverpassphrase'
In this case, 25CAF99E is the server key with the ever-secret passphrase ‘serverpassphrase’. The 3B70D27E specifies our master key which will also be used as an encryption key. The backup sets can be opened by anyone who has either of the two corresponding private keys.
The next section defines what to backup and where to do it.
SOURCE='/'
TARGET='ftp://127.0.0.20/backups/myapp'
TARGET_USER='bobmarley'
TARGET_PASS='isthislove'
SOURCE can be set to a specific directory and doesn’t have to be root. Here, we use root and rely on an exclusion file to select which directories are actually included. Use this if you want to backup across multiple directories like var and etc.
# DUPLY EXCLUSION CONFIGURATION
# file: /etc/duply/myapp/exclude
- **/.svn
- **/.subversion
+ /var/myapp
+ /etc/myapp
- **
Going back to the conf file, we specify what to do during a duply purge command.
# Keep for...
MAX_AGE=6M
# Number of full backups to keep. Must issue one at least every month.
MAX_FULL_BACKUPS=10
MAX_FULLBKP_AGE=1M
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
During the first backup, duply will export the server keys into the profile directory. It might be a good idea to back this folder up onto CD/DVD and store it safe as well. However, if you ever lose it – or you just have way too many servers to track these - you can still restore from the master key.
Cron
Setup an automated schedule. Given our MAX_FULLBKP_AGE, we’ll prune about once a month.
10 15 1 * * /usr/bin/duply myapp backup_ purge --force 2>&1 >> /var/log/duply/myapp.log
10 3 * * * /usr/bin/duply myapp backup 2>&1 >> /var/log/duply/myapp.log
Be sure to setup a logrotate as the output can get quite verbose.
Pre and Post Scripts
To dump a database before the actual backup is run, duply has a pre script
#!/bin/sh
# file: /etc/duply/myapp/pre
echo 'Some very important mysql stuff'
Listing and Restoring Files
You can list the contents of the set. For the most recent
duply myapp list
or if you want something older, say from a week ago
duply myapp list 7D
Restoring is as easy as
duply myapp restore ~/restored
or if you want something older, say from a week ago
duply myapp restore ~/restored 7D
Notes
Sadly, a compromised host will mean its target bucket is also vulnerable. There's really nothing to do other than periodically archiving the backup sets to an offline storage (or at least semi-offline like Amazon glacier).
I also suggest separate FTP accounts for each server to protect them from each other.
This guide assumes
- You are running Debian
- You want to backup selective parts of your system to remote storage
- The remote archive store is relatively unsecured
- A desire to be able to verify/restore files in time
- Backup purging
Installation
Start by installing the required packages
sudo apt-get install –y duplicity duply ncftp
Duplicity, and duply as a result, can have targets that use rsync, ftp, s3, scp, webdav. I’ll use FTP as that’s most easily available.
Encryption and Signature Keys
Duply is pretty flexible with how you setup encryption. For our purposes, we have the following requirements:
- Protect a cluster’s backup from a compromised origin server
- Have the ability to open any backup with a master key
- Have the ability to automatically verify a backup set from the server
Let’s start by creating a master key
gpg --gen-key
Accept the default key type, bits, and expiry. Fill in the name, email as you see fit. Use a passphrase and note it down somewhere very safe. At the end, it’ll create a key and automatically add it to your keyring. For example,
gpg: key 3B70D27E marked as ultimately trusted
public and secret key created and signed.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
pub 2048R/3B70D27E 2013-10-31
Key fingerprint = 476A 1D6E 2C3E B4AC 0D94 5EA2 1A3D F71A 3B70 D27E
uid Bob Marley <[email protected]>
sub 2048R/317C2259 2013-10-31
Note the ID next to the pub line. 3B70D27E is the key ID we’ll be using in this example. We’ll need to export this public key
gpg ––armor ––export 3B70D27E > master-archive-enc.pub
We’ll also export the private key and store it in a very secure place – preferably offline on a CD/DVD. Delete it from the ring too just to be sure. You really don’t need this private key except in extreme situations so feel free to dust off those old ICBM launch code plastic cases and place it in there.
gpg ––armor ––export-secret-keys 3B70D27E > master-archive-enc.key
gpg ––delete-secret-keys 3B70D27E
On the servers you wish to backup, import your master public key and trust it
gpg --import master-archive-enc.pub
gpg --edit-key 3B70D27E
> Trust
> 5 = I trust ultimately
> quit
Then generate the encryption key for the server
gpg --gen-key
We don’t need to export these. Leave both public and private keys in the GPG keyring.
Profiles
Create a profile using the command
/usr/bin/duply myapp create
This will create a profile folder in /etc/duply/myapp with a skeleton conf file. There’s a lot to take in, so we’ll outline a simpler configuration file. Let’s start with declaring our encryption keys.
# MINIMAL DUPLY CONFIGURATION
# file: /etc/duply/myapp/conf
GPG_KEY='25CAF99E,3B70D27E'
GPG_PW='serverpassphrase'
In this case, 25CAF99E is the server key with the ever-secret passphrase ‘serverpassphrase’. The 3B70D27E specifies our master key which will also be used as an encryption key. The backup sets can be opened by anyone who has either of the two corresponding private keys.
The next section defines what to backup and where to do it.
SOURCE='/'
TARGET='ftp://127.0.0.20/backups/myapp'
TARGET_USER='bobmarley'
TARGET_PASS='isthislove'
SOURCE can be set to a specific directory and doesn’t have to be root. Here, we use root and rely on an exclusion file to select which directories are actually included. Use this if you want to backup across multiple directories like var and etc.
# DUPLY EXCLUSION CONFIGURATION
# file: /etc/duply/myapp/exclude
- **/.svn
- **/.subversion
+ /var/myapp
+ /etc/myapp
- **
Going back to the conf file, we specify what to do during a duply purge command.
# Keep for...
MAX_AGE=6M
# Number of full backups to keep. Must issue one at least every month.
MAX_FULL_BACKUPS=10
MAX_FULLBKP_AGE=1M
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
During the first backup, duply will export the server keys into the profile directory. It might be a good idea to back this folder up onto CD/DVD and store it safe as well. However, if you ever lose it – or you just have way too many servers to track these - you can still restore from the master key.
Cron
Setup an automated schedule. Given our MAX_FULLBKP_AGE, we’ll prune about once a month.
10 15 1 * * /usr/bin/duply myapp backup_ purge --force 2>&1 >> /var/log/duply/myapp.log
10 3 * * * /usr/bin/duply myapp backup 2>&1 >> /var/log/duply/myapp.log
Be sure to setup a logrotate as the output can get quite verbose.
Pre and Post Scripts
To dump a database before the actual backup is run, duply has a pre script
#!/bin/sh
# file: /etc/duply/myapp/pre
echo 'Some very important mysql stuff'
Listing and Restoring Files
You can list the contents of the set. For the most recent
duply myapp list
or if you want something older, say from a week ago
duply myapp list 7D
Restoring is as easy as
duply myapp restore ~/restored
or if you want something older, say from a week ago
duply myapp restore ~/restored 7D
Notes
Sadly, a compromised host will mean its target bucket is also vulnerable. There's really nothing to do other than periodically archiving the backup sets to an offline storage (or at least semi-offline like Amazon glacier).
I also suggest separate FTP accounts for each server to protect them from each other.
