Features you'd want in a bash backup script?

[MannDude]

I'm working on a backup script as a learning project. I already use it to maintain backups for vpsboard and other projects. As of right now it's relatively simple and just creates a compressed local backup (WWW data and MySQL), rsyncs it to a remote server every 2 hours where it is archived and categorized by date/time. When the script re-runs it removes the most recent LOCAL backup (only one local backup is stored) and does it's thing again.

I'm going to work on an installer script that prompts the user to set certain variables, such as directory paths to be backed up, frequency of backups, etc, etc. Another feature will be checking available disk space on the backup server and removing the oldest archived backup when needed to create room for recent backups.

Any other features you think it should include?

This is a learning project for myself but after I get it more refined I'd be happy to share it with the community so that folks here can use it to backup their own work.

 

[drmike]

This sounds awesome!

Very typical need...

Disk space checks on both ends,  rolling stamped version maybe with some clean up of ancient dumps, ummm some easy way to "deploy" the backups to a fresh install....

 

[MannDude]

This sounds awesome!

Very typical need...

Disk space checks on both ends,  rolling stamped version maybe with some clean up of ancient dumps, ummm some easy way to "deploy" the backups to a fresh install....

Hmm, the restores would be interesting. That may come at some point. I'm still a bash newb so we'll see.

I think I can add a feature that sends email reports every XX hours. Maybe after every backup is made to confirm, but also reporting current disk usage.

Example:

Your backup was a success! Details are below:


website-backup-9-29-2013-20:00.tar.gz (102MB) was successfully backed up to backupserver-hostname

Local server has 48GB free space available.

Backup server has 12GB free space available.

Or something along those lines.

 

[scott2020]

Maybe also consider including email in the backup for those running their own pop3/IMAP server.

 

[vampireJ]

There is a backup script that I used before that does this also (which escapes me right now). But instead of sending a backup of the whole server- it only sends the delta or changed files.

 

[jcaleb]

versioning

 

[splitice]

hashing on the destination server and comparison after transfer (even if its in the protocol, so many things can go wrong).

Encryption for transport and storage is also a must, preferably asymmetric with a verifiable signature.

 

[TheLinuxBug]

Use openssl to encrypt your back-ups.  While it isn't the strongest encryption, it is better than none if you intend to be storing your back-ups on an unsecure service or even in an OpenVZ container. 


tar -zcvf - <directory or files to backup> | openssl des3 -salt -k <encryption password/phrase> > /path/to/output.tar.gz.enc

Then to restore use this bash script:


#/bin/bash
openssl des3 -d -salt < $1 | tar -zxvf -

save this as something like: decryptbackup.sh

(remember to chmod +x decryptbackup.sh)

Then to restore your backups you would do: decryptbackup.sh <backupfile.tar.gz.enc>

You will then be prompted for the password.

Hopefully this helps.

Cheers!

 

[ultimatehostings]

How about a GUI?

 

[TheLinuxBug]

You may also wish to include tar.gz over ssh to the target back-up server. A good resource for this is: http://blog.bravi.org/?p=259 this way you do not have to locally have the space for the back-up to be created in, instead you can pipe the back-up directly to your back-up server.

Also, if you want to do full volume images, you could consider incorporating ddrescue:

http://www.dpreview.com/forums/post/39120721

http://www.gnu.org/software/ddrescue/ddrescue.html

apt-get install ddrescue

If are making the tar.gz locally and you want to create a checksum of it so you know you have the full uncorrupted image on your destination you can use md5sum to make an md5 sum or cksfv  to create an sfv.

I personally suggest sfv:

apt-get install cksfv

Create SFV:

cksfv <file> > <filename.sfv>

Check SFV:

cksfv -g <filename.sfv>

Cheers!

 

[raindog308]

One that is often forgotten is to version and backup configuration.

Examples:

• dpkg -l or rpm -qa
• ifconfig -a
• df -h and mount
• chkconfig (is there a Debian equiv? I don't know - shows what runs at various run levels)

etcetera.  You can get some or all of that from a backup of /etc, but it's easier to just diff a dpkg -l and install what's missing or have a chkconfig output rather than have to dig through rc scripts.

A good backup set should have:

- database backups in a restorable format.  If MySQL, I like to do one backup (mysqldump) per DB plus one --all-databases.  That way if I only need to restore one DB, it's easy.  Lather rinse repeat for PostgreSQL, Mongo, Oracle, whatever.

- configuration backup, as above

- the filesystem backup.  You can exclude lots - you're not going to restore /usr, though you might want what's in /usr/local.  You're not going to restore /tmp, etc.  But it's always better to do "everything by default"

Ideally, you have each of those versioned by day.  Versioned is really important - you want to be able to look back a few days after you're hacked  How far back?  How big is your backup system...

Also consider pulling rather than pushing backups.  If your script does a backup and then scps or rsyncs it somewhere, it's flawed, because if your server is compromised, your backups will shortly follow.  Better to have a remove server pull via rsync or some other technology (I mean proper rsync, not rsync over ssh).  If your backup server is compromised, you lose your backups.  If your primary server is compromised, you lose your primary server.  But one does not automatically mean the other, unlike push backups.

 

[perennate]

$databaseString = shell_exec('mysql --user=root --password=IjKYWGtXkD92y0VD -e "SHOW DATABASES;" | tr -d "| " | grep -v Database');
$databases = explode("\n", $databaseString);

foreach($databases as $db) {
    if($db != "information_schema") {
        exec("mysqldump --skip-lock-tables --force --opt --user=root --password=IjKYWGtXkD92y0VD --databases $db > /root/backupdata/$db.sql");
    }
}

(no that's not my password)

 

[CodyRo]

Use openssl to encrypt your back-ups.  While it isn't the strongest encryption, it is better than none if you intend to be storing your back-ups on an unsecure service or even in an OpenVZ container. 


tar -zcvf - <directory or files to backup> | openssl des3 -salt -k <encryption password/phrase> > /path/to/output.tar.gz.enc

Then to restore use this bash script:


#/bin/bash
openssl des3 -d -salt < $1 | tar -zxvf -

save this as something like: decryptbackup.sh

(remember to chmod +x decryptbackup.sh)

Then to restore your backups you would do: decryptbackup.sh <backupfile.tar.gz.enc>

You will then be prompted for the password.

Hopefully this helps.

Cheers!

Great advice but if you'd want stronger encryption use something like AES-256 and have a certificate handy: http://stackoverflow.com/a/12233688

You could use a keypair although for larger files it's unsavory at times.

 

[wlanboy]

• Roundrobin on different backup locations (based on free disk space left)
• list functionality to see what versions on what locations are available
• automated restore (including installation of packages)
• possibility to run plugins (like run all .sh files in subfolder plugins) for backup and for restore

 

[MannDude]

Thanks for the suggestions everyone. I'll try to do some of the easier ones, as this is a learning project for me

 

[splitice]

I would like to extend my suggestion, ensure it is on github (or similar social coding site -- atleast once in a reasonable state) and try for a modular or extendable architecture (e.g supply scripts to be run before download, after download, before decrypt, after decrypt, .... on complete etc). This way, although you will not be able to meet all the particulars suggested here. All should be possible (and you can use them as an idea of the hooks required).

So many people have made their own scripts. But very few have made easily extendable (and documented) architectures that I am aware of.

 

[Raymii]

Since i discovered Bacula and Duplicity i stopped rolling my own shizzle. Cliënt/server model, incrementals, a lot of backends, encryption and both are very fast...

 

[peterw]

Anyone able to create a duplicity tutorial?

 

[mikho]

Anyone able to create a duplicity tutorial?

Like this ?

http://www.lowendguide.com/3/backup/basic-website-and-database-backups-with-duplicity/

 

[BuyCPanel-Kevin]

Well every great backup system has a restore function, perhaps that's something you could implement.