Do you test your backups?

[Raymii]

​My Nagios monitors if the backups are running (Duplicity), both on the client and on the backup servers. However, I do find stuff to go hay-wire sometimes. I therefore have a monthly recurring calendar item to check backups. Has saved my ass a few times. The test is a simple restore of one or more files from one or more servers. If I have some spare time, I restore a backup to a full new server.

​

​

​Do you guys test backups? If so, often? Any horror stories?

 

[yylzcom]

Check my backups randomly, whenever I have time.

I have a backup server on ramnode, which backup all my three website-servers(from hostigation/highspeedweb) with rsync. After synced, it compress the whole thing and upload to Dropbox.

I DO found something wrong with my backup server and backups, but I was very lucky that main servers never have any problem. 

I will test backups tonight by restore it to a whole new server.

 

[AlphaNine_Vini]

Most of the backups are compressed in .tar file. Yes, it is possible if a auto backup generated is corrupted. Then if you are so unlucky that at same time your main node goes down. You have rely on old backup you took last week or month. But there tools available that can repair a corrupted file. It cant repair whole the data. It good if you make a shadow of your existing server. So if the main server goes down. It will have an secondary option to keep things running. 

 

[blfr]

​​Do you guys test backups? If so, often? Any horror stories?

It has been my New Year's resolution for two years in a row now. Third time's the charm.

 

[Joshua-Epic]

We test backups randomly once a week on random servers just to ensure everything is running smoothly. 

 

[IntegralHost]

I think it would be helpful if you can create a shell script something like ( http://forums.asmallorange.com/topic/13649-shell-script-to-monitor-file-changes/ ) to monitor the file change and send notification to your email.

 

[softsyshosting]

We have reporting enabled for Idera backups which notifies our staff for any failed backups. For scripted snapshot backups of Hyper-V VM's, we have custom script to report any backup failures every morning.

We do have few customers who routinely ensure that their servers are backed up successfully by performing BMR on separate test servers..

 

[Kakashi]

Our backups are verified automatically every week and then manually every other week Backups are only as good as your last restore.

 

[William]

I don't do incremental, only full and that by rsync (daily) as archive and GPG encrypt that - I do not verify them. I don't see much way for this backup to fail without being obvious (zero file size and alike)

 

[TekStorm - Walter]

I always check the back ups on a regular basis and in some case have aback up of the back ups if its important enough that i will lose my shirt if thing do go hay wire. I have had things go to shit and i lost everything because my back ups were stored in the same server and when that went bust i lost both, so now i have a back up kept in another area just in case that things go bad again.

 

[Geekion]

i wont say that i check it daily because i will be lien but i check it every week 

 

[vpsadm]

There are two aspects to backup checking:

* Verify the entire backup process end to end.

If your provider disappears, can you restore your backup to a new provider?

Can you restore after many backups?

(In other words, don't check the first backup only, but make sure that your incremental backup process does not change things in a way that prevents you from restoration.)

* Verify the integrity of ongoing backups.

Are you sure your latest backup captured all of the correct, desired changes?

Did your backup preserve data integrity? Were your databases backed up in a known restorable state?

 

[DomainBop]

I don't do incremental, only full and that by rsync (daily) as archive and GPG encrypt that - I do not verify them.

Basically the same method here.  Company "Internet" gets backed up to 3 cheap no-raid dedicated servers at 3 different providers and company "intranet" gets backed up to 3 other cheap no-raid dedicated servers at 3 different providers.

My personal files these days get duplicated to RunAbove Object Storage or GreenQloud StorageQloud..

 

[tonyg]

I have not performed a test backup in a while.

However, I implemented an md5 checksum of both the original tar.gz and subsequent encrypted files in both the original source and remote locations.

That does give me some piece of mind.

 

[GS-Dylan]

I just recently got done setting up and tweaking my new backup system so I have been monitoring it pretty close to make sure all is well.

 

[Aurimas]

In my opinion backups should be checked once in a week. That's what I do as well. Otherwise you might lose your data, it is better to do it once in a while though

 

[Husky]

Usually, the web hosting copany shedules a backup of the servers once a week. You need to talk to your web hosting company for this.

I think you misunderstood the question somehow... It's not how often do you back up, it's "Do you test your backups"

 

[lbft]

Usually, the web hosting copany shedules a backup of the servers once a week. You need to talk to your web hosting company for this.

The question was asked in a VPS forum - many VPS services are unmanaged and do not have guaranteed backups (and even then, many providers will only restore backups in case of a problem on their end - not "oh crap, I accidentally deleted an important file"). Even with crappy shared hosting you should not be relying solely on provider backups.

But the ultimate question is testing your backups, something that you probably can't easily do if you rely only on a provider's backups of your data.

 

[vpsadm]

Usually, the web hosting copany shedules a backup of the servers once a week. You need to talk to your web hosting company for this.

Sometimes the hosting company claims that they are backing up their customers' VPSs on a regular basis (daily or weekly), but they don't actually do it. What does "talk to your web hosting company" do for you in that instance?

TMZHosting (TMZVPS) deleted all of the data in one of their customer's managed VPSs, and then they admitted that they had not been maintaining the regular backups that they had promised their customers, nor had they notified their customers that the backups had not been maintained for some period of time. The incident happened only a few weeks ago. You have to read a few pages into this thread, but eventually all the ugly details come out:

http://www.webhostingtalk.com/showthread.php?t=1449130

Now that this messy incident has been made public, I wonder what TMZHosting has done to formally notify their other customers about the promised backups that do not exist?

 

[vpsadm]

I hate to repeat what I already posted in #12 above, but it is so important that I will repeat a summary here:

• The data owner is responsible for assessing the value of the data and determining what safeguards are necessary to protect it. In this context, the data owner is the VPS customer. 
• If the data is valuable, then the data owner must take responsibility for the backups. This is a vital part of disaster recovery and business continuity planning.
• The data owner is responsible for ensuring that that entire backup and restoration process will work as expected in case it must be called upon in a real disaster.
• The data owner is responsible for ensuring that ongoing backups are being performed correctly as planned.

Allow me to add clarifications:

• Not all data has the same value. If your VPS is a hobby and you can afford to lose it, then maybe backups are not necessary. 
• It is up to you to determine the value of your data and the acceptable risk. If your backups cost more than your data is worth, then reevaluate your backup process. Perhaps you can accept the risk of not fully testing your backup process. 
• As the data owner, you have responsibility for your data, but you do not have to do the work. It is perfectly acceptable to hire others to maintain your backups, run your tests, etc. When you hire people, it is your responsibility as the data owner to verify that they are doing the work you are paying them to do, and they are doing it correctly.