data backups

Z3-Masterd

Hello friends,

I was hoping to start a conversation about your (organization's) data backup strategy. My friend and I were chatting over a few beers when he told me that his organization of +3,000 users relied solely on local drives and a couple of RAID arrays in some file servers in lieu of backing up data.

The most recent outfit I was with performed full tape backups every Friday evening and did incremental backups Mon-Thurs. The major achilles heel in their strat was that they overwrote all existing data every week by reusing the same tapes, so if a user accidentally deleted an important file from the file server two weeks ago and no one noticed within the week, there was no possible chance of retrieving the file from our backups.

What backup strategies are you using? What mediums do you store the data on, and how far back do your archives go? Any horror stories?

colby_ar

We use SAN replication and snapshots for our primary backups and versioning. Our primary SAN gets synchronously replicated to our failover SAN in our secondary server room. The failover SAN gets asynchronously replicated to another SAN in a remote data center. Most of the snapshots are automatic at regular intervals and are kept for varying lengths of time.

This covers about 50TB of medical images and related data, around 25TB of server backups and client machine images, and an additional 10TB for miscellaneous data and database backups.

Several of the most critical databases are also regularly backed up to tape and those tapes are stored in a bank vault in the building next door.

rsutton

We have a few different backup strategies. For things in the data center running on VMware we use VDP to take daily backups. We keep these for 30 days. For on-premise servers we use Acronis to take full image backups every 2 weeks, and incrementals every night. We also keep these for 30 days. If the server happens to be a file server we also use Shadow Copies for further backups/quick restores. If the client has multiple file servers we use DFS to replicate the data (not a backup but does provide redundancy). Our servers in our datacenter use a SAN configured with RAID 50. On-premise servers generally use RAID 5.

I'm a big fan of having multiple layers of redundancy and backups. I also recommend treating backup failures as critical and resolving these issues asap.

PurpleIT

First, to your friend, I have to say, "RAID is NOT a backup solution". Repeat that until he has scared himself enough to go to his boss and demand something be done.

I currently am doing a GFS rotation direct to tape with a 7 day retention time on the dailies, 4 weeks for the weeklies and one year on the monthlies. I take the December monthly and put it in the data safe as a "year end edition" in case we need something years down the road. I will be putting a disk to disk to tape solution in place soon and will experiment with replicating the D2D backups to a warm-site for disaster recovery.

My network drives are continuously replicated to two additional sites which is really more for disaster recovery than backup purposes, and on my file servers I have Condusiv's Undelete (formerly Diskeeper) which does a great job keeping versions of MS office files and serves as a network-aware recycle bin so most of the time when someone deletes important files I can retrieve them in a matter of seconds. It does nothing for the users who drag and drop a folder however...

As for horror stories, most of them involve not keeping an eye on the backup jobs to ensure they are completing successfully or not testing restore procedures (including disaster recovery) and finding out that your data is A), not backed up and , even if it were, you can't get it to restore properly.

As rsutton said, I prefer to treat backups a critical and resolve any issues ASAP. This makes support contracts essential (especially if using Symantec), and test, test, test!

it_consultant

We use Commvault as our backup strategy which hooks into our SAN replication.

We had nearly a horror story, a cluster disk went offline during a large scale VM move and it ruined all the VMs. They were dev machines and we are in the middle of a huge release - I was able to easily get them out of backup (Commvault supports all the popular hypervisors). Without that backup, we would have been many thousands of man hours in the hole.

RAID is not a backup solution, SAN migration is more of a disaster recovery slash business continuance strategy than a backup strategy - although I think both of those things should still be done.

lsud00d

DPM is my current customers backup solution. It's a D2D2T (disk to disk to tape) implementation which is great for the HyperV infrastructure and not affecting server performance at all.

Most data is backed up to disk 3 times a day and stored on a tiered SAN configuration for 3-5 days. Weekly backups occur every Saturday, Monthly backups occur the first Saturday of every month, and then there are other retention periods that occur every 3 months on up to 3 years. Tapes are rotated off-site and I have a pretty well-oiled machine via a redesigned 5 week rotation schedule.

Every time we have needed to use it, it's pulled through. It can be a cranky baby sometimes and VSS is either your best friend or worst enemy, but that's the nature of the game.