Monday, October 19, 2009

Distributed File System Replication (DFS-R) and Volume Shadow Copy (VSS) for Backups

So, we have been doing an some testing on a new backup solution at our company and I wanted to see if anyone had any input. So far the testing has gone well, but I wanted to make sure we are not missing anything before we implement this in our enterprise.

We wanted off-site backups without having to carry the physical media off-site, but still wanted version control. So, what I decided to do was replicate our file servers with DFS-R and use VSS to provide version control at the primary location.

My thought process is that off-site storage of backups and version control are separate processes that get lumped together only because most backup vendors provide both in the same package. However, they really fulfill two different requirements. The ability to restore a file to a point in time (Version Control), and the ability to recover from a major disaster or hardware failure (Off-site Backup).

I started by setting up DFS-R internally at our primary site from one of our file servers to a Windows Storage Server 2003 R2 NAS. Then, I added a server at an alternate location to the replication group that would replicate from the NAS device during non-peak hours. Both locations have fairly high-speed internet connections (10 Mbps), so conceptually we are able to replicate just under 66 GB of data overnight (assuming 8 to 5 work day). This ignores the compression and byte-level replication aspects of DFS so instead of actual rates being less than conceptual rates we can replicate at close to or significantly more than conceptual rates depending on what types of files are on the server.

So, this takes care of disaster recovery, but leaves us in a world of hurt if we accidently delete some files, a file is accidently overwritten, or a file gets corrupted. This is where VSS comes into play. If we set up volume shadow copy, we can recover from deleted or changed files. Now, I know that people will complain because you could lose your versioning if you lose your server, but for most people the cost benefits of this backup solution should outweigh this negative. Also, I have come up with a few other ways to guard against this loss. The first way only works in a virtual environment, but if you are running virtual servers, you can set up your VSS volume on a disk located on a NAS or SAN device that does not host your server disks. Another solution would be to implement VSS in two locations. The last solution would be to use traditional backup technology to back up your VSS volume (You would not have to take this off-site).

I really like this solution for our backups and so far it has been very low maintenance. There are some concerns with the stability of VSS, for example, I have read it can be wiped out by disk defragmenting. However, I have had many issues with traditional backups also so I find that for me the risks are outweighed by the benefits. Besides, spending zero money on backup software makes me happy (Ok, zero money is stretching the truth, I am still working on a solution for SQL Server backups and Exchange).

Up Next: DFS Namespaces

6 comments:

  1. Hi,
    have you succeded putting on your DFSR/VSS infrastructure?
    I am thinking of doing the same here at my site but I have a question: did you enable Shadow Copies on both replication servers ?
    And, if so, when midified data is copied from host A to host B, does VSS Service on host B make a backup copy of newly modified files?
    I hope the question is clear... and thank you for your help
    THX
    Carlo
    http://www.happysysadm.com/

    ReplyDelete
  2. Carlo,

    We usually only set up VSS at the primary site, but it does work even if changes are made at the secondary site and replicated. We tested it pretty extensively. The only issues we have had with DFS are that it increases resource utilization quite a bit on our systems with millions of small files and occassionally the service locks up and replication stops so you need to check the reports regularly.

    -David

    ReplyDelete
  3. Since it's been a couple of years since you posted this, I wonder if you can tell me how well it's working for you? We happen to have been doing something similar for a couple years, but we've found it to not be as reliable as we had hoped. I actually found this blog post while searching for a way to make it so that file deletions are not replicated through DFS. We are using VSS, but we discovered some time back that it had somehow been turned off - we were lucky we didn't need it at that time. We actually had a server die on us, and discovered that somehow some of the files had not been replicated. We managed to recover them from the hard drive of the dead server, but again we were lucky. We used to use tape backups, and it's looking like we might have to go back to it, unless we can find a way to make the DFS/VSS configuration more robust. Thanks!

    ReplyDelete
  4. Have you discovered a small program called mirrorflder that can work in RAID1 or mirror in real time across WAN's or Sites, at specific times, throtle configurations, can check the CRC32 of each file, and much moreand the best of all, at an folder level and costs 25 pounds.you can even mirror an entire OS Volume with it.I have been using it for a few years now for my external HDD to be replicated to my 2 internal HDD's eah time it's connected to the computer and it does it automatically with no mess at all.
    Here's the link for the interested ones:
    http://www.techsoftpl.com/backup/

    ReplyDelete
  5. This is scary, I wouldn't rely on it as your only backup. Take this scenario: An employee gets the CryptoLocker virus. All files are encrypted. Restoring the entire volume from VSS previous versions barely ever works... But all of the encrypted files were replicated by VSS, and are encrypted in your backup too.

    ReplyDelete
    Replies
    1. Daniel, I appreciate you taking the time to comment. I wrote this post in 2009 so it is pretty dated. However, I am glad you mentioned this scenario. At the time, I discussed the loss of the server holding the shadow copies and some methods to recover from that scenario, but since CryptoLocker didn't appear until around 2013 I did not cover that scenario.

      In any case, there are probably many more options today for inexpensive, reliable, online offsite backups. However, you can recover from many versions of CryptoLocker if you have shadow copies enabled. More current versions attempt to delete your shadow copies, but I am assuming this only succeeds if the ransomware is executing with admin privileges.

      Anyway, I definitely think that CryptoLocker needs to be considered when designing a backup solution today. While you may still be able to recover with this solution, it is not something I tested and I wouldn't even bother today with the backup options available.

      Thanks again for the comment.

      Delete