rsync + robocopy + vshadow = fast, free, time machine-like backup

In my role as Maitland Christian School’s IT manager, I’ve been working on a non-proprietary, enterprise-grade backup system that will eventually replace our current ShadowProtect installation.

There are a few reasons for this:

  1. ShadowProtect costs the school $1k+ annually (before adding licenses for various servers I’m still deploying). That’s a lot of money for what is essentially a file copying tool, given we don’t need its instant-restore features in this environment.
  2. I’m not a fan of needing to install and run proprietary tools to explore backup snapshots. It’s slow (you typically need to mount one snapshot at a time and wait for the software to resolve incremental snapshots back to the latest full copy), risky (you depend absolutely on your software vendor to get your files back) and inflexible (for example, you can’t allow users to explore their own snapshots).
  3. ShadowProtect is a resource hog.
  4. Excellent free (or zero-cost) alternatives exist.

As of today, I’ve completed testing the ingredients of our new backup system, which are as follows.

1. rsync

Given this is primarily a Windows environment, it’s unfortunate to depend on software that isn’t NTFS-aware, but none of the usual suspects offer an equivalent to rsync’s --link-dest option, which creates full snapshots using hard links. This option makes it possible to create Time Machine like replicas of entire filesystems as they appear at a given point in time, without duplicating files that haven’t changed since the last sync. Furthermore, individual replicas can be deleted ad-hoc without damaging others. Awesome!

2. robocopy

To minimise the time and scripting involved in restoring data when needed (and also to make it safe for end-users to – hypothetically! – access snapshots straight off our backup server), it’s important to retain NTFS permissions and ownership. Here’s the command that does it (run immediately after rsync):

robocopy SOURCE DEST /mir /copy:sou /is /it /xl /xx

Note that if you don’t use /mir, directory permissions won’t be copied. Annoying but true. At least NTFS permissions get attached to hard links and not the underlying files, though! (Not being an NTFS guru, I wasn’t sure if this would be the case or not.)

3. vshadow

This ensures that files don’t change while you’re copying them. Microsoft explain it all here.

I’ll let you know if/when I have backup scripts that could be used elsewhere!