As much as I love Apple’s Time Machine, it’s a hard drive pig. If not carefully watched, the little porker will use every spare byte of free space it can. What is particularly obnoxious, however, is that you might not realize you have a problem until it is too late and you’re backup drive is filled to capacity.
Take my situation as an example. I have a single MacBook Pro notebook with a 250 GB hard drive. Most of my files are text based and on the smallish side. In comparison, my networked backup is a hefty 1.5 terabytes. The combination of small hard drive and large backup drive had me thoroughly convinced that I wouldn’t have to worry about free up space for years.
I was wrong.
Because of the size of the backup drive, I like to keep other files on it – mostly music and video files – so that I have a duplicate copy. But earlier this week, I got a nasty surprise while trying to add an album I had just downloaded from Amazon Mp3. The Mac informed me the backup drive was full.
As you might guess, I found this to be very confusing. How could the drive be full? Sure … I had three or four hundred gigabytes of music and video files on it, but there was no way that the Time Machine backup could be over a terabyte in size … Could it?
This situation didn’t smell right, so I decided to investigate. I mounted the backup drive and tracked down the Time Machine sparsebundle and confirmed the impossible. My Time Machine Backup was a whopping 1.15 terabytes worth of disk space. “How in the world could the backup be so large?”, I asked myself. “Time Machine is supposed to be an incremental system. 1.15 terabytes is big enough to hold every bit and byte on my computer four and a half times over!”
First, I got annoyed; then, I got angry. What really tipped the scale toward seething fury, however, was failing to find any straightforward way of getting the space back. Yet another spectacular example of Apple’s “simple over useful” approach to computer design!
After the first bout of obscenities, I came to a simple conclusion: I could publicly express my dissatisfaction with Apple’s product line or I could go about trying to find a solution. Publicly spouting off was unlikely to help much, so I opted for the latter option. What follows is a brief summary of what I learned.
In part 1 of this article, I shared a few of the frustrations and reasons why I decided to write my own backup utility rather than submit to the tyranny of currently available solutions. While some might find those ruminations interesting, the vast majority are probably far more interested in the end result. There is a reason why “Get to the point” is one of the most important sentences in the English language.
Here’s the short version: After becoming tremendously frustrated by the state of backup on Linux, I decided to take matters into my own hands and create my own tool.
And though I only want a few things, I want that tool to do each very well. First, I’m looking for a solution that can incrementally backup over the network and let me restore a file from an existing snapshot. Second, those snapshots should be compressed, encrypted and secure. Third, it should be easy to browse old backups for existing files and restoration should be a one-click affair. Fourth, I want a backup system that can protect me from disaster, carelessness and pathological stupidity.
It is a terrible thing to realize that you are stuck in a rut. Being in a rut effectively means that you’ve stopped advancing and life has evolved to monotony. No one likes to be around people in ruts, but it’s even worse to discover that you are personally trapped in one. And, most unfortunately, I am in a rut.
Don’t believe me? Take a look at the home page of this blog. You will likely notice that a full six of the ten most recent posts have dealt with one subject: backing up your computer. That’s pretty conclusive evidence of a rut.
Now, backing up your computer is a very important thing to do; you should do it regularly and have a plan. But … well … it’s boring. Talking, thinking and writing about nothing but backup is dull. As one of the doctors I work with likes to say, “That isn’t sexy. If I’m going to spend any time with it – women, food, wine; it doesn’t matter – it should be sexy.”
He’s got a valid point, backup is not “sexy” and I’d like to write about things that are, at least for a while. This, therefore, will be my last post on backups, archives, or servers for the relatively foreseeable future (technology is just too cool to lay it aside for too long). But before doing that, I want summarize where I ended up in my quest for the ultimate backup system.
Backup on Mac is taken care of, I use Time Machine to a Samba share. More adventurous persons than I might even say that this arrangement approaches sexy. It’s convenient, fast, and robust. It even covers disaster recovery.
But the third major operating system, Linux, is a bit of the odd-man out. Certainly, you can find some excellent backup systems, Back In Time is one such example. With a bit of work, you can even tweak it so that it is almost perfect. But it’s the “almost perfect” and closely related cousins (“mostly useful” and “good enough”) that are the problem. They have those stupid qualifiers – almost, mostly, enough – bolted on.
Any time you hear a qualifier, you can rest assured that you aren’t going to like what follows. Consider the rather innocuous phrase, “that may be a problem.” Here, the term “may,” makes an already bad situation much worse. Instead of specifying some probability of problemhood, it all but guarantees it. Positive qualifiers are just as bad.
As a result, it angers me that nearly every backup program available for Linux requires some kind of qualifier. It shouldn't be like this. Linux is a brilliant operating system in practically every way. It is highly integrated, wonderfully modular and tremendously easy to extend. So … after finding that nearly every backup utility in existence has failed to meet my needs, I found the situation intolerable and decided to do something about it.
The most recent versions of Microsoft Windows, Vista and Windows 7, include a wonderfully useful tool called Volume Shadow Copies. You can think of Volume Shadow Copies as insurance against momentary stupidity or negligence.
Consider that in any given day, the typical computer user (namely me) works with a lot of files. These include data, images, and text. As part of the workflow, I may be editing and combining changes to a document from many people. Over time, this can result in a great deal of cruft. Thus, while I’m working, I try and maintain some semblance of organization by applying edits to the most recent version and keeping a semi-automated log of the changes that have been made. Older versions of the document will typically be backed up in the subversion repository on my server, or in a dedicated archive.
However, in the process of shuffling and moving the digital detritus, occasionally I have accidentally deleted the wrong file. Which, invariably, happens while away from the backup server. Losing work is obnoxious, frustrating and embarrassing; thus, my healthy appreciation for Volume Shadow Copies.
On a regular schedule, Windows takes a picture of how your drive looks at that moment and saves it away. So, should you ever need to restore a lost file, you can use the Volume Shadow copies to do so. It’s even a relatively straightforward process. Simply right click on the folder you need to access and select “Properties.” Then, click on the “Previous Versions” tab and you will find a list of every snapshot that the computer has taken.
But while all versions of Windows have Volume Shadow Copies, Microsoft decided that only those who purchase the most expensive versions (Business, Enterprise and Ultimate) get the ability to use them. Home users are out of luck. The Shadow Copy service is still there, but you need a third party program to get at the stored information.
The open source Shadow Explorer is one such option. It’s free and gets the job done. But I recently stumbled upon a second alternative that is worth mentioning, Time Traveler, developed by Bears on the Loose Software.
While reading about Time Traveler, I learned something very interesting. Microsoft considers Volume Shadow Copies to be essential to the operation of Windows. More than a few of the internal services like System Restore and File Backup make extensive use of it. Support is even built-in to Windows Explorer (the file managing utility, not the internet browser). If you know the proper url, you can navigate to where the previous versions are stored on the hard drive and work with them like any other file.
But, bizarrely, Microsoft didn’t connect any of these technologies. They didn’t make it easy for the end-user to take advantage of their hard work. Sure, right clicking is easy enough … but … that’s the problem. It’s like saying that something is “good enough,” or “useful enough.” The “enough” is a qualifier, it implies that the solution is merely passable rather than excellent.
The problem with Microsoft’s implementation of Previous Versions is that you have to add the qualifier. Of course a good tool should be invisible, at least until you need it. But Microsoft’s Previous Versions is too invisible. It forgets that you should also be able to find said tool quickly and efficiently. To use Microsoft’s Previous Versions (assuming you even have the right version of Windows), you first have to load a context menu, then you have to go to the all-encompassing “Properties” option and find the right tab. Only after three unnecessary clicks can you actually review your shadow copies. And what happens if you don’t know where your lost file once lived? There’s no way to actually search through the archive.
You see, it’s “good enough.” And because there’s a gaping hole in the integration, others can make some money by patching it; enter Time Traveler. Bears on the Loose took the final step that Microsoft should have taken: they integrated Shadow Copies into the rest of the Windows.
The program does one thing, it points Windows explorer to the url where the relevant volume shadow copy lives. Simple, huh? Remember, Windows Explorer already has support for browsing the shadow copies built-in. It just needs a little help getting there. And this works on every version of Windows: Basic, Home, Home Premium, Business, Enterprise and Ultimate.
Time Traveler represents “Previous Versions” as it should be. Invisible until needed, then easy to access and use. It’s loaded by clicking on View –> Explorer Bar -> Time Traveler. Alternatively, you can hit Control + T to toggle the slider on and off.
When loaded, it subtly clues you into what previous versions are available via a convenient timeline. You can open that point in time by moving the slider bar. Hence the name, you “travel back in time.” Windows Explorer does the rest of the work.
But even if Time Traveler makes the navigation of shadow copies easier, that one simple action doesn’t justify the $20 that Bears on the Loose charges for it. I would say that it’s the program’s “other” feature that makes it valuable. Namely, it makes it really easy to configure and manage the Volume Shadow Copy Service.
While I am aware that managing the Volume Shadow Copy Service is possible with the Control Panel, I’m not exactly sure how it’s done. Frankly, I’m not even sure where to begin. I once found a few options when I was looking for something else, but I can’t remember where I saw them. Time Traveller, however, takes all of those available settings and puts them on the same configuration page. Want to change how often the computer takes snaphosts? You can do that. Want do change how much hard drive space is used to store them? You can do that too. Want to manually delete or protect old snapshots? The settings for that are also available.
Sure, Time Traveler isn’t what anyone would call groundbreaking software. But that doesn’t mean that it isn’t valuable. It transforms Volume Shadow Copies from a system tool into a user tool, primarily by linking Microsoft’s own technologies together in a more cohesive manner. I hope that someone at Microsoft takes a good look at how Time Traveler works. It represents “Previous Versions” done right.
Despite all of its strengths, Back In Time has one major weakness: it doesn’t support backup over a network. But luckily, it runs on Linux,the single most customizable operating system in the world! As a result even though Back In Time might not officially support a backup to a remote computer, you can still make it work through a little bit of spit and grit.
The magical sauce is the Fuse SSH file system. This kernel module allows for Linux to mount a remote share as a local drive. And by doing so, Back In Time thinks that it is working with a local folder even though the data might be sent across the network or internet. To make it work, though, you’ll have to work at the command line and do a bit of programming. But, don’t panic because it isn’t that hard. It amounts to a few shell commands and about six lines of code. Below, I’ll show you how in seven simple steps.
Linux is a bit of a funny creature. In many ways, it’s the most integrated and full featured operating system in existence. Take software development, for example. You couldn’t ask for a more integrated, well thought out solution. The tools are absolutely first rate, the system is stable, and because the platform is open source, most of the rough edges get worn off rather quickly.
But there’s also a downside to open source. You see, everyone tends to scratch their own itch first. Thus, while there are a bevy of tools that perfectly meet the needs of a few (most notably their authors), they often miss the needs of the many. That is a problem, because the masses don’t necessarily have the skill set required to adapt an already existing program so that it fits them. If anything, this is why Microsoft is valuable. They create software that more or less handles the needs of the masses.
And as might be expected, this is also the state of backup on Linux. It is no understatement to say that you can get your hands on the very best backup tools available, for free. All it requires is that you use some derivative of the following:
sudo apt-get install toolname
Unfortunately, while you can find the very best tools, that doesn’t necessarily mean that you will be able to use them. Rather that be written with the end-user in mind, they were created for programmers and system administrators. You know, the type of people who wonder why anyone would bother with a user interface. After all, it’s so much faster to do everything from the command line.
That’s not to say that there aren’t contenders. In fact, a quick search on Google will reveal several: sbackup, Flyback, and TimeVault all look promising. But if you look at their project pages, you will notice something disconcerting: in each case, the programs are old and there isn’t much recent activity. This is typically a bad sign for an open source project. No activity usually means that the program is dead and unlikely to advance further. And that is really too bad, because sbackup, Flyback and TimeVault are good programs with potential. But, like many other open source options, they have a number of very rough edges.
Maybe that’s why I got so excited when I saw a new contender, called “Back In Time,” when it was recently featured on Lifehacker. At the time, I thought, “Have I finally found a backup program I can really use?” Now, after four months of using it full time, I can answer that question: Yes! Without a doubt!
At it’s World Wide Developer’s conference in June of 2006, Apple released a product that changed the way that a great many people think about backup: Time Machine. While I cringe at the thought, I need to descend into the fawning adoration public relations speak that masquerades as critical coverage of Apple Products. (Actually, forget that, here’s how Apple describes their backup system.)
Time machine is a breakthrough automatic backup that’s built right into Mac OS X. It keeps an up-to-date copy of everything on your Mac – digital photos, music, movies, TV shows, and documents. Now, if you ever have the need, you can easily go back in time to recover anything … Time Machine takes care of it … Automatically, in the the background. You’ll never have to worry about backing up again.
General sarcasm and bitterness aside, Time Machine really is a spectacular piece of kit. Sure, you can very successfully imitate a Time Machine experience using the tools within Windows Vista or Linux (or via third party tools such as Norton Ghost). Even so, Time Machine is just just about the perfect combination of simple, powerful, and integrated. And like most Apple products, when used within the Apple eco-system is a lovely experience.
However, if you migrate too far out of the walled garden, Time Machine isn’t quite so nice to work with. Actually, it can be a bit demanding and temperamental. For example, it requires its own formatted hard drive or the ready availability of a specialized Apple router (called a TimeCapsule). Alternatively, it can be a bit flaky; when I was backing up to a local hard drive, it would often quit with an indecipherable error. Luckily, however, these limitations are pretty easy to overcome. In this article, I will look briefly at how to setup Time Machine so that it works with a simple home server running Samba and Subversion.