Apolitically Incorrect » Linux Backup

Upgrading from Time Drive 0.1

Rob Oakes — Mon, 05 Oct 2009 19:47:42 +0000

When it comes to most things, starting fresh is a blessing. The reason for this is rather simple, when starting over you don’t have to worry about baggage. After all, baggage is only valuable when on holiday; otherwise, it just slows everyone down.

This is especially true for software. Over time, computers tend to accumulate a rather potent type of digital baggage that can be very difficult to get rid of. And that digital garbage results in inconsistencies that can cause enormous – and usually unforeseen – problems.

However, even though starting fresh is usually the best option, that isn’t always true. Sometimes, it’s better to risk the problems and incompatibilities. For example, starting over may mean that you destroy hours worth of customization, or that you lose work already created because the older version are not compatible with the new.

Unfortunately, the general rule is also somewhat true of Time Drive. So, if you were one of those stalwart and brave individuals who decided to experiment with Time Drive 0.1, this post is for you.

In the last few days, I have been in touch with a number of people who have experienced a number of said inconsistencies and problems. And while several of these problems ended up providing insight on mistakes made during development, some of the others were changed on purpose. That is to say, the so called “bug” was actually a feature.

After fielding a couple of particularly angry e-mails, however, I thought that it might be good do a formal write up that describes how to work around these incompatibilities. And while no one likes to squash bugs or fix things that previously worked, rather fortunately, every one of these problems can be overcome with a little bit of effort and patience.

Reset Your Preferences

Right. Now that the tone has been set, it might be good to talk briefly about the changes that caused the majority of the problems. As explained in Time Drive 0.2 overview, Time Drive has a completely new settings system. In order to support Amazon S3 and a few of the more advanced Duplicity settings, these changes were necessary; but at the same time, they introduced the aforementioned issues.

Here’s the bad news: as a general rule, Time Drive 0.2 can’t read the settings file from 0.1.0 to 0.1.5. It might try, but the results are not going to be pretty. The list below includes just a few examples:

If running Time Drive on a version of Python 2.5, you may experience sudden user interface freezes or unexplained errors.
The program might just refuse to do anything. For example, it may not let you open the configuration pane, run a backup, or a restore option.
The computer might suddenly start to lie to you, for example, it could say that a backup completed successfully when it really failed. Or that a file was restored, when instead it got sucked into some miscellaneous electrical chaos somewhere.

In all of these cases, there is only one thing to do. You’ll need to delete the old Time Drive settings file, which is found in the user configuration folder for Linux and Cygwin (Windows) users:

/home/Username/.config/Oak-Tree/Time Drive.conf

And in the Library/Preferences folder for users of Mac OS X:

/Users/Username/Library/Preferences/us.oak-tree.Time Drive.plist

After you find and delete that file, then you can launch Time Drive in the normal fashion. This will create a brand new settings file of the same name. Unfortunately, you will need to re-specify your backup location, re-add your folders or exclusion criteria, and reset your preferences. But once that’s done, you can pick up right where you left off.

Beware of Spaces

While the new preferences system is the biggest cause of upgrade problems, it wasn’t the only culprit. The next largest cause of problems happened to be us, the developers. In addition to the aforementioned “bugs” that were really features, we also inadvertently created new ones. The best example would probably be how Time Drive deals with spaces.

The underlying issue doesn’t have anything to do with Time Drive, actually, but is due to quirks in Duplicity. Put simply: Duplicity doesn’t like spaces in the backup destination. At all. It does okay when backing up to a secondary hard drive or USB drive, but that’s about it. SSH, FTP, WebDav and Amazon S3 backups will all fail. Or worse, it will cutoff the destination name at the first space, causing a raft of other problems. (/home/Jon Adams would suddenly become /home/Jon).

So … to fix this issue, we decided that we simply wouldn’t allow spaces in the backup destination any longer. For most remote connections, this was never much of an issue, since who ever heard of a space in an ftp or http url? But it did become a problem if one of your folders had a space in the name.

Let’s consider, for a second, that you added a folder named “My Documents” to your list of backup folders. Time Drive 0.1 would try and store all of the backup data at “/path/to/archive/My Documents”. If that data was sent over ssh, ftp, or webdav(s), you got all kinds of other obnoxious errors. But, if you happened to choose a local folder as your backup destination, it would work.

Time Drive 0.2.0 solved this problem by automatically replacing all spaces in the folder destination name with a dash. Thus, instead of storing your backup data at “My Documents”, it would be stored at “/path/to/archive/My-Documents" (note the dash).

But while this fixed the remote backup issue, it caused problems for people who made local backups. After the upgrade, Time Drive wasn’t might not have been able to find the previous backup sets, and would start all over again; an action that some people found annoying or confusing.

This problem has now been corrected. Time Drive 0.2.1 and later allow spaces for local backup paths. Remote backup urls will still have any space replaced by a dash, however.

Cron No Longer Works

A third very common complaint was that backup jobs through cron would no longer work correctly. Like the issue described above, this “bug” was actually a feature. For automatic backups, Time Drive uses the built-in Linux/Mac OS X cron daemon. Based upon the option that you specified, it would add then schedule itself for daily, weekly, or monthly backups and provide a link to the “timedrive-backup” script.

Well the location of that script moved. Previously, it lived in the “timedrive” folder. But in Time Drive 0.2, we changed how the program gets installed. It now uses a proper Python distutils script (setup.py) to copy the main program (time-drive) and the backup script (timedrive-backup) into usr/local/bin. The new location of the backup script results in a broken cron job.

To fix this problem, simply go into the preferences dialog of Time Drive and change the backup frequency to something other than what it is now, then, click the “Ok” button. You can then go change it back. This will cause the crontab to be updated with the correct location of “timedrive-backup”. And, like always, Time Drive on Windows doesn’t support automatic backups, since Windows doesn’t have a cron daemon.

Summary

And there you have it, three of the most common upgrading complaints. But please keep in mind, this list is far from exhaustive. Should you run into other problems, please leave a note in the comments and I will get back to with something that may represent timeliness. I may even be helpful, but as those encounters have yet to happen, overall that is yet to be seen.
______________________________________________________________

(Miscellaneous Request: When making comments, please be polite. I can certainly understand that you may be frustrated. After all, Time Drive might have just destroyed your only backup of your novel! Even so, rudeness probably not be tolerated;unless it happens to be deviously clever, that is. Cleverness and wit will always be tolerated, and may even be encouraged.)

Time Drive 0.2 – A “More S” Release: More Stable, More Secure, More Settings and Now Supports Amazon S3 Storage

Rob Oakes — Thu, 24 Sep 2009 18:41:57 +0000

The first time that you attempt to do something, it’s pretty much a guarantee that it’s going to suck. This doesn’t necessarily need to be a bad thing. Shows like America’s Funniest Home Videos and MXC have found dozens of way to cash in on the humiliation of their participants. (And what better exemplifies pure suck than a golf ball to the groin?)

It, therefore, shouldn’t come as any surprise that creative pursuits are no exception to the general rule of suckiness. After all, you have to overcome inexperience and ineptitude to produce anything. The only way to ensure that a release doesn’t suck is to finish a first draft and revise heavily. Which requires a great deal of work.

Yet … as interesting as that might be, this isn’t a post about the creative process. It’s about Time Drive and I should probably admit that Time Drive 0.1 had a few … rough … edges. Sure, it mostly worked, but it was new software and did too many strange things to declare anything other than a “work in progress.” But Time Drive 0.1 was a first release and first releases suck.

Second releases, however, offer a chance to clean things up, refine the bleeding edge, and otherwise deliver the goods. Maybe that’s why I’m so excited to announce the release of Time Drive 0.2. This version of Time Drive is a great improvement over it’s predecessor. So much so that Time Drive 0.2 is hereby dubbed the “More S” release: more stable, more secure, more settings, and Amazon S3 storage.

In the remainder of this post, I’ll attempt to justify such a silly name by taking a look at a few of those new features.

More Stable and Secure

First up, I should probably say something about the program’s stability and security. But before doing that …

Isn’t it a bit ironic that stability and security have become major selling points? Go ahead and look at the advertising for any major piece of software. References to stability and security – or their closely related siblings, reliability and refinement – will feature prominently.

As an example, let’s analyze the promotional materials for the most recent release of Mac OS X, Snow Leopard. In the banner advertisement below, stability and reliability form the cornerstone of the brainwashing message: “Refinements, your Mac gets faster and easier to use.” And in the text, we catch all the buzzwords: simplicity, reliability, security, speed. In short, they’ve attempted to make make old things new by refining them.

Time Drive has gone through a similar type of rewrite. With the generous help of Philippe Delodder, all of the core classes have been reformulated and reorganized: they’re new again. The end result is that all things are improved, spiffed and otherwise sped up. In short, Time Drive no longer resembles a prototype as much as a proper foundation upon which to grow.

More Settings

And nowhere is this more apparent than in the new advanced settings dialog. In version 0.1 Time Drive handled all the important things that a backup program should. It was easy to specify which folders you wanted to backup and where the archive was to be stored. But that was about it.

The first version didn’t let you fully use the power of Duplicity to specify what you considered important. What if you didn’t want to backup hidden files or settings? Or what if you wanted your work files to be stored in a different location than your home files? Or what if you want to create a new backup every couple of months?

Time Drive 0.2 gives you that degree of control without losing the simplicity. Just like before, you get started by telling the program where you want to store the archive and which folders to include. If that’s all you need, then you’re done. Click “Ok” and let Time Drive do it’s thing.

But why settle for simple when you can micromanage? Need to refine the settings for an individual folder, highlight it from the list and press the “Advanced” button. From there, you have easy access to things you don’t want included in the backup, fine control over where it gets kept, and even the ability to remotely manage the backup sets.

What You Exclude Can Be as Important as What You Include. Despite the fact that we should keep redundant copies of our data, sometimes there are things you just aren’t worried about. For example, you might be a Python developer and want to use Time Drive to backup your source code. The source code is important, but whenever a python script gets run, the python interpreter creates a compiled file (“.pyc”). These files are necessary for the code to run, but they don’t contain any valuable or new information. They just take up space. So, let’s leave them out them by adding “.pyc” to the exclude list. Just press the “Add” button and type the offending characters into the message box. Want to leave “.pyc” files out of all of your backups? You can do that too from the main exclude list.

Different Storage Locations. Now, what if you want your work related backups to be kept on your company’s ftp server instead of your local hard? Time Drive has you covered there as well. Just click on the “Archive Settings” tab and tell Time Drive where you want the folder’s archive to live. You can even specify a folder specific encryption key should you so choose.

Manage Your Backup Sets. Perhaps the only downside of creating frequent and redundant backups is that old files might take up needed hard drive space. Sure, managing backup sets by hands is possible … but who wants to do that? Time Drive 0.2 makes it easy to delete your old backup sets when they’re no longer needed, thereby saving valuable hard drive space.

Amazon S3 Storage

Which finally brings me to the biggest new addition to Time Drive, support for Amazon S3: a reliable, simple and inexpensive way to save your files over the internet. When it appeared a few years ago, S3 introduced a radically new idea: only pay for the storage you actually use. At the time, other providers charged a flat rate for a set amount of storage, regardless of whether you needed it or not. While prices ran the gamut of reasonable to outrageous, you were still paying for something that you might not be using.

Using S3, however, it is possible to have offsite and redundant backups while knowing that you are only getting charged for what you use. Moreover, the rates are quite reasonable, about 0.15 cents per gibabyte of data per month. For example, I currently keep all of my source code files and pictures backed up through Time Drive. In total, it amounts to about 15 GB of space. Amazon charges me about $2.25 a month for the space, which is substantially less than any other provider I’ve been able to find. Sure, there are plans that are slightly more expensive and provide more space; but that’s just it, I don’t need that additional space. I’m probably not going to use it. Which makes Amazon a really good fit for my needs. It’s cheap, it’s convenient and it’s available everywhere. Most importantly though, it’s offsite, so I’m prepared in case some sort of disaster does strike. And Time Drive works with it seamlessly.

Conclusion

In summary, I think that Time Drive handily earns it’s “More S” label. It’s a big improvement over the previous version and steps up the stability, the number of settings, and the types of storage backends that it works with. But it still remains elegant and easy to use. In short, it’s one step closer to the overall goal of “Backup Done Right.”

Head on over to the project’s home page and check it out.

Backup for Linux, Done Right - Part 2: Time Drive 0.1

Rob Oakes — Fri, 07 Aug 2009 17:39:48 +0000

In part 1 of this article, I shared a few of the frustrations and reasons why I decided to write my own backup utility rather than submit to the tyranny of currently available solutions. While some might find those ruminations interesting, the vast majority are probably far more interested in the end result. There is a reason why “Get to the point” is one of the most important sentences in the English language.

Here’s the short version: After becoming tremendously frustrated by the state of backup on Linux, I decided to take matters into my own hands and create my own tool.

And though I only want a few things, I want that tool to do each very well. First, I’m looking for a solution that can incrementally backup over the network and let me restore a file from an existing snapshot. Second, those snapshots should be compressed, encrypted and secure. Third, it should be easy to browse old backups for existing files and restoration should be a one-click affair. Fourth, I want a backup system that can protect me from disaster, carelessness and pathological stupidity.Time Drive is based on the excellent command line program, Duplicity though it was also inspired by another backup utility, Back In Time. Because it uses Duplicity for its backend, Time Drive plugs into the same industrial strength utilities used to backup and restore enormous sets of enterprise data. It just removes the industrial level complexity.

Like Back In Time for Linux and Time Machine for Mac OS X, Time Drive not only keeps a copy of the most recent file, but it remembers how that file looked at many points in the past. Put another way, it creates an incremental snapshot which lets you restore to a previous version should you so desire.

The Time Drive main window is designed to be simple and functional. Options related to backing up files are on the left, options related to restoring files are on the right. If you want to restore from a remote archive which is not currently specified in your settings, you can do so through the “Advanced Restore Options.”

Over the Network or Across the World

Backups can be made to remote servers and web services that support any one of the following protocols: file transfer protocol (ftp), secure shell (ssh), secure copy (scp), rsync, webdav (http), and secure webdav (https). Time Drive also supports backups to a locally attached drive or mounted volume. This includes shares that are mounted with virtual filesystems like sshfs, smbfs, and s3fs.

You can configure the remote backup by clicking on the general tab of the settings pane. Then, move the option to “Use a remote directory (SCP, WebDav or FTP).” From the drop down list, choose the communications protocol.

If making a backup to ftp or webdav (http), enter the name of the host and the path to your backup folder as a standard web url. It should look like:

computer.name.com/path/to/folder

For example, the backup path to my webdav share is:

backup.oak-tree.us/RobOakes/Backup

If backing up by scp or ssh, you will need to use a second slash after the server name, for example:

backup.oak-tree.us//RobOakes/Backup

Next, enter the username and password that you use to access the server and some sort of passkey in the “Encrypt Files” box. (While future versions of Time Drive will allow users to choose whether or not they wish to encrypt their files, this has not yet been implemented.) When finished, you can check the connection by clicking on the “Check Connection” button to the right of url text box. If Time Drive is able to connect to the store, a little green check mark will appear.

Restoring Files

Should you delete a file that you needed, you can quickly browse or search the archive to find it and restore it. Need to restore a bunch of files? You can add as many as needed to the queue and then restore them with a single click. The restore dialog is accessed by clicking on the “Add files” button of the main window. The restore dialog itself allows you to browse the folders available in your backup, and any subfolders of the archive.

But what if you need access to a file in your work archive and you happen to be at home? Time Drive makes it easy to brose and restore from any archive, anywhere. Simply type in the url and click “Add files” to enter the restore files dialog.

Set It, Then Forget It

While Time Drive can be configured to work many different ways, file backup should ultimately be a very simple affair. True, it’s nice to have more advanced tools available, but most things in life aren’t really all that complicated.

As a result, setting up Time Drive for the first time is designed to be as easy as possible. First, pick where you want to perform your backups: either to the local machine, or across the network. Next, enter in a passkey to encrypt your backup files and keep them safe. Finally, choose which folders you want to see included in the job. When finished, click “Ok.”

Because each folder is a run as a separate duplicity script, you can always re-add that folder at a later time and Time-Drive will update the existing snapshot with the most recent information. All of the existing history will be there should you ever need to access it.

Time Drive also makes it easy to schedule automated backups through the use of the powerful crontab utility available on most *nix machines. Because it uses cron, there is no need to have a dedicated background process. (Though cron must be running for scheduled backups to work.)

Automated backups are configured by opening the settings dialog and navigating to the “Options” tab. Then, just pick how frequently you would like the program to run: every hour, day, week or month. When done, select “Ok.”

Notifications and Feedback

When looking at other solutions, I quickly became frustrated by the way which they handled user notifications. In some cases, the program provided a level of detail that was simply overwhelming. In others, it didn’t provide any feedback whatsoever; even when there was a problem.

With Time Drive, I’ve tried to split the difference. It uses the notification system of Ubuntu 9.04 and will let a user know when new backup jobs start and it finishes, if there is a problem, it will let you know that too. But for those who would rather not be bothered with such trivial things, the notifications can be easily disabled through the settings tab.

Beta Software

Even though Time Drive is relatively full featured and offers a convenient backup package, it is still beta software. That means that it needs to be tested, beat upon and otherwise tortured. As you do so, I would ask that you be a bit patient and mind the potholes. If it doesn’t quite meet your needs, please leave a bug report or feature request on the bugtracker page. Be as specific as possible and I will try to incorporate your ideas into the next release.

Conclusion

With Time Drive, I finally have a backup utility for Linux that does things right. It can backup to a computer on the network or across the internet; keeps incremental snapshots of only the files that have changed; makes file restoration a one-click affair; and keeps my personal information safe. What’s better, no tweaks or hacks are required to make it work the way that I want. In short, I finally have something that “Just Works” without having to stick some stupid qualifier in front.
______________________________________________________________

If you are interested in downloading, testing or using Time Drive, you can find additional information on the project homepage. This includes installation instructions for Time Drive, Duplicity and PyQt in addition to the program documentation.

Backup for Linux, Done Right- Part 1: A Mini Melodrama

Rob Oakes — Fri, 07 Aug 2009 04:58:08 +0000

It is a terrible thing to realize that you are stuck in a rut. Being in a rut effectively means that you’ve stopped advancing and life has evolved to monotony. No one likes to be around people in ruts, but it’s even worse to discover that you are personally trapped in one. And, most unfortunately, I am in a rut.

Don’t believe me? Take a look at the home page of this blog. You will likely notice that a full six of the ten most recent posts have dealt with one subject: backing up your computer. That’s pretty conclusive evidence of a rut.

Now, backing up your computer is a very important thing to do; you should do it regularly and have a plan. But … well … it’s boring. Talking, thinking and writing about nothing but backup is dull. As one of the doctors I work with likes to say, “That isn’t sexy. If I’m going to spend any time with it – women, food, wine; it doesn’t matter – it should be sexy.”

He’s got a valid point, backup is not “sexy” and I’d like to write about things that are, at least for a while. This, therefore, will be my last post on backups, archives, or servers for the relatively foreseeable future (technology is just too cool to lay it aside for too long). But before doing that, I want summarize where I ended up in my quest for the ultimate backup system.

Backup on Mac is taken care of, I use Time Machine to a Samba share. More adventurous persons than I might even say that this arrangement approaches sexy. It’s convenient, fast, and robust. It even covers disaster recovery.

Backup on Windows is also covered. The built-in file backup is easy to use and works well. Moreover, setting up a disaster recovery system is relatively painless.

But the third major operating system, Linux, is a bit of the odd-man out. Certainly, you can find some excellent backup systems, Back In Time is one such example. With a bit of work, you can even tweak it so that it is almost perfect. But it’s the “almost perfect” and closely related cousins (“mostly useful” and “good enough”) that are the problem. They have those stupid qualifiers – almost, mostly, enough – bolted on.

Any time you hear a qualifier, you can rest assured that you aren’t going to like what follows. Consider the rather innocuous phrase, “that may be a problem.” Here, the term “may,” makes an already bad situation much worse. Instead of specifying some probability of problemhood, it all but guarantees it. Positive qualifiers are just as bad.

As a result, it angers me that nearly every backup program available for Linux requires some kind of qualifier. It shouldn't be like this. Linux is a brilliant operating system in practically every way. It is highly integrated, wonderfully modular and tremendously easy to extend. So … after finding that nearly every backup utility in existence has failed to meet my needs, I found the situation intolerable and decided to do something about it.

I wrote my own.

Backgrounds and Backends

Fixation and ruts can make you do silly things like that. Instead of just accepting the limitations of an existing situation, a fixated person will demand that the world bend to their expectations. While this sometimes leads to great advances, more often it results in interpersonal disasters of epic scale. And the general rule is, the more menial the detail, the larger the scandal. (At this point, it might be argued that backup to an external hard drive versus across the network is a rather menial detail. To all such detractors, all I can say is: stuff it. This is my story.)

Basic Needs

Luckily, however, the story of my backup utility isn’t nearly so sordid. This is probably because my needs are actually pretty simple. I need a backup utility to do just a few things, but I require that it does them well:

It should create versioned snapshots of my drive. This lets me restore a file to any number of past states.
The utility should only transfer the portions of the file or directory that have changed. This makes the backup operation quick and efficient.
I should be able to backup over a network or across the internet. (Very important!)
Restoring a file should be quick, easy and painless. A backup is only as good as your ability to get at the information. And like it or not, information will be lost due to disaster, carelessness or miscellaneous stupidity.

Being a somewhat clever person, I didn't to avoid duplicating as much work as possible. As a result, I started creating my program by reviewing the application that comes closest to doing what I want: Back In Time. The Back In Time user interface is simple and elegant. More importantly, though, the configuration options make sense. Whether you want to run a backup, give your snapshot a name, or just get rid of it; you don’t have to go hunting to find the controls. Additionally, Back In Time makes good use of existing open source programs to actually do the backing up. Though they might be a bit old, rsync and the Unix copy command, cp, are a formidable duo.

But I quickly discovered something unfortunate, Back In Time really couldn’t be tweaked to meet all of my requirements. You see, it utilizes a technique called hard-linking to create space efficient repositories. But hard links only work as long as they are on the same hard drive. Because I want to store my files on the network or across the internet, that isn't good enough.

Duplicity

So, I started to look elsewhere for other options, which is when a colleague pointed me toward a command line script called Duplicity. In a striking bit of irony, while the GUI tools of Linux backup are all fundamentally flawed, the command line versions are the best in existence. They easily accomplish all of the requirements on my list, and do a great many things I’ve never even heard of. As a result, if you are willing to roll up your sleeves and hit the learning curve, there is nothing that you can’t accomplish. And, as you might deduce from my glowing introduction, Duplicity makes short work of my requirements list.

Can it do incremental backup? Yes. What about backup over the network? Ditto. In fact, it handles most protocols I’ve heard of, and quite a few that I haven’t. Should incremental through ssh, webdav, ftp, smb, or imap be insufficient, it also supports backup to Amazon S3. File restoration is also relatively painless, though you have to work through the command line interface to do so.

And that’s only for starters, Duplicity also supports encryption via passphrase or passkey, compression of the files into archives, and local caching of file signatures so that backup operations are lightning quick. I decided relatively quickly that Duplicity would be the ideal backend for my utility.

Frontend and Features

But while a solid foundation is a great starting point, it can only get you so far. After all, you don’t brag up the brickwork and reinforced concrete of your newly purchased home, you’re far more likely to show off the kitchen and home theater.

For a program, the points of interest are going to be the user interface. Which makes two lessons learned while working on LyX-Outline very important: 1) It’s really hard to put together an interface that is both intuitive and uncluttered. 2) I’m really bad at it. Both points prompted me to act on the advice of a famous painter:

Good artists copy; great artists steal.

Thus, I decided to steal the user interface from Back In Time. (It’s open source, which makes it okay.) And speaking from a practical standpoint, why wouldn’t I? The developer spent a great deal of time working out how he wanted his program to work. It would be silly to duplicate that effort. Moreover, the fact that I was able to rave about it for nearly 2800 words means that it wasn't wasted effort. As you browse the menus and configuration panes of my derivative tool, you will probably notice that it bears a striking resemblance to other programs. A few things will be “innovatively different,” but not many. Just remember, I stole the interface. Wholesale. But I’m a discriminating thief, and only stole the best parts.

Which brings us to the real point of interest of this post and my program: the feature list. In part 2 of this article, we’ll take a look at my little creation, which I have unimaginatively dubbed “Time Drive.”

Back In Time (Part 2): Over the Network and Across the World

Rob Oakes — Tue, 21 Jul 2009 01:15:03 +0000

Despite all of its strengths, Back In Time has one major weakness: it doesn’t support backup over a network. But luckily, it runs on Linux,the single most customizable operating system in the world! As a result even though Back In Time might not officially support a backup to a remote computer, you can still make it work through a little bit of spit and grit.

The magical sauce is the Fuse SSH file system. This kernel module allows for Linux to mount a remote share as a local drive. And by doing so, Back In Time thinks that it is working with a local folder even though the data might be sent across the network or internet. To make it work, though, you’ll have to work at the command line and do a bit of programming. But, don’t panic because it isn’t that hard. It amounts to a few shell commands and about six lines of code. Below, I’ll show you how in seven simple steps.

Step 1: Prepare a Local Mount Point and Set Permissions

First, we first need to create a mount point. A mount point is a blank folder that the operating system uses to connect the local computer to the remote server. I typically like to have my mount points in the /media folder. If the mount point is in the /media folder, it will show a little icon on the desktop when active.

So, open up the terminal and type the following:

sudo mkdir /media/Backup

This command will create a new folder (Backup) in the media folder. As is, we could then mount the SSH file system to this mount point, but to do so would require that we run the command as the super user (root). It’s not necessarily good to run things as root. So, let’s change the folder’s owner and permissions so that we can mount it without requiring the super user:

sudo chown –R /media/Backup
sudo chmod a+x /media/Backup

In the first command above, replace with the username that you use to log-on to Linux. In my case, it would be: roakes. The second command properly sets the permissions so when it comes time to mount the remote share, the computer won’t prompt for the root password.

Step 2: Install the SSH File System (SSHFS)

Now that we have a valid mount point, let’s install the software packages for SSHFS. If using Ubuntu, type the following into your system terminal:

sudo apt-get install sshfs

Step 3: Get the IP Address or Url of the Remote Computer

With the mount point set and software installed, we need to do a bit of planning. “Proper preparation prevents pitifully poor performance” and all that. First, we need to know where the data will be stored. Will we be using a simple home server or will we be storing that data on a remote server over the internet? Also, will you be able to connect to the server using the secure shell (SSH) protocol? If so, what is the destination folder?

We will use this information to construct a connection string. It will have the form:

@Machine.Location:/path/to/backup/folder

SSH to a Machine on the Local Network

If we will be backing up to a computer on the local network, we need to know it’s IP address so that we can establish an SSH connection. To find this, go to the server and log-on. Then type ifconfig into the terminal window. This will return a block of text similar to that seen below, which was copied from my server. I have added a series of x to some of the information.

eth0 Link encap:Ethernet HWaddr 00:xx:xx:xx:e0:93
inet addr:xxx.xxx.1.106 Bcast:xxx.xxx.1.255 Mask:255.255.255.0
inet6 addr: fe80::xxx:xxxx:xxxx:xxxx/64 Scope:Link
UP BROADCAST RUNNING MULTICAST xxx:1500 Metric:1
RX packets:26460698 errors:23 dropped:145 overruns:17 frame:0
TX packets:14444204 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2320464437 (2.1 GB) TX bytes:2079312151 (1.9 GB)
Interrupt:3 Base address:0xd800

lo Link encap:Local Loopback
inet addr:127.0.0.x Mask:255.0.0.0
inet6 addr: ::x/xxx Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:9308 errors:0 dropped:0 overruns:0 frame:0
TX packets:9308 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4232264 (4.0 MB) TX bytes:4232264 (4.0 MB)

The IP address can be found on the second line below eth0, which starts : inet addr. In my case, the IP address is: xxx.xxx.1.106. The xxx.xxx will have numeric values that are specific to your network. To connect to the machine, I use a connection string that looks like:

@xxx.xxx.1.106:/path/to/backup/folder

SSH to a Machine on the Internet

If you will be connecting to another computer on the internet, then instead of using an IP address to connect, you will instead be using a url. You can get this information from the information page of your backup service, or from the network system administrator. To connect to the remote machine, you will use a connection string that looks like:

@machine.url:/path/to/backup/folder

To remotely access my outside of the network computer, I use:

roakes@remote.oak-tree.us:/media/Linux-Backup

Step 4: Test the Connection

Once I have the connection string, I can now test SSHFS and make sure that it can mount to my mount point (Step 1). To mount the remote drive, type:

sshfs @Machine.Location:/path/to/backup/folder /media/Backup

Be sure to replace the part in italics with your connection string. The first time that you connect, you will be prompted to accept the server’s key and be asked for your password. After you enter your user credentials, it should mount the file system to /media/Backup. Open up the location in finder and make sure that you can create folders and move files onto the remote machine.

You can unmount the volume by typing (no root required):

fusermount –u /media/Backup

Step 5: Use a Passkey Instead of a Password

While a password is acceptable when you want to connect manually, we want the backup script to run automatically. That means no password prompt.

Luckily, however, it’s easy to configure Linux so that it uses a passkey instead of a password. This will allow your computer to connect automatically instead of prompting you for your user credentials each time that you want to run a backup or restore a file. To do so:

Generate a key on your computer by typing ssh-keygen at the terminal. You will be prompted for the file where you want to save the key. Just use the default (/home/username/.ssh/id_rsa). It will then ask for a passphrase, leave this empty.
Next, append the public key that you just created to the authorized keys file on the remote server. To do so, type:

cat ~/.ssh/id_rsa.pub | ssh @Machine.Location ‘cat >> .ssh/authorized_keys'

Now when you use the command from Step 4 above, it will automatically mount without prompting for a password.

Step 6: Put It All Together With Custom Scripts

Now that we have the connection string and have configured a passkey instead of a password, we are ready put all the pieces together in two custom scripts. The first will be used for automating the backups and the second will be used for opening the GUI. There are three things in particular that need to happen:

The computer mounts the remote drive to the appropriate point in the file system.
Either the user interface or automated backup routine runs.
The remote store is unmounted when the program finishes and exits.

Creating an Automatic Backup

Below is a template for a script that can be used to start a backup job:

sshfs @Machine.Location:/path/to/backup/folder /media/Backup
backintime -b
fusermount -u /media/Backup

The first line mounts the remote folder, the second launches the command line version of backintime and tells it to make a new snapshot, and the last line then unmounts the remote drive after Back In Time has finished. Save this file to someplace in your home file as “Data-Backup.sh”. I like to keep all of my scripts together in a folder imaginatively called “Applications.”

Anytime that you want to run an automated backup, you can do so by typing:

bash /path/to/folder/Backup-Data.sh

Launching the GUI

Now that we have a script for running the backup to the remote location, we are half way done. But as a backup is only as good as ability to access the data, we also need a second script. This will be used for launching the program, retrieving files, or making changes to the configuration. Again, we follow the same steps as the automatic backup script. Below is the template:

sshfs @Machine.Location:/media/backup/Backup/Linux-Backup /media/Backup
backintime-gnome
fusermount -u /media/Backup

The first line mounts the remote share, the second line launches the GUI version of Back In time, and the third unmounts the volume after you exit. Save this as “GUI-Back-In-Time.sh” in your scripts folder.

Step 7: Easy Access and Automation

With your two custom scripts, you have all of the tools needed to backup and restore files to your remote computer. But launching the terminal every time you need to do one of these jobs can be a bit of a pain. And because we are backing up to a remote location, we can’t use the scheduler built in to Back In Time. So, to make things easier, let’s:

Create a custom menu launcher for easy access
Automate the backup process using Gnome Schedule and the user crontab file.

Create a Custom Menu Launcher for Easy Access

To make it easier to access our custom script, let’s create a custom launcher by using the Applications->System Tools->Main Menu utility.

After the utility launches, click on the “System Tools” sub-menu and then push the “New Item” button. In the pop-up, specify the type of launcher as “Application” and the name as “Back In Time.” For the command, use:

bash /path/to/scripts/folder/GUI-Back-In-Time.sh

Add a description to remind you what Back In Time does and, if you want, you can customize the icon by clicking on the picture at left. When finished, press “Close.” This will add your custom launcher to the main menu.

Automated Backup: Cron Jobs and Gnome Schedule

After all that preliminary work, we’re finally able to automate our backup. We’ll accomplish this through the use of the Linux cron daemon. Cron is a program that allows for tasks to be executed at a certain time. You can create cron tasks to update a website, or automatically download files, or perform system maintainence tasks.

Cron tasks are specified in a file called the “crontab.” Each user has a crontab file. There is also a system crontab that runs under the root user.

While editing the cron file by hand is easy enough, we will be using a special GUI called “Gnome Schedule.” To install it, type the following command in the terminal:

sudo apt-get install gnome-schedule

After the installation finishes, you can launch the utility by going to Applications->System Tools->Scheduled Tasks.

To create a new scheduled task, click on the “New” button and then choose “Recurrent Task” from the drop down list. A recurrent task is run at a specified interval: every second, minute, hour, month, or year. If you want to be fancy, you can specify which hour of the day that you would like it execute.

For the command, use the following:

bash /path/to/scripts/folder/Data-Backup.sh

When finished, press “Add.”

Conclusion

Congratulations! Through the use of the sshfs kernel module, you've now configured Back In Time to more or less duplicate all of the behavior of Time Machine. We've fooled it into thinking that it's working on the local drive, when in reality that archive may be sitting across the network or across the world. And though the instructions provided here were specific to the SSH file system extension, extensions can also be found for Amazon Simple Storage Solutions (S3), WebDav, FTP and Samba. This makes it possible to backup your data to almost any service or destination imaginable. Even better, through the use of scripting, it is possible to automate the process so that it reaches “set it and forget it” simplicity.

Back In Time (Part 1): Linux Backup Made Easy

Rob Oakes — Tue, 21 Jul 2009 01:12:13 +0000

Linux is a bit of a funny creature. In many ways, it’s the most integrated and full featured operating system in existence. Take software development, for example. You couldn’t ask for a more integrated, well thought out solution. The tools are absolutely first rate, the system is stable, and because the platform is open source, most of the rough edges get worn off rather quickly.

But there’s also a downside to open source. You see, everyone tends to scratch their own itch first. Thus, while there are a bevy of tools that perfectly meet the needs of a few (most notably their authors), they often miss the needs of the many. That is a problem, because the masses don’t necessarily have the skill set required to adapt an already existing program so that it fits them. If anything, this is why Microsoft is valuable. They create software that more or less handles the needs of the masses.

And as might be expected, this is also the state of backup on Linux. It is no understatement to say that you can get your hands on the very best backup tools available, for free. All it requires is that you use some derivative of the following:

sudo apt-get install toolname

Unfortunately, while you can find the very best tools, that doesn’t necessarily mean that you will be able to use them. Rather that be written with the end-user in mind, they were created for programmers and system administrators. You know, the type of people who wonder why anyone would bother with a user interface. After all, it’s so much faster to do everything from the command line.

The result is that while the Linux backup tools may be powerful, they also have a learning curve steep enough to bang your head against. And that is an absolute shame. In general, I (and a great deal of humanity in general) am against activities that require me to learn new things or otherwise grow in what might end up being a painful direction. Even if it is for a good cause. (And I can think of few more important than making sure thatmy personal information is safe and secure.) Ideally, I want backup to be easy. I want to “Set it and forget it.” Time Machine for Mac OS X allows me to do this, as does the Windows Vista File Backup. But at first pass, such a solution for Linux is woefully absent.

That’s not to say that there aren’t contenders. In fact, a quick search on Google will reveal several: sbackup, Flyback, and TimeVault all look promising. But if you look at their project pages, you will notice something disconcerting: in each case, the programs are old and there isn’t much recent activity. This is typically a bad sign for an open source project. No activity usually means that the program is dead and unlikely to advance further. And that is really too bad, because sbackup, Flyback and TimeVault are good programs with potential. But, like many other open source options, they have a number of very rough edges.

Maybe that’s why I got so excited when I saw a new contender, called “Back In Time,” when it was recently featured on Lifehacker. At the time, I thought, “Have I finally found a backup program I can really use?” Now, after four months of using it full time, I can answer that question: Yes! Without a doubt!

Back In Time

Back in time is the full package. It includes a robust backend that makes incremental snapshots easy. You’ll also find a simple to use restoration system and a settings pane that actually make sense. What’s more, the defaults are actually useful! As a particularly nice touch, there are even separate packages for KDE and Gnome, so that the program will natively blend with whatever desktop you happen to have installed.

Installation

On Ubuntu, you can install Back In Time by downloading the packages for your Linux distribution. As noted above, you can find two different user interfaces available: one for KDE and another for Gnome. Don’t fret about downloading one over the other. They are identical in terms of functionality. (Note: The screenshots you see in this article were taken from the Gnome user interface.)

Configuration

Thanks to the well-thought out defaults, getting started with Back In Time is easy and straightforward. The first time you launch the program, you are prompted to configure the differenct backup options. (Should you ever want to change these, you can access the same pane by clicking on the “Settings” button.) In the configuration window, you will see several different tabs. Each tab corresponds to a different part of the backup job. There are the general settings which describe where you want your backups to be saved and how often you want the backup to run. And there are settings for what you want to include, exclude, and how many snapshots you want to keep. Back In Time can back up to any attached storage device, including a USB hard drive or flash drive. Unfortunately, however, it doesn’t work with network drives or with optical media.

The include tab lets you specify which folders you would like to have added to the backup. Since Back In Time runs in two different modes (user and root), I’ve been using each mode to accomplish a slightly different goal. The user mode is ideal for backing up files and settings, while the root mode is ideal for backing up the system itself (similar to the way in Time Machine works on Mac OS X).

By using the different modes in this way, I can keep user fuke backups are kept in one location and the system information in another. Thus, if I should lose my system, I can quickly and easily restore my system using the "System" backup and then copy back my files. I avoid duplicating data through the “Exclude” tab. In the root mode, I exclude the “home” directory, where all of the files and profiles are kept. But since I am backing up my home directory through the user mode, this doesn’t really matter.

Incremental backups are most valuable when run frequently. Time Machine creates snapshots every hour for more or less “maximum” protection. And as I learned recently (after accidently deleting my entire source code directory for LyX-Outline), hourly backups can be a tremendous lifesaver. After my "accident," I was able to restore the directory to the way it had looked just 15 or so minutes before. In doing so, I recovered a number of important changes that hadn’t yet been committed to Subversion. At that moment, I was tremendously grateful that Back In Time could be configured for hourly snapshots.

But over time, the side effect of hourly snapshots is a huge number of unnecessary backups clog the drive. After all, I don't need to know what my drive looked like during every hour of 2009. It's here that we see another fine example of Back In Time's overall polish. The auto-remove settings provide a way to quickly and efficiently deal with too many snapshots. Back in time can be configured to prune the number of backups to something that is slightly more reasonable.

Restoring Files

File restoration is dead simple as well. You can quickly browse through backed up files and folders using the restore pane of the GUI. If you need a version from an older snapshot, simply select it from the “snapshots” menu on the left. Once you have selected the files you are interested in, click “Restore.” They will be copied to their old locations.

Limitations

But even though Back In Time is probably the best user-oriented backup solution I’ve found for Linux, it still isn’t perfect. (Though really, really close.) There is at least one major limitation: lack of a network based backup.

As I’ve said before, I prefer to keep my backups on my home server. There are a number of reasons for this, but the most important is that I don’t need to remember to turn my backup drive on. When it’s time to automatically back up, it just happens. Unfortunately, Back In Time does not natively support network or remote backups. (Fortunanately it runs on Linux, the most customizable operating system in the world, and there happens to be a workaround.)

Conclusion

Back In Time is a wonderful piece of software. In fact, it’s the only backup program on Linux I’ve found that really meets the dead simple needs of the typical computer user. It is built on the solid foundation of rsync, allows for incremental backups, has a well though out GUI and defaults, and generally “Just Works © .”

Even its greatest weakness (the lack of a network based backup option) is more of an inconvenience than a truly show stopping fault, since the same thing can readily be accomplished using a Linux kernel extension and a simple script. In Part 2 of this article, I will show you how.

Apolitically Incorrect » Linux Backup

Upgrading from Time Drive 0.1

Reset Your Preferences

Beware of Spaces

Cron No Longer Works

Summary

Similar Posts:

Time Drive 0.2 – A “More S” Release: More Stable, More Secure, More Settings and Now Supports Amazon S3 Storage

More Stable and Secure

More Settings

Amazon S3 Storage

Conclusion

Similar Posts:

Backup for Linux, Done Right - Part 2: Time Drive 0.1

Over the Network or Across the World

Restoring Files

Set It, Then Forget It

Notifications and Feedback

Beta Software

Conclusion

Similar Posts:

Backup for Linux, Done Right- Part 1: A Mini Melodrama

Backgrounds and Backends

Basic Needs

Duplicity

Frontend and Features

Similar Posts:

Back In Time (Part 2): Over the Network and Across the World

Step 1: Prepare a Local Mount Point and Set Permissions

Step 2: Install the SSH File System (SSHFS)

Step 3: Get the IP Address or Url of the Remote Computer

SSH to a Machine on the Local Network

SSH to a Machine on the Internet

Step 4: Test the Connection

Step 5: Use a Passkey Instead of a Password

Step 6: Put It All Together With Custom Scripts

Creating an Automatic Backup

Launching the GUI

Step 7: Easy Access and Automation

Create a Custom Menu Launcher for Easy Access

Automated Backup: Cron Jobs and Gnome Schedule

Conclusion

Similar Posts:

Back In Time (Part 1): Linux Backup Made Easy

Back In Time

Installation

Configuration

Restoring Files

Limitations

Conclusion

Similar Posts: