Back In Time (Part 2): Over the Network and Across the World

 | July 20, 2009 9:15 pm

Staircase at the Vatican Museum Despite all of its strengths, Back In Time has one major weakness: it doesn’t support backup over a network.  But luckily, it runs on Linux,the single most customizable operating system in the world!  As a result even though Back In Time might not officially support a backup to a remote computer, you can still make it work through a little bit of spit and grit.

The magical sauce is the Fuse SSH file system.  This kernel module allows for Linux to mount a remote share as a local drive.  And by doing so, Back In Time thinks that it is working with a local folder even though the data might be sent across the network or internet.  To make it work, though, you’ll have to work at the command line and do a bit of programming.  But, don’t panic because it isn’t that hard.  It amounts to a few shell commands and about six lines of code.  Below, I’ll show you how in seven simple steps.

Step 1: Prepare a Local Mount Point and Set Permissions

First, we first need to create a mount point.  A mount point is a blank folder that the operating system uses to connect the local computer to the remote server.  I typically like to have my mount points in the /media folder.  If the mount point is in the /media folder, it will show a little icon on the desktop when active.

So, open up the terminal and type the following:

sudo mkdir /media/Backup

A remote server share mounted as a local drive using Samba FS.This command will create a new folder (Backup) in the media folder.  As is, we could then mount the SSH file system to this mount point, but to do so would require that we run the command as the super user (root).  It’s not necessarily good to run things as root.  So, let’s change the folder’s owner and permissions so that we can mount it without requiring the super user:

sudo chown –R <Username> /media/Backup
sudo chmod a+x /media/Backup

In the first command above, replace <Username> with the username that you use to log-on to Linux.  In my case, it would be: roakes.  The second command properly sets the permissions so when it comes time to mount the remote share, the computer won’t prompt for the root password.

Step 2: Install the SSH File System (SSHFS)

Now that we have a valid mount point, let’s install the software packages for SSHFS.  If using Ubuntu, type the following into your system terminal:

sudo apt-get install sshfs

Step 3: Get the IP Address or Url of the Remote Computer

With the mount point set and software installed, we need to do a bit of planning.  “Proper preparation prevents pitifully poor performance” and all that.  First, we need to know where the data will be stored.  Will we be using a simple home server or will we be storing that data on a remote server over the internet?  Also, will you be able to connect to the server using the secure shell (SSH) protocol?  If so, what is the destination folder?

We will use this information to construct a connection string.  It will have the form:

<Username>@Machine.Location:/path/to/backup/folder

SSH to a Machine on the Local Network

If we will be backing up to a computer on the local network, we need to know it’s IP address so that we can establish an SSH connection.  To find this, go to the server and log-on.  Then type ifconfig into the terminal window.  This will return a block of text similar to that seen below, which was copied from my server.  I have added a series of x to some of the information.

eth0 Link encap:Ethernet HWaddr 00:xx:xx:xx:e0:93
inet addr:xxx.xxx.1.106 Bcast:xxx.xxx.1.255 Mask:255.255.255.0
inet6 addr: fe80::xxx:xxxx:xxxx:xxxx/64 Scope:Link
UP BROADCAST RUNNING MULTICAST xxx:1500 Metric:1
RX packets:26460698 errors:23 dropped:145 overruns:17 frame:0
TX packets:14444204 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2320464437 (2.1 GB) TX bytes:2079312151 (1.9 GB)
Interrupt:3 Base address:0xd800

lo Link encap:Local Loopback
inet addr:127.0.0.x Mask:255.0.0.0
inet6 addr: ::x/xxx Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:9308 errors:0 dropped:0 overruns:0 frame:0
TX packets:9308 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4232264 (4.0 MB) TX bytes:4232264 (4.0 MB)

The IP address can be found on the second line below eth0, which starts : inet addr.  In my case, the IP address is: xxx.xxx.1.106.  The xxx.xxx will have numeric values that are specific to your network.  To connect to the machine, I use a connection string that looks like:

<Username>@xxx.xxx.1.106:/path/to/backup/folder

SSH to a Machine on the Internet

If you will be connecting to another computer on the internet, then instead of using an IP address to connect, you will instead be using a url.  You can get this information from the information page of your backup service, or from the network system administrator.  To connect to the remote machine, you will use a connection string that looks like:

<Username>@machine.url:/path/to/backup/folder

To remotely access my outside of the network computer, I use:

roakes@remote.oak-tree.us:/media/Linux-Backup

Step 4: Test the Connection

Once I have the connection string, I can now test SSHFS and make sure that it can mount to my mount point (Step 1).  To mount the remote drive, type:

sshfs <Username>@Machine.Location:/path/to/backup/folder /media/Backup

Be sure to replace the part in italics with your connection string.  The first time that you connect, you will be prompted to accept the server’s key and be asked for your password.  After you enter your user credentials, it should mount the file system to /media/Backup.  Open up the location in finder and make sure that you can create folders and move files onto the remote machine.

You can unmount the volume by typing (no root required):

fusermount –u /media/Backup

Step 5: Use a Passkey Instead of a Password

While a password is acceptable when you want to connect manually, we want the backup script to run automatically.  That means no password prompt.

Luckily, however, it’s easy to configure Linux so that it uses a passkey instead of a password.  This will allow your computer to connect automatically instead of prompting you for your user credentials each time that you want to run a backup or restore a file.  To do so:

  1. Generate a key on your computer by typing ssh-keygen at the terminal.  You will be prompted for the file where you want to save the key.  Just use the default (/home/username/.ssh/id_rsa).  It will then ask for a passphrase, leave this empty.
  2. Next, append the public key that you just created to the authorized keys file on the remote server.  To do so, type:
  3. cat ~/.ssh/id_rsa.pub | ssh <Username>@Machine.Location ‘cat >> .ssh/authorized_keys'

Now when you use the command from Step 4 above, it will automatically mount without prompting for a password.

Step 6: Put It All Together With Custom Scripts

Now that we have the connection string and have configured a passkey instead of a password, we are ready put all the pieces together in two custom scripts.  The first will be used for automating the backups and the second will be used for opening the GUI.  There are three things in particular that need to happen:

  1. The computer mounts the remote drive to the appropriate point in the file system.
  2. Either the user interface or automated backup routine runs.
  3. The remote store is unmounted when the program finishes and exits.

Creating an Automatic Backup

Below is a template for a script that can be used to start a backup job:

sshfs <Username>@Machine.Location:/path/to/backup/folder /media/Backup
backintime -b
fusermount -u /media/Backup

The first line mounts the remote folder, the second launches the command line version of backintime and tells it to make a new snapshot, and the last line then unmounts the remote drive after Back In Time has finished.  Save this file to someplace in your home file as “Data-Backup.sh”.  I like to keep all of my scripts together in a folder imaginatively called “Applications.”

Anytime that you want to run an automated backup, you can do so by typing:

bash /path/to/folder/Backup-Data.sh

Launching the GUI

Now that we have a script for running the backup to the remote location, we are half way done.  But as a backup is only as good as ability to access the data, we also need a second script.  This will be used for launching the program, retrieving files, or making changes to the configuration.  Again, we follow the same steps as the automatic backup script.  Below is the template:

sshfs <Username>@Machine.Location:/media/backup/Backup/Linux-Backup /media/Backup
backintime-gnome
fusermount -u /media/Backup

The first line mounts the remote share, the second line launches the GUI version of Back In time, and the third unmounts the volume after you exit.  Save this as “GUI-Back-In-Time.sh” in your scripts folder.

Step 7: Easy Access and Automation

With your two custom scripts, you have all of the tools needed to backup and restore files to your remote computer.  But launching the terminal every time you need to do one of these jobs can be a bit of a pain.  And because we are backing up to a remote location, we can’t use the scheduler built in to Back In Time.  So, to make things easier, let’s:

  1. Create a custom menu launcher for easy access
  2. Automate the backup process using Gnome Schedule and the user crontab file.

Create a Custom Menu Launcher for Easy Access

To make it easier to access our custom script, let’s create a custom launcher by using the Applications->System Tools->Main Menu utility.

To add a custom launcher to our user script, launch the "Main Menu" utility.

After the utility launches, click on the “System Tools” sub-menu and then push the “New Item” button.  In the pop-up, specify the type of launcher as “Application” and the name as “Back In Time.”  For the command, use:

bash /path/to/scripts/folder/GUI-Back-In-Time.sh

Add a description to remind you what Back In Time does and, if you want, you can  customize the icon by clicking on the picture at left.  When finished, press “Close.”  This will add your custom launcher to the main menu.

To create a custom launcher, press the new item button.

Automated Backup: Cron Jobs and Gnome Schedule

Gnome System Scheduler - Automated BackupAfter all that preliminary work, we’re finally able to automate our backup.  We’ll accomplish this through the use of the Linux cron daemon.  Cron is a program that allows for tasks to be executed at a certain time.  You can create cron tasks to update a website, or automatically download files, or perform system maintainence tasks.

Cron tasks are specified in a file called the “crontab.”  Each user has a crontab file.  There is also a system crontab that runs under the root user.

While editing the cron file by hand is easy enough, we will be using a special GUI called “Gnome Schedule.”  To install it, type the following command in the terminal:

sudo apt-get install gnome-schedule

After the installation finishes, you can launch the utility by going to Applications->System Tools->Scheduled Tasks.

To create a new scheduled task, click on the “New” button and then choose “Recurrent Task” from the drop down list.  A recurrent task is run at a specified interval: every second, minute, hour, month, or year.  If you want to be fancy, you can specify which hour of the day that you would like it execute.

For the command, use the following:

bash /path/to/scripts/folder/Data-Backup.sh

Gnome Schedule makes it easy to automatically run the custom backup script.

When finished, press “Add.”

Conclusion

Congratulations!  Through the use of the sshfs kernel module, you've now configured Back In Time to more or less duplicate all of the behavior of Time Machine.  We've fooled it into thinking that it's working on the local drive, when in reality that archive may be sitting across the network or across the world.  And though the instructions provided here were specific to the SSH file system extension, extensions can also be found for Amazon Simple Storage Solutions (S3), WebDav, FTP and Samba.  This makes it possible to backup your data to almost any service or destination imaginable.  Even better, through the use of scripting, it is possible to automate the process so that it reaches “set it and forget it” simplicity.

Similar Posts:

6 Responses to “Back In Time (Part 2): Over the Network and Across the World”

[...] and Photography « Why Bother With a Personal Website? Back In Time (Part 2): Over the Network and Across the World [...]

Rob wrote a comment on September 5, 2009

Good article. The one thing I'm unsure about is the use of a virtual file system (sshfs). As far as I know hard linking isn't possible. Would this not remove one of the main benefits of using BackInTime?

Rob Oakes wrote a comment on September 6, 2009

@Rob. Unfortunately, yes. In the past few months or so, I've just found that the hack described in the BackInTime article mostly works, rather than being a truly reliable solution. This is one of the big reasons that I've begun writing my own backup utility, called Time Drive. The idea is to approximate the BackInTime feature set as closely as possible. You can find more information at:

http://www.oak-tree.us/blog/index.php/science-and-technology/time-drive

Rob wrote a comment on September 7, 2009

Yeah found your app after reading the article. Very cool. Especially to get it on LifeHacker.

How did you get around the problem I mentioned in your app? Access SSH directly?

I've been playing with Ruby and written my own little script. Things I came across:

1) I looked at using rsync to do all of the linking etc... for me so as to avoid SSHFS. It has a parameter like --links or --hard-links etc... and also --backup. I moved away from that though (couldn't be bothered with the experimentation :) ).

2) In the end I did a mixed approach. I used your suggestion of SSHFS. This was so that Ruby could create any necessary directories, and also delete any older ones as part of the cleanup my script does. This was easier than using the net-ssh lib which I was having problems with. I then use rysnc -e "ssh" to sync over normal ssh, and also an ssh command to do the "cp" hard linking.

Not an ideal solution as it's a bit mixed but it's working for me. I may well take a look at your app though if it's pure SSH. I should really try and get the net-ssh lib in Ruby working then all my problems would be solved, but given I've got it working I'm a little bored/tired of it for now. I should really have done some upfront unit tests for it too.

Is your app written in Python?

Rob Oakes wrote a comment on September 7, 2009

@ Rob. Thanks! I thought that being featured by Lifehacker was cool too. Just shows how geeky I am, but it's always been a bit of a goal. The program is still really new (0.1.x), so it has a number of rough edges. But I'm trying to get those filed off as quickly as possible.

"How did you get around the problem I mentioned in your app? Access SSH directly?"

Currently, Time Drive is just a front end to Duplicity, so it uses it to handle all of the actual backup. But to answer your question, you are correct, it uses ssh directly and avoids hard links. Rather than create a hard line like the cp/rsync strategy of BackInTime, instead duplicity creates a compressed/encrypted tar file archive. Subsequent backups are then added to the directory as encrypted diffs of the changed files. It then retrieves information about the backup by parsing the file signatures.

The result of this setup is that you can't browse the archive directly, but need to use either duplicity or a utility like Time-Drive to parse it. While I prefer this sort of scheme (since I use FTP as a redundant backup and I am not about to post unencrypted files on even a secure FTP site), I know that there are many others who do not.

"Is your app written in python?"

Yes, both Time-Drive and Duplicity are pure python (with some PyQt thrown in for the GUI).