Backup, Share and Sync – Part 1: Introduction

 | January 22, 2009 6:12 pm

On any given day, I engage in a lot of different activities which can generate a great deal of digital stuff.  The digital stuff might include notes, pictures, drawings, drafts, scientific data, or dozens of other little things.  What’s more, the random bits often contain very important information about both my digital and often real life.  About two years ago, though, I had a disaster.  My main computer hard drive failed and with it went all of my data.  I lost vacation pictures of my family, my entire digital music collection, drafts of two major papers, as well as the only copy of a book I was writing.

To say that I was completely devastated would be an understatement.  It took me nearly six months to repeat much of the work that was lost.  I was never able to recover either my pictures or the music, and I’ve never restarted work on that book.  Shortly after that crash, I decided to get serious about keeping a good backup.  Rebuilding a digital life from scratch is something I never want to do again; it was extremely painful.

To avoid another disaster, I studied out all of the different ways that I could keep a redundant set of backups that would ensure that all of my digital information would be available to me wherever I happened to be.  After a little bit of research and a great deal of tinkering, I put together a system that works very well for my needs.  Using my little system, I am able to keep my computers in sync, backed up, and access files that I need over the network and internet.  My system marries both software and online services and helps me be more productive and worry less about losing important information.  Coincidentally, every service and piece of software is absolutely free to use (and many are OpenSource), so the barrier to entry is pretty low.

In this series of articles, I will describe what software and hardware I use to backup, share and sync my files.  I will provide instructions on what materials you will need, how to setup the software and hardware, and how to configure the services on your various devices.  Lastly, since the setup can be somewhat complicated, I will try and provide tips that might help you avoid problems and snags that I discovered along the way.

In this article, we will quickly overview the setup as well as describe how to prepare a simple file server and install the core software that makes digital nirvana possible.  Later articles in this series will look at software configuration and setup on both the server as well as on the clients.

The Big Picture

Server Software

At the center of my configuration is a simple Linux file server (based on Ubuntu) which runs Subversion and Samba.  Keeping subversion repositories allows me to work on source code, documents, or just about anything that happens to be text based from a variety of computers.  I can make changes and then commit those changes back to a central storage.  It’s a pretty cool system for a writer too (more details on that in a later post).  Samba is the piece of software that lets my Linux server talk with my Mac and PC.  Think of it as the glue that binds the whole configuration together.  Below is a more technical summary.

  • Subversion - Subversion is an open source version control system. Using subversion, it is possible to record the history of source files (computer code) and documents. It manages the files and directories over time. A tree of files is placed into a central repository which acts much like an ordinary file server, except that it remembers every change ever made to its files and directories.
  • Samba - Samba is a free software re-implementation of the SMB/CIFS networking protocol, originally developed by Australian Andrew Tridgell. It is capable of providing file and print services for Microsoft Windows clients and it is capable of integrating with a Windows Server Domain. Samba is standard on Apple's Mac OS X Server, all versions of Linux, and it is commonly used as a basic system service on other Unix based operating systems as well. Using Samba, it is possible to set up a shared network drive which may then be used for backup or file sharing.

Online Services

In addition to Subversion and Samba, I also make use of several important online services.  These help to synchronize my work amongst the many computers I might be using and ensure that I have an online, redundant backup system.  For syncing, I use WIndows Live Mesh.  For offsite backup, I use a Linux utility called Duplicity for and Amazon S3 for storage.

Client Software

While i use several programs on the client side, these aren’t nearly so important as the software on the server.  For accessing the subversion repositories, I use TortoiseSVN.  For backup, I use Time Machine on the Mac and the built-in backup client on Windows Vista.  On Linux, I use rsync.  While there are certainly other technologies which are more configurable, these have worked very well for my needs.  In later articles, we will be looking at the desktop software in detail.

Materials : What you will need

Computer Hardware – Server

While a subversion and file server can be created from nearly any piece computing equipment, the amount of horsepower necessary is fairly minimal. You might a have large number of old computer parts lying around the house (I know I do), it might be good to scavenge some of those parts and put them to a good use. You could also find a cheap PC online or through a local reseller.  Many businesses are often willing to part with their old PCs for next to nothing as well.  At the bare minimum, you need a computer that can turn on and boot.  Linux does an excellent job with older hardware drivers.

Since I happened to have an old PC lying around, this tutorial will explain how to use that PC with two attached USB hard disks.  I suppose at this point that I should throw in a disclaimer.  The setup described here is intended for home use only. While the software that we are going to install runs the server farms of the biggest corporations in the world, the hardware (that I’m using) probably wouldn't get you nearly so far.

USB Hard Drives

I will be using USB external drives (so that they can easily be moved from one computer to another), and will be formatting them with the NTFS file format (the one most commonly used by Microsoft Windows). Should I need or want to move the hard disk into a new computer, they can be read by most Windows and Linux machines. Mac OS X even has read support for NTFS drives.  Also, should the primary hard drive of the machine fail, the data on the USB drives will be left intact.

While there are other formats which can be read by Linux, Mac and Windows (most notably the Fat16 or Fat32 format), I would recommend against using them. Configuring the read and write permissions for these volumes can be very frustrating. In contrast, I had no problems with the NTFS formatted drives.

Server Operating System

The base operating system of the server will Ubuntu Server. While just about any flavor of Linux will do, I chose Ubuntu for two important reasons:

  1. Familiarity. I have used their Desktop Edition (www.ubuntu.com) extensively and I like it.
  2. Ease of Administration. Ubuntu Server makes use of a package management utility (a program that allows additional software to be downloaded, installed [and later uninstalled] with great ease) called apt-get. Apt-get is based on the Debian (.deb) standard, and as a result, handles dependencies internally. I have previously used other distributions like SuSE or RedHat which make use of other package management systems that lack the ease and elegance of apt-get.

Setting Up the Server

Step 1: The base system

To install the system, you will need the following:

  1. The Ubuntu Server CD, which you can obtain from the Ubuntu Website.  When you are prompted to select the version of the software, make sure that you choose Ubuntu Server 8.04 rather than the newer 8.10.  The newer version includes a version of Samba that doesn’t work as well with Windows and Mac than the older version.
  2. An internet connection
  3. The PC which you intend to use as the server and additional hard disks. I used an old Dell tower and an two separate, one terabyte external hard drive. This configuration offers me a few tangible benefits.
    • I will not be mirroring (RAID) the system. As a result, if the main computer dies, the data should remain untouched on the external drive.
    • It might be necessary to backup other computer files to the large drive. The ability to power-off the server and disconnect the drive is a large benefit.

For right now, leave the USB hard drives detached from the server.  We will add and configure them after the basic operating system has been installed.

Once you have all of the files, it is time to get started. Insert the Discs into the drive and turn the system on. The onscreen wizard will help you set your computers name, create an initial user, and determine which programs you would like to run on your server.

When you are prompted to select the software packages you would like to include on the server, choose the following:

  1. LAMP Server
  2. OpenSSH server
  3. Samba File server

clip_image002

Each of these software packages will be important to getting the server correctly configured.

The LAMP server contains the basic functionality which Subversion will use for http access. OpenSSH allows for you to remotely access the server, simplifying the rest of the steps in this guide. Samba serves as the basis of the file sharing and backup server.

After you have selected the necessary software, hit the enter key. The installation process will then be completed and the base software packages will be installed and configured to their default settings.

Step 2: Set up a secure shell connection (SSH)

After the installation has finished and the computer has been restarted, you will be able to begin the setup necessary for your file share and subversion (SVN). Rather than having to stand at the server and type the commands in manually (which works just fine), I like to open up a terminal connection (called a secure shell or SSH) to the computer. This allows me to finish the process from my desktop computer.

In order to set up the shell, you will need to know the server's IP address on your home network. You can do this by typing ifcongif into the terminal at the server console. Ifconfig will return something similar to what you see below, which I copied from my server. I have added a series of x to some of the information.

eth0 Link encap:Ethernet HWaddr 00:xx:xx:xx:e0:93
inet addr:xxx.xxx.1.106 Bcast:xxx.xxx.1.255 Mask:255.255.255.0
inet6 addr: fe80::xxx:xxxx:xxxx:xxxx/64 Scope:Link
UP BROADCAST RUNNING MULTICAST xxx:1500 Metric:1
RX packets:26460698 errors:23 dropped:145 overruns:17 frame:0
TX packets:14444204 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2320464437 (2.1 GB) TX bytes:2079312151 (1.9 GB)
Interrupt:3 Base address:0xd800

lo Link encap:Local Loopback
inet addr:127.0.0.x Mask:255.0.0.0
inet6 addr: ::x/xxx Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:9308 errors:0 dropped:0 overruns:0 frame:0
TX packets:9308 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4232264 (4.0 MB) TX bytes:4232264 (4.0 MB)

The IP address can be found on the second line below eth0, which starts : inet addr. In my case, my IP address is:

xxx.xxx.1.106

The xxx.xxx will have numeric values that are specific to your network.

At your computer, open a terminal session (on Mac OS X) or a Console if using Linux. If you are using Windows, you do not have an SSH built in and will need to download a program called PuTTY from the internet. You can find the download files here.

Create a secure shell to the server. This can be done by typing in:

ssh YourUserName@xxx.xxx.1.106

Where YourUsername is the username that you set up previously and xxx.xxx.1.106 is the IP address of the server. If you have set everything up correctly, you will now be connected to the server and should see a window similar to that shown below.

clip_image002[6]

Getting Access to the Root Username

Once you have managed to successfully connect to the server via ssh, you are ready to begin configuring the software. Much of the configuration needs to be done as the administrator or root. By default, Ubuntu does not enable the root account. Instead, they encourage that you run the commands by invoking the command:

sudo command

Unfortunately, this will only work for some of the most basic administrative tasks. Other tasks, such as mounting and unmounting disks require that you be the root user. There is a convenient workaround. Type:

sudo bash

You will be prompted to enter your account password, after which you will be given root access. Now you can do what you need. When you no longer need root access, just type exit and you will be taken back to the standard log-on.

Step 3: Software Installation

Once you are connected to the server, it's necessary to download some additional software and edit configuration files. Configuration files in Linux are simple text documents, as a result, they can be edited using any text editor.

3.1 : Choosing Your Text Editor

In a stock Ubuntu server installation, there is no desktop environment, and as a result, a graphical text editor is not included. You will need to use a command line editor line editor like Vi or Emacs. Vi is included in the standard Ubuntu installation. If you would like to use emacs, you will need to download it. I personally prefer to use vi.

Because it is a command line editor, take a minute and learn a few of the navigation commands. While it might appear extremely intimidating at first glance, most of them are fairly intuitive, with perhaps a few exceptions. Here are few things to keep in mind when using vi.

  1. There are two discrete modes: text input and navigation. In text navigation mode, you can use the arrow or shortcut keys to move around the screen. When in text input mode, the arrow keys no longer work for navigation.
  2. You enter text input mode by typing i (insert) or o (add a new line of text).
  3. You exit text input mode by hitting escape.
  4. You can save and exit the program by typing the semicolon character and then x (:x). Many other important functions are accessed in the same way.

Here are some of the commands which I use most frequently:

Command

Action

Arrow Keys

Navigation

i

Text Input Mode

o

Add a new line of text and enter input mode.

Esc

Exit text input mode

: x

Save File and Exit Program

Control - F

Page Down

To edit the configuration files, you will need to be logged on as the root user.

3.2 : Download and Install Packages

In addition to the packages that have already been set up, you will need to download some additional software. These include the packages necessary for subversion, the subversion http software, and a backup program so that you can create a redundant, offsite backup.

To install the software necessary for subversion, type:

sudo apt-get install subversion libapache2-svn

For the off-site backup program, we will be using a utility called Duplicity, it can be installed by typing:

sudo apt-get install duplicity

Conclusion

After you have finished downloading the additional packages, all the software needed has been installed.  In the next article in this series, the USB hard disks will be added and I will show to configure them.  I will also show how to create a Samba share that can be accessed by any computer on the network.

Reference Documents

  1. Create a Simple File Server (Based on Ubuntu) - http://www.howtoforge.com/ubuntu-home-fileserver
  2. Ubuntu - Community Documentation - http://help.ubuntu.com/community/Subversion

Similar Posts:

11 Responses to “Backup, Share and Sync – Part 1: Introduction”

[...] Art and Photography « Backup, Share and Sync – Part 1: Introduction [...]

[...] (all 250 GB of it), reformat, and then copy the data back.

Apolitically Incorrect » A Pitiful and Pathetic Story sent a pingback on February 11, 2009

[...] … If I wanted to be productive, I probably could be.

[...] machine, this isn’t necessarily the best or most efficient way to work.

[...] the introduction of Samba, we have taken another step towards computer backup, sharing and sync bliss.

Apolitically Incorrect » Backup and Archive sent a pingback on March 1, 2009

[...] the estimated lifespan is between five and seven years.

[...] In this article, I will look briefly at how to setup Time Machine so that it works with a simple home server running Samba and [...]

[...] While the instructions here assume that you will be making your backup to a simple home server running Samba and Subversion, they will work with most network attached storage devices [...]

[...] [...]

[...] Older versions of the document will typically be backed up in the subversion repository on my server, or in a dedicated [...]