Apolitically Incorrect » Programming

Time Drive 0.3: Better, Easier, More Refined

Rob Oakes — Mon, 26 Oct 2009 16:16:56 +0000

Imagine how awesome it would be if this announcement read: “Time Drive has been completely rewritten from scratch (yet again) to take better advantage of the paradigms of modern computing! Version 0.3 has hundreds of updates and new features which will make your life easier and more fulfilled!”

There's just one little problem … such a hyper inflated announcement wouldn't necessarily be true. (Marketing hyperbole, I never knew thee!) The truth is this: Time Drive is a simple backup program that does a good job of reliably backing up your data. It offers a nice list of potential backup options: from an attached hard drive, to a computer over the network, or across the internet. It makes it easy to search for and restore a lost file. In short, Time Drive seeks to change the world by making an act of computer maintenance more convenient.

But the real test of a program isn’t how well it works, but how easy it is to fix when broken. A good program does what you want, but a better program helps you get back on track when things go wrong. Back when I was looking at other backup programs available for Linux, this was my number one frustration. Most of the applications would work (for the most part), but I could never troubleshoot or repair problems when they happened. There just wasn’t enough information available.

For an example, let’s take SBackup. It’s a lovely little program, except you have no way of knowing if it is working. It doesn’t keep log files, it doesn’t notify you if a backup job failed. It doesn’t let you know if it is running. Its simplicity is actually symptomatic of a flaw: it’s incomplete.

These were problems that I desperately wanted to avoid with Time Drive. And version 0.3 includes a number of refinements that solve these issues while at the same time making make it better, easier and more refined. In the rest of this post, I’ll explain why.

Better

There’s a very good reason why Time Drive works with the Ubuntu notifications system and has progress bar notifications for nearly everything. From the very first, I wanted the program to notify the user of what it was doing. But in versions 0.1 and 0.2, there was an essential part missing: some kind of program log. It was a big omission, and several people complained rather vocally about it.

Logs and Notifications

And it's the first thing that was added to version 0.3. Time Drive now logs its activities and will notify you of errors that happen behind the scenes. If a backup job fails silently, you can put on your detective hat and go find out why. If your internet connection cuts out while using cron to automatically backup your computer, you’ll get a warning the next time that you run the program. Backup statistics, problems, errors, even love notes; they’re all there. To open the log, just click on the blue information button on the main user interface.

Progress Indicators for Absolutely Everything

In addition to the log, we fixed a second oversight. Versions 0.1 and 0.2 lacked an indicator that told you how far a backup job had progressed. In 0.3, that problem’s been solved.

Easier

Great. Time Drive does a better job of letting you know about problems, but what about fixing them? For example, what will version 0.3 do when a backup job fails, or your internet connection cuts out at the wrong time? When these things happen, Duplicity leaves unneeded files on the backup drive. (Files, by the way, which are a pain to remove.) How does Time Drive make repair jobs easier?

Well … I’m glad that you asked. Time Drive now has a cleanup function. If it detects a mess or unneeded files, the program will automatically go about restoring things to right. And if you happen to be one of those obsessively tidy people, you can manually run the same tool by going to the advanced settings pane for the corresponding folder.

More Refined

So much for better notifications and easier management of messes. But what about the promised refinements? Where can you look to find those?

For starters, take a look at the settings for remote connections. All of the input boxes have been reworked for cleaner input. While Time Drive and Duplicity have always supported non-standard port numbers, it wasn’t necessarily clear how how to configure them. Now, it is.

More importantly, though, the user interface adapts to your choices. If you choose to make a backup to an attached drive, you aren’t going to be asked for a server and port. If you choose FTP, however, those options will be there. Best of all, you no longer have to keep track of which types of connections require two slashes in the connection url and which one uses just one. Time Drive will do the configuration and accounting for you!

Looking Toward the Future

And those are only the changes that you will see. There’s a lot more that has happened behind the scenes to make version 0.3 the best release of Time Drive ever. You might call these “future enhancements.” They include:

International Support. If you don’t speak English, Time Drive now has you covered. Dutch, French and Spanish translations are in the works. Want to see Time Drive translated to Chinese? Come help out!
Documented API. You know that a software program has grown up and reached maturity when you have a documented API.
Automated Installation and Upgrades for Linux Users. One of the best things about using Linux is apt-get. It keeps your entire system up to date with the most recent software, and the Time Drive project at Launchpad now has it’s own PPA.

Additional Announcements

As you can see, Time Drive is managing to evolve and add new features at a pretty good clip. But as we do so, it’s also time to face a pretty harsh reality. An easy to install and compatible Windows version simply isn’t going to happen.

Producing a version of Time Drive for Windows has always been extraordinarily hard, there are just too many *nix dependencies. To use it requires Cygwin, a compatible version of PyQt and a great deal of black magic and prayer. Changes that are compatible with Mac OS X and Linux break the Windows version, and vice versa. After months of wrestling with it, I’ve decided that It’s just not worth the headache. So, from version 0.3 onward, we will not be targeting Windows as a development platform.

For those of you waiting with baited breath for a Windows release, my apologies. But I should point out that Time Drive isn’t really needed on Windows. There are a number of fantastic backup programs available. The built-in backup is actually quite good in, and if that doesn’t suit your fancy there are some really fantastic commercial alternatives.

Conclusion

In summary, version 0.3 is a nice, incremental release of Time Drive. It patches some important holes and does a much better job of notifying users of what’s going on. It’s easier to manage your backups and clean up disasters. Lastly, it’s more refined with a bunch of spectacular back end improvements. So while version 0.3 may be an incremental improvement, we’ve got a foundation that will let us release even better versions in the future. This announcement might not be filled with marketing speak and exaggerated claims, but you can bet that future announcements will do a used car salesman proud.

Installing Duplicity on Mac OS X

Rob Oakes — Wed, 07 Oct 2009 19:20:40 +0000

While Mac OS X shares many things in common with other Unix operating systems, it also has a couple of missing parts. This can make installing software more difficult when compared to Linux or other *nix variants, and unfortunately, Duplicity happens to be one of those cases.

But just because it is more difficult to install Duplicity on Mac doesn’t mean that it isn’t worthwhile. Duplicity is one of the best command line backup programs available anywhere. Using the same program, you can backup to a local hard drive, FTP server, over SSH or even to Amazon S3. It uses the rsync algorithm, which means that backups happen quickly and only the parts of files that actually changed get copied. Subsequently, backups are smaller.

Nor does it mean that installing Duplicity is actually all that hard, it’s just a bit tedious. Like many other *nix programs, Duplicity requires a number of additional programs (called dependencies) that you will have to search out and install separately. To make that process easier, this article explains how. Below, you’ll find links to the download pages and instructions for how to compile each of Duplicity’s dependencies on Mac OS X. The instruction set has been tested both on Mac OS X 10.5 (Leopard) and Mac OS X 10.6 (Snow Leopard).

XCode

The first dependency you will need is a rather large one. Several of the software packages need to be compiled from source. But unfortunately, Mac OS X doesn’t install the compilers by default. Thus, you will either need to install them from the Mac OS X installation CD, or by getting the XCode tools from the Apple Developer Connection (ADC). Note: ADC requires that you have an Apple ID – which is free – to log in into the site.

After you log-in, download the XCode developer tools package. Once XCode has finished downloading, run the package installer. It will ask you to select the destination drive and what components of the software you would like to install.

Choose the Develoepr Tools Essentials, System Tools and Unix Development Support options. This will automatically install and configure the GNU tool chain and the GCC compiler collection. Once it has finished, you can access the tools by going to the “/Developer/” folder on the root of the drive.

After XCode is finished installing you are ready to build and install the other necessary dependencies.

Note: All other steps in this tutorial will be run from the command line terminal.

GnuPG

GnuPG handles the encryption and decryption of tar archive files. If it isn’t installed and configured correctly, Duplicity (and by extension) Time Drive simply won’t work. On Linux and other Unix operating systems, GnuPG is typically included as part of the default installation. This is not true on Mac OS X, however. As a result, you will need to download, compile and install it from source.

You can find the source files on the project download page or the associated ftp site. Duplicity will work with version GnuPG version 1.4 or 2.0. The biggest difference between the two is that version 2.0 is slightly more modular, and as a result, has a speed advantage. From a practical standpoint, however, unless you are running large, frequent backups, you are not likely to notice any difference. I have been able to compile both without any difficulty on both Leopard and Snow Leopard, so the choice is largely one of preference.

After downloading the most recent version, extract the archive by typing:

tar –xf gnupg-x.x.x.tar.gz2

Where x.x.x is the version of GnuPG that you downloaded (at the time of this writing, it is version 2.0.12). This will create a new directory called gnupg-2.0.12. From that directory, you can configure, compile, test and install:

cd gnupg-2.0.12
./configure
make; sudo make install

When finished, return to the working directory by typing:

cd ..

GnuPG Interface – Python Module

Now that you have built and installed GnuPG, you will next need to install the Python bindings so that duplicity can make use it. You can get these from the GnuPG Interface Module project page, specifically from the download section. Grab the most recent version (0.3.2). To install, extract the folder, enter the setup directory and then use python to run the “setup.py” script:

tar -xf GnuPGInterface-0.3.2.tar.gz
cd GnuPGInterface-0.3.2
sudo python setup.py install

NcFTP

NcFTP enables Duplicity to make backups to FTP servers. The download files can be found on NcFTP’s download page, which lists both the NcFTPd Server and the NcFTP client. We want the most recent version of the client source code (version 3.2.3).

Like GnuPG, we need to configure, compile and install:

tar –xf ncftp-3.2.3-src.tar.bz2
cd ncftp-3.2.3
./configure
make; sudo make install

Boto – Amazon S3 Support

To use Amazon as a backup destination, you need to grab the Boto python module. Boto provides a structured and integrated interface to most of the features offered by Amazon Web services, most importantly: the Simple Storage Service (S3). You can get the most recent version (1.8d) from the Boto downloads page on Google Code.

From there, the installation procedure is similar to that for the GnuPG python interface module. Extract the archive, enter the program folder and use python to run the setup script.

tar -xf boto-1.8d.tar.gz
cd boto-1.8d
sudo python setup.py install

Duplicity

Once you have installed GnuPG, the GnuPG python interface, NcFTP and Boto, you are ready to install Duplicity. Because it is a python program, the steps are identical to those of Boto and the GnuPG python interface. Download the source code from the project home page (keeping in mind that Time Drive requires at least version 0.6.04); then, extract it the same way that you did the previous libraries and enter the source code directory:

tar –xf duplicity-0.6.04.tar.gz
cd duplicity-0.6.04

Finally, run the python setup script:

sudo python setup.py install

Duplicity will byte compile and be copied to the python packages directory.

When finished, you can test the installatin by typing:

duplicity –help

If successful, you will see the text from the duplicity manual page.

Install Time Drive On Ubuntu and Other Debian Linux Distributions

Rob Oakes — Mon, 05 Oct 2009 22:23:01 +0000

Since version 0.2.2 of Time Drive, installing Time Drive and keeping it up date has gotten a great deal easier. Instead of requiring that you install from source, we now offer Debian based packages through the Time Drive Personal Package Archvie (PPA). This article will briefly describe how to install Time Drive using that PPA.

Background

But before leaping headlong into the nitty gritty, let me provide a bit of background about packages and repositories. Unlike Windows or Mac OS X, software on Linux is organized into a container called a package. The package contains the files needed to run the program in addition to artwork and configuration information. Any given program may require many other programs to be installed before it can run. These other programs are referred to as dependencies. Ubuntu and uses a tool for managing packages and dependencies called Apt.

The benefits of using Apt over manually installing things is that Apt figures out all of the dependencies automatically. Further, it keeps everything up to date by periodically scanning the online repository (PPA) and downloading any updates. In practice, Apt works much like the Windows Update service, only better, since it monitors every piece of software installed on your computer.

Installation Instructions

To install Time Drive using the PPA, we need to do the following:

Add the Time-Drive-Devel and Duplicity PPA to your Ubuntu Software Sources.
Configure Ubuntu so that it will trust both Time-Drive-Devel and the Duplicity packages.
Install Duplicity and Time Drive

Step 1: Add the Package Repositories to the Ubuntu Software Sources

Open the Ubuntu Software Sources by going to the System –> Administration –> Software Sources. Then, select the “Third-Party” software tab of the application. “Software Sources” requires root access, therefore, you will be prompted to type in your administrator password.

We need to add two package repositories to the list, one for Duplicity and the other for Time Drive. Depending on the version of Ubuntu that you are using, copy and paste the following lines (one at a time) into the “Add Sources” dialog. After each line, press the “Add Source” button.

For Ubuntu Karmic Kaola (9.10):

deb http://ppa.launchpad.net/duplicity-team/ppa/ubuntu karmic main # Duplicity – Ubuntu 9.10

deb http://ppa.launchpad.net/time-drive-devel/stable/ubuntu karmic main # Time Drive – Ubuntu 9.10

For Ubuntu Jaunty Jackolope (9.04):

deb http://ppa.launchpad.net/duplicity-team/ppa/ubuntu jaunty main # Duplicity – Ubuntu 9.04

deb http://ppa.launchpad.net/time-drive-devel/stable/ubuntu jaunty main # Time-Drive – Ubuntu 9.04

For Ubuntu Intrepid Ibex (8.10):

deb http://ppa.launchpad.net/duplicity-team/ppa/ubuntu intrepid main # Duplicity – Ubuntu 8.10

deb http://ppa.launchpad.net/time-drive-devel/stable/ubuntu intrepid main # Time-Drive – Ubuntu 8.10

When finished, press the “Close” button. Ubuntu should then notify you that the software information is out of date and needs to be updated. Instead of clicking “Reload,” click the “Close” button. We will refresh Apt from the command line in Step 3.

Step 2: Configure Apt to Trust Time Drive and Duplicity Packages

After adding the repository information, you will then need to add the repository key to your system’s list of trusted keys. This is most easily done by running the update-launchpad shell script. (To download, right click on the link and select “Save As”.) When run, the script automatically downloads any needed signing keys and automatically add them to the “Trusted Sources” list.

To run the script, go to Accessories –> Terminal and navigate to where you saved the file, then type:

cd /path/to/folder
sudo bash update-launchpad.sh UbuntuVersion

Be sure to substitute the correct Ubuntu code name for UbuntuVersion in the second step – karmic for 9.10, jaunty for 9.04, or intrepid for 8.10 – otherwise, you will get an error. I happen to use Ubuntu 9.04, therefore, for my setup, I would type the following:

sudo bash update-launchpad.sh jaunty

Step 3: Install Duplicity and Time Drive

Finally, we are ready to install Time Drive and Duplicity. From the command line, refresh the package lists by typing:

sudo apt-get update

Then, to install time-drive and duplicity, type:

sudo apt-get install time-drive duplicity

Application Updates

As noted above, whenever a new version of Time Drive, Duplicity, or another dependency is added to the package repository, your system will automatically be updated. However, should you upgrade your system to a newer version of Ubuntu, you will need to return to this page to add the repository for that version.

__________________________________________________

Note: These instructions were modified from this tutorial on WineHQ.

Time Drive 0.2 – A “More S” Release: More Stable, More Secure, More Settings and Now Supports Amazon S3 Storage

Rob Oakes — Thu, 24 Sep 2009 18:41:57 +0000

The first time that you attempt to do something, it’s pretty much a guarantee that it’s going to suck. This doesn’t necessarily need to be a bad thing. Shows like America’s Funniest Home Videos and MXC have found dozens of way to cash in on the humiliation of their participants. (And what better exemplifies pure suck than a golf ball to the groin?)

It, therefore, shouldn’t come as any surprise that creative pursuits are no exception to the general rule of suckiness. After all, you have to overcome inexperience and ineptitude to produce anything. The only way to ensure that a release doesn’t suck is to finish a first draft and revise heavily. Which requires a great deal of work.

Yet … as interesting as that might be, this isn’t a post about the creative process. It’s about Time Drive and I should probably admit that Time Drive 0.1 had a few … rough … edges. Sure, it mostly worked, but it was new software and did too many strange things to declare anything other than a “work in progress.” But Time Drive 0.1 was a first release and first releases suck.

Second releases, however, offer a chance to clean things up, refine the bleeding edge, and otherwise deliver the goods. Maybe that’s why I’m so excited to announce the release of Time Drive 0.2. This version of Time Drive is a great improvement over it’s predecessor. So much so that Time Drive 0.2 is hereby dubbed the “More S” release: more stable, more secure, more settings, and Amazon S3 storage.

In the remainder of this post, I’ll attempt to justify such a silly name by taking a look at a few of those new features.

More Stable and Secure

First up, I should probably say something about the program’s stability and security. But before doing that …

Isn’t it a bit ironic that stability and security have become major selling points? Go ahead and look at the advertising for any major piece of software. References to stability and security – or their closely related siblings, reliability and refinement – will feature prominently.

As an example, let’s analyze the promotional materials for the most recent release of Mac OS X, Snow Leopard. In the banner advertisement below, stability and reliability form the cornerstone of the brainwashing message: “Refinements, your Mac gets faster and easier to use.” And in the text, we catch all the buzzwords: simplicity, reliability, security, speed. In short, they’ve attempted to make make old things new by refining them.

Time Drive has gone through a similar type of rewrite. With the generous help of Philippe Delodder, all of the core classes have been reformulated and reorganized: they’re new again. The end result is that all things are improved, spiffed and otherwise sped up. In short, Time Drive no longer resembles a prototype as much as a proper foundation upon which to grow.

More Settings

And nowhere is this more apparent than in the new advanced settings dialog. In version 0.1 Time Drive handled all the important things that a backup program should. It was easy to specify which folders you wanted to backup and where the archive was to be stored. But that was about it.

The first version didn’t let you fully use the power of Duplicity to specify what you considered important. What if you didn’t want to backup hidden files or settings? Or what if you wanted your work files to be stored in a different location than your home files? Or what if you want to create a new backup every couple of months?

Time Drive 0.2 gives you that degree of control without losing the simplicity. Just like before, you get started by telling the program where you want to store the archive and which folders to include. If that’s all you need, then you’re done. Click “Ok” and let Time Drive do it’s thing.

But why settle for simple when you can micromanage? Need to refine the settings for an individual folder, highlight it from the list and press the “Advanced” button. From there, you have easy access to things you don’t want included in the backup, fine control over where it gets kept, and even the ability to remotely manage the backup sets.

What You Exclude Can Be as Important as What You Include. Despite the fact that we should keep redundant copies of our data, sometimes there are things you just aren’t worried about. For example, you might be a Python developer and want to use Time Drive to backup your source code. The source code is important, but whenever a python script gets run, the python interpreter creates a compiled file (“.pyc”). These files are necessary for the code to run, but they don’t contain any valuable or new information. They just take up space. So, let’s leave them out them by adding “.pyc” to the exclude list. Just press the “Add” button and type the offending characters into the message box. Want to leave “.pyc” files out of all of your backups? You can do that too from the main exclude list.

Different Storage Locations. Now, what if you want your work related backups to be kept on your company’s ftp server instead of your local hard? Time Drive has you covered there as well. Just click on the “Archive Settings” tab and tell Time Drive where you want the folder’s archive to live. You can even specify a folder specific encryption key should you so choose.

Manage Your Backup Sets. Perhaps the only downside of creating frequent and redundant backups is that old files might take up needed hard drive space. Sure, managing backup sets by hands is possible … but who wants to do that? Time Drive 0.2 makes it easy to delete your old backup sets when they’re no longer needed, thereby saving valuable hard drive space.

Amazon S3 Storage

Which finally brings me to the biggest new addition to Time Drive, support for Amazon S3: a reliable, simple and inexpensive way to save your files over the internet. When it appeared a few years ago, S3 introduced a radically new idea: only pay for the storage you actually use. At the time, other providers charged a flat rate for a set amount of storage, regardless of whether you needed it or not. While prices ran the gamut of reasonable to outrageous, you were still paying for something that you might not be using.

Using S3, however, it is possible to have offsite and redundant backups while knowing that you are only getting charged for what you use. Moreover, the rates are quite reasonable, about 0.15 cents per gibabyte of data per month. For example, I currently keep all of my source code files and pictures backed up through Time Drive. In total, it amounts to about 15 GB of space. Amazon charges me about $2.25 a month for the space, which is substantially less than any other provider I’ve been able to find. Sure, there are plans that are slightly more expensive and provide more space; but that’s just it, I don’t need that additional space. I’m probably not going to use it. Which makes Amazon a really good fit for my needs. It’s cheap, it’s convenient and it’s available everywhere. Most importantly though, it’s offsite, so I’m prepared in case some sort of disaster does strike. And Time Drive works with it seamlessly.

Conclusion

In summary, I think that Time Drive handily earns it’s “More S” label. It’s a big improvement over the previous version and steps up the stability, the number of settings, and the types of storage backends that it works with. But it still remains elegant and easy to use. In short, it’s one step closer to the overall goal of “Backup Done Right.”

Head on over to the project’s home page and check it out.

Filing Bugs for Time Drive or LyX-Outline

Rob Oakes — Sun, 23 Aug 2009 04:30:59 +0000

Even though software developers try and create a perfect product, every program has its flaws. These can be show stopping, or merely annoying; but they are unavoidable. Even so, unavoidability doesn’t mean that they should be tolerated. Which is where users bear some responsibility. If you want a better program, you need to provide feedback to the developer.

Given how important feedback is, I find it hugely ironic that many of the software programs used to solicit such feedback (called bugtrackers), are horrifically difficult to use. Perhaps the single best (worst?) example is the system used by the Mozilla foundation (famed creators of Firefox and Thunderbird): Bugzilla. The interface is backward, it’s impossible to find anything, and getting to the same page twice requires a minor miracle.

Knowing this upfront, I tried to find a very easy to use program for collecting feedback on LyX-Outline and Time Drive. And I thought I had succeeded when I came across Flyspray, an open source bug tracking system. While still remaining powerful, Flyspray is fairly easy to use for both developers and users. Unfortunately, though, it hasn’t been quite easy enough.

You see, even though it’s pretty simple to open a new bug report, getting to the right place isn’t the most intuitive process in the world. A number of people have expressed confusion about where to go to complain about Time Drive. As a result, I thought i would write a brief guide.

Step 1: Go to the Bug Tracker Website

To get started, you need to head to the Time Drive and LyX-Outline bug tracker website. It can be found at:

http://www.oak-tree.us/bugtracker

Step 2: Go to the Appropriate Project Page

When you load the page for the first time, you will see a list of the currently open bugs for all active projects. Depending how busy I’ve been, this list could be quite long, or quite short. What you won’t find on this page, however, is an easy way to add new bugs. To do that, you’ll need to navigate to the project specific page.

Look at the very left hand side of the tool-bar. Underneath the username box, you will see a drop down list that says “All Projects.” This magical list is the main way that you navigate my Flyspray based bugtracking system.

For example, let’s say that Time Drive just corrupted your vacation photos and you want me to know this and how angry it made you. To communicate these thoughts (bonus points if you can do so without swearing), you need to go to the Time Drive project page. This is as easy as clicking on the down arrow and choosing “Time Drive” from the options. After you change the option, it is likely the page will refresh, and the main navigation menu will change. (Though on some browsers, most notably Opera, you will need to click the “Switch” button.)

Now, instead of the utterly boring, “Overview” and “Tasklist” options, you should be presented with a slightly more exciting menu. This includes the options to take a look at the “Roadmap” and more importantly, “Open a new Task anonymously.”

Step 3: Complain Loudly, Thoroughly and Convincingly

This is where you need to complain. Tell me exactly what you think of me, my creations, and my professionalism. Rant about how Time Drive corrupted your carefully maintained comic book collection, or how LyX-Outline destroyed your thesis. This information is important, it helps me to improve the programs. And getting it off your chest might help you too. Nothing is more soothing than a good rant. Especially when the person on the other end really wants your input.

Squeaky wheels tend to greased. Thus, consider this positing as my official permission to squeak. But to be most effective, you need to make your noise in a place where it will get noticed. For Time Drive and LyX-Outline, this happens to be at their respective project pages.

Time Drive 0.1.5: Incremental Changes

Rob Oakes — Fri, 14 Aug 2009 15:19:10 +0000

It’s been an interesting couple of days. I was rather honored to see that Lifehacker did a short highlight of Time Drive, which I thought was pretty cool. It’s always been one of my goals to have something featured in Lifehacker or Gizmodo, and now I’m going to have to scratch that off the list of goals. But that’s okay, I’ve got other things to fill the void. Like … how exactly does one get invited to present at TED?

On another note … while I knew that I would see some kind of traffic bump due to the article in Lifehacker, I wasn’t necessarily prepared for the magnitude. In mathematics, there is this thing called a step function. It’s where you move from one value to another more or less instantaneously. It looks like a step, hence the name. Sure, It may not actually exist, since even very dramatic shifts still have a non vertical slope; but even so, the change in my traffic might as well be a step-function. Between yesterday and today, I’ve had more visits to this site than I’ve had in much of the rest of the year combined. I think that’s kind of cool, though it probably won’t last.

(This might be a good time to say that I am actually rather proud of my “lackluster” web traffic. Though it might not necessarily be that impressive, it is, nevertheless, mine. I’ve worked hard for it, and I revel in the fact that some 40 to 50 people each day find the unorganized garbage of my mind intoxicating. Some of them even come back!)

But as interesting as that might be, traffic stats is probably not why you're here. Good thing, since I’ve got announcements.

Time Drive Update

As of this moment, Time Drive has been updated to version 0.1.5 and is available on the project page.

Let me start by saying what this release isn’t. I really wanted to write a blog post and say, “Time Drive is available, and now it’s really easy to install!” In the past few days, this has been by far the most requested “feature” and I’ve tried hard to deliver. (Honest!) Unfortunately, creating a binary installer for Windows and a self contained package for Mac has proven … illusive. I’m not really a programmer and the inner workings of py2app and py2exe are tremendously frustrating. I have made some serious headway, but stand-alone installers are still “forthcoming.”

But a lot of things are new, and some of them are even shiny! Moreover, the relatively huge interest has gotten me excited about the program, and I’ve spent some time refining things that didn’t quite work right. “So …” you ask, “What’s different?”

New Stuff

For starters, the Snapshot History on the main window actually works now. If you want to restore a previous version of a file, you can select it from the list and click the add button (+) to open the archive browser. It will automatically load the snapshot as it looked on the selected day and time.

If you can’t find your file there, don’t despair, from within the archive browser, you can easily navigate forward and backward between snapshots by using the arrow buttons or by dragging the slider to the appropriate place. Once you have it sitting on the snapshot of interest, hit the refresh button to reload the file list. This is starting to get closer to the overall vision of the program that I originally stole. Now, you can actually use Time Drive to keep tabs on how your files change.

I’ve also been trying to get things cleaned up so that the program runs better. I went back and modified the code so that all long running processes use their own threads. This should help the UI from freezing and lets you do several things at once.

I also cleaned up the way that Time Drive interfaces with cron. Now it works on Mac without requiring tweaks. In the relatively near future, I’ll write a proper backend for Windows so that is uses the Windows Vista task scheduler. Installing cron on Cygwin is just too much of a pain.

Looking into the future, I foresee support for Amazon S3, pruning and naming snapshots, and advanced folder options. There may be other changes as inspiration strikes. Expect additional announcement soon.

Conclusion

Well, that’s it. Why this may be a bit of a minor addition, there’s still some nice stuff. (I know, I’m biased.) Head over the download page and check it out.

Backup for Linux, Done Right - Part 2: Time Drive 0.1

Rob Oakes — Fri, 07 Aug 2009 17:39:48 +0000

In part 1 of this article, I shared a few of the frustrations and reasons why I decided to write my own backup utility rather than submit to the tyranny of currently available solutions. While some might find those ruminations interesting, the vast majority are probably far more interested in the end result. There is a reason why “Get to the point” is one of the most important sentences in the English language.

Here’s the short version: After becoming tremendously frustrated by the state of backup on Linux, I decided to take matters into my own hands and create my own tool.

And though I only want a few things, I want that tool to do each very well. First, I’m looking for a solution that can incrementally backup over the network and let me restore a file from an existing snapshot. Second, those snapshots should be compressed, encrypted and secure. Third, it should be easy to browse old backups for existing files and restoration should be a one-click affair. Fourth, I want a backup system that can protect me from disaster, carelessness and pathological stupidity.Time Drive is based on the excellent command line program, Duplicity though it was also inspired by another backup utility, Back In Time. Because it uses Duplicity for its backend, Time Drive plugs into the same industrial strength utilities used to backup and restore enormous sets of enterprise data. It just removes the industrial level complexity.

Like Back In Time for Linux and Time Machine for Mac OS X, Time Drive not only keeps a copy of the most recent file, but it remembers how that file looked at many points in the past. Put another way, it creates an incremental snapshot which lets you restore to a previous version should you so desire.

The Time Drive main window is designed to be simple and functional. Options related to backing up files are on the left, options related to restoring files are on the right. If you want to restore from a remote archive which is not currently specified in your settings, you can do so through the “Advanced Restore Options.”

Over the Network or Across the World

Backups can be made to remote servers and web services that support any one of the following protocols: file transfer protocol (ftp), secure shell (ssh), secure copy (scp), rsync, webdav (http), and secure webdav (https). Time Drive also supports backups to a locally attached drive or mounted volume. This includes shares that are mounted with virtual filesystems like sshfs, smbfs, and s3fs.

You can configure the remote backup by clicking on the general tab of the settings pane. Then, move the option to “Use a remote directory (SCP, WebDav or FTP).” From the drop down list, choose the communications protocol.

If making a backup to ftp or webdav (http), enter the name of the host and the path to your backup folder as a standard web url. It should look like:

computer.name.com/path/to/folder

For example, the backup path to my webdav share is:

backup.oak-tree.us/RobOakes/Backup

If backing up by scp or ssh, you will need to use a second slash after the server name, for example:

backup.oak-tree.us//RobOakes/Backup

Next, enter the username and password that you use to access the server and some sort of passkey in the “Encrypt Files” box. (While future versions of Time Drive will allow users to choose whether or not they wish to encrypt their files, this has not yet been implemented.) When finished, you can check the connection by clicking on the “Check Connection” button to the right of url text box. If Time Drive is able to connect to the store, a little green check mark will appear.

Restoring Files

Should you delete a file that you needed, you can quickly browse or search the archive to find it and restore it. Need to restore a bunch of files? You can add as many as needed to the queue and then restore them with a single click. The restore dialog is accessed by clicking on the “Add files” button of the main window. The restore dialog itself allows you to browse the folders available in your backup, and any subfolders of the archive.

But what if you need access to a file in your work archive and you happen to be at home? Time Drive makes it easy to brose and restore from any archive, anywhere. Simply type in the url and click “Add files” to enter the restore files dialog.

Set It, Then Forget It

While Time Drive can be configured to work many different ways, file backup should ultimately be a very simple affair. True, it’s nice to have more advanced tools available, but most things in life aren’t really all that complicated.

As a result, setting up Time Drive for the first time is designed to be as easy as possible. First, pick where you want to perform your backups: either to the local machine, or across the network. Next, enter in a passkey to encrypt your backup files and keep them safe. Finally, choose which folders you want to see included in the job. When finished, click “Ok.”

Because each folder is a run as a separate duplicity script, you can always re-add that folder at a later time and Time-Drive will update the existing snapshot with the most recent information. All of the existing history will be there should you ever need to access it.

Time Drive also makes it easy to schedule automated backups through the use of the powerful crontab utility available on most *nix machines. Because it uses cron, there is no need to have a dedicated background process. (Though cron must be running for scheduled backups to work.)

Automated backups are configured by opening the settings dialog and navigating to the “Options” tab. Then, just pick how frequently you would like the program to run: every hour, day, week or month. When done, select “Ok.”

Notifications and Feedback

When looking at other solutions, I quickly became frustrated by the way which they handled user notifications. In some cases, the program provided a level of detail that was simply overwhelming. In others, it didn’t provide any feedback whatsoever; even when there was a problem.

With Time Drive, I’ve tried to split the difference. It uses the notification system of Ubuntu 9.04 and will let a user know when new backup jobs start and it finishes, if there is a problem, it will let you know that too. But for those who would rather not be bothered with such trivial things, the notifications can be easily disabled through the settings tab.

Beta Software

Even though Time Drive is relatively full featured and offers a convenient backup package, it is still beta software. That means that it needs to be tested, beat upon and otherwise tortured. As you do so, I would ask that you be a bit patient and mind the potholes. If it doesn’t quite meet your needs, please leave a bug report or feature request on the bugtracker page. Be as specific as possible and I will try to incorporate your ideas into the next release.

Conclusion

With Time Drive, I finally have a backup utility for Linux that does things right. It can backup to a computer on the network or across the internet; keeps incremental snapshots of only the files that have changed; makes file restoration a one-click affair; and keeps my personal information safe. What’s better, no tweaks or hacks are required to make it work the way that I want. In short, I finally have something that “Just Works” without having to stick some stupid qualifier in front.
______________________________________________________________

If you are interested in downloading, testing or using Time Drive, you can find additional information on the project homepage. This includes installation instructions for Time Drive, Duplicity and PyQt in addition to the program documentation.

Backup for Linux, Done Right- Part 1: A Mini Melodrama

Rob Oakes — Fri, 07 Aug 2009 04:58:08 +0000

It is a terrible thing to realize that you are stuck in a rut. Being in a rut effectively means that you’ve stopped advancing and life has evolved to monotony. No one likes to be around people in ruts, but it’s even worse to discover that you are personally trapped in one. And, most unfortunately, I am in a rut.

Don’t believe me? Take a look at the home page of this blog. You will likely notice that a full six of the ten most recent posts have dealt with one subject: backing up your computer. That’s pretty conclusive evidence of a rut.

Now, backing up your computer is a very important thing to do; you should do it regularly and have a plan. But … well … it’s boring. Talking, thinking and writing about nothing but backup is dull. As one of the doctors I work with likes to say, “That isn’t sexy. If I’m going to spend any time with it – women, food, wine; it doesn’t matter – it should be sexy.”

He’s got a valid point, backup is not “sexy” and I’d like to write about things that are, at least for a while. This, therefore, will be my last post on backups, archives, or servers for the relatively foreseeable future (technology is just too cool to lay it aside for too long). But before doing that, I want summarize where I ended up in my quest for the ultimate backup system.

Backup on Mac is taken care of, I use Time Machine to a Samba share. More adventurous persons than I might even say that this arrangement approaches sexy. It’s convenient, fast, and robust. It even covers disaster recovery.

Backup on Windows is also covered. The built-in file backup is easy to use and works well. Moreover, setting up a disaster recovery system is relatively painless.

But the third major operating system, Linux, is a bit of the odd-man out. Certainly, you can find some excellent backup systems, Back In Time is one such example. With a bit of work, you can even tweak it so that it is almost perfect. But it’s the “almost perfect” and closely related cousins (“mostly useful” and “good enough”) that are the problem. They have those stupid qualifiers – almost, mostly, enough – bolted on.

Any time you hear a qualifier, you can rest assured that you aren’t going to like what follows. Consider the rather innocuous phrase, “that may be a problem.” Here, the term “may,” makes an already bad situation much worse. Instead of specifying some probability of problemhood, it all but guarantees it. Positive qualifiers are just as bad.

As a result, it angers me that nearly every backup program available for Linux requires some kind of qualifier. It shouldn't be like this. Linux is a brilliant operating system in practically every way. It is highly integrated, wonderfully modular and tremendously easy to extend. So … after finding that nearly every backup utility in existence has failed to meet my needs, I found the situation intolerable and decided to do something about it.

I wrote my own.

Backgrounds and Backends

Fixation and ruts can make you do silly things like that. Instead of just accepting the limitations of an existing situation, a fixated person will demand that the world bend to their expectations. While this sometimes leads to great advances, more often it results in interpersonal disasters of epic scale. And the general rule is, the more menial the detail, the larger the scandal. (At this point, it might be argued that backup to an external hard drive versus across the network is a rather menial detail. To all such detractors, all I can say is: stuff it. This is my story.)

Basic Needs

Luckily, however, the story of my backup utility isn’t nearly so sordid. This is probably because my needs are actually pretty simple. I need a backup utility to do just a few things, but I require that it does them well:

It should create versioned snapshots of my drive. This lets me restore a file to any number of past states.
The utility should only transfer the portions of the file or directory that have changed. This makes the backup operation quick and efficient.
I should be able to backup over a network or across the internet. (Very important!)
Restoring a file should be quick, easy and painless. A backup is only as good as your ability to get at the information. And like it or not, information will be lost due to disaster, carelessness or miscellaneous stupidity.

Being a somewhat clever person, I didn't to avoid duplicating as much work as possible. As a result, I started creating my program by reviewing the application that comes closest to doing what I want: Back In Time. The Back In Time user interface is simple and elegant. More importantly, though, the configuration options make sense. Whether you want to run a backup, give your snapshot a name, or just get rid of it; you don’t have to go hunting to find the controls. Additionally, Back In Time makes good use of existing open source programs to actually do the backing up. Though they might be a bit old, rsync and the Unix copy command, cp, are a formidable duo.

But I quickly discovered something unfortunate, Back In Time really couldn’t be tweaked to meet all of my requirements. You see, it utilizes a technique called hard-linking to create space efficient repositories. But hard links only work as long as they are on the same hard drive. Because I want to store my files on the network or across the internet, that isn't good enough.

Duplicity

So, I started to look elsewhere for other options, which is when a colleague pointed me toward a command line script called Duplicity. In a striking bit of irony, while the GUI tools of Linux backup are all fundamentally flawed, the command line versions are the best in existence. They easily accomplish all of the requirements on my list, and do a great many things I’ve never even heard of. As a result, if you are willing to roll up your sleeves and hit the learning curve, there is nothing that you can’t accomplish. And, as you might deduce from my glowing introduction, Duplicity makes short work of my requirements list.

Can it do incremental backup? Yes. What about backup over the network? Ditto. In fact, it handles most protocols I’ve heard of, and quite a few that I haven’t. Should incremental through ssh, webdav, ftp, smb, or imap be insufficient, it also supports backup to Amazon S3. File restoration is also relatively painless, though you have to work through the command line interface to do so.

And that’s only for starters, Duplicity also supports encryption via passphrase or passkey, compression of the files into archives, and local caching of file signatures so that backup operations are lightning quick. I decided relatively quickly that Duplicity would be the ideal backend for my utility.

Frontend and Features

But while a solid foundation is a great starting point, it can only get you so far. After all, you don’t brag up the brickwork and reinforced concrete of your newly purchased home, you’re far more likely to show off the kitchen and home theater.

For a program, the points of interest are going to be the user interface. Which makes two lessons learned while working on LyX-Outline very important: 1) It’s really hard to put together an interface that is both intuitive and uncluttered. 2) I’m really bad at it. Both points prompted me to act on the advice of a famous painter:

Good artists copy; great artists steal.

Thus, I decided to steal the user interface from Back In Time. (It’s open source, which makes it okay.) And speaking from a practical standpoint, why wouldn’t I? The developer spent a great deal of time working out how he wanted his program to work. It would be silly to duplicate that effort. Moreover, the fact that I was able to rave about it for nearly 2800 words means that it wasn't wasted effort. As you browse the menus and configuration panes of my derivative tool, you will probably notice that it bears a striking resemblance to other programs. A few things will be “innovatively different,” but not many. Just remember, I stole the interface. Wholesale. But I’m a discriminating thief, and only stole the best parts.

Which brings us to the real point of interest of this post and my program: the feature list. In part 2 of this article, we’ll take a look at my little creation, which I have unimaginatively dubbed “Time Drive.”

Introducing LyX-Outline 0.1

Rob Oakes — Wed, 13 May 2009 22:16:12 +0000

Writing anything - whether it be a book, or a short story or an angry letter to your boss - is substantially more than starting from the first idea moment of inspiration and continuing to the final draft. Rather, writing involves a fair number of idea fragments, fleeting moments of inspiration, and a tremendous number of dead ends.

It is incredibly unfortunate that most writing software, however, is geared to organizing and structuring the document after most of the hard work has already happened. It simply assumed that most of the planning and layout has already happened and the author is ready to string words together. Unfortunately, this assumption overlooks one important truth: ideas are best defined as they are expressed. Thus, it's usually about the time that the a writer sits down to compose the draft, that the document's true structure becomes apparent. In my own case, this often leads to a flurry of reorganization. And it’s during the restructuring that the real battle begins.

When in full creative passion, I am typically working with three or four different programs all at the same time: OneNote is open so that I can access my ideas, Word is there to start collecting the somewhat finished text, and I'll also probably be using a MindMapper so that I can see a visual representation of the document structure. The tools are separate and don't communicate with one another. Thus, a change made in one place needs to be made everywhere. And all too often, that I end up fighting the word-processor and the notetaker, and the mind mapper. It is tremendously frustrating to battle the tools of your adopted trade. Luckily, I am not alone in my frustration.

In the past everal years, a number of programs have become available that leave the linear model of writing behind. On the Mac platform, one such tool looms above the others: Scrivener. Central to Scrivener's function are two important metaphors: that of the outline and that of the corkboard. And it works really, really well. There are just a few minor problems. First: Scrivener is only available for Mac and Scrivener's lead developer has made it clear that there won't be versions for other platforms. Second: Scrivener was really designed with creative writing in mind. Thus, while it can be used for long and complicated documents, this is a slightly less than straightforward process. Last, to create said fancy documents, Scrivener requires the raw use of a markup language (and all of the associated headaches that come with it).

The document processor, LyX, however, excels in many areas where Scrivener falls short. It is built upon the mature and robust underpinnings of LaTeX, the typesetting language of choice in the science and engineering. And more importantly, it is easy to use (where LaTeX most decidedly is not). But it fails in the same way as Microsoft Word and other word processors, it is a linear writing tool and doesn’t offer a great deal of work-flow flexibility. That is where LyX-Outline comes in.

LyX-Outline is a marriage between Scrivener's organizational tools and LyX's typesetting tools.

Looking for Flexibility

While you can download and play with LyX-Outline, please reember that is still a forthcoming add-on to the LyX. And while you find it useful, this release is an alpha level technical preview. Right now, you can view the structure of your document at a glance in the corkboard, move things around, poke about in the outliner, and even jot your winded thoughts with the very basic plain text editor.

But even though some of the functionality is lacking, you can start to explore. And as you do so, one of the first things you might notice is that LyX-Outline has been designed with flexibility in mind. Nearly all of the tools can be docked, ripped away, or hidden if they are not needed. The work environment should adapt to your work habits, not the other way round. Whenever the writer needs to adapt to the environment, a moment of productivity and a spark of life has been lost.

A Quick Overview

The main interface consists of three main elements. This includes the document map, the editor pane, and the dock tools. As described above, the document map and the dock tools can be relocated, docked or left free floating (depending on your preference). And if you don't find them useful, they can quickly be hidden.

The document map lets you see a hierarchical list of every item in your project. Selecting one of the items displays its content's in the editor pane. The editor pane is the main text processing portion of LyX-Outlineand can be toggled between a text editor (plain text only at the moment) and a corkboard view.

In addition to the main editor and corkboard, there is an additional corkboard and outliner tool available as a dockable widgets. Just as in Scrivener, you can view the pieces of your documents in many way. Every item is a document, and an index card, and an outline point. They are intimately tied together and any change made to the order or content of one will be reflected everywhere. The document map is used to navigate the content of the main window.

The dock tools, however, can be controlled separately. You can view the entire document, or filter to a specific level in the hierarchy via a drop-down menu. In the future, filtering by search term, keyword, document structure, or tag will also be possible.

Item Summary and Organization

The health of any written work can also be greatly improved by connecting a simple synopsis to a larger chunk of text. This makes it much easier to check the flow of your work at a glance . A summary can easily be added to any individual item in the collection by double clicking on it's index card in the corkboard or the synopsis column in the outliner. The summary text is independent from the full text and can include pertinent information that helps you structure the document. For now, the synopsis must be entered by hand; however, future versions will allow for automatic generation of the synopsis from the text.

Of Outlines and Index Cards

Scrivener (and LyX-Outline as it's dedicated clone) utilizes multiple metaphors. The Corkboard may be useful in some instances, but will be less helpful in others. Ditto for the outliner. Either way, it is important to know that both the corkboard and the outliner display sub-items. If you are using the main corkboard, these sub items will be connected to the active item in the document map. If you are using the dock tool, they can be filtered via the drop-down menu. From the corkboard, you can then edit items by double clicking on either their title, or their summary. You can reorder them through drag and drop. From the outliner, you can add new items or sub items, or delete them from the document. You can also change their order (by using the up or down arrow buttons) or change their indentation level (with the left and right arrow buttons).

Minding the Potholes

And that is a very quick tour of the Outline modules for LyX. As noted above, this release of LyX-Outline is a prototype and it is still very far from its intended target. Thus, if it doesn't meet your needs right out of the box, please be patient. The mantra of OpenSource is "release early, release often." In this case, I have erred on the side of early and as you might suspect, there are some dangerously rough edges. These include:

Unfinished interfaces. Most of your manipulation will probably need to happen in the outliner pane. This includes adding or removing new document nodes, as well as indenting or un-indenting the document selection. The final version will include ubiquitous drag and drop.
A crappy editor. The current editor is plain text is a placeholder only. Ultimately, LyX-Outline will be integrated with the LyX editor and have access to all of it's very powerful features. It is important to keep that future in mind.
No export option. While you can save and open past documents, there is currently no way to actually get them out of the program.

In it's current incarnation, this prototype is valuable for pecking out thoughts and playing with the organizational tools. I am releasing it so that people can see how the tools might look in their final form and provide feedback. Please do so. Download the source, use it as you can, and let me know what you think. What things work? What things don’t? What features would you as users like to see?

Update: You can download the source code and find installation instructions on the LyX-Outline project page.

Cross Platform GUI Development with Python and Qt

Rob Oakes — Tue, 12 May 2009 21:16:18 +0000

One of computer programmers many fantasies (or at least one of my fantasies) is a cross-platform codebase that is easy to maintain. I want to write an application once and then be able to deploy in on any platform in existence. It should integrate into the destination environment in a native manner, but still be similar enough to the other platforms so that issues can be addressed in a way that doesn’t hinder overall progress.

Part of this dream can be fulfilled by using Python and standard library, it is also possible to do much of it by using IronPython and .Net (through the OpenSource Mono framework). However, in each case, you are forced to make a tradeoff. Python and the standard library are wonderful, if you are willing to accept a speed hit. IronPython and Mono also work well, as long as you avoid using any of the Windows specific libraries. Unfortunately, many of neatest .Net features (like Windows Presentation Foundation) are only available on Windows.

Okay, there is always GTK+ or wxWidgets (the same toolkits used for the Gnome desktop), but GTK+ is written in C and while much of the miscellaneous nastiness has been abstracted away, it is still one of the hardest toolkits to work with. Oh, and the documentation sucks. (When any documentation can be found at all.) Thus, we arrive at the final contender: Qt. Qt has had a long and controversial history involving license issues, passion, and miscellaneous geek pride. In fact, it’s due to Qt and some rather divisive decisions made by its developers that we even have alternative toolkits at all. But, despite that history, Trolltech’s recent decision to release the framework under the permissive LGPL license (rather than the noxious GPL) changes all that. In many ways, Qt might just become the cross-platform toolkit of choice

Of course, it's a given that Qt is tremendously powerful with rich functionality. Applications that are written in Qt are fully cross-platform and behave the same way wherever they are deployed. It uses native widgets whenever possible and has a sophisticated theming system that allows it to follow platform conventions without significantly modifying the underlying code. More importantly, Qt is backed by a passionate developer community and a well funded corporation (Nokia). It is true, you get what you pay for. Qt is well designed and the documentation is simply fantastic. And unlike GTK+ or wxWidgets, the entire toolkit is written in proper object oriented C++.

I first started experimenting with the Qt framework about two months ago, after deciding to tackle a major add-on to one of my favorite writing programs, LyX. As I have said elsewhere, I am not the world’s most

gifted computer programmer (far from it actually), but Qt actually makes it pretty easy to get yourself up and running in a reasonable amount of time. If I were forced to do most of my GUI programming in Qt and C++, that would be an acceptable solution. However, that is completely unnecessary. You see, Qt has one of the most tightly integrated Python bindings available, through the user of PyQt from Riverbank software. Qt plus Python is, quite simply, programming perfection. So while I did some mild experimenting with the Qt frameworks in C++, I have spent much of the past few months doing the heavy work of learning the framework via the use of Python.

In the process, I have made a great many mistakes and a great many more discoveries. And since the final step in any discovery is to share your findings, I will be doing that through this series of articles. Of more interest to most, however, I will also be introducing the fruits of my labors: a somewhat functional prototype of my outliner add-on for LyX.

Part 0: Thoughts on Licenses, Setup and Installation

But before I bore you over the benefits of abstraction and dynamically typed object oriented programming, I should first dispense with some of the nuances and details of setting up a system to run Qt. This also, unfortunately, must include a brief discussion of licenses. While most general computer users likely do not care, the license under which a given piece of software is released is tremendously important. In the case of a most OpenSource software, it determines what rights the users receive and what the developers retain. In some cases, it may even determine if the developer is able to charge for his or her work.

This is why Trolltech’s decision to switch from the GPL (a popular and very restrictive OpenSource license) to the more permissive LGPL is a very big deal. As alluded to above, it allows for commercial developers to link against the Qt libraries without legal fears. Unfortunately, however, the new licensing is only for the most recent version of the toolkit (version 4.5). In addition, while Qt is available under a more permissive license, PyQt is not. PyQt is only available under the terms of the GPL or a separately purchased commercial license. For people who will use PyQt for in-house work or for end users, the distinction is a purely philosophical matter. But if you are a developer who intends to distribute your code (or attempt to charge for it), it is something that you need to consider carefully. Unless you buy a commercial license, PyQt can only be used for GPL related development. The good news, however, is that it is perfectly possible to utilize a commercial PyQt license in conjunction with the LGPLed Qt. Further, the costs for a commercial PyQt license are quire reasonable, about $400 dollars per developer. While this might sound like a lot, compare it to the more than $5000 that Trolltech charges for a cross-platform commercial Qt license.

Installation, Setup and Configuration

Now that some of the grit has been sorted, let’s get down to the the minutiae of setting up a PyQt development environment. This guide will focus on using the most recent version of the framework (Qt 4.5.1). Since the procedure varies slightly based upon your platform, separate instructions will be provided for Mac OS X, Linux and Windows.

There are four separate pieces of software that you will need to download and install on your system. 1) The Qt Framework, which you can get from the Trolltech website, 2) the Python interpreter and related libraries, 3) SIP, which is used to generate Python bindings to C and C++ code, and 4) PyQt. In general, the installation procedure goes like this:

Download and install Qt via the package installer for your platform from the Trolltech homepage. In order to avoid manually compiling the libraries, I would recommend downloading the package entitled “Qt SDK: The Complete Development Environment.”
Install the GNU compiler tools for Windows (MinGW). If you use the complete Qt4 package in step 1, this will happen automatically. (The GCC tools are typically provided as a package for Linux and can be installed from the Mac OS X system CD or by downloading the XCode developer tools)
If there isn’t a version of Python provided with your OS, download, install and configure Python. For Windows users, be sure to select the 2.5.4 Windows installer. (Both Linux and Mac OS X ship with a Python interpreter and can skip this step).
Download and manually compile the SIP library from Riverbank Computing. Note: Because we will be using the most recent version of Qt, you will need version 4.8 or greater. As of this writing, SIP 4.8 was still in development, thus get the most recent development snapshot.
Download, manually compile and install the PyQt source code for your platform. Note: Again, because we are using the most recent version of Qt, you will need to download the development snapshot. Be sure that the version on the bindings is 4.5 or greater.

While the general procedure may look quite intimidating at first glance, it is actually fairly simple. The platform specific guides below explain how to do everything step by step. There is some variability between platforms, so be sure to select the guide for your OS.

Linux
Windows - Qt, MinGW, Python, SIP and PyQt
Mac OS X - Qt, XCode, GCC, Autoconf, SIP and PyQt on Mac OS X

Installation of PyQt on Mac OS X

Rob Oakes — Tue, 12 May 2009 21:12:17 +0000

Note: The installation instructions in this article apply to Mac OS X 10.5 (Leopard) and Qt 4.5.3. While it is possible to get Qt 4.5.3 to run on Snow Leopard, it requires compiling it from source. Instructions for doing so can be found in the comments below. It is important to note, however, that there are some minor issues with the 4.5 framework on Snow Leopard (including mutlithreaded memory leaks).

You can find instructions for how to install PyQt on Snow Leopard using the Qt 4.7 frameworks at http://blog.oak-tree.us/index.php/2010/05/27/pyqt-snow-leopard.

To install PyQt on Mac OS X, it is also necessary to install and configure the GCC compilers and the Qt framework in addition to SIP and PyQt. The easiest way to get the most current version of the GCC compilers and tools is by downloading the XCode tools package from Apple at the Apple Developer Connection (ADC). ADC requires that you have an Apple ID and account, which are both free. After you log-in, download the XCode developer tools package, which is a bit hefty (nearly 1 GB in size) and may take a while to download depending on your internet speed.

XCode and Qt

While waiting for the XCode tools to download, you can also begin downloading the Qt binary installer from the Trolltech downloads site. As with Windows, I would advise that you you select the complete SDK (which also include pre-compiled binaries) rather than the source release. If you only download the Qt framework, you will need to compile the source yourself which is both time intensive and is not worth the additional headache.

Once XCode has finished downloading, run the package installer. It will ask you to select the destination drive and what components of the software you would like to install.

This will automatically install and configure the GNU tool chain. Once it has finished, you can access the tools by going to the “/Developer/” folder on the root of the drive.

After XCode has finished, run the Qt installer.

The Qt installer package will automatically place the tools and other things that you will need in the same Developer folder where the XCode tools and frameworks are located. You will need to make a note of the path for qmake. The default installation site for qmake is “/usr/bin/qmake-4.5”.

SIP and PyQt

SIP is a program that makes it easy to create C++ bindings program for Python. You can find the source code at Riverbank’s SIP download page. Since we will be using the 4.5 version of PyQt, you will need to download the latest developer snapshot (version 4.8 or greater). Be sure to get the Linux/Unix source code, rather than the Windows source. You will also need the latest source code snapshot for PyQt 4.5, which is also available from Riverbank.

After you have finished downloading the source files, move them to a folder in your Users directory. I have a special directory entitled “Applications” where I keep the source code for programs that I have manually compiled. Note: The rest of the steps will be done from the Mac OS X terminal.

After you have moved the source code for both SIP and PyQt to this new directory, extract it by using the tar utility with the x and f options (tar –xf):

cd ~/Applications
tar -xf sip-4.8-snapshot-20090430.tar
tar -xf PyQt-mac-gpl-4.5-snapshot-20090507.tar

After you have expanded the files, it might be a good idea to rename the directories to something more manageable (like sip-4.8 and PyQt4.5:

mv sip-4.8-snapshot-20090430 sip-4.8
mv PyQt-mac-gpl-4.5-snapshot-20090507 PyQt4.5

First, we need to compile and install SIP. The default configuration will move the compiled files to a directory where Leopard can’t read them. So, we will manually need to specify the destination directory using the –d flag:

cd sip-4.8
python configure.py -d /Library/Python/2.5/site-packages

After the configuration is finished, run make and sudo make install:

make
sudo make install

Once SIP has finished installing, we need to repeat the process for PyQt. From the sip-4.8 directory, chagne to PyQt:

cd ..
cd PyQt4

Next, run the configuration script specifying the path to your version of qmake and the installation directory for the python bindings:

python configure.py -q /usr/bin/qmake-4.5 -d /Library/Python/2.5/site-packages/

Then compile and install:

make
sudo make install

Since Qt is a rather large framework, it may require between 15 and 30 minutes to fully compile.

Installation of PyQt on Windows

Rob Oakes — Tue, 12 May 2009 18:00:38 +0000

On Windows, we first need to install the pre-compiled Qt binary package from the Trolltech downloads page. Select the LGPL/Free tab and choose the “Qt SDK: Complete Development Environment” for Windows. It is a fairly hefty package (167 megabytes) and may take a while to transfer. The installer includes the core Qt libraries, several important C++ development tools, and an integrated development environment (IDE) called Qt creator. Luckily, the same installer will work on Windows XP, Windows Vista and Windows 7.

Qt and MinGW

When you run the installer, it will ask for the installation destination. To avoid future problems, this folder should not include any spaces. The default is “C:\Qt\2009.02”, but because I like to keep the root directory of my drive as clean as possible, I changed this to “C:\Qt\Windows”. To begin the installation, press “Install.”

The installer will also download and configure the GNU compiler toolset for Windows (also known as MinGW). By default, MinGW will be installed in a subdirectory under the main Qt directory. In this example, it was installed to “C:\WINDOWS\Qt\mingw”. Note this file path, it will be important to the steps below.

Python

After you finish with the Qt framework, the next step is to download and install Python. You can get the Python interpreter and other associated files from the Python release download page. While the newest version is Python 3.0.1, I would recommend that you download and install Python 2.5.4 instead. When trying to set up SIP with Python 2.6.2 and Python 3.0.1, I experienced a number of errors. These were resolved by using the x86 version of Python 2.5.

When you first run the installer, it will ask whether you want to install Python for all users or for just the current users. Make sure that “install for all users” is selected. Like Qt, you should install Python to a directory without spaces. The default install location will be “C:\Python25”, though I typically change it to “C:\Windows\Python25” so that it matches the installation directory for Qt.

After the Python installer has finished running, you will need to modify one of the Windows environment variables. On Windows XP, the easiest way to access the environment variables to right click on “My Computer” and select “Properties” from the drop own menu. When the “System Properties” window launches click on the “Advanced” tab and then press the “Environment Variables” button. (If you are using Windows Vista or the Windows 7 beta, the easiest way to access the Environment Variables is to go to Control Panel and Search for “Path” in the search bar. All other steps remain the same.)

From the “System Variables” list, select the Path Variable and then press “Edit”.

In the variable value box, type in the path to the Python folder you just installed, the path to the “bin” folder of MinGW (which should be under the MinGW folder from above), and last the path to a Qt program called qmake (which is located in the qt\bin folder of the Qt installation) . You should separate these values from the other paths (and from each other) by a semi-colon. If you have followed all of the instructions in this tutorial, you will add:

C:\Windows\Qt\mingw\bin;C:\Windows\Python25;C:\Windows\Qt\qt\bin

And your path variable should now look something like:

%SystemRoot%\System32; C:\Windows\Qt\mingw\bin; C:\Windows\Python25; C:\Windows\Qt\qt\bin

Press “Ok” to close the dialog box and then “Ok” again to close first the Environment Variables and then the “System Properties Pane.” Now that you have changed the Path variable, you need to restart your computer so that the changes can take effect.

SIP and PyQt

Prior to compiling PyQt, you will first need to compile and install SIP, which can be found at Riverbank Software. Though the stable version of the software is currently (4.7.9), you will need to download the latest Windows source code snapshot of version 4.8 (Qt 4.5 doesn’t work with earlier versions of SIP). (Update: Alternatively, you can also find a copy of the 4.8 snapshot here.) You will need the most recent source code snapshot of PyQt 4.5, which is also available from Riverbank. (Update: Multiple people seem to have had problems installing the 4.5 snapshot from the Riverbank website. As a matter of convenience, you can find the April 30, 2009 version here. This is the same version that was used in the writing of this article.)

After you have finished downloading the source files, extract them to the Qt installation directory (C:\Windows\Qt in this tutorial). This might also be a good time to rename the directories to something shorter than the default snapshot names. I have changed mine to PyQt4 and sip-4.8.

The rest of the steps will be run from the command line prompt (cmd.exe) and must be run with administrator privileges to work properly. Start by going to the start menu and then choosing the run command; then type “cmd.exe” and press enter. (If you are on Windows Vista, you can open the command prompt by typing “cmd.exe” in the search dialog of the start menu. To run with administrator privileges, right click on the top program choice and select “Run as administrator” from the context menu.)

At the command line, first, go to the Qt installation directory by typing:

cd C:\Windows\Qt

Then go to the SIP directory:

cd sip-4.8

Prior to compiling, we need to create a proper configuration file:

python configure.py –p win32-g++

After the configuration is done, compile the new file by typing:

mingw32-make

This will take some time. After the files have finished compiling, you can install them by typing:

mingw32-make install

The process is repeated for PyQt. Change to the PyQt4 source directory by typing:

cd ..
cd PyQt4

Then configure the make files, compile and install by typing:

python configure.py –p win32-g++
mingw32-make
mingw32-make install

Since Qt is a rather large framework, it may take between 15 and 30 minutes to fully compile.

Update: If you only want to run PyQt programs, the installation process can be greatly simplified by using the automated PyQt installer for Windows. The installer will automatically install a copy of the Qt framework and the PyQt bindings. You will need to install Python separately, however.

Installation of PyQt on Linux (Ubuntu)

Rob Oakes — Tue, 12 May 2009 17:52:17 +0000

In comparison to Windows and Mac OS X, installing PyQt on Linux is trivially easily. It is likely that all of the software necessary for Python development is likely available in your favorite distribution repository. If you are using the most recent version of Ubuntu 9.04 (and who isn't), then, all you need to do is open up your terminal and type:

sudo apt-get install python-qt4 qt4-dev-tools python-qt4-dev build-essential pyqt4-dev-tools

The package manager (apt-get) will take care of downloading and installing any dependencies. Once the installation routine finishes, you will have a fully working installation of Qt 4.5 and PyQt 4.5 without any compilation, configuration, or other miscellaneous headaches. The qt4-dev-tools, pyqt4-dev-tools and python-qt4-dev packages install important tools used for creating user-interfaces. The build-essential package installs the GNU compilers and autoconf tool chain. If you will only be running PyQt programs, these additional packages are unnecessary.

Creating the Perfect Writing Tool: A Proposal

Rob Oakes — Thu, 05 Mar 2009 02:32:33 +0000

In addition to nouns, verbs, and adjectives; there are other tools which supplement the writer's arsenal. some writers, this is a moleskin notebook and ballpoint pen. For others, writing is synonymous with the usage of computer word processors, such as Microsoft Word. What is a given, however, is that the choice is intensely personal. If you ask ten different writers, "What is the perfect tool?" Expect ten different answers.

For those who create software, however, this is big problem. Programs can't be customized to the individual whims of every writer. (A more diverse or idiosyncratic group does not exist.) It is necessary to select a cross section of features that meet the needs of most people. Unfortunately, however, this has resulted in some serious compromises and an emphasis on the technical process of writing, often to the exclusion of the creative process.

The Problem

Like most writers, I have some strong feelings about the tools I use; and like most writers, I have never found one that fits my needs. But this is not for a lack of trying. In fact, I’ve been searching “perfect tool” for a very long time, at least since I was a senior in high school.

At that time, I enjoyed writing things out in longhand, with separate drafts copied over from the first iteration. I fancied myself a budding novelist and fervently believed that such an exercise would stimulate my creativity.

A decade later, however, I have become impatient. Now, I prefer to peck out my thoughts at the keyboard. Sometime in college, I realized that writing by hand is slow and while it can focus the mind on word choice, it's a plodding and frustrating exercise. In contrast, typing at the computer is quick; and since I can type nearly as fast as I can think, it allows the text to keep pace with my thoughts.

Moreover, working on the computer is organic. There isn't a discrete first, second, and final draft; instead, I can add, modify or change my writing as I form the ideas. If I want to jump back and change a word or sentence, I do so. Pen and paper isn't nearly so adaptable and many ideas are lost before they can be captured and expanded.

The Limitations of Existing Tools

But even though I prefer to write at the computer, I haven’t been able to find a writing program that lets me creatively adapt the structure and flow of my document. I have tried nearly everything available, but nothing fully meets my needs.

Consider Microsoft Word, for example. Even with all of its power, there are a number of things that it doesn't handle very well. Ever tried to work with a complex Word document, numbering into the hundreds or thousands of pages? If not, I don't recommend that you do. Large Word documents cause the program to slow down or crash and they have a tendency to become corrupted. There is nothing more disconcerting than finding out one of your book chapters is no longer usable due to bit rot.

Introducing LyX: The Document Processor

Luckily, there may a potential solution. The open source document processor, LyX, meets many of Word’s shortcomings. LyX encourages people to approach writing based on the structure and content of the document, rather than the appearance. Thus, it isn't a "What You See Is What You Get" so much as a "What You See Is What You Mean" (WYSIWYM).

When used right, LyX focuses the user on writing the content. The program handles the mechanics of layout and output. To do so, LyX leverages the LaTeX typesetting system, which is used heavily in the engineering and science disciplines to publish dissertations, grant proposals, journal articles, and books.

When I first found LyX three or four years ago, it fundamentally changed the way that I worked. Before, I manually formatted everything in a given document. While this approach is okay for simple letters or lists, it quickly breaks down in anything longer than a few pages.

In contrast, LyX makes extensive use of tags that tell the program what a given piece of text is. The title is enclosed by a "title" tag, book sections by the "section" tag, chapter headings by the "chapter" tag, block quotes by the "quote" tag, and paragraph text by the "body" tag. The program then applies formatting rules globally and in a consistent manner. Text within LyX has a meaning and will be processed in a specific way.

And thus we have the first important component of a perfect writing environment: the program is aware of what content is. Such awareness allows for extremely complicated written pieces to be assembled in a relatively easy fashion.

A Sloppy Process

Despite its strengths, though, LyX is an incomplete solution. It focuses on the mechanical aspects of writing and ignores what is arguably the more important component: the creative process. Using LyX, I can assemble a true monstrosity of a manuscript replete with footnotes, endnotes, references, tables, lists of figures, indices, appendices, and all of the other components of something unreadable. Yet such elements are mere after-thoughts rather than the heart and soul of the work. The center is all about ideas.

Which brings me to the serious limitations inherent in most word processing programs: joining ideas together is a sloppy process and they simply don’t facilitate the sloppiness.

Every writer is creative in a different way. I personally start by making a miscellaneous brain dump. Everything relevant to the topic is listed out and then I start to type additional thoughts which seem relevant. After a while, I run out of steam which prompts research, reference collection, reference reading and chicken scratch notes. Then I groan, swear a little, go for a walk, and kick ideas around in my head.

When I come back to the computer, I move things around even further and excise any bits that now seem irrelevant (carefully saving them for later). Usually by this point, I have probably started over a few times and hopefully clarified my goals. If the process works, I get a slightly better fix on direction I intend to take and, eventually, a first draft magically seems to appear. That is when the real fun starts.

Unfortunately, LyX was designed to work linearly. You start at the beginning and continue to the end. All of the words and ideas are supposed to come marching out like perfect little soldiers on parade day. Using such a tool can be extremely frustrating to someone who doesn't think in a linear fashion: I want the freedom to be intuitive!

And in the name of freedom, I will probably use of four or five programs spanning two operating systems: OneNote for data collection, KDissert or FreeMind for MindMapping, and Word or LyX for the actual writing. Then I go back and update my files in the other programs.

There should be a better way! I want to use the same tool from beginning to end, not continually move between the word processor, the outliner, the mind mapper, and the note taker.

A Potential Solution: Scrivener

There just happens to be a program on Mac OS X that was apparently made for scattered people: Scrivener. Scrivener employs a number of simple (yet profoundly powerful) metaphors that let you manage the anarchy of creativity. These include:

An intuitive and simple map of the project (similar to Word's Outline View).
A corkboard that lets you create any number of "index cards" with a title and short summary of the card's contents.
An outline view where the entire structure of your document can be seen and manipulated through drag and drop.

Scrivener leverages its corkboard metaphor and outliner by linking the "index card" to a real piece of text. Thus, the index card actually represents the physical chunk of text in the underlying draft. The index cards can be moved, nested and set into folders and any changes made to the index cards are then instantly applied to the overall structure of the document. Pretty cool, huh? A given index card can represent a section, paragraph, chapter, or sentence. Even better is that cards can be split or joined together easily.

The practical result is that the entire draft is instantly fluid. You can experiment with new ideas, split away sections of text for later use, and re-arrange things as inspiration strikes you. It substantially enhances the organic process already provided by the computer. I know a surprising number of people who have moved to the Mac platform just so that they can use Scrivener.

While other programs (like Semantik, formally known as KDissert) use similar metaphors, the use of a "summary" field is what makes Scrivener stand apart. The summary begins life as an outline point and later becomes the synopsis of the section. It can keep notes, goals, or thoughts in addition to a regurgitation of what is in the text. And no other program uses anything like it. Therefore, while you can replicate many of Scrivener's best features through Word or LyX's outline view, you are stuck manipulating the entire text (which becomes impractical and unwieldy very quickly).

The Outliner. Scrivener includes a high quality outliner that allows individual elements to be repositioned via drag and drop. Each outline item is connected to the text in the actual draft; therefore, changes in the outline view are immediately reflected in the structure of the document. Scrivener's outliner is particularly effective via the inclusion of a "synopsis" field that contains a summary of the text. This summary field isn't part of the full draft, and as a result, can begin life as an outline point that reflects the goal of the section. In contrast, the outline view in Microsoft Word uses the text of the actual document. In long or complicated documents, it is a slightly less than effective way to add new ideas.

The Corkboard. As an alternative to the outliner, Scrivener also provides a "corkboard" where parts of the document can be moved around on index cards. Each card has a title and summary that is is linked to a block of text in the document. Unlike the outliner, however, the corkboard only deals with one navigational element of the document. When at the top level, the cards might represent the parts or chapters of a book. When working on the corkboard of a book chapter, they might represent sections or individual scenes. There is no limit to the number of cards or nesting that can be done.

Scrivener's Singular Strength

Though Scrivener handles the creative process beautifully, it isn't a serious word or document processor. On the program's website, the primary developer states that Scrivener is a "personal writing tool." As a result, it seems uniquely geared to creative writing; that is to say, novels and screenplays. Not many novels or screenplays have a need for a bibliography, index, appendix, footnotes, or endnotes. As a result, Scrivener cannot handle the minutiae of complicated documents.

Just forget support for hyperlinks, cross-links, cross-references, figures, legends or graphics. They're outside the developer's stated need and are unlikely to appear. That's too bad, because it makes Scrivener worthless to those who need support for these features. Microsoft Word has them and LyX has them, OpenOffice pretends that it has them. However, Word, LyX, OpenOffice and other word processors completely lack the creative tools that make Scrivener a joy to use.

A Glorified Text Editor. The weakest component of Scrivener is the actual word processor. Advanced (and even not so advanced) features like footnotes, endnotes, cross references of links are completely unsupported. It is simply unsuited to writing a complicated document like an article or cross-referenced book.

A Combined Approach

The simplicity of Scrivener's word processor causes a semi-crisis. If given my choice, I would compose nearly everything in it for the convenience of its outliner and index cards. However, it isn't up to supporting the type of heavy lifting I am often asked to do. While I could use it as my creative crutch, that worsens the overall problem. Instead of four programs across two platforms (Window/Linux), I would be using five programs across three platforms; not what I'm aiming for. I want to simplify my work process, not make it more complicated.

I've tried to find what I want in the software of others and determined that it doesn't exist. The fallback is to develop it myself by turning to opensource solutions. Since LyX already meets my needs as a document processor, it is a great place to start. As a front-end to LaTeX, it automatically supports the advanced features I need. It is cross-platform, which means that the writing environment would be the same regardless of whether I am working on Windows, Mac OS X, or Linux. Additionally, it has deep integration with revision control systems, which greatly simplify collaboration. Last, it is available under the GNU public license, which means that I can access the source code. LyX combined with the same creative tools found in Scrivener would be nearly perfect.

Almost There: LyX's Existing Features

After a careful review of LyX, it becomes obvious that much of the functionality required for a robust outliner and corkboard is already present. At left is the outline panel in LyX. The outline is automatically generated from the heading information in a given document and any child documents. The particular example shown here is series of three essays kept in four different files: a master document and three child documents.

LyX already recognizes blocks of text and understands their contextual meaning. In fact, you can already specify the order of any element and what position it holds. Further, individual text elements can be moved from one child document to another. In other words, nearly all of the background elements for a Scrivener-like outliner or corkboard are already present.

Form and Function

What is missing is the ability to assign a summary or "synopsis" to a given chunk of text. This would ideally exist as associated meta-data and affiliated with a particular tag. Additionally, it might prove convenient to associate other meta-data in addition to the summary.

To add the "creative elements," most development could focus on creating a user interface that blends the existing outline view with the functional equivalents of the corkboard and outliner from Scrivener. Convenient drag and drop of document sections from within the document map, outliner, or corkboard would allow text to be moved and positioned. The outliner would show the complete document with the option to expand or collapse a given section. The corkboard, in contrast, would show only the elements contained in one particular outline node.

The figures below show an example of what such a user-interfaces might look like. The first figure shows the outliner and the second shows the corkboard. (Though the index cards and outline points of the example correspond to the chapters and sections of the document map, this should not necessarily be required in an ideal implementation. Instead, the user would be able to determine which index cards correspond to sections and which correspond to a specific piece of evidence, argument, or narrative. That is to say, the user should be allowed the freedom to create the outline structure, rather than have it imposed.)

Proposal

I propose to create an add-on for LyX which would combine the best features of Scrivener with the already powerful capabilities of LyX. As described above, this would consist of two inter-related tools:

1) The Outliner. The outliner would be an expandable panel where the structure and rough content of a document could be laid out prior to beginning the first draft. Specific features of the Outliner would include:

a. Each outline point would have room for a title, synopsis and type of content (drawn from the list provided by the LaTeX class).

b. Individual nests in the outline can be expanded or collapsed.

c. Elements can be dragged and dropped from one point to another. The underlying document will be updated to reflect any changes.

d. Though most points in the outline might correspond to headings (e.g. book parts, chapters, sections, or subsections), this would not be required. Instead, any block of text could be assigned to a point in the outline. It would behave in precisely the same manner as other outline elements.

2) The Corkboard. Housed in the same panel as the Outliner would also be a digital Corkboard which would display a small "Index card" with the title of an outline element in addition to the synopsis. However, in contrast to the outline, only the individual elements for a selected point in the document map would be shown. In the example below, the sections of a specific chapter, "Eragon Shadeslayer: Sociopath" are shown with their corresponding summaries. Changes in the order of the cards would be reflected in the underlying draft.

Conclusion

The proposed additions to LyX would further expand its other excellent features, making it the only document/word processor with a powerful set of creative tools. By associating blocks of text with additional meta-data (like a synopsis), working with the document structure would become a visual and interactive process, thereby allowing for much greater flexibility. Indeed, such changes would bring LyX one step closer to writing perfection.

Note: A PDF copy of this article can be found here.

IronPython in Action

Rob Oakes — Mon, 09 Feb 2009 20:46:53 +0000

There is an old adage, often used and much abused, “Give a man a fish and he will eat for a day. Teach a man to fish and he will never want for food.” And while this old saying is extremely tired and very much cliché, it is a surprisingly apt way for me to begin a review of “IronPython in Action,” a programming book by Michael J. Foord and Christian Muirhead.

“IronPython in Action” delves into the use of scripting (or dynamic) languages and how they merge with Microsoft’s enormous .Net platform. For readers of this site, it shouldn’t be any secret that I have been working hard to teach myself Python. And while that particular adventure is better documented elsewhere, “IronPython in Action" has been central to it. Indeed, it has been my primary reference manual and guide.

It’s been nearly six months and it’s far past time to write a review of the guide. But this review is going to be a little bit different than others that you might read. If you are looking for a more traditional summary, you might be better served by Craig Murphy’s exhaustive breakdown. Rather than detail the contents and render an opinion, I would like to first tell a story and then focus on a much narrower question: Is IronPython valuable for people who are new to Python or .Net programming? More to the point, can it be used to teach a novice “how to fish?”

Backgrounds and Purposes

To understand my take on Foord’s book, it is important to understand my background. You see, I am not a computer science person. I am not a programmer. I am first and foremost a research scientist and then an engineer. Most of the time, I am concerned about refining and extending the technology of others; not creating it from nothing. And while I occasionally need to write a bit of code, it usually falls into a realm that most real programmers treat with disdain: scripting. This explains why, for the past five years or so, I have had an ongoing (if rather tempestuous) love affair with a technical computing toolbox called Matlab.

During those years, I have helped The Mathworks Company (creators of Matlab) to line their pockets and strengthen their stranglehold as the de-facto language for technical computing, I have also had a long string of much-wiser computer people tell me that I should consider OpenSource alternatives. (Anyone who is at all familiar with computers should be aware of the OpenSource/Proprietary divide. If you’re not, it is very similar to the Windows/Mac divide; except substitute flame-throwers for flame-bait.) OpenSource certainly has its advantages: it’s free, you can audit the code, there are a huge number of well written extensions, it will never go away, and it has a very enthusiastic community. It also has some severe weaknesses: documentation sucks, there’s no technical support, and it has an overly enthusiastic community. I was particularly hung up on the bit about sucky documentation.

And then, about 18 months ago, something happened that changed my opinion. Instead of trying to apply existing tech to new problems, I and my research group found ourselves looking at some very complex issues that no one had really studied before. It didn’t take long to realize that our existing tool set was completely inadequate. The group desperately needed alternatives to Matlab and I found myself burdened with a rather unpleasant research project. I needed to find and screen said alternatives. This started with Perl, proceeded to Ruby, and eventually ended up at Python.

I quickly decided that I liked what I saw. Python was fast, flexible, and had a very simple (yet elegant) syntax that could be taught to students and technicians familiar with Matlab. There was a large number of extensions available and it was cross platform. There was, however, a major problem: Python lacked bindings for the Insight Toolkit (ITK), an essential framework for people who work with medical imaging data. Cue despair. But … after yet more research, I found an acceptable workaround. While Python might lack bindings, C# does not. Coincidentally, there happened to be a Python implementation which allows for C# code to be used like native Python code: IronPython.

More importantly, however, all of the advantages were retained. IronPython is Python. The syntax still works, many of the extensions will load and run, and (through the use of Mono) IronPython is cross platform. There are additional benefits as well: .Net grants access to an enormous number of very high quality C# libraries. Thus, after careful consideration (lasting about 15 or 20 seconds), I made the decision to move our nascent code from Matlab to Python. There was just one problem, I and essentially everyone else would need to learn Python and .Net from scratch, which might prove a formidable challenge.

Learning IronPython

This is where “IronPython in Action” makes its first appearance. The book was first recommended by a friend familiar with my panic-stricken, "make things up as I go along" style. He said it was well written and would be a good first read for someone with some (but not a whole lot) of computer programming experience. I thanked him for his suggestion and purchased an electronic copy of the work in progress via the Manning Early Access Program (MEAP).

At the time, I had a list of very specific needs:

Learn Python, its frameworks, syntax, and shortcomings.
Get a whirlwind introduction to .Net (Mono), its architecture and how IronPython fit into the big scheme of .Net dynamic languages.

“IronPython in Action” fulfilled both, after a fashion, in addition to several more that I didn’t recognize as needs. Of these unrecognized needs, the most important important was facing my own ignorance. You see, IronPython helped me start to fill a few of the holes in my computer science education. Being an engineer and not a programmer, I always considered myself above actually learn how to code. I much preferred a “practical” view of coding best be articulated as, “Make it work! Now!” This “practical” view was most often delivered to an underling, at the top of my voice, prior to a major deadline. And, as might be guessed, it didn’t always lend itself to an elegant, reusable, or even a useful end-product.

Fortunately, Python (by design) encourages all of the above: elegance, use and reuse. More importantly though, Foord and Muirhead stress these concepts from the moment that you open their book. Rather than delve immediately into the gritty of examples, Chapters 1 to 3 insist on providing a comprehensive overview and background to .Net, Python and how the two integrate. To return to the fishing analogy: instead of baiting the hook and getting the reader started immediately, Foord instead opts to describe the tackle and trade tools and how they are used.

This can be quite frustrating to someone who wants to deal with the “practical” world. After all, I want to code, not re-learn how to think! Except … that is exactly what I needed to do. As I read through “IronPython in Action,” I found myself drawn to the many footnotes (29 in the second chapter alone) and references so that I could learn more about a given topic. Instead of tackling syntax, frameworks and shortcomings I found myself reading a great deal about object oriented, procedural, and meta programming. More fatally, instead of focusing on “just making it work,” I was entertaining radical thoughts like “how does it work” and “can it be improved.” Such introspection approaches sacrilegious among “practical” people.

Thus, when the authors do begin to provide examples during Part 2 of the book, the reader is in a much better position to make good use of them. Essential lessons, the equivalent of baiting hooks and making a cast, have already been covered and Foord is able to begin sharing insights into how to successfully “fish.” And there are plenty of said insights covering most major topics of interest: Windows Forms, Web Development, and even Windows Presentation Foundation (WPF). The chapters covering system administration are just a bonus.

An Education

At every point, Foord and Muirhead seem to hold to another old adage, “Rather than ‘teach,’ I would prefer to help you learn.” In doing so, they encourage and empower their readers to explore well beyond the examples and techniques covered within the pages of “IronPython in Action.”

In case it isn’t clear, I heartily recommend that anyone interested in learning Python or .Net purchase this book. It certainly has its idiosyncrasies, but it still manages two roles very successfully: 1) introductory text and 2) reference manual. While novice readers (like myself six months ago) will find tremendous value in reading the book from cover to cover, more experienced hands will find value in skimming individual chapters. As someone who has written technical documentation, I can testify that such a feat is extremely difficult to pull off.

While this style is particularly valuable to the novice, it might not work to the benefit of everyone. More advanced readers might desire to see more extensive/complicated examples or a more thorough overview of both the Python and .Net frameworks. But ... I suspect that these individuals will be few, after all, that information is provided by the respective Python Standard Library and Microsoft documentation libraries.

In conclusion, I can only congratulate the authors on this excellent introductory work. While I initially set out with a desire to learn a skill, I instead received an education. In the process, I got excited about programming, Python, and OpenSource. To answer my own question: Can IronPython in Action be used to teach a novice how to “fish?”

Most definitely.

Apolitically Incorrect » Programming

Time Drive 0.3: Better, Easier, More Refined

Better

Logs and Notifications

Progress Indicators for Absolutely Everything

Easier

More Refined

Looking Toward the Future

Additional Announcements

Conclusion

Similar Posts:

Installing Duplicity on Mac OS X

XCode

GnuPG

GnuPG Interface – Python Module

NcFTP

Boto – Amazon S3 Support

Duplicity

Similar Posts:

Install Time Drive On Ubuntu and Other Debian Linux Distributions

Background

Installation Instructions

Step 1: Add the Package Repositories to the Ubuntu Software Sources

Step 2: Configure Apt to Trust Time Drive and Duplicity Packages

Step 3: Install Duplicity and Time Drive

Application Updates

Similar Posts:

Time Drive 0.2 – A “More S” Release: More Stable, More Secure, More Settings and Now Supports Amazon S3 Storage

More Stable and Secure

More Settings

Amazon S3 Storage

Conclusion

Similar Posts:

Filing Bugs for Time Drive or LyX-Outline

Step 1: Go to the Bug Tracker Website

Step 2: Go to the Appropriate Project Page

Step 3: Complain Loudly, Thoroughly and Convincingly

Similar Posts:

Time Drive 0.1.5: Incremental Changes

Time Drive Update

New Stuff

Conclusion

Similar Posts:

Backup for Linux, Done Right - Part 2: Time Drive 0.1

Over the Network or Across the World

Restoring Files

Set It, Then Forget It

Notifications and Feedback

Beta Software

Conclusion

Similar Posts:

Backup for Linux, Done Right- Part 1: A Mini Melodrama

Backgrounds and Backends

Basic Needs

Duplicity

Frontend and Features

Similar Posts:

Introducing LyX-Outline 0.1

Looking for Flexibility

A Quick Overview

Item Summary and Organization

Of Outlines and Index Cards

Minding the Potholes

Similar Posts:

Cross Platform GUI Development with Python and Qt

Part 0: Thoughts on Licenses, Setup and Installation

Installation, Setup and Configuration

Similar Posts:

Installation of PyQt on Mac OS X

XCode and Qt

SIP and PyQt

Similar Posts:

Installation of PyQt on Windows

Qt and MinGW

Python

SIP and PyQt

Similar Posts:

Installation of PyQt on Linux (Ubuntu)

Similar Posts:

Creating the Perfect Writing Tool: A Proposal