Apolitically Incorrect » Computer

Hannibal, Napoleon, and Joseph Charles Minard

Rob Oakes — Mon, 22 Feb 2010 21:49:11 +0000

No study of the history of scientific communication can be complete without mention of Joseph Charles Minard, a 19th Century French civil engineer and cartographer.

At the end of his life, Minard created two very famous examples of statistical charts, called flow maps, that every scientist, engineer and student should be familair with. The first showed Hannibal’s crossing of the Alps (218 BC, Second Punic War), and the second describes Napoleon’s disastrous invasion of Russia (1812-1813).

Both examples are beautiful works of art and masterful examples of evidence. But they are also more than that, they tell cohesive and interesting stories. In this post, I thought it might be interesting to take a closer look at the history of Hannibal and Napoleon, and highlight the ways which Minard’s charts help us to explain their eventual outcome.

(Note: High resolution, PDF versions of the two maps are available for download. These versions have been translated from the original French. To download, either click on the images, or here for the Hannibal invasion of Northern Italy, and here for the French Invasion of Russia.)

Hannibal Invasion of Italy (218 BC, Second Punic War)

Hannibal was perhaps the single greatest threat to Roman power to ever live. At the same time that Roman legions marched unopposed through most of the Mediterranean world, Hannibal lead a force of twenty-five thousand soldiers into the heart of Roman territory. He then spent the next fifteen years occupying portions of the Italian peninsula and generally wreaking havoc. But even though he ravaged the Roman countryside, Hannibal was never successful in subduing Rome. Ironically, this might be due to one of his greatest exploits: the overland crossing of the Alps.

Hannibal departed his home base in Spain with nearly 90,000 troops (one of the largest armies that had ever been fielded at that point in history) and had a number of early successes. He marched north over the Pyrenee mountains and into Gaul (modern day France). He subdued the Celtic tribes and crossed the Rhone River before the Romans could take measures to stop his advanced. He forged important alliances and found unexpected friends in his fight against the Romans.

After crossing the Rhone, however, Hannibal’s luck began to change. He received word that a large Roman force was marching from the Mediterranean coast to intercept and destroy him. As the force outnumbered him by a significant margin, Hannibal was faced with a difficult choice. Should he stay at his current location and engage the Roman army or retreat and prepare a defense?

Not content with either choice, Hannibal ultimately chose a third option; which, ironically, would have enormous consequences for his overall campaign. To evade the Romans, Hannibal took the unprecedented action of turning inland and marching over the Alps. (It is thought that he probably crossed over the valley of the Drome and south of the Col de Montgenevre or near the Col de Mont Cenis.) This had never been done before – indeed many thought it impossible – and has long been praised as a brilliant tactical decision.

Minard’s reconstruction of Hannibal’s journey shows the journey from Spain, through Transalpine Gaul and eventual arrival in Italy, allows it to be seen a slightly different light, however. Through use of a flow line, Minard shows how the strength of Hannibal’s army waned through the march, with a disastrous loss of twenty thousand men while passing over the Alps. This steady loss of strength, subtly reinforced by Minard’s map, helps to explain why Hannibal failed to subdue Rome.

Hannibal’s brilliant tactical decision was also a strategical disaster. It came at a dramatic loss of men, war elephants, and material with the most disastrous loss being the destruction of his siege engines. Without the siege engines, Hannibal was never able to successfully attack Rome. And eventually the Romans realized this and changed their tactics. Instead of engaging Hannibal directly, they instead fought a long-pitched war of attrition and eventually destroyed the one resource he couldn’t replace: his men. Thus, even though crossing the Alps helped Hannibal to win a few stunning victories, the decision eventually caused him to lose the war.

Napoleon’s Invasion of Russia (1812-1813)

Like in the case of the Hannibal, Minard’s depiction of the 1812-1813 invasion of Russia also tells an extraordinary story. It combines information about the size of the army, geography, sub-campaigns and temperature into a single narrative. This time, though, it is anything but subtle.

Note: Because of the elegant management of all this information – at any time showing the army’s direction, size, and loss relative to the start – some statisticians and data visualization experts (such as Edward Tufte, Etieene Jules Marey and Howard Wainer) have declared Minard’s depiction of Napoleon’s Invasion of Russia to be the greatest numerical graphic ever created.

From the very beginning, the army met with tremendous hardship. The cold temperatures and lack of available resources took an enormous toll. Napoleon had planned on scavenging for supplies to support his massive army, but the Russians had destroyed all of their crops and burned their villages so that Napoleon wouldn’t be able to make use of them. As a result, the army was heavily reliant on slow supply trains from Europe that were unable to keep pace with its quick advance. Hundreds of thousands died from starvation and exposure, greatly outnumbering those lost in combat.

Minard’s map shows this trend of devastating loss. Napoleon’s Grand Army starts out as a wide, brown line to the left of the map. As the army progresses across Russia, the number of men decreases dramatically. A few ribbons of men are dispatched to other targets, but by far the majority succumb to the elements.

Eventually, a substantially reduced force (100,000 men, about 20% of those who began the campaign) arrived in Moscow. What they found did little to improve their circumstances. The city had been abandoned and burned to the ground. No food was available. The only available option was to return to France.

Demoralized and broken, the Grand Army began a slow retreat to the west. Their progress is shown in the black. As they crossed the vast Russian wilderness, the temperature turned bitter cold (shown in the bottom third of the map). When the army arrived at the Niemen river, there were a scant 10,000 troops. More than 400,000 had died.

Such a devastating loss of life crippled Napoleon’s young empire and left him open to invasion by his enemies. He was defeated later in 1813 and exiled to the island of Elba.

Conclusion

Minard’s graphics are successful because they demonstrate powerful trends and place evidence within its proper context. In the case of Hannibal, it shows how he began in a position of strength and then undermined himself through a brilliant tactical innovation. In the case of Napoleon, it shows how distance and temperature ravaged an army and destroyed an emperor.

The clear evidence and thoughtful presentation define the story, they show where things are headed and help to foreshadow the inevitable conclusion. For this reason, they are powerful examples of clear communication.

____________________________________________________

Unrelated Thoughts: Re-Visioning Minard with Open Source Tools

Since first seeing these charts as part of an undergraduate statistics course, I’ve been obsessed with them. I even have a poster sized copy of the Russian invasion (available from Graphics Press) hanging in my office. Given this obsession, it should come as no surprise that I’ve wanted to include the maps in my book.

There is, however, just one problem; there is not anything new to say about them. Edward Tufte has held up Minard as a “Da Vinci of Data” for the better part of thirty years. Nearly every one of his books includes a copy of the maps (in the case of Beautiful Evidence, his most recent work, there is an entire chapter devoted to the graphics; including eight different reproductions). Other than, “They’re beautiful, study them carefully, read Tufte’s analysis,” there just isn’t that much to say.

That is, until I started looking for examples of scientific illustration that could be reproduced using the vector illustration program, Inkscape. Any such example should have simple lines and clean integration of text and images. I’m not sure that a better description of Minard’s work exists.

The two images above (and attendant PDFs) were my first pass at re-drawing the maps. And I have to say, I’m rather happy with the result. For the most part, I was able to get a very faithful reproduction (with the exception of the type face, which I deliberately changed because I hate script fonts), and only required a couple of hours worth of work.

I think that I may have found my book examples. It allows me to include one of the best statistical charts of all time, talk about flow maps, and even include some interesting how-tos which show how the graphics can be built in an automated fashion. Which, come to think of it, might just count as a new contribution after all.

Acknowledgements and Further Reading

More information on the book can be found at the publisher’s website.

The source images for the two maps included “Re-Visions of Minard” by Michael Friendly (1999) and the reproductions in “Beautiful Evidence” by Edward Tufte. Translation for the Hannibal map was adapted from those available on Edward Tufte’s website (Dawn Finley) and sources available at Re-Visions of Minard. Translation for the Russian campaign map were taken from the reproduction in Edward Tufte’s “Beautiful Evidence”, Wikimedia Commons and sources available from Re-Visions of Minard.

Additionally, the Re-Visions of Minard Website include a number of interesting adaptations of the Napoleon graphic that might be of interest, including both Matlab and GGPlot2 reconstructions.

Typeset Your Curriculum Vitae – Part 3: Automatically Generate a List of Publications

Rob Oakes — Wed, 02 Dec 2009 15:19:44 +0000

Publications are the currency of ideas. Through them the experts, thinkers and dreamers of this world can share their thoughts and insights. A good publication is not only influential, but it’s even capable of shifting the course of a whole society, as Martin Luther King demonstrated with his “Letter from a Birmingham Jail”.

Since publications are so important to the dissemination of knowledge, there is a rather high expectation that an academic author should publish prolifically. The mantra “Publish or Perish” is not just a clever quip, but a very serious way of life.

It is ironic, then, that the most prolific of academic writers can suffer from a surprising problem: it can be very difficult to keep track of all of their work. Yet, an up to date CV is very important. After all, publishing your work in influential journals is an important first step toward establishing tenure!

Members of a research team or those who collaborate outside of their institution experience this same problem, only more so. Such a person may work on many projects at once, but only have direct responsibility for one or two of them. This places the researcher in the unenviable position of trying to track the work of others. This situation becomes even more complicated if the collaborator refuses to play by the rules of common decency.

It would be nice, for example, if the primary author of a publication would notify the co-authors of its progress, or when it has been submitted. But … that doesn’t always happen. Academic researchers are busy people and soliciting feedback from all of your collaborators can be difficult … and there is a tendency for difficult things to go undone. Thus, if you don’t follow what your team mates are working on, it is quite possible that an abstract might have gotten submitted while your back was turned.

To stay on top of the “delightful chaos”, you need to have some kind of system. Personally, I keep my list of projects and publications in three places. The first (and perhaps most important) is the hand-written list in my experimental notebook. Any time I hear about a new project, it gets added to this list. I keep track of what I’ve contributed, what papers or abstracts have been created from the data, and what their status is. When I know that an abstract or paper has been accepted, I then create an entry for the item in my bibliography manager. Once in the bibliography manager, I can cite the reference in other documents such as proposals or related papers.

About once a year, I go through the tedious process of updating my CV. This typically involves manually sorting through both my project list and my reference database and account for new items or reconcile differences. Every time I do this, it’s painful; and because I’ve historically formatted the reference list by hand, it’s not uncommon for a typo to sneak its way in or for an author to accidentally get left off of a citation. These mistakes are never intentional, but they do happen.

When I find such an error in the reference database, I fix it. But since I often import these references from websites, the errors tend to be few and far between. Moreover, my reference database is something that I use every day; as a result, it gets a lot of scrutiny. My CV, on the other hand, gets updated much less frequently and errors tend to persist longer.

For a very long time, I’ve wanted to automate the process. Instead of keeping three separate lists – active projects, reference database, and CV – I’d prefer to keep only one (or two). But I’ve never found a really satisfactory way of doing so. Or at least I hadn’t found a system until quite recently.

In my last review of different ways to typeset a CV, I came across an interesting article by Dario Taraborelli. In it, he described how to create a CV based on the standard “article” document class. It was well designed, elegant, simple and attractive. From his work, I created the xetexCV document class. Additional research turned up an add-on module that makes it convenient to automatically generate a list of publications. So, for the first time in a great while, I have finally found a way to automatically generate a publications list in a simple and automated manner. In this article, I will demonstrate how that is done.

The Basic Requirements

For any automated solution to be successful, it needs to meet three primary criteria:

When generating a list of references, bibliographic details should be pulled from the database. This ensures that the information is always up to date.
The citation style must be customizable. I want the ability to specify whether my CV includes page numbers and hyperlinks. Not all citation styles do.
The publication list should be broken into different categories. Book chapters should not be listed with peer-reviewed journal articles, nor should abstracts be listed amongst the books.

cvsplitbib and xetexCV

All three of these requirements can be met by using the xetexCV document class and the cvsplitbib style (both are provided in this zipped directory). cvsplitbib is a version of regular splitbib, a package that makes it possible to create categories in your bibliography. While all of the main features work as described in the splitbib manual, the macros have been changed to use the \cvsection and \cvsubsection macros of the xetexCV document class. This ensures that formatting and indentation of the publications list is consistent with the rest of the document.

This combination also has an additional benefit. Because cvsplitbib works entirely through LaTeX, any bibtex citaiton style can be used. In the example below, I’ve used the the “plain” style, though it would have been just as easy to use ama, MLA, chicago or any other style in my distribution.

Note: There is an important difference between cvsplitbib and splitbib. In the standard version of splitbib, it is possible to create subsubcategories (a second level of nested references). Cvsplitbib does not support this as xetexCV does not define a \cvsubsubsection macro.

Creating a Categorized List

Creating different bibliography sections is achieved through the use of the “category” environment. The arguments of the environment define the title of the section while the \SBentries macro is used to specify the citation key of any reference that should be included in that particular list.

Note: Citation keys should be added as a list separated by commas, any spaces will result in an error.

The following example creates a categorized bibliography with three different sections (Novels, Children’s Literature and Comics, Graphic Novels). The bibliography data is pulled from the file “NeilGaiman-Publications.bib”, part of the xetexCV examples (see below):

\begin{category}[A]{Novels}
\SBentries{Gaiman2001-American-Gods,Gaiman2005-Anansi,
Gaiman1990-Good-Omens,Gaiman1996-Neverwhere,
Gaiman1999a-Stardust,Gaiman1985-Ghastly}
\end{category}

\begin{category}[B]{Children’s Literature}
\SBentries{Gaiman1998a-Goldfish,Gaiman2008-Graveyard}
\end{category}

\begin{category}[C]{Comics and Graphic Novels}
\SBentries{Gaiman1994b-Angela,Gaiman1993-Angels,
Gaiman1989-Black-Orchid,Gaiman1993a-
Books-Magic,Gaiman1994a-Death-Living,Gaiman1997-Death-Time-Life,
Gaiman2000a-Green-Lantern,Gaiman2003-Marvel-1602,
Gaiman1999-Midnight-Days,Gaiman1994-Anthology-Virus,
Gaiman2000-End-World,Gaiman1992-Signal-to-Noise,
Gaiman1998b-Smoke-Mirrors,Gaiman1998-Inkeeper,
Gaiman1991-Sandman,Gaiman1987-Violent-Cases}
\end{category}

Any reference that is not included in a specified category will be added to a fourth category, titled: “Miscellaneous”.

Generating the Bibliography

Once the various references have been categorized, you can generate the bibliography as you would in any other document:

\nocite{*}\bibliographystyle{plain}
\bibliography{NeilGaiman-Publications}

Here the \nocite{*} command is used to list all of the publications in the .bib database. \bibliographystyle{plain} specifies the BibTeX style to use, and \bibliography{Database-Name} tells BibTeX which database file to pull the information from. Part of the formatted output from the example is shown below.

Examples and Class Files

Document Class and Examples (.zip). This file contains the document class, cvsplitbib package, examples and other supporting files.
Neil Gaiman (Automatic Bibliography). LaTeX Source, PDF Output, Reference Database
Albert Einstein. LaTeX Source, PDF Output
Isaac Newton. LaTeX Source, PDF Output

______________________________________________________

Acknowledgements and Further Reading

The xetexCV document style is based on work from Dario Taraborelli’s website. The cvsplitbib package is a modified version of splitbib, by Nicolas Markey.

This article is part 3 of a four part series. Part 1 introduces the xetexCV document class and describes its use. Part 2 dissects the code and explains how it works. Part 4 describes how to use the document class and a corresponding layout file with LyX

Typeset Your Curriculum Vitae – Part 2: Extending and Customizing an Existing Document Class

Rob Oakes — Mon, 30 Nov 2009 18:54:16 +0000

Many first-time users of LaTeX often mistakenly look at the language as a a type of glorified word processing software – albeit a particularly complicated one. While such an analogy may be apt in helping new users become acclimatized to the language, it suffers from a rather nasty problem: LaTeX isn’t a word processor.

If anything, LaTeX shares more in common with a programming languages than any type of application. In fact, the document processing system is really nothing more than a bunch of re-usable pieces of programming called macros. Everything is a macro. That includes the commands that every user is familiar with: \title{}, \section{}, \subsection{}; in addition to the internal formatting commands that allows LaTeX to function. (Most of the macros were originally created or packaged by Leslie Lamport as a way of making TeX – the typesetting system created by Donald Knuth – easier to work with.)

This has some rather practical consequence; because everything in LaTeX is a macro, it is far more extensible than a word processor could ever hope to be. If you require a feature that doesn’t yet exist, it typically isn’t all that difficult to add it. And when your extension is packaged inside a style or class, you can use those customizations in anything that you want to write.

But though creating macros isn’t particularly complicated, it is a different beast than just using the stock macros for writing. This is not surprising, the craft of design is inherently different than the craft of writing. There are different conventions to follow and different topics to obsess about. In the first article of this series, I introduced the xetexCV document class, which is one example of where I decided to don the designer hat.

But before you get too far down the road of customizing and extending, there are a some important things that you need to know. These include the general conventions used when working with document classes, their internal anatomy, an understanding of how macros are created, and how to handle formatting and layout challenges. In this article, I will look at these issues more in detail, particularly as they pertain to xetexCV. In the process of reviewing these topics, I will also explain some of my design choices.

Understanding Conventions

When trying to customize or extend a document class, perhaps the very first thing to understand is that there are certain conventions. The document “LateX 2E for class and package designers” describes them this way:

LaTeX has three types of commands.

There are the author commands, such as \section, \emph and \times: most of these have short names, all in lower case.

There are also the class and package writer commands. Most of these have long mixed-case names such as the following:

\InputIfFileExists \RequirePackage \PassOptionsToClass

Finally, there are the internal commands used in the LaTeX implementation, such as \@tempcnta, \@ifnextchar and \@eha: most of these commands contain @ in their names, which means they should not be used in documents, only in class and package files.

These conventions help to better separate procedural formatting from the descriptive tags used in writing a document. But it also makes it easier to identify what different sections of the code do. Sections that are responsible for importing and configuring packages will typically use mixed case names, while variables will contain the @ symbol, and user facing macros will be in lower case.

The Internal Anatomy of a Document Class

The next important thing to understand is that all document classes have a few well defined components. That is to say, they have an internal anatomy which contributes to their function and physiology. The internal bits include the class description and required statements, a block that imports and configures packages which new features will depend on, and a section where new macro commands are created or existing commands are redefined.

Class Description and Required Statements

\ProvidesClass

The very first such section gives the package name and description. Also included is a definition of the most recent version of LaTeX that the document class will work with. The relevant code from xetexCV is shown below:

\NeedsTeXFormat{LaTeX2e}
\ProvidesClass{xetexCV}[2009/11/30 – Modern looking résumé which uses
the xetex typesetting system]
…

Here, the \NeedsTeXFormat{LaTeX2e} command is the earliest version of LaTeX that the document class will work with. \ProvidesClass{xetexCV} identifies the class name (xeteCV) and everything in square brackets is the document description.

\LoadClassWithOptions

Generally speaking, there are two major types of LaTeX document classes: those that are free standing, and those that are variations of other document classes. The standard classes (article.cls, report.cls, and book.cls) are examples of the former. They contain all of the code needed to compile and work without additional extensions. xetexCV, is an example of the second type.

To load an existing class as the foundation, you can use the \LoadClassWithOptions macro:

…
\LoadClassWithOptions{article}
…

xetexCV uses “article” for its foundation. For that reason it is likely that other packages and styles will work without conflict.

Building on the Foundation

\RequirePackage

After the base class has been loaded, the new package is fully functional, if identical to the foundation class. The next step in the customization process, then, is to begin adding new features and modifying old ones. In the case of xetexCV, there were several changes which I wanted to make. First, I wanted to use an 8 1/2 inch by 11 inch paper (US letter) by default. I also wanted to set a 1 inch margin. Next, I wanted to use the XeTeX typesetting system and a pair of advanced OpenType fonts.

These changes are made through the use of other LaTeX packages. In a document class, add-on modules are loaded through the \RequirePackage command. Below is the related code from xetexCV:

…
\RequirePackage{fontspec}
\RequirePackage{xunicode}
\RequirePackage{xlxtra}

\RequirePackage{graphicx}
\RequirePackage[colorlinks]{hyperref}

\RequirePackage{ifthen}
…

The first three packages (fontspec, xunicode, and xlxtra) are used by XeTeX to set fonts and provide unicode support. fontspec, in particular, is very important. It provides convenience macros that make it easy to change the the various font families: \setmainfont{fontname}, \setromanfont, \setsansfont, and \setmonofont. It also provides a way to manipulate font mappings, color, scaling, and inter-word spacing. graphicx is used to add images to the document and manipulate their size.

hyperref makes it possible to add clickable hyperlinks and bookmarks. The [colorlinks] option specifies that these special links should be rendered in a different color than other text. The hyperref package is then configured using the \hypersetup{} macro. By default, I’ve configured xetexCV to use blue for urls and webpages and black for filenames:

\hypersetup{linkcolor=blue,citecolor=blue,filecolor=black,urlcolor=blue}

The ifthen package provides several macros that make it easier to apply conditional formatting. These will be discussed in more detail below.

Creating New Features and Modifying Old Ones

The last major section of a document class is where new macros are defined and old ones are modified. As might be expected, this section typically accounts for the majority of the code.

\def and \newcommand

There are two major commands used to define macros and variables: \def and \newcommand. \def is a TeX primitive while \newcommand is a LaTeX macro that checks for name clashes or other problems. In many cases, they may be used interchangeably or in concert. Consider the code block below which defines the \cvname macro:

…
\def\@cvname{\relax}
\newcommand{\cvname}[1]{\gdef\@cvname{#1}}
…

\@cvname (according to the LaTeX conventions) is a variable/macro which should only be used from inside the LaTeX document class. Consequently, it needs a user-facing macro that can be used to assign it a value. This is what \cvname does. In this example, the first line of code creates the variable and assigns it the LaTeX equivalent of a null value (\relax). The second line is a little more interesting. It defines a macro called \cvname with one argument [1]. Then, it takes the \cname argument and assigns it to \@cvname.

A second, related example can be seen in the definitions of \@cvimage and \cvimage:

…
\def\@cvimage{\relax}
\newcommand{\cvimage}[1]{\gdef\@cvimage{#1}}
…

Again, the first line of code creates a variable (\@cvimage) with a null value, and the second line creates the user facing macro (\cvimage). In later commands, the internal variables are used for layout and formatting (see below).

\renewcommand

If I need to modify a particular piece of code, those changes can be made using the \renewcommand macro. This is similar to over-riding a method in an object-oriented programming language like C++ or Python. Everything that hasn’t been changed continues to work as before. In xetexCV, I wanted to modify the \section and \subsection macros. To do so requires \@startsection and six arguments:

\@startsection {NAME}{LEVEL}{INDENT}{BEFORESKIP}{AFTERSKIP}{STYLE}

Generic command to start a section.

NAME : e.g., “subsection”
LEVEL : a number, denoting the depth of the section in the table of contents
INDENT : Indentation of the heading from the left margin
BEFORESKIP : Space to leave above the heading
AFTERSKIP : Space to leave after the heading
STYLE : Commands to set the font and style of the heading

I, therefore, copied over the definitions from article.cls and modified the spacing and fonts:

\renewcommand\section{\@startsection
{section}{1}{\z@}%
{-3.5ex \@plus –1ex
\@minus -.2ex \vspace{1mm}}%
{0.5mm}%
{\sffamily\large\bfseries}}

\renewcommand\subsection{\@startsection
{subsection}{1}{\z@}%
{-3.5ex \@plus -1ex \@minus -.2ex}%
{3.0mm}%
{\sffamily\mdseries}}

But while I chose to completely rewrite my section definitions, much the same thing could have been accomplished through the sectsty package. Despite the ease of sectsty, I felt that it would be better to define my styles from scratch. sectsty often throws errors when used with other document classes, like scrbook.

Rules for Creating New Macros

The third important thing to understand when writing a document class is when you should add new macros rather than modifying the behavior of old ones. In trying to make this decision, it is good to think about how changing a macro’s behavior will influence other document features. In cases where a change might cause other features to break, it is better to add a new macro.

Managing Complex Formatting – \cvsection and \cvsubsection

While I was redefining the \section and \subsection macros, I ran into one such difficulty. I wanted to set one indentation level for the header and another for the body text. Further, I wanted to include a decorative line following the header. Yet, no matter how I chose to define the STYLE or AFTERSKIP arguments, I couldn’t quite achieve the formatting I wished (shown below).

I finally decided that the formatting was too complex and required a new macro that could be used instead of \section and \subsection. These new additions became \cvsection and \cvsubsection:

\newcommand{\cvsection}[1]{\leftskip 0cm
\section*{#1}\decorativeline\marginpar{\vspace{0.3ex}}
\leftskip 116pt}

As in the previous examples, \newcommand was used to define a code block with a single argument – the text to be formatted). Then, I specified the procedure that I wanted to apply to the text

The left text margin is moved so that it is flush with the page margin (\leftskip 0cm)
A non-numbered section header is added (\section*{#1})
The decorative line is included (\decorativeline)
The text indentation is returned to the previous value (\leftskip 116pt).

The definition of \cvsubsection is similar, with two exceptions. First, the \cvsubsection style does not include a decorative life. The second major difference is more substantial. I found the vertical spacing preceding the \subsection definition to be too large, I therefore moved the entire text block up by 0.2 cm (\vspace{-0.2cm}). This was ultimately easier than trying to find a better value for the BEFORESKIP argument:

\newcommand{\cvsubsection}[1]{\leftskip 0cm \vspace{-0.2cm}
\subsection*{#1}\vspace{1.0mm} \leftskip 116pt}

Note: When given a negative value, \vspace moves up the block of text by the specified amount.

Layout and Formatting

One of the very best features of TeX is the automatic layout algorithm that it uses to position content. In nearly all cases, allowing for LaTeX to manage spacing and layout will result in a better looking document than if you try and do so yourself. There will be circumstances, however, where it is necessary to manually specify the position of photos or text in a more precise manner.

In xetexCV, there are two such cases. In the first, special handling is required so that the CV contact information is correctly labeled and typeset. Additional formatting commands are required so that the images and text of the title section have a “balanced” feel to them. All of this processing occurs in the \makecvtitle macro.

Conditional Formatting – \ifthenelse

While it may appear simple, the formatting of the xetexCV contact section (which includes the institution, address, phone, fax, email and website information) is rather involved. First, in order to simplify the use of the document class, I chose to add this information through the use of special macros (\email, \website\, \cvname, etc.). Because the information is added through tags, this means that logical rules must be set up to handle to handle the layout. What should happen, for example, if a fax number or website isn’t supplied?

The obvious answer is that the information should be left out of the contact details. That also means that the label (“Fax:”, “Phone:”, “Email:”) should be omitted. To accomplish this goal requires conditional formatting, better known to programmers as the “if, then, else” loop. “If” a condition exists, “then” do one thing, “else” do another.

In LaTeX, conditional formatting is performed by the “ifthen” package and uses the \ifthenelse macro.

\ifthenelse takes three arguments:

\ifthenelse{CONDITION}{IF TRUE}{ELSE}

CONDITION : The test condition, often expressed as \equals{VARIABLE}{VALUE}
IF TRUE : The code that should be executed if the condition is satisfied
ELSE : What to do if the condition is false

At several instances in xetexCV, \ifthenelse is used to test if there is a null (or empty) value for a particular tag. In the cases where it finds a null value, then no formatting commands are executed. If, however, there is a value, then text is added. A few of the simpler examples are shown below:

\ifthenelse{\equal{\@phonenumber}{\relax}}
{}{Phone: \texttt{\@phonenumber}\\}
\ifthenelse{\equal{\@faxnumber}{\relax}}
{}{Fax: \texttt{\@faxnumber}\\[0.2cm]}
\ifthenelse{\equal{\@email}{\relax}}
{}{Email: \href{mailto:\@email}{\@email}\\}
\ifthenelse{\equal{\@website}{\relax}}
{}{Website: \href{\@website}{\@website}\\}

In each of these cases, the \equal macro is used to check if the pertinent variable (specified by the \@ symbol) still has a value of \relax (\equal{\@variablename}{relax}).

Note: \relax can be thought of as an equivalent to null in other language.

If the value is still set to \relax, then nothing is done (denoted by the empty diamond brackets). If there is a non-null value, however, then the appropriate label and formatted text is added to the document.

While these examples are relatively simple, the same principle can be used to for very complicated formatting. \makecvtitle, for example, uses a similar conditional statement to see if the user has specified an image. If not, then it will use one formatting style for the entire title section. If so, then it uses the layout shown in the examples.

Balancing Images and Text – \makecvtitle

In addition to the conditional formatting issues, \makecvtitle must also wrestle with the best way of creating a single, unified title block. The CV name, contact information and photo should all be included. Moreover, subsequent sections should be added after the end of the photo, rather than wrapping around it. One of the easiest ways to create such a unified block of content is to use the minipage environment.

The minipage is extremely flexible. It can have text, formatting and image within a self-enclosed space. Furthermore, they can be embedded within one another. By creating a series of embedded minipages (as shown below), you can group related content. You can additionally specify how wide you want each minipage environment to be, or you can let the LaTeX engine make those determinations.

Note: An additional advantage of minipage is that any footnotes added to text within the minipage are part of that environment instead of the main document. This is useful for adding footnotes to figures or text. Accomplishing the same thing in Microsoft Word without tedious manual formatting is impossible.

I’ve used this strategy to ensure that text containing the contact details is properly aligned against the cv photo. The minipage environment from xetexCV is shown below:

\begin{minipage}{6in}
\begin{minipage}{114pt}
\resizebox*{100pt}{!}
{\includegraphics{\@cvimage}}
\end{minipage}
\begin{minipage}{4in}
\ifthenelse{\equal{\@institution}{\relax}}
{}{\bfseries\@institution\\}
\mdseries\@contactaddress\\[0.2cm]
\ifthenelse{\equal{\@phonenumber}{\relax}}
{}{Phone: \texttt{\@phonenumber}\\}
\ifthenelse{\equal{\@faxnumber}{\relax}}
{}{Fax: \texttt{\@faxnumber}\\[0.2cm]}
\ifthenelse{\equal{\@email}{\relax}}
{}{Email: \href{mailto:\@email}{\@email}\\}
\ifthenelse{\equal{\@website}{\relax}}
{}{Website: \href{\@website}{\@website}\\}
\end{minipage}
\end{minipage}

As you may notice, there are three embedded minipages here (the main minipage and two sub pages). The main minipage is set to span the entire width of the text. However, the first embedded minipage is only 114 pt in width (about 1.6 inches). Within this first box, I place the cvphoto, resized to a width of 100 pt:

\resizebox*{100pt}{!}
{\includegraphics{\@cvimage}}

Note: The {!} argument in \resizebox allows the image to stretch proportionally in the vertical direction.

The second minipage is allowed to take up the rest of the paper width and includes the conditional formatting for the contact details.

Conclusion

Through the use of custom formatting, packages and macros, I’ve created a document that is substantially different from the foundation class (article.cls). Despite those differences, nearly every major customization was made using simple LaTeX macros which many users will be familiar with. Further, because it is built on a solid foundation, the ability to use other add-on packages is retained.

Extending an existing document class does not need to be difficult. It is possible to use many of the commands used for writing a document. Additionally, the modifications could be packaged as a style (.sty) and used to enhance any content that I might write in the future.

This flexibility is one of the reasons why LaTeX is such a good candidate for the typesetting of scientific and technical material. It’s also why LaTeX is a good candidate for creating design heavy documents, such as a curiculum vitae.

______________________________________________________

Acknowledgements and Further Reading

Additional information on the structure of document classes can be found in the article “Rolling your own Document Class: Using LaTeX to keep away from the Dark Side” by Peter Flynn of Silmaril Consultants.

A second article called “How to develop your own document class – our experience” describes additional techniques for documents that might have non-standard formatting. It outlines several common mistakes and ways to get avoid them.

This article is part 2 of a four part series. Part 1 introduces the xetexCV document class and describes its use. Part 3 shows how to generate a publications list through the use of a .bib database and custom BibTeX style. Part 4 describes how to use the document class and a corresponding layout file with LyX.

Typeset Your Curriculum Vitae – Part 1: The xetexCV Document Class

Rob Oakes — Wed, 25 Nov 2009 04:02:27 +0000

Very few documents are more personal than a curriculum vitae (CV). A CV lists a person’s educational history, who they’ve worked for and what they’ve accomplished. Moreover, a CV is frequently used to judge a person’s inherent worth and value (or at least exploitability). A quality curiculum vitae matters, a lot.

For that reason, a CV not only needs to include all the pertinent information of a person’s life, but it also needs to look good. An attractive CV with good spacing and contrast leaves a positive impression and makes it easier to find information. When laid out correctly, a reviewer might just find themselves scouring past accomplishments for interesting tidbits: “I didn’t realize that this applicant organized a lecture series with Patch Adams and other notables, that’s interesting!”

There are many dedicated CV classes which can be used to typeset a CV with LaTeX. Unfortunately, they all share one thing in common: they’re complicated. To ensure that the spacing and fonts are correct, far too often a document based on these classes will devolve into a jumbled mess of tags and obscure page layout commands.

I’ve tried to typeset my CV with LaTeX a number of times, and in nearly every case, I’ve been unhappy with the output. Sure, the end product is attractive enough; but it doesn’t quite match the CV that I have in my head. To really arrive at my coveted ideal of perfection, I would need to manually specify the font and position of nearly every element on the page; which is why I’ve normally used a dedicated layout program (such as Adobe InDesign) for the job.

This is unfortunate, however, since a CV based on LaTeX would be much easier to keep current. Additionally, a well designed document class and corresponding BibTeX style would allow me to use my existing bibliographic database to automate the process.

Thus, any time that I need to make major updates, I always check the internet for information and new LaTeX classes. While conducting one such review a few days ago, I came across a very interesting article and set of examples by Dario Taraborelli of the University of Surrey. In his write-up, Dr. Taraborelli expresses frustrations which are similar to my own; but instead of complaining and throwing his hands up in exasperation, he went on to solve the problems in an elegant way.

Rather than use or modify one of the existing CV classes (as I would have done), Taraborelli opted to extend the article class by adding additional features. Perhaps the most important addition was the use of XeTeX and the \fontspec package, which allows for OpenType or expert fonts to be used and other advanced features. The end result of this work is an attractive and flexible document template.

Dr. Taraborelli’s example also shows how easy it can be to extend an existing document class to meet a more specific need. As I read through the example code, I realized that it would be straightforward to build upon his work and create something that would suit my particular aesthetic.

So … I did, and this series is the result of my efforts.

In the remainder of this article, I will introduce the new CV document class I created (called xetexCV) and provide a few examples of how to work with it. In part 2, I will dissect the class code and explain how it works. I will also provide hints on how it might be modified. In part 3, I will show how to generate a publications list through the use of a .bib database and custom BibTeX style. In Part 4, I will describe how the document class and corresponding layout file can be used with LyX.

xetexCV – An Attractive and Easy to Use CV Document Class

Downloads and Installation

All of the files needed to use the class (including xetexCV.cls, examples and supporting files) can be found here. To install, copy to an appropriate folder in your tex path and refresh the tex database:

sudo texhash

Package Requirements

XeTeX Typesetting Engine. For users of Mac OS X and Windows, XeTeX is included in the MacTeX and MikTeX distributions (respectively). Users of Linux, however, will need to install the texlive-xetex package
Fontspec package (included with MacTeX, MikTeX or texlive-xetex)
OpenType Expert Fonts. By default, the document class uses Fontin and Fontin sans.

Curiculum Vitae Examples

Albert Einstein. LaTeX Source, PDF Output
Isaac Newton. LaTeX Source, PDF Output

xetexCV Class Usage

To use the xetexCV class, there are a few important macros. These include:

The \makecvtitle macro and title block tags
The \cvsection and \cvsubsection macros
The \years tag for hanging notes

\makecvtitle

The title block is created through the use of the \makecvtitle macro. The name, institution and contact information are all specified by descriptive tags. The example below shows what these are and how they are used:

\documentclass{xetexCV}

\cvname{Albert Einstein}
\cvimage{Albert-Einstein.jpg}
\institution{Institute for Advanced Study}

\contactaddress{Einstein Drive\\
Princeton, New Jersey 08450
United States of America}

\phone number{609-734-8000}
\faxnumber{609-924-8399}
\email{a.einstein@ias.edu}
\website{http://www.ias.edu/spfeatures/einstein}

\makecvtitle

\cvsection and \cvsubsection

Sections and subsections can be added through the \cvsection{section name} and \cvsubsection{subsection name} macros. The following codeblock creates the “Publications” section header and “Peer Reviewed Journals” subhead seen at right:

\cvsection{Publications}
\cvsubsection{Journal Articles}

…

Hanging Notes

Hanging notes can be specified through the use of the \years macro. This is helpful for typesetting education, employment history, and awards. The following example produces the output seen below:

\cvsection{Appointments Held}

University of Bern \years{1908-1911}\\
University of Zurich \years{1911-1912}\\
Charles University of Prague \years{1912-1914}\\
Prussian Academy of Sciences, Berlin \years{1914-1932}\\
University of Leiden \years{1920-1930}\\
Institute for Advanced Study, Princeton \years{1932-1955}

Choosing a Font

To change the font, use the \fontspec package and appropriate \settext macro. The example below shows how to change the main font and sans font to Warnock Pro and Frutiger LT Std (commercial fonts available as part of the Adobe Font Folio 11):

\setmainfont[Ligatures={Common}, Numbers={OldStyle}]{Warnock Pro}
\setsansfont{Frutiger LT Std}

All of the examples shown here were typeset with Warner Pro and Frutiger. By default, however, the document class uses Fontin and Fontin sans, which are available as free downloads. Another excellent font alternative is Gentium Basic. Because XeTeX can use OpenType fonts, however, you can use any font installed on your computer.

Other Features

Font features such as ligatures and glyph variants can be accessed through the fontspec package
The examples and class are unicode compatible. This means means that you can write in Chinese, Cyrillic or Greek without additional packages

______________________________________________________

Acknowledgements and Further Reading

As noted above, this document class is based on a similar example available from Dario Taraborelli’s website. Other examples of how to create a resume or CV are available from Matthew Boedicker and Kamil Wójcicki.

Customizing LyX: Character Styles and the LyX Local Layout

Rob Oakes — Sat, 14 Nov 2009 21:00:00 +0000

Imagine for a minute that you’re writing a book or technical manual. Let’s say it’s a book on technology, maybe the open source tools used for scientific writing (to randomly pick an example). As you write this book, you realize that you need some way to cue the reader into different parts of the text.

For instance, you might want all definitions to appear in bolded text so that a reader pick out key terms quickly. Or you might want code examples to appear in a different font than the regular text, again, so they’re easy to find. What’s the best way to do this?

Sure, you could just bold the definitions, or manually change the font for the code examples. But that’s painful! Changing typeface and size every time that you have a section of code will eventually result in a lot of lost time. Moreover, you might make a mistake, which destroys your consistency and makes your writing look unprofessional. There must be a better way!

Thankfully, there is. It’s through the consistent use of styles.

Styles Defined

A style is a collection of formatting commands with a meaningful name. Examples in Microsoft Word include “Heading 1”, “Body”, and “Figure”. When you use them consistently, styles allow for “logical markup” of a document. That is to say, you tell the computer what the section, paragraph or text corresponds to and then it handles the formatting. For this reason, styles form the basis of the “What You See Is What You Mean (WYSIWYM)” philosophy used by LyX and other “document processors”.

There are three major types of styles: section styles, paragraph styles and character styles. You are likely familiar with section and paragraph styles, as they populate the main drop down box of LyX’s main toolbar. The most common section styles are: chapter, section, subsection. Examples of paragraph styles include: standard text, quotations, and code. But the third main type of style, “a character style” may be a stranger. It isn’t as common as it’s two older brethren and easier to lose track of; which is a huge shame, because it is just as important.

Character styles, as the name implies, work on individual characters within the text of a paragraph. To extend the examples given in the introduction, the most efficient way to bold a definition would be to use a pre-defined character style called “definition”. Ditto for an in-paragraph sample of code.

“Now wait a minute,” you’re probably saying, “I’m not sure that I follow the difference between character styles and other text formatting commands like ‘bold’, ‘italic’, or ‘underline’!”

That is perfectly alright, the distinction can be a bit hazy. Nonetheless, it is still key in understanding the difference between “logical markup” and formatting. Let me see if I can clear it up.

As described above, a style is typically used to describe what a part of the document is, rather than how it should look. Subsequently, you are unlikely to see a style called “bold”, “italic” or “center”. Those terms describe how the text should appear; which is to say, they are formatting commands. Bolded text uses a heavier line stroke and centered text appears in the middle of the page. A style, on the other hand, is used to describe a “definition”. You then associate that style with how you want “definition” to look. It could be numbered, bolded and centered; or it might be larger than the surrounding text and italicized. The appearance is determined by a “style sheet”.

Note: Since the use and customization of section and paragraph styles is well explained in the LyX help documentation, this article will focus on character styles.

Using Character Styles in LyX

LyX provides some common character styles through the use of its “Logical Markup” module. These include styles for nouns, emphasizing text, and code – which may be useful for highlighting filenames or programming syntax within the body of the paragraph.

The character styles are not loaded by default, however. Instead, they come as part of an optional module. You can use them by going to the “Document->Settings->Modules” pane and adding “Logical Markup” to your document.

Once the “Logical Markup” module has been loaded, you use of apply a particular character style through the “Edit->Text Styles” menu, or by first highlighting a block of text and selecting “Text Style” from the right click context menu.

Note: The emphasize style is distinct in that it can also be toggled on and off through the “Ctrl+E” keyboard shortcut.

After you’ve formatted a block of text with character style, it will appear differently from the surrounding text. It will not only use a different font, but will also be surrounded by a set of hash marks. The example below shows a document with two instances of the “Code” character style. Optionally, character styles have a label which you be toggled on and off. The screen shot below shows what this looks like. The first instance, “TEXMFLOCAL”, has the label toggled off while the instance, “/var/lib/texmf/tex/latex”, shows the same character style with the label visible.

Customizing Character Styles

But what if you need access to other character styles? Perhaps you’re writing a novel and would like to have a dedicated style for specifying when a character is talking to himself (called internal dialogue)? Or you’d finally get back to working on your textbook with bolded definitions. Such styles aren’t defined by the “Logical Markup” module, or any module at all for that matter. Nor are they likely to be used so frequently that it is worth trying to modify the built-in modules or create a new one to support them.

Luckily, there is a solution to this problem. In the first article in this series, I looked at how to create a template for an NIH grant proposal from an existing LaTeX document class. In that article, the secret to using the “nih.cls” document class lay in its corresponding LyX layout file (“nih.layout”). Also in that article, I explained how LyX layout files primarily contain information concerned with on-screen presentation.

Note: For more detail, see: Customizing LyX – Part 1, Understanding the Big Picture.

As a first approximation, what I said in that article is mostly true. But, naturally, the reality is more complex. LyX layout files can also contain information about character styles. This includes both information about how they should be presented on-screen and how the underlying LaTeX typesetting language should process them. Moreover, that information can either live in a global layout file (like nih.layout) or in the .lyx file you are working on (meaning that it can be conveniently added to a template). In the remainder of this post, I will show how to create custom character styles by modifying the local layout.

Example: Creating a “User Interface” Character Style

As you might have guessed, the example introduced above isn’t so random. For the past several weeks, I have been working on a book about scientific and technical writing. The book will cover a number of programs and technologies including LyX, LaTeX, BibTeX, Mendeley, Scribus and Kile.

Since it is geared to a somewhat technical audience, it includes both instructions on how to use program’s UI and code samples. For my code examples, I can use the internal LyX character style. But I’ve found that I would also like to highlight references to the UI. There isn’t a pre-defined UI character style.

Fortunately, I not the first person writing a book with UI references. If you look at the LyX user manuals, you will see that the LyX documentation team created a character style specifically for highlighting LyX menu commands. It uses a sans serif font and I think it is rather attractive. But, how do you go about adding it to another document?

It’s actually pretty straightforward:

Since there isn’t a UI to modify the local layout in LyX, open your document in a text editor
Create a local_layout section
Define how you want your character style to appear (both on screen and in the LaTeX output)

Step 1: Open your document in a text editor

As explained above, custom character styles live in the local_layout section (enclosed with the “\begin_local_layout” and “\end_local_layout”) near the beginning of a LyX document. Unfortunately, there is not currently a user interface pane within LyX to edit this section. As a result, open the document in your favorite text editor. The figure shows what this looks like for a file called “Book Chapter – Technical.lyx”.

Note: There is nothing to be afraid of, LyX’s file format is relatively easy to parse; even for human beings.

“Book Chapter – Technical.lyx” has three major sections:

Document: enclosed by “\begin_document” and “\end_document” (off-screen)
LaTeX Preamble: enclosed by “\begin_preamble” and “\end_preamble”
Local Layout: enclosed by “\begin_local_layout” and “end_local_layout” (off-screen)

I have added some text to the LateX preamble in this example. The command tells LyX to use the “Latin Modern” fonts instead of the LaTeX default fonts, which I find to be rather ugly:

% set fonts for nicer pdf view
\IfFileExists{\lmodern.sty}{\usepackage{lmodern}}{}

Step 2: Create a Local Layout Section

If it doesn’t already exist, insert a local_layout section. For organizational purposes, I like for it to appear following the preamble. Below is the full text of the layout section for “Book Chapter – Technical.lyx”:

\begin_local_layout
    Format 7
        InsetLayout    CharStyle:UserInterface
        LyxType               charstyle
        LabelString           userinterface
        LatexType             command
        LatexName             userinterface
        Font
            Family              Sans
        EndFont
        Preamble
            \newcommand*{\userinterface}[1]{{\sffamily #1}}
        EndPreamble
    End
\end_local_layout

In the example, there is some template code (also known as boilerplate), and values that you can modify in order to better customize the appearance of your character style. The boilerplate includes “Format 7”, LyXType charstyle”, and “LatexType command”. These values will be constant for all character styles, but everything else can be customized.

Step 3: Define how you want the character style to look

When customizing the appearance of your character style, the first thing to do is to decide what you want to call your style. This is done by modifying the “InsetLayout” argument. Since my book project covers many different programs, I’ve decided that I will use this character style for any reference to the ui. That includes menus, button names, or dialog box text. As a result, I’ve called it “CharStyle:UserInterface”. This is what appears when I go to the “Edit->Text Style” menu and in the dialog box that appears when I right click.

Next, you need to decide what you would like the “label” to be. The label appears when the “show label” option is toggled on. In this example, it’s “userinterface”.

I also need to associate my character style with a LaTeX name. Again, I’ve opted to call it “userinterface”.

The next block of options, “Font”, specify how the text will look on screen. You could be extremely ornate specifying text size, typeface, and font weight. In this case, however, I want to go with something simple. I’m just going to use the “Sans” font family for on-screen to display. Boring, but functional.

Note: A complete list of available options can be found in the LyX help documentation: “Customizing LyX: Features for the Advanced User”. See section 5.3.6.

The Preamble block tells LaTeX how it should process the character style. Again, I don’t want to do anything fancy, so I’m just going to use the “Sans Serif” font family. This is accomplished by the following LaTeX command:

\newcommand*{\userinterface}[1]{{\\sffamily #1}}

The “\newcommand” syntax tells LaTeX about my custom style, while everything after is specifying the text and the font family. “\ssfamily” refers to the sans serif font family. Other options I could have used include the roman font family (“\rmfamily”), the typewriter family (“\ttfamily”), medium text (“\mdseries”), italic (“\itshape”), or small caps (“\scshape”).

Note: For a more complete list with sample output, see here (Table 4.5).

When finished, save your document and exit the text editor. When you load the document back into LyX, your style should appear in the “Edit->Text Style” menu. The screenshot below shows how the “User Interface” character style looks in use.

Miscellaneous Things and Conclusion

Like the example in the previous article, I like to save documents with custom character styles as templates. This means that I can create additional documents very easily by going to the “File” menu and choosing the “New from template” option. The new document will have all of the template’s settings, including the LaTeX preamble and the custom character styles. It also means that I can keep separate templates for technical documents, scientific articles, book chapters and fiction. Each has character styles defined that make sense for that type of publication.

Character styles make writing long documents much easier. It’s so much better to label a piece of text as “code” or “userinterface” than it is to remember the exact formatting specifications. Moreover, it makes the document more consistent and provides an easy path to change formatting at a later date. As a first time author, I have no complaints about anything that makes my writing easier, more organized, and flexible.

Statistics With R – Part 1: An Old Dog Learns New Computing Tricks

Rob Oakes — Mon, 09 Nov 2009 02:21:20 +0000

When doing math or numerical analysis, the knowledge of the technique is far too often tied to the tool performing the calculation. Consider an engineer whose understanding of the Fast Fourier transformation is inseparably tied to the fft function in Matlab. Of course this hypothetical engineer understands what the results mean (more or less) but may not be able to duplicate his analysis if Matlab were taken away.

In most cases, it is likely that no deeper understanding will be required. But what happens if the computer makes a mistake? Or the program becomes unavailable? Both situations are entirely possible. Computer algorithms aren’t perfect and occasionally arrive at results make little sense; and hardware has been known to fail.

When the engineer understands how the computer arrived at the answer, however, he can recognize, understand, and ultimately correct those cases where the results are unexpected. This is an important reality check that can prevent costly disasters later down the line. Or, if the hardware is unavailable, he can use an alternative tool or software package to duplicate the analysis.

But while such a situation can arise with any type of numerical software, it’s most likely to happen to users of a statistical package. I find this extremely ironic since a proper understanding of statistics is essential to live in the modern world. (Much more so than an understanding of the Fast Fourier transform, at any rate.) The rules of probability, the normal curve, correlation, and multivariate statistics can have a direct impact on how we live our lives. They are used in making important decisions in finance, medicine, science and government. A misunderstanding of stats and the methods of science (from which statistics is inseparable), underlies the most divisive issues of our day: abortion, stem cell research, and global warming.

Moreover, neither side has a monopoly on ignorance or misunderstanding. People fail to distinguish between correlation and causality, or insist in using the word “average” as a slur. Nearly as bad are those that – like the hypothetical engineer described above – only understand statistics within the narrow context of their stats package. Casual statisticians are nearly as dangerous as the wholly uninformed.

The Statistical Package for the Social Sciences (SPSS), is one of the biggest perpetrators of this crisis. Which is hugely ironic, because I happen to love SPSS. SPSS is probably the first statistical package that has placed advanced statistical methods within the grasp of the novice user. I’ve been a happy user for nearly a decade (ever since I was introduced to the program in high school). But there is no doubt that I’ve come to understand statistics within the context of SPSS and its GUI.

Please don’t misunderstand me, I have a pretty good grasp of basic statistics. I can sling probability with the best of them and take relish in describing when to use the Fischer Exact test instead of a Chi-Square; but advanced statistics are a completely different matter. Advanced stats scare me. I can certainly use these more complicated methods. I’ve analyzed and written about multi-variate models and even ventured into Analysis of Variance (ANOVA). But I have to rely on SPSS and the aid of my institution’s biostatistician to help me recognize when there is a problem.

Which is why, in a time of tight budgets, losing the institution’s SPSS license has been a crushing blow to my productivity. (Whoever made that decision should be hauled out and shot!) Because I don’t have my statistics software any more, there are certain aspects of my job that are much more difficult to do. And unfortunately, there is only logical conclusion to draw: I’ve become a victim of the statistical ease of SPSS.

Open Source Alternatives

I went through a similar experience about a year ago. At the time, I had become increasingly frustrated with the restrictions, licensing fees, and limitations of the Matlab technical computing language. After one particularly infuriating meeting, I decided that I had had enough and was going to do something about it. In the months that followed, I spoke with friends and colleagues, and experimented with every alternative I could get my hands on. I looked at Octave (the “Open Source Matlab”) and Ruby, before eventually settling on a combination of Python and PyQt to meet my needs. The result of these changes has been tremendously positive. Python is both easier to use and far more powerful than Matlab could ever hope to be. Not only am I happier and more productive, but so are those who work with me.

It is, therefore, logical that when I lost my statistical language of choice that I would look to open source to provide an alternative. Fortunately, the Open Source community delivers not one alternative to SPSS, but two: Gnu PSPP and R.

Gnu PSPP

As the name implies, PSPP has one simple goal: to clone SPSS in every way that matters. It can perform descriptive statistics, T-tests, linear regression and non-parametric tests. It has an easy to easy to use and relatively intuitive GUI. It can use SPSS syntax and read SPSS data files. It supports an obscene number of variables and cases (about a billion). It interoperates with Gnumeric and OpenOffice. Finally, it’s fast.

Aside from its horribly ugly icon, PSPP would appear to deliver exactly what I want and need. Except, you might have noticed that this article is titled “Statistics with R”, not “Statistics with PSPP”. Obviously, I chose to go with the second alternative. But why?

PSPP works as advertised. I found it able to deal with nearly all of the old SPSS data files and syntax that I threw its way. But, the program suffers from the problem of all clones everywhere: it’s greatest aspiration is to be a copy of something else. That is to say, it seeks to be “Good Enough”, and therein lies the problem, I don’t want a tool that is good enough. I want to use excellent software, even if it’s different or requires me to learn new things. Even if I have to pay for it.

I’m not trying to pick on or be unfair to PSPP. It meets an important need in the free software landscape. It just doesn’t fit in my with my desires or preferences very well.

The R Statistical Project

This is where R steps into the picture. Whereas PSPP is “aimed at statisticians, social scientists and students requiring fast convenient analysis of sampled data (emphasis added)”, R is the software that most statisticians actually use. When I contacted the statistician at my institution to ask, “What statistical software should I use? I’m looking at R and PSPP.”

He responded, “Oh that’s easy. Use R. There will be a learning curve, but it’s much more powerful and capable than even SPSS or SAS.”

As I’ve started to explore the feature set and available modules, it readily becomes apparent as to why. R is a huge language. There are thousands of packages that cover every type of statistics I’ve ever heard of, and many more I haven’t.

Even better, people have gone to great lengths to incorporate R into other tools. It has a set of excellent python bindings and interoperates very well with LyX and LaTeX. As just a single example, using the Sweave document class, you can use R to easily embed code in reports and other documents that need to be updated on a very frequent basis. This allows for these publications to be generated on demand with the most recent data. The only other place I’ve seen the equal to this feature is within the proprietary universe of Microsoft Office and SQL Server.

Easing Into R

Indeed, if R can be said to have a major weakness, it would be that it is too full featured and capable. Particularly for someone who is a statistical novice. The sheer number of packages and options available is absolutely overwhelming. Moreover, the reference material is distributed in nature. Like other open source tools, you can find answers to your questions; but you need to be intelligent about how you ask them.

As I’ve explored R, there have been quite a few painful moments. This isn’t because R is more difficult than SPSS or SAS, but rather because it is tremendously different. As an example, consider the differences in how the programs work with data.

Both SPSS and SAS use one main data structure, the data set. A data set can be thought of as a big spreadsheet where the variable names are kept in columns and the individual observations are kept in rows (SPSS calls them cases). In contrast, R uses may types of data structures. It has an a two dimensional array that is similar to the data set, but it is possible to use one dimensional arrays of data (similar to vectors), or three dimensional arrays (which might contain extremely complex data). The added options raise very complicated questions: What sorts of statistical calculations are done on a three dimensional arrays? Why are they necessary? How do I need to code my data so that I can take advantage of R’s advanced features?

Those aren’t issues that a user of SPSS or SAS even needs to consider. But with R, they present themselves before you even begin to use the program; and there is no centralized source of information to help you figure out the answers. The result of too many options and too little information is unproductive agony.

Series Introduction

Which is why I decided to write this series. R does some spectacular things, it is excellent software. But there are some things you need to know before using it. Here are just a few examples:

What user interfaces are available for R and which should you use?
How can you use R to summarize data and do basic comparisons?
How does R handle moderately advanced statistics like one-way ANOVA and non-parametric tests? What about mult-variate statistics and regression analysis?
How does R work with other programs? How can you format your output into publication quality figures?
What support does LyX and LaTeX offer for users of R?

The purposes of these articles are to address the concerns of the novice statistician or scientist. I will try and avoid jargon and other indecipherable terms. I will ensure that the examples are interesting and relevant. But most importantly, I will try and to help build a deeper statistical foundation. I know exactly what it feels like to be the “hypothetical” engineer who has become too reliant on his tools. As long as the tool is nearby, you’re fine. But when that tool is taken from you, be prepared for a world of hurt.

Until fairly recently, I’ve been in that world of hurt. You might just say that these articles are my way of explaining how I got out.

Customizing LyX: Create an NIH Grant Proposal Template

Rob Oakes — Mon, 02 Nov 2009 22:21:54 +0000

LyX is a wonderful writing program. It’s easy to use and produces beautifully typeset output. More importantly, though, it lets an author focus on the content and structure of his writing; rather than the formatting. It isn’t so easy to customize, though, which limits its usefulness in a big way. What if you need to create a new layout or take advantage of one of the thousands of specialized LaTeX styles? How, exactly, do you go about doing that?

That’s why this article was written. Recently, I was asked to help with a National Institutes of Health (NIH) R21 grant proposal. After some talk amongst the different investigators, it was decided that we would use LaTeX and LyX to draft it. Unfortunately, we hit a rather substantial hurdle early in the process: LyX doesn’t have an NIH grant template.

After additional debate, we decided to proceed with LyX anyway. But in the process, I found myself saddled with an additional job. In addition to responsibilities as research flunky and copy editor, I was tasked with creating a LyX and LaTeX template for our NIH grant. This article will summarize the steps I took and describe how to create a custom template using an available style on CTAN.

Note: All of the files in this tutorial can downloaded here (.zip).

Understanding the Big Picture

Before laying out the procedure, I will start with the theory. That includes understanding how LyX and LaTeX interact, as well as what files they use for their respective jobs.

An Introduction to LyX and LaTeX

As you probably already know, LyX is a document preparation system which supports LaTeX, DocBook, and other typesetting technologies. Writing with LyX is easier than using regular LaTeX or DocBook because it abstracts away the markup language and most of the associated pain. Want to insert a citation? Just hit Insert –> Citation. There’s no fussing with markup tags or macro commands. The same goes for inserting figure legends, tags, labels, footnotes or any other relatively complex formatting. LyX takes care of all the minutiae for you.

When you want to customize LyX, however, all of the ugliness and complexity gets shoved right back into your face. Instead of just writing and letting the program sort out the details, all of a sudden you are transformed from a user into a collaborating developer. You need to know how the underlying technologies work. What’s the difference between a LaTeX class and a style sheet? What does a LyX layout document do? How does that differ from a LyX template document? How do you install them and get the system to recognize that they’re present?

To get a handle on these questions, you need to understand that LyX and LaTeX accomplish two different (though complementary) goals. LaTeX is largely in the job of creating printed documents (typesetting), whereas LyX is about helping you to write and organize the document (on-screen display).

I like to think of LyX as an editorial assistant while LaTeX is the production staff. The editorial assistant organizes the various elements into a more logical layout, puts references and other material at your fingertips, and finally helps collate everything into a package that the production people can make sense of. Once the package has been assembled, the production staff then goes about laying it all out on paper, making sure that references are all numbered and formatted correctly, and otherwise handling the business of making the document beautiful.

LaTeX Classes and Styles

To go about its work, LaTeX uses a couple of important files: document classes (.cls) and styles (.sty). As mentioned, the entire purpose behind LaTeX is that you should be able to focus on the content of your writing without being distracted by its visual presentation. Classes and styles are what make this possible. As you are writing, you tell LyX (and by extension LaTeX) whether a block of text is a chapter, figure legend, table, or footnote. When ready, you can then transform the markup (which makes sense to a computer) into something that makes sense to a human being.

LaTeX classes are the first step in the transformation. They contain important information about what the structure of the document should be and where things should go. You might even say that a class file tells a document “what” it is. Subsequently, the most common types of document classes are articles, reports, and books.

A LaTeX style, in contrast, tells the file “how” it should look. If a class says, “You’re a book,” then a style will say, “you should have one inch margins” and “the chapter heading should be in Arial 16 point font with 0.2 inches of space above and below them”. Any given document will only have a single document class, but may use many different styles.

LyX Layout and Template Files

In contrast, a LyX layout file corresponds to a LaTeX class and tells LyX how to draw things on screen; and this is where things can get a little confusing. Even though a LyX layout is supposed to look like the printed page, this isn’t always true. For example, the final document may have two or three columns; but on-screen the LyX document will only have one. What is true, however, is that a LyX layout file will contain information about what document elements are available. If you’re writing a book, for example, the LyX layout will have an option called “Chapter”; even though the fonts and spacing used won’t always be exactly the same.

The final fie type that you need to know about is called a template. Whereas a layout file largely contains information about the on-screen display, the template file has information about typesetting and presentation. In many ways it inherits both the properties of the LaTeX .cls and .sty files and the LyX layout properties. This means that you can add both LaTeX and LyX information to it.

The figure below shows these relationships graphically.

Creating a Custom LyX Template – NIH Grant Proposal

With the definitions out of the way, here is the procedure for creating a custom document template. There are four basic steps:

Download the necessary document classes and styles
Install the files into the proper folder of your LaTeX distribution
Create an appropriate layout file corresponding to the document class or style
Create a template file with additional details and information

Step 1: Download the necessary styles and templates

The very first step is to locate the document class that you need to use. There are many good resources to do this. The first is CTAN, which is probably the most comprehensive resource available. (You can also find very good package lists on Wikipedia and through Google.) Since I already know the name of the document class I need (NIH), I will download it from the associated CTAN page.

Step 2: Install the files into the proper folder for your LaTeX distribution

Once you have saved the files onto your computer, you need to place them in one of several locations. If all users of the computer need it, install it to the system wide LaTeX tree. If, however, you are the only person who will be using the package, you can place it in your own “user” folder.

The location of the system tree varies depending on which version of LaTeX you happen to be running. It is typically defined by the “TEXMFLOCAL” variable. For individuals using Ubuntu Linux, you will find it at “/var/lib/texmf/tex/latex”. For users of other LaTeX distributions or operating systems, you will need to consult the documentation files. The “user” tree is located in your own user folder. For users of Linux, this will be “/home/username/texmf/tex/latex”. Note: Be sure to replace “username” with your appropriate login information.

To install the package, unpack the archive and copy all of the contents to the appropriate location. After, you will need to update the LaTeX database by running texhash.

sudo texhash

Step 3: Create an appropriate style layout for LyX

Now that the package has been installed, we need to create a layout file so that LyX can make use of it. This is a two step process:

Edit the document class and style so that it only contains general elements
Generate a new LyX layout file

Step 3.1: Edit the document class and style to only contain general elements

Working with LaTeX classes in LyX is slightly different than how you would work with them in a tool like Kile or TeXMaker. As already mentioned, LyX abstracts a lot of the technical detail of LaTeX away. However, when using a tool like Kile, you directly interact with the LaTeX code. When working this way, it can be convenient to place commonly used elements into the document class or style. The NIH files that we downloaded and installed in the steps above do this. (The page headers and footers contain the name of the primary investigator and the reference number for the grant.)

As you might guess, however, this isn’t a good thing if you are trying to create a general template. The document class and style files should only contain the elements that are going to applied to all of the documents. This shouldn’t include dates specific to a given revision, the name of a principal investigator, or the title of the grant. We want to install the LaTeX files and then forget about them, which is simply impossible if they contain such specific information.

As a result, before we go further, we need to redact this information from the nih.cls file. But even though we are removing it, don’t throw it away! Stick it in a spare text file for the time being. It is still needed so that the final document will match NIH style conventions. We will add it back to the template created in Step 4.

In the original files from CTAN, open up nih.layout in your favorite text editor and cut out the following text:

% preamble stuff
\newcommand{\nih@PIname}{Oakes, Robert Stilman}
\newcommand{\piname}[1]{\renewcommand{\nih@PIname}{#1}}
\makeatletter

% constants
\newcommand{\nih@sillysize}{\scriptsize}

…

\newtheorem{proposition}{Proposition}
\newtheorem{lemma}{Lemma}

In addition to redacting the information above, you will also want to add the following line close to the end of nih.cls. It will ensures that alphabetic section labels will be used instead of numeric labels:

\def\thesection{\Alph{section}}

Note: In the files for this tutorial, I have already redacted the unnecessary text from the nih.cls file and added it to the NIH Grant Proposal template. I have also added the section label code.

Step 3.2: Generate a new LyX layout file

The easiest way to create a new layout file is to modify one of the existing examples. If you look closely at the NIH document class (nih.cls), you will notice that it is based on the LaTeX “article” class. As we try and create an appropriate nih.layout, let’s start by modifying article.layout.

When you first open the file, you will see the following:

# Do not delete the line below; configure depends on this
# \Declare LaTeXClass{article}
…
Format 11
Input stdclass.inc

SecNumDepth             3
TocDepth                3

NoStyle Chapter
NoStyle Chapter*

Style Part
    Align                 Left
    AlignPossible         Left
    TopSep                2
    BottomSep             1.5
    Font
      Size                Larger
    EndFont
End
…

As you study the layout, you should start to see a few general sections. For example, the second line tells LyX that it is using the LaTeX class “article”. “Input stdclass.inc” tells LyX to import the standard list of document elements. Whenever you see a line that begins “NoStyle”, that is telling LyX to remove those particular section types from the available list of options since they aren’t supported by the document class.

Pay close attention to the section block that begin with “Style”. This tells LyX how to display that particular document element on screen. It includes information about how the text should be aligned (“Align”, “AlignPossible”) and what font should be used (“Font”, “Size”, “EndFont”).

To make article.layout work with nih.cls, we need to make two modifications. First, we need to change the \DeclareLaTeXClass and then we need to add a new “Style” block.

First, let’s change the \DeclareLaTeXClass. Add the following:

\DeclareLaTeXClass[nih]{proposal (NIH)}

In this example, nih is the name of the LaTeX document class (nih.cls) and the information in curly brackets is a description which will make it easier to find in the LyX document settings pane.

Next, we need to add a “Style” block that describes the “Section” element of the nih document class. By default NIH sections are labeled alphabetically: “A Specific Aims”, “B Background”, “C Methods”, etc. In contrast, the article labels its sections numerically: “1 Specific Aims”, “2 Background”, “3 Methods”, etc. If we don’t modify the standard labeling, it will be displayed incorrectly on screen

Luckily, it’s an easy fix. Add the following to article.layout:

Style Section
    LabelType             Counter
    LabelCounter          section
    LabelString           “\Alph{section}”
End

After you’ve made these changes, resave the file as “nih.layout”. Then copy the file to either the main LyX directory layouts folder (typically LyXDir/layouts) or to your LyX user directory (typically found at .lyx/layouts/). For LyX to see and use the new files, you will need to run Tools –> Reconfigure.

Note: A full version of the nih.layout file is available as part of the archive package for this tutorial.

Step 4: Create a LyX template file

With a fully functioning LyX layout (nih.layout), you can now create a template file that has all of the needed elements. As mentioned above, a LyX template file is nothing more than a regular LyX file. So, start by creating a new LyX document (File –> New).

After the new document is open, the very first thing that you should do is change the document class. Go to Document –> Settings –> Document Class. Then from the drop down menu, select “proposal (NIH)”.

Next, you need to add the information removed from nih.cls to the LaTeX Preamble. Go to Document –> Settings –> LaTeX Preamble, and paste the text into the provided space. While doing this, you may want to modify the name of the PI that will be shown in the document header. You make this change by adding your name to the line that reads:

\newcommand{\nih@PIname}{Last Name, First Name Middle}

When done, press “Close”. You now have a fully functional, NIH compliant grant proposal template.

But while that’s the end of the essential boilerplate, you may want to include a few additional goodies. Wouldn’t it be nice, for example, if you had access to clickable hyperlinks? Or if you were be able to use the bookmarks sidebar to navigate the document? For these and other desirables, add the following to the preamble:

% Hyperlink Options
\usepackage{ifpdf} % part of the hyperref bundle
\ifpdf % if pdflatex is used

% Use True Type Fonts Instead of Older LaTeX Fonts
\IfFileExists{lmodern.sty}{\usepackage{lmodern}}{}

\fi % end if pdflatex is used

% for correct jump positions when clicking on a link to a float
\usepackage[figure]{hypcap}

This code enables the hyperref package. Then, go to Document –> Settings –> PDF Properties and enable hyperref support in the options. While changing the PDF properties, I personally like to customize the colors of the links. I think that document links and cited references should be black; while urls and files should be blue. To make this change, add the following to the “additional options” line:

linkcolor=black, citecolor=black, urlcolor=blue, filecolor=blue

Using the Template

If you’ve followed all of the steps in this tutorial, you will now have a fully functioning NIH grant template. Now might be a good time to save it in the same location where you keep other templates. (If you don’t have a templates folder, you might want to create one.) I’ve saved mine as “NIH Grant Proposal.lyx”. From here on out, whenever I want to create a new NIH document, I simply choose New from template from the File menu and load the appropriate version.

Now that my template is set up, I’m back to an automated wonderland. LyX and LaTeX keep track of the references, labels, footnotes, formatting and the other minutiae that make complicated documents so painful. I’m free to focus on the ideas, voice, and message; which is exactly how it should be.

Create a Unified Inbox in Gnome Evolution

Rob Oakes — Tue, 27 Oct 2009 07:43:00 +0000

I have a serious love-hate relationship with Linux. I love the fact that it’s free and open source. I love the fact that it can breathe new life into old hardware. I love the fact that it’s easy to extend. I love the fact that it has a vibrant and passionate user community.

What I do not love is that many open source programs are incomplete. They can do most everything that you need, but never get around to adding the one or two features that prevent them from being finished, polished and exceptional. I’ve ranted about this before, back when I was trying to find the perfect backup program.

Well … I’m at it again; except this time, I’m looking for the perfect email program.

For a long time, I thought that I had found it in Thunderbird 3. Thunderbird is powerful, highly configurable and easy to use. Moreover, alpha and beta builds hint at changes that will make it even better. The most obvious of these is a universal Inbox. This is one area where I should give Apple some credit. They figured out a long time ago that it’s better to have all your emails in the same place, regardless of what account they may belong to. It makes things easier if the communication from your your work account, home account, webmaster account, software development account, or hobby account all live in the same place. You just read the email and respond, the program makes sure that it gets routed along the proper pipes.

This makes it really easy to apply selective filters. On my home computer, I only subscribe to those email accounts for home related things. On the work computer, I subscribe to work related accounts. On my laptop, I have all of them.

But most people use their email client for much more than just sending messages. About ten years ago, Microsoft showed that calendar and tasks should also be part of the email application. It’s more convenient when the tools that you use to organize your life are in the same place as those you use to communicate. After all, who wants to use three or four apps when you can use one?

Yet, on both these fronts, open source has been pretty slow to catch up. The integrated inbox just appeared in Thunderbird. And while there is a calendar component, it’s pretty limited. It does alright with a local calendar, but a calendar chained to a single computer is mostly worthless. Thunderbird can’t connect to an Exchange server, where your business calendar is likely to live. It can’t talk to the family’s calendar, which might be kept on a consumer service like Yahoo or Google. And while it has experimental support for CalDav, that’s the problem, the support is still experimental. I’ve found that tasks or appointments don’t update reliably, and it only works when you are connected to an internet connection.

Perhaps worst of all, Thunderbird can’t talk to your cell phone. If you’re anything like me, my cell phone is my second memory. If I don’t have it written down and in front of me, I’m going to miss it or fail to do it. As a result, a robust cell phone sync is a must. Taken together, these limitations and rough edges are just too much. I need an alternative, even though I love what Thunderbird does for my email.

This is where the venerable Gnome Evolution steps in. In contrast to Thunderbird, which is promoted as an email application, Evolution is a collaboration app. That is to say, it takes calendar, contacts, tasks and notes seriously. CalDav? Got it covered. Google? Ditto. Yahoo calendar? Yep. Microsoft Exchange? You bet.

They have also given some serious thought to integration and user experience. Evolution is deeply integrated with the Gnome desktop. The desktop drop down calendar shows your appointments. Email and notifications are integrated into the dashboard. You don’t even need to have the application open!

But Evolution crashes and burns as an email client. It lacks all of the subtle tricks that Thunderbird, Mail.app and Microsoft Outlook use to keep organized. As just one example, the folder list is overwhelming and it’s hard to know when you’ve received new mail. Thunderbird and Mail.app solve these problems through their unified inbox. Microsoft Outlook solves it by having a “favorites” area where you can create shortcuts to folders that you ca re about. When you receive a new email message, this shortcut is bolded. It’s not quite as effective as the universal inbox, but at least it’s something. Evolution has nothing.

As a result, even though I’ve loved the calendaring and integration with the desktop, I was terribly afraid I would have to dump yet another application because it offered a substandard experience. Luckily, I found a workaround.

There is a pretty simple way to create a unified inbox in Evolution. It was so easy, in fact, that I was surprised that the Evolution developers haven’t taken the time to make it a standard part of the program.

The magic happens through the search folders. In a nutshell, you create a search folder that looks through all of your account inboxes for either read or unread email messages. Presto, a unified inbox!

Start by right clicking on the “Search Folders” entry in the folders list, then select “New Folder.” It will load a dialog similar to the one seen below. You will need to add two conditions. Set the first condition to “Status Is Read” and the second to “Status Is Not Read” as shown in the screenshot below. When done, press the “Ok” button.

The same thing can be done for your drafts, or outbox folders. Pretty slick, huh? What I find incredible is that such a simple hack makes such a huge difference in usability. Instead of separately monitoring six different email account settings, I can check just a single folder for new things. It transforms Evolution from a pain in the ass to a pretty decent little email client.

Do you have your own Evolution hacks? If so, let’s hear about them in the comments.

Time Drive 0.3: Better, Easier, More Refined

Rob Oakes — Mon, 26 Oct 2009 16:16:56 +0000

Imagine how awesome it would be if this announcement read: “Time Drive has been completely rewritten from scratch (yet again) to take better advantage of the paradigms of modern computing! Version 0.3 has hundreds of updates and new features which will make your life easier and more fulfilled!”

There’s just one little problem … such a hyper inflated announcement wouldn’t necessarily be true. (Marketing hyperbole, I never knew thee!) The truth is this: Time Drive is a simple backup program that does a good job of reliably backing up your data. It offers a nice list of potential backup options: from an attached hard drive, to a computer over the network, or across the internet. It makes it easy to search for and restore a lost file. In short, Time Drive seeks to change the world by making an act of computer maintenance more convenient.

But the real test of a program isn’t how well it works, but how easy it is to fix when broken. A good program does what you want, but a better program helps you get back on track when things go wrong. Back when I was looking at other backup programs available for Linux, this was my number one frustration. Most of the applications would work (for the most part), but I could never troubleshoot or repair problems when they happened. There just wasn’t enough information available.

For an example, let’s take SBackup. It’s a lovely little program, except you have no way of knowing if it is working. It doesn’t keep log files, it doesn’t notify you if a backup job failed. It doesn’t let you know if it is running. Its simplicity is actually symptomatic of a flaw: it’s incomplete.

These were problems that I desperately wanted to avoid with Time Drive. And version 0.3 includes a number of refinements that solve these issues while at the same time making make it better, easier and more refined. In the rest of this post, I’ll explain why.

Better

There’s a very good reason why Time Drive works with the Ubuntu notifications system and has progress bar notifications for nearly everything. From the very first, I wanted the program to notify the user of what it was doing. But in versions 0.1 and 0.2, there was an essential part missing: some kind of program log. It was a big omission, and several people complained rather vocally about it.

Logs and Notifications

And it’s the first thing that was added to version 0.3. Time Drive now logs its activities and will notify you of errors that happen behind the scenes. If a backup job fails silently, you can put on your detective hat and go find out why. If your internet connection cuts out while using cron to automatically backup your computer, you’ll get a warning the next time that you run the program. Backup statistics, problems, errors, even love notes; they’re all there. To open the log, just click on the blue information button on the main user interface.

Progress Indicators for Absolutely Everything

In addition to the log, we fixed a second oversight. Versions 0.1 and 0.2 lacked an indicator that told you how far a backup job had progressed. In 0.3, that problem’s been solved.

Easier

Great. Time Drive does a better job of letting you know about problems, but what about fixing them? For example, what will version 0.3 do when a backup job fails, or your internet connection cuts out at the wrong time? When these things happen, Duplicity leaves unneeded files on the backup drive. (Files, by the way, which are a pain to remove.) How does Time Drive make repair jobs easier?

Well … I’m glad that you asked. Time Drive now has a cleanup function. If it detects a mess or unneeded files, the program will automatically go about restoring things to right. And if you happen to be one of those obsessively tidy people, you can manually run the same tool by going to the advanced settings pane for the corresponding folder.

More Refined

So much for better notifications and easier management of messes. But what about the promised refinements? Where can you look to find those?

For starters, take a look at the settings for remote connections. All of the input boxes have been reworked for cleaner input. While Time Drive and Duplicity have always supported non-standard port numbers, it wasn’t necessarily clear how how to configure them. Now, it is.

More importantly, though, the user interface adapts to your choices. If you choose to make a backup to an attached drive, you aren’t going to be asked for a server and port. If you choose FTP, however, those options will be there. Best of all, you no longer have to keep track of which types of connections require two slashes in the connection url and which one uses just one. Time Drive will do the configuration and accounting for you!

Looking Toward the Future

And those are only the changes that you will see. There’s a lot more that has happened behind the scenes to make version 0.3 the best release of Time Drive ever. You might call these “future enhancements.” They include:

International Support. If you don’t speak English, Time Drive now has you covered. Dutch, French and Spanish translations are in the works. Want to see Time Drive translated to Chinese? Come help out!
Documented API. You know that a software program has grown up and reached maturity when you have a documented API.
Automated Installation and Upgrades for Linux Users. One of the best things about using Linux is apt-get. It keeps your entire system up to date with the most recent software, and the Time Drive project at Launchpad now has it’s own PPA.

Additional Announcements

As you can see, Time Drive is managing to evolve and add new features at a pretty good clip. But as we do so, it’s also time to face a pretty harsh reality. An easy to install and compatible Windows version simply isn’t going to happen.

Producing a version of Time Drive for Windows has always been extraordinarily hard, there are just too many *nix dependencies. To use it requires Cygwin, a compatible version of PyQt and a great deal of black magic and prayer. Changes that are compatible with Mac OS X and Linux break the Windows version, and vice versa. After months of wrestling with it, I’ve decided that It’s just not worth the headache. So, from version 0.3 onward, we will not be targeting Windows as a development platform.

For those of you waiting with baited breath for a Windows release, my apologies. But I should point out that Time Drive isn’t really needed on Windows. There are a number of fantastic backup programs available. The built-in backup is actually quite good in, and if that doesn’t suit your fancy there are some really fantastic commercial alternatives.

Conclusion

In summary, version 0.3 is a nice, incremental release of Time Drive. It patches some important holes and does a much better job of notifying users of what’s going on. It’s easier to manage your backups and clean up disasters. Lastly, it’s more refined with a bunch of spectacular back end improvements. So while version 0.3 may be an incremental improvement, we’ve got a foundation that will let us release even better versions in the future. This announcement might not be filled with marketing speak and exaggerated claims, but you can bet that future announcements will do a used car salesman proud.

Reclaiming Free Space from a Time Machine Backup

Rob Oakes — Mon, 19 Oct 2009 03:24:09 +0000

As much as I love Apple’s Time Machine, it’s a hard drive pig. If not carefully watched, the little porker will use every spare byte of free space it can. What is particularly obnoxious, however, is that you might not realize you have a problem until it is too late and you’re backup drive is filled to capacity.

Take my situation as an example. I have a single MacBook Pro notebook with a 250 GB hard drive. Most of my files are text based and on the smallish side. In comparison, my networked backup is a hefty 1.5 terabytes. The combination of small hard drive and large backup drive had me thoroughly convinced that I wouldn’t have to worry about free up space for years.

I was wrong.

Because of the size of the backup drive, I like to keep other files on it – mostly music and video files – so that I have a duplicate copy. But earlier this week, I got a nasty surprise while trying to add an album I had just downloaded from Amazon Mp3. The Mac informed me the backup drive was full.

As you might guess, I found this to be very confusing. How could the drive be full? Sure … I had three or four hundred gigabytes of music and video files on it, but there was no way that the Time Machine backup could be over a terabyte in size … Could it?

This situation didn’t smell right, so I decided to investigate. I mounted the backup drive and tracked down the Time Machine sparsebundle and confirmed the impossible. My Time Machine Backup was a whopping 1.15 terabytes worth of disk space. “How in the world could the backup be so large?”, I asked myself. “Time Machine is supposed to be an incremental system. 1.15 terabytes is big enough to hold every bit and byte on my computer four and a half times over!”

First, I got annoyed; then, I got angry. What really tipped the scale toward seething fury, however, was failing to find any straightforward way of getting the space back. Yet another spectacular example of Apple’s “simple over useful” approach to computer design!

After the first bout of obscenities, I came to a simple conclusion: I could publicly express my dissatisfaction with Apple’s product line or I could go about trying to find a solution. Publicly spouting off was unlikely to help much, so I opted for the latter option. What follows is a brief summary of what I learned.

The Technical Details

The good news is that you can reclaim your drive space. It’s not even very hard. Here’s what you need to do:

Delete any and all unnecessary backups from the sparse bundle image. Time Machine is tremendously aggressive in making sure that you’re protected and while this might make sense for the near past, do you really need that backup from 9 months ago?
Mount your AirDisk, Samba share or Time Capsule as a local drive.
Use the hdiutil to compact the sparse bundle.

Step 1: Delete Backups from the Sparse Bundle

Start by loading Time Machine from the root of your main hard drive. (In my case, the hard drive is named “RobOakes-Mac”.) Next, select the snapshot that you are interested in deleting from the timeline at right. Finally, press the gear button and select the “Delete Backup” option.

Time Machine will ask if you want to proceed. Confirm that you do, indeed, want to permanently remove the selected backup by pressing the “Ok” button.

At this point – if you’re using Snow Leopard – you may run into a rather obnoxious bug. For reasons that make little sense, the confirmation dialog box doesn’t always appear. Instead, the user interface might freeze and you’ll be treated to the spinning beach ball of death. If this happens, be patient. It might take a minute or two for Time Machine to go about it’s business. (Since this problem doesn’t exist on regular Leopard, go ahead and curse Apple’s grandiose, false, and self-serving rhetoric about parallelism. It might even make you feel better, I know it helps me.)

When the damn beach ball finally goes away, just hit enter. Apparently, the dialog box really is there, you just can’t see it. So, use your imagination and pretend.

Finally, you will be prompted to enter an administrator password.

When you’re all finished deleting snapshots, click on the main “Cancel” button. This will take you back to your desktop.

If everything worked as planned, you should see something similar to the screenshot below. There will be a “Delete One Backup” indicator for each of the backups you removed. Important: Wait for the indicators to finish before proceeding to the next step.

Step 2: Mount the Disk Using Finder

Since we will be using a command line utility to compact the sparsebundle, you will need to mount your backup disk before proceeding. While this can be done a variety of ways, the easiest is to open up the finder and select the backup drive from the list under “Shared”. (When it is mounted, you will see a little eject symbol next to the computer name.)

Now open up a command line terminal and go to the /Volumes/ folder.

cd /Volumes/

Spend a bit of time looking in the various subdirectories until you find the correct .sparsebundle file. The sparsebundle file will have the general form of Computername_MACaddress.sparsebundle. Note the name of this file, it will be important in Step 3.

Step 3: Use the Command Line to Shrink the Sparsebundle Image

Once you’ve found your sparsebundle, use the following command to reclaim your free space (be sure to replace SparsebundleName with the name of your sparsebundle):

hdiutil compact SparsebundleName

Depending on just how your fat your Time Machine backup has gotten, this might take anywhere from a few minutes to a few hours. Regardless, the good news is that it works! After aggressively cutting away old backups, I was able to get my 1.15 terabyte porker down to a much more reasonable 500 gigabytes.

“An Ounce of Prevention”

While the procedure described above will let you fix the problem, it is far from an ideal solution. For starters, it’s only a temporary fix. Time Machine still has a gluttonous appetite for space and you’ll need to repeat the above steps whenever you want some of it back. For me, that works out to every six months. Further, the procedure is time consuming. The obnoxious ui freezes, spinning beach balls, backup deletions, and sparse bundle shrinking requires more time than anyone should have to spend on this problem.

This is a clear case of “an ounce of prevention is worth a pound a cure.” Rather than resorting to digital liposuction, it’s much better to simply limit how big the sparse bundle can become. This is no more complicated than using the “-size” flag when you create it. But if you (like me) overlooked that small step, then welcome to the club. Let’s just hope that Apple implements a more intelligent way to managing Time Machine backup size in the relatively near future!

Installing Duplicity on Mac OS X

Rob Oakes — Wed, 07 Oct 2009 19:20:40 +0000

While Mac OS X shares many things in common with other Unix operating systems, it also has a couple of missing parts. This can make installing software more difficult when compared to Linux or other *nix variants, and unfortunately, Duplicity happens to be one of those cases.

But just because it is more difficult to install Duplicity on Mac doesn’t mean that it isn’t worthwhile. Duplicity is one of the best command line backup programs available anywhere. Using the same program, you can backup to a local hard drive, FTP server, over SSH or even to Amazon S3. It uses the rsync algorithm, which means that backups happen quickly and only the parts of files that actually changed get copied. Subsequently, backups are smaller.

Nor does it mean that installing Duplicity is actually all that hard, it’s just a bit tedious. Like many other *nix programs, Duplicity requires a number of additional programs (called dependencies) that you will have to search out and install separately. To make that process easier, this article explains how. Below, you’ll find links to the download pages and instructions for how to compile each of Duplicity’s dependencies on Mac OS X. The instruction set has been tested both on Mac OS X 10.5 (Leopard) and Mac OS X 10.6 (Snow Leopard).

XCode

The first dependency you will need is a rather large one. Several of the software packages need to be compiled from source. But unfortunately, Mac OS X doesn’t install the compilers by default. Thus, you will either need to install them from the Mac OS X installation CD, or by getting the XCode tools from the Apple Developer Connection (ADC). Note: ADC requires that you have an Apple ID – which is free – to log in into the site.

After you log-in, download the XCode developer tools package. Once XCode has finished downloading, run the package installer. It will ask you to select the destination drive and what components of the software you would like to install.

Choose the Develoepr Tools Essentials, System Tools and Unix Development Support options. This will automatically install and configure the GNU tool chain and the GCC compiler collection. Once it has finished, you can access the tools by going to the “/Developer/” folder on the root of the drive.

After XCode is finished installing you are ready to build and install the other necessary dependencies.

Note: All other steps in this tutorial will be run from the command line terminal.

GnuPG

GnuPG handles the encryption and decryption of tar archive files. If it isn’t installed and configured correctly, Duplicity (and by extension) Time Drive simply won’t work. On Linux and other Unix operating systems, GnuPG is typically included as part of the default installation. This is not true on Mac OS X, however. As a result, you will need to download, compile and install it from source.

You can find the source files on the project download page or the associated ftp site. Duplicity will work with version GnuPG version 1.4 or 2.0. The biggest difference between the two is that version 2.0 is slightly more modular, and as a result, has a speed advantage. From a practical standpoint, however, unless you are running large, frequent backups, you are not likely to notice any difference. I have been able to compile both without any difficulty on both Leopard and Snow Leopard, so the choice is largely one of preference.

After downloading the most recent version, extract the archive by typing:

tar –xf gnupg-x.x.x.tar.gz2

Where x.x.x is the version of GnuPG that you downloaded (at the time of this writing, it is version 2.0.12). This will create a new directory called gnupg-2.0.12. From that directory, you can configure, compile, test and install:

cd gnupg-2.0.12
./configure
make; sudo make install

When finished, return to the working directory by typing:

cd ..

GnuPG Interface – Python Module

Now that you have built and installed GnuPG, you will next need to install the Python bindings so that duplicity can make use it. You can get these from the GnuPG Interface Module project page, specifically from the download section. Grab the most recent version (0.3.2). To install, extract the folder, enter the setup directory and then use python to run the “setup.py” script:

tar -xf GnuPGInterface-0.3.2.tar.gz
cd GnuPGInterface-0.3.2
sudo python setup.py install

NcFTP

NcFTP enables Duplicity to make backups to FTP servers. The download files can be found on NcFTP’s download page, which lists both the NcFTPd Server and the NcFTP client. We want the most recent version of the client source code (version 3.2.3).

Like GnuPG, we need to configure, compile and install:

tar –xf ncftp-3.2.3-src.tar.bz2
cd ncftp-3.2.3
./configure
make; sudo make install

Boto – Amazon S3 Support

To use Amazon as a backup destination, you need to grab the Boto python module. Boto provides a structured and integrated interface to most of the features offered by Amazon Web services, most importantly: the Simple Storage Service (S3). You can get the most recent version (1.8d) from the Boto downloads page on Google Code.

From there, the installation procedure is similar to that for the GnuPG python interface module. Extract the archive, enter the program folder and use python to run the setup script.

tar -xf boto-1.8d.tar.gz
cd boto-1.8d
sudo python setup.py install

Duplicity

Once you have installed GnuPG, the GnuPG python interface, NcFTP and Boto, you are ready to install Duplicity. Because it is a python program, the steps are identical to those of Boto and the GnuPG python interface. Download the source code from the project home page (keeping in mind that Time Drive requires at least version 0.6.04); then, extract it the same way that you did the previous libraries and enter the source code directory:

tar –xf duplicity-0.6.04.tar.gz
cd duplicity-0.6.04

Finally, run the python setup script:

sudo python setup.py install

Duplicity will byte compile and be copied to the python packages directory.

When finished, you can test the installatin by typing:

duplicity –help

If successful, you will see the text from the duplicity manual page.

Glass Houses and Stones

Rob Oakes — Tue, 06 Oct 2009 20:24:48 +0000

Regular readers of this blog might accuse me of having a deep seated resentment against iPhone, Mac OS X and Apple in general. The only problem, of course, is that resentment is the wrong word. Disillusionment and disgust are much more accurate.

You see, purchasing a Mac computer was one of the single biggest disappointments of my young technical life. I had been promised so much!

If you read the ramblings of online pundits or dedicated Apple purists, you will know that switching to a Mac brings a Zen like state to your computing. It will make you more productive, more creative, more organized, more intelligent and possibly even more attractive.

Except after nearly three years of owning one and using it more or less daily, I’ve come to a simple conclusion: my MacBook Pro, in addition to being a lovely paperweight, is a computer. Nothing more, and quite possibly a whole lot less. (Were it just a computer, I might even be able to use it the way that I want, instead of capitulating to the desires of a mega corporation.)

In fact, I’ve further decided that there is only one possible way that you can possibly claim that a Mac is easier to use than a PC (short of using mind altering chemicals, that is). You must choose to stay within Apple’s suffocating glass greenhouse and allow Apple to decide what you can do and dictate precisely how you will do it. The Apple experience demands nothing less.

Want to use Time Machine to back up to a network storage unit different than their ticking time bomb? Sorry, you can’t do that. “It’s not supported.”

Want to run that piece of software that worked just fine until you installed Apple’s latest glorified service pack? Sorry, that isn’t going to happen, either. “Backwards compatibility prevents us from creating innovative and utterly amazing ™ new user experiences.”

Or want to use that iPhone program that was approved at the highest levels, and then rejected without explanation? “We just can’t allow that. It could result in user confusion.”

It’s either Apple’s way or no way, even when Apple’s way is pathologically stupid. Yet, there is no lack of iCult members who are positively giddy to be treated like iTools!

In contrast, when something goes wrong on a PC, people – rightly, might I add – blame Microsoft. Microsoft makes a disgusting amount of money from their software; and in a sane world, money buys accountability. We pay the CEOs of large corporations obscene salaries and even more ludicrous bonuses to fix problems. If there’s a malfunction, someone is reassuringly responsible; if there’s a disaster, someone is handily available to be lynched.

Except, reality breaks down within the Church of Apple. If a Mac user has a problem, you can rest assured that she will blame herself. You just know that a technical glitch couldn’t possibly be because Apple made a mistake, or the product contains a flaw. Apple merchandise is loving crafted and precisely engineered! And the omnisicient Steve Jobs thinks of absolutely everything!

Is it really so hard to see that Apple’s technical accomplishments represent the pinnacle of human accomplishment? Or that every contact with the Holy Church is divinely sublime?

It is positively convenient to drive 50 miles to the nearest Apple store, wait for more than an hour because the iDisciples can’t keep to their appointment system, and lose your computer for a week and a half because a computer repair service doesn’t stock hard drives. You get to talk to a human being, who will insult you to your face rather than over the phone! If you can’t get it to work, that’s your problem. You’re obviously not smart or cool enough to be an Apple person.

In Apple’s pristine little world, it’s just inconceivable that Apple’s products might not be nearly so desirable as the punditocracy claims. It’s blasphemy of the highest order, requiring that thorough penance to be administered by the all-too enthusiastic congregation of assorted hippies, losers and online freaks. Any individual who so much as implies something negative about Apple deserves the accusations of bias – defined as anything less than a total willingness to sacrifice their firstborn’s blood on the iAltar – that will plague them for the rest of their public life.

After all, Apple has never done anything to encourage resentment or anger. They’re far too busy voiding warrantees, sabotaging relationships and having a party to promote the thousands of invisible (albeit refined) features and APIs of their near-perfect operating system. As a result, it’s simply incorrect to assert that I resent Apple. Until such time as they do something improper, I’ll just have to classify my feelings as disillusionment and disgust.

Install Time Drive On Ubuntu and Other Debian Linux Distributions

Rob Oakes — Mon, 05 Oct 2009 22:23:01 +0000

Since version 0.2.2 of Time Drive, installing Time Drive and keeping it up date has gotten a great deal easier. Instead of requiring that you install from source, we now offer Debian based packages through the Time Drive Personal Package Archvie (PPA). This article will briefly describe how to install Time Drive using that PPA.

Background

But before leaping headlong into the nitty gritty, let me provide a bit of background about packages and repositories. Unlike Windows or Mac OS X, software on Linux is organized into a container called a package. The package contains the files needed to run the program in addition to artwork and configuration information. Any given program may require many other programs to be installed before it can run. These other programs are referred to as dependencies. Ubuntu and uses a tool for managing packages and dependencies called Apt.

The benefits of using Apt over manually installing things is that Apt figures out all of the dependencies automatically. Further, it keeps everything up to date by periodically scanning the online repository (PPA) and downloading any updates. In practice, Apt works much like the Windows Update service, only better, since it monitors every piece of software installed on your computer.

Installation Instructions

To install Time Drive using the PPA, we need to do the following:

Add the Time-Drive-Devel and Duplicity PPA to your Ubuntu Software Sources.
Configure Ubuntu so that it will trust both Time-Drive-Devel and the Duplicity packages.
Install Duplicity and Time Drive

Step 1: Add the Package Repositories to the Ubuntu Software Sources

Open the Ubuntu Software Sources by going to the System –> Administration –> Software Sources. Then, select the “Third-Party” software tab of the application. “Software Sources” requires root access, therefore, you will be prompted to type in your administrator password.

We need to add two package repositories to the list, one for Duplicity and the other for Time Drive. Depending on the version of Ubuntu that you are using, copy and paste the following lines (one at a time) into the “Add Sources” dialog. After each line, press the “Add Source” button.

For Ubuntu Karmic Kaola (9.10):

deb http://ppa.launchpad.net/duplicity-team/ppa/ubuntu karmic main # Duplicity – Ubuntu 9.10

deb http://ppa.launchpad.net/time-drive-devel/stable/ubuntu karmic main # Time Drive – Ubuntu 9.10

For Ubuntu Jaunty Jackolope (9.04):

deb http://ppa.launchpad.net/duplicity-team/ppa/ubuntu jaunty main # Duplicity – Ubuntu 9.04

deb http://ppa.launchpad.net/time-drive-devel/stable/ubuntu jaunty main # Time-Drive – Ubuntu 9.04

For Ubuntu Intrepid Ibex (8.10):

deb http://ppa.launchpad.net/duplicity-team/ppa/ubuntu intrepid main # Duplicity – Ubuntu 8.10

deb http://ppa.launchpad.net/time-drive-devel/stable/ubuntu intrepid main # Time-Drive – Ubuntu 8.10

When finished, press the “Close” button. Ubuntu should then notify you that the software information is out of date and needs to be updated. Instead of clicking “Reload,” click the “Close” button. We will refresh Apt from the command line in Step 3.

Step 2: Configure Apt to Trust Time Drive and Duplicity Packages

After adding the repository information, you will then need to add the repository key to your system’s list of trusted keys. This is most easily done by running the update-launchpad shell script. (To download, right click on the link and select “Save As”.) When run, the script automatically downloads any needed signing keys and automatically add them to the “Trusted Sources” list.

To run the script, go to Accessories –> Terminal and navigate to where you saved the file, then type:

cd /path/to/folder
sudo bash update-launchpad.sh UbuntuVersion

Be sure to substitute the correct Ubuntu code name for UbuntuVersion in the second step – karmic for 9.10, jaunty for 9.04, or intrepid for 8.10 – otherwise, you will get an error. I happen to use Ubuntu 9.04, therefore, for my setup, I would type the following:

sudo bash update-launchpad.sh jaunty

Step 3: Install Duplicity and Time Drive

Finally, we are ready to install Time Drive and Duplicity. From the command line, refresh the package lists by typing:

sudo apt-get update

Then, to install time-drive and duplicity, type:

sudo apt-get install time-drive duplicity

Application Updates

As noted above, whenever a new version of Time Drive, Duplicity, or another dependency is added to the package repository, your system will automatically be updated. However, should you upgrade your system to a newer version of Ubuntu, you will need to return to this page to add the repository for that version.

__________________________________________________

Note: These instructions were modified from this tutorial on WineHQ.

Upgrading from Time Drive 0.1

Rob Oakes — Mon, 05 Oct 2009 19:47:42 +0000

When it comes to most things, starting fresh is a blessing. The reason for this is rather simple, when starting over you don’t have to worry about baggage. After all, baggage is only valuable when on holiday; otherwise, it just slows everyone down.

This is especially true for software. Over time, computers tend to accumulate a rather potent type of digital baggage that can be very difficult to get rid of. And that digital garbage results in inconsistencies that can cause enormous – and usually unforeseen – problems.

However, even though starting fresh is usually the best option, that isn’t always true. Sometimes, it’s better to risk the problems and incompatibilities. For example, starting over may mean that you destroy hours worth of customization, or that you lose work already created because the older version are not compatible with the new.

Unfortunately, the general rule is also somewhat true of Time Drive. So, if you were one of those stalwart and brave individuals who decided to experiment with Time Drive 0.1, this post is for you.

In the last few days, I have been in touch with a number of people who have experienced a number of said inconsistencies and problems. And while several of these problems ended up providing insight on mistakes made during development, some of the others were changed on purpose. That is to say, the so called “bug” was actually a feature.

After fielding a couple of particularly angry e-mails, however, I thought that it might be good do a formal write up that describes how to work around these incompatibilities. And while no one likes to squash bugs or fix things that previously worked, rather fortunately, every one of these problems can be overcome with a little bit of effort and patience.

Reset Your Preferences

Right. Now that the tone has been set, it might be good to talk briefly about the changes that caused the majority of the problems. As explained in Time Drive 0.2 overview, Time Drive has a completely new settings system. In order to support Amazon S3 and a few of the more advanced Duplicity settings, these changes were necessary; but at the same time, they introduced the aforementioned issues.

Here’s the bad news: as a general rule, Time Drive 0.2 can’t read the settings file from 0.1.0 to 0.1.5. It might try, but the results are not going to be pretty. The list below includes just a few examples:

If running Time Drive on a version of Python 2.5, you may experience sudden user interface freezes or unexplained errors.
The program might just refuse to do anything. For example, it may not let you open the configuration pane, run a backup, or a restore option.
The computer might suddenly start to lie to you, for example, it could say that a backup completed successfully when it really failed. Or that a file was restored, when instead it got sucked into some miscellaneous electrical chaos somewhere.

In all of these cases, there is only one thing to do. You’ll need to delete the old Time Drive settings file, which is found in the user configuration folder for Linux and Cygwin (Windows) users:

/home/Username/.config/Oak-Tree/Time Drive.conf

And in the Library/Preferences folder for users of Mac OS X:

/Users/Username/Library/Preferences/us.oak-tree.Time Drive.plist

After you find and delete that file, then you can launch Time Drive in the normal fashion. This will create a brand new settings file of the same name. Unfortunately, you will need to re-specify your backup location, re-add your folders or exclusion criteria, and reset your preferences. But once that’s done, you can pick up right where you left off.

Beware of Spaces

While the new preferences system is the biggest cause of upgrade problems, it wasn’t the only culprit. The next largest cause of problems happened to be us, the developers. In addition to the aforementioned “bugs” that were really features, we also inadvertently created new ones. The best example would probably be how Time Drive deals with spaces.

The underlying issue doesn’t have anything to do with Time Drive, actually, but is due to quirks in Duplicity. Put simply: Duplicity doesn’t like spaces in the backup destination. At all. It does okay when backing up to a secondary hard drive or USB drive, but that’s about it. SSH, FTP, WebDav and Amazon S3 backups will all fail. Or worse, it will cutoff the destination name at the first space, causing a raft of other problems. (/home/Jon Adams would suddenly become /home/Jon).

So … to fix this issue, we decided that we simply wouldn’t allow spaces in the backup destination any longer. For most remote connections, this was never much of an issue, since who ever heard of a space in an ftp or http url? But it did become a problem if one of your folders had a space in the name.

Let’s consider, for a second, that you added a folder named “My Documents” to your list of backup folders. Time Drive 0.1 would try and store all of the backup data at “/path/to/archive/My Documents”. If that data was sent over ssh, ftp, or webdav(s), you got all kinds of other obnoxious errors. But, if you happened to choose a local folder as your backup destination, it would work.

Time Drive 0.2.0 solved this problem by automatically replacing all spaces in the folder destination name with a dash. Thus, instead of storing your backup data at “My Documents”, it would be stored at “/path/to/archive/My-Documents” (note the dash).

But while this fixed the remote backup issue, it caused problems for people who made local backups. After the upgrade, Time Drive wasn’t might not have been able to find the previous backup sets, and would start all over again; an action that some people found annoying or confusing.

This problem has now been corrected. Time Drive 0.2.1 and later allow spaces for local backup paths. Remote backup urls will still have any space replaced by a dash, however.

Cron No Longer Works

A third very common complaint was that backup jobs through cron would no longer work correctly. Like the issue described above, this “bug” was actually a feature. For automatic backups, Time Drive uses the built-in Linux/Mac OS X cron daemon. Based upon the option that you specified, it would add then schedule itself for daily, weekly, or monthly backups and provide a link to the “timedrive-backup” script.

Well the location of that script moved. Previously, it lived in the “timedrive” folder. But in Time Drive 0.2, we changed how the program gets installed. It now uses a proper Python distutils script (setup.py) to copy the main program (time-drive) and the backup script (timedrive-backup) into usr/local/bin. The new location of the backup script results in a broken cron job.

To fix this problem, simply go into the preferences dialog of Time Drive and change the backup frequency to something other than what it is now, then, click the “Ok” button. You can then go change it back. This will cause the crontab to be updated with the correct location of “timedrive-backup”. And, like always, Time Drive on Windows doesn’t support automatic backups, since Windows doesn’t have a cron daemon.

Summary

And there you have it, three of the most common upgrading complaints. But please keep in mind, this list is far from exhaustive. Should you run into other problems, please leave a note in the comments and I will get back to with something that may represent timeliness. I may even be helpful, but as those encounters have yet to happen, overall that is yet to be seen.
______________________________________________________________

(Miscellaneous Request: When making comments, please be polite. I can certainly understand that you may be frustrated. After all, Time Drive might have just destroyed your only backup of your novel! Even so, rudeness probably not be tolerated;unless it happens to be deviously clever, that is. Cleverness and wit will always be tolerated, and may even be encouraged.)

Time Drive 0.2 – A “More S” Release: More Stable, More Secure, More Settings and Now Supports Amazon S3 Storage

Rob Oakes — Thu, 24 Sep 2009 18:41:57 +0000

The first time that you attempt to do something, it’s pretty much a guarantee that it’s going to suck. This doesn’t necessarily need to be a bad thing. Shows like America’s Funniest Home Videos and MXC have found dozens of way to cash in on the humiliation of their participants. (And what better exemplifies pure suck than a golf ball to the groin?)

It, therefore, shouldn’t come as any surprise that creative pursuits are no exception to the general rule of suckiness. After all, you have to overcome inexperience and ineptitude to produce anything. The only way to ensure that a release doesn’t suck is to finish a first draft and revise heavily. Which requires a great deal of work.

Yet … as interesting as that might be, this isn’t a post about the creative process. It’s about Time Drive and I should probably admit that Time Drive 0.1 had a few … rough … edges. Sure, it mostly worked, but it was new software and did too many strange things to declare anything other than a “work in progress.” But Time Drive 0.1 was a first release and first releases suck.

Second releases, however, offer a chance to clean things up, refine the bleeding edge, and otherwise deliver the goods. Maybe that’s why I’m so excited to announce the release of Time Drive 0.2. This version of Time Drive is a great improvement over it’s predecessor. So much so that Time Drive 0.2 is hereby dubbed the “More S” release: more stable, more secure, more settings, and Amazon S3 storage.

In the remainder of this post, I’ll attempt to justify such a silly name by taking a look at a few of those new features.

More Stable and Secure

First up, I should probably say something about the program’s stability and security. But before doing that …

Isn’t it a bit ironic that stability and security have become major selling points? Go ahead and look at the advertising for any major piece of software. References to stability and security – or their closely related siblings, reliability and refinement – will feature prominently.

As an example, let’s analyze the promotional materials for the most recent release of Mac OS X, Snow Leopard. In the banner advertisement below, stability and reliability form the cornerstone of the brainwashing message: “Refinements, your Mac gets faster and easier to use.” And in the text, we catch all the buzzwords: simplicity, reliability, security, speed. In short, they’ve attempted to make make old things new by refining them.

Time Drive has gone through a similar type of rewrite. With the generous help of Philippe Delodder, all of the core classes have been reformulated and reorganized: they’re new again. The end result is that all things are improved, spiffed and otherwise sped up. In short, Time Drive no longer resembles a prototype as much as a proper foundation upon which to grow.

More Settings

And nowhere is this more apparent than in the new advanced settings dialog. In version 0.1 Time Drive handled all the important things that a backup program should. It was easy to specify which folders you wanted to backup and where the archive was to be stored. But that was about it.

The first version didn’t let you fully use the power of Duplicity to specify what you considered important. What if you didn’t want to backup hidden files or settings? Or what if you wanted your work files to be stored in a different location than your home files? Or what if you want to create a new backup every couple of months?

Time Drive 0.2 gives you that degree of control without losing the simplicity. Just like before, you get started by telling the program where you want to store the archive and which folders to include. If that’s all you need, then you’re done. Click “Ok” and let Time Drive do it’s thing.

But why settle for simple when you can micromanage? Need to refine the settings for an individual folder, highlight it from the list and press the “Advanced” button. From there, you have easy access to things you don’t want included in the backup, fine control over where it gets kept, and even the ability to remotely manage the backup sets.

What You Exclude Can Be as Important as What You Include. Despite the fact that we should keep redundant copies of our data, sometimes there are things you just aren’t worried about. For example, you might be a Python developer and want to use Time Drive to backup your source code. The source code is important, but whenever a python script gets run, the python interpreter creates a compiled file (“.pyc”). These files are necessary for the code to run, but they don’t contain any valuable or new information. They just take up space. So, let’s leave them out them by adding “.pyc” to the exclude list. Just press the “Add” button and type the offending characters into the message box. Want to leave “.pyc” files out of all of your backups? You can do that too from the main exclude list.

Different Storage Locations. Now, what if you want your work related backups to be kept on your company’s ftp server instead of your local hard? Time Drive has you covered there as well. Just click on the “Archive Settings” tab and tell Time Drive where you want the folder’s archive to live. You can even specify a folder specific encryption key should you so choose.

Manage Your Backup Sets. Perhaps the only downside of creating frequent and redundant backups is that old files might take up needed hard drive space. Sure, managing backup sets by hands is possible … but who wants to do that? Time Drive 0.2 makes it easy to delete your old backup sets when they’re no longer needed, thereby saving valuable hard drive space.

Amazon S3 Storage

Which finally brings me to the biggest new addition to Time Drive, support for Amazon S3: a reliable, simple and inexpensive way to save your files over the internet. When it appeared a few years ago, S3 introduced a radically new idea: only pay for the storage you actually use. At the time, other providers charged a flat rate for a set amount of storage, regardless of whether you needed it or not. While prices ran the gamut of reasonable to outrageous, you were still paying for something that you might not be using.

Using S3, however, it is possible to have offsite and redundant backups while knowing that you are only getting charged for what you use. Moreover, the rates are quite reasonable, about 0.15 cents per gibabyte of data per month. For example, I currently keep all of my source code files and pictures backed up through Time Drive. In total, it amounts to about 15 GB of space. Amazon charges me about $2.25 a month for the space, which is substantially less than any other provider I’ve been able to find. Sure, there are plans that are slightly more expensive and provide more space; but that’s just it, I don’t need that additional space. I’m probably not going to use it. Which makes Amazon a really good fit for my needs. It’s cheap, it’s convenient and it’s available everywhere. Most importantly though, it’s offsite, so I’m prepared in case some sort of disaster does strike. And Time Drive works with it seamlessly.

Conclusion

In summary, I think that Time Drive handily earns it’s “More S” label. It’s a big improvement over the previous version and steps up the stability, the number of settings, and the types of storage backends that it works with. But it still remains elegant and easy to use. In short, it’s one step closer to the overall goal of “Backup Done Right.”

Head on over to the project’s home page and check it out.

Apolitically Incorrect » Computer

Hannibal, Napoleon, and Joseph Charles Minard

Hannibal Invasion of Italy (218 BC, Second Punic War)

Napoleon’s Invasion of Russia (1812-1813)

Conclusion

Unrelated Thoughts: Re-Visioning Minard with Open Source Tools

Acknowledgements and Further Reading

Similar Posts:

Typeset Your Curriculum Vitae – Part 3: Automatically Generate a List of Publications

The Basic Requirements

cvsplitbib and xetexCV

Creating a Categorized List

Generating the Bibliography

Examples and Class Files

Acknowledgements and Further Reading

Similar Posts:

Typeset Your Curriculum Vitae – Part 2: Extending and Customizing an Existing Document Class

Understanding Conventions

The Internal Anatomy of a Document Class

Class Description and Required Statements

\ProvidesClass

\LoadClassWithOptions

Building on the Foundation

\RequirePackage

Creating New Features and Modifying Old Ones

\def and \newcommand

\renewcommand

Rules for Creating New Macros

Managing Complex Formatting – \cvsection and \cvsubsection

Layout and Formatting

Conditional Formatting – \ifthenelse

Balancing Images and Text – \makecvtitle

Conclusion

Acknowledgements and Further Reading

Similar Posts:

Typeset Your Curriculum Vitae – Part 1: The xetexCV Document Class

xetexCV – An Attractive and Easy to Use CV Document Class

Downloads and Installation

Package Requirements

Curiculum Vitae Examples

xetexCV Class Usage

\makecvtitle

\cvsection and \cvsubsection

Hanging Notes

Choosing a Font

Other Features

Acknowledgements and Further Reading

Similar Posts:

Customizing LyX: Character Styles and the LyX Local Layout

Styles Defined

Using Character Styles in LyX

Customizing Character Styles

Example: Creating a “User Interface” Character Style

Step 1: Open your document in a text editor

Step 2: Create a Local Layout Section

Step 3: Define how you want the character style to look

Miscellaneous Things and Conclusion

Similar Posts:

Statistics With R – Part 1: An Old Dog Learns New Computing Tricks

Open Source Alternatives

Gnu PSPP

The R Statistical Project

Easing Into R

Series Introduction

Similar Posts:

Customizing LyX: Create an NIH Grant Proposal Template

Understanding the Big Picture

An Introduction to LyX and LaTeX

LaTeX Classes and Styles

LyX Layout and Template Files

Creating a Custom LyX Template – NIH Grant Proposal

Step 1: Download the necessary styles and templates

Step 2: Install the files into the proper folder for your LaTeX distribution

Step 3: Create an appropriate style layout for LyX

Step 3.1: Edit the document class and style to only contain general elements

Step 3.2: Generate a new LyX layout file

Step 4: Create a LyX template file

Using the Template

Similar Posts:

Create a Unified Inbox in Gnome Evolution