Apolitically Incorrect » Cool Stuff

Temples and Timelines

Rob Oakes — Thu, 25 Feb 2010 21:55:14 +0000

Newsweek has a fascinating article about an archaeological site at Göbekli Tepe in Turkey that is well worth a look.

The site is the oldest religious temple ever discovered. Preliminary carbon dating has determined that some of the artifacts date from 9,400 BC, which makes the place about 11,500 years old. (Which, just to be clear, is 7000 years before the Great Pyramid and 6500 years before Stonehenge.) The article further explains:

The ruins are so early that they predate villages, pottery, domesticated animals and even agriculture – the first embers of civilization. … [It] may be the very first thing that human beings ever built.

And yet, the site is amazing. The pillars show beautiful stone carvings and there are examples of sophisticated engineering techniques. The stone circles are nearly 30 yards across with pillars that stand more than 17 feet tall. Many of the stones (some weighing up to 50 tons) were first quarried and then transported half a kilometer to the site, where they were erected. What staggers me, though, is that the stone circles were roofed.

This quote from Ian Hodder, head of archaeology at Stanford University, summarizes my response pretty well:

[Göbekli Tepe] is unbelievably big and amazing, at a ridiculously early date. The huge stones and fantastic, highly refined art [changes everything]. It overturns the whole apple cart. All our theories were wrong.

This doesn’t happen often. Scientists don’t admit mistakes and call for established theories to be overturned. But when faced with such a revolutionary piece of evidence, you have little choice.

Göbekli is literally an outlier in every way. It shows engineering, organization, and artistic sophistication that seems to materialize out of nowhere. The only other comparable examples won’t appear for five thousand years.

To really put this in perspective, consider the timeline below. Arrayed across the bottom axis are the reigns of several ancient civilizations: the Chinese, Romans, Egyptians and Mesopotamians. In addition to this information, I’ve also placed the approximate dates of the the ice age, stone age and examples of religious and cultural monuments (the oldest of which dates to about 3500 BC).

When compared with Göbekli, the great civilizations and monuments of the ancient world seem to to huddle in an upstart mob at the right of the chart. Even the very oldest of the examples, a Mesopotamian palace, is separated from Göbekli by the same span of time that divides the ancient age from the modern day.

Such an amazing and sophisticated example at such an early date, literally, boggles my mind. It’s absolutely amazing. And, paradoxically, the amazement and wonder helps to explain why Göbekli has remained essentially unknown. A discovery of this magnitude demands enormous attention and dedication. It takes almost as much as it gives, particularly from those that discovered it; and not every scientist is willing to give that kind of commitment. Thus, I completely understand the response of the man who discovered the site.

[Unable to interpret what he saw], the [American] archeologist who stumbled on [on the site] in the 1960s simply walked away.

But, even so, the evidence at Göbekli has the potential to completely transform the history of civilization. And I, for one, look forward to seeing what emerges.

Note: You can view a high resolution PDF of the timeline by clicking on the image, or here.

Hannibal, Napoleon, and Joseph Charles Minard

Rob Oakes — Mon, 22 Feb 2010 21:49:11 +0000

No study of the history of scientific communication can be complete without mention of Joseph Charles Minard, a 19th Century French civil engineer and cartographer.

At the end of his life, Minard created two very famous examples of statistical charts, called flow maps, that every scientist, engineer and student should be familair with. The first showed Hannibal’s crossing of the Alps (218 BC, Second Punic War), and the second describes Napoleon’s disastrous invasion of Russia (1812-1813).

Both examples are beautiful works of art and masterful examples of evidence. But they are also more than that, they tell cohesive and interesting stories. In this post, I thought it might be interesting to take a closer look at the history of Hannibal and Napoleon, and highlight the ways which Minard’s charts help us to explain their eventual outcome.

(Note: High resolution, PDF versions of the two maps are available for download. These versions have been translated from the original French. To download, either click on the images, or here for the Hannibal invasion of Northern Italy, and here for the French Invasion of Russia.)

Hannibal Invasion of Italy (218 BC, Second Punic War)

Hannibal was perhaps the single greatest threat to Roman power to ever live. At the same time that Roman legions marched unopposed through most of the Mediterranean world, Hannibal lead a force of twenty-five thousand soldiers into the heart of Roman territory. He then spent the next fifteen years occupying portions of the Italian peninsula and generally wreaking havoc. But even though he ravaged the Roman countryside, Hannibal was never successful in subduing Rome. Ironically, this might be due to one of his greatest exploits: the overland crossing of the Alps.

Hannibal departed his home base in Spain with nearly 90,000 troops (one of the largest armies that had ever been fielded at that point in history) and had a number of early successes. He marched north over the Pyrenee mountains and into Gaul (modern day France). He subdued the Celtic tribes and crossed the Rhone River before the Romans could take measures to stop his advanced. He forged important alliances and found unexpected friends in his fight against the Romans.

After crossing the Rhone, however, Hannibal’s luck began to change. He received word that a large Roman force was marching from the Mediterranean coast to intercept and destroy him. As the force outnumbered him by a significant margin, Hannibal was faced with a difficult choice. Should he stay at his current location and engage the Roman army or retreat and prepare a defense?

Not content with either choice, Hannibal ultimately chose a third option; which, ironically, would have enormous consequences for his overall campaign. To evade the Romans, Hannibal took the unprecedented action of turning inland and marching over the Alps. (It is thought that he probably crossed over the valley of the Drome and south of the Col de Montgenevre or near the Col de Mont Cenis.) This had never been done before – indeed many thought it impossible – and has long been praised as a brilliant tactical decision.

Minard’s reconstruction of Hannibal’s journey shows the journey from Spain, through Transalpine Gaul and eventual arrival in Italy, allows it to be seen a slightly different light, however. Through use of a flow line, Minard shows how the strength of Hannibal’s army waned through the march, with a disastrous loss of twenty thousand men while passing over the Alps. This steady loss of strength, subtly reinforced by Minard’s map, helps to explain why Hannibal failed to subdue Rome.

Hannibal’s brilliant tactical decision was also a strategical disaster. It came at a dramatic loss of men, war elephants, and material with the most disastrous loss being the destruction of his siege engines. Without the siege engines, Hannibal was never able to successfully attack Rome. And eventually the Romans realized this and changed their tactics. Instead of engaging Hannibal directly, they instead fought a long-pitched war of attrition and eventually destroyed the one resource he couldn’t replace: his men. Thus, even though crossing the Alps helped Hannibal to win a few stunning victories, the decision eventually caused him to lose the war.

Napoleon’s Invasion of Russia (1812-1813)

Like in the case of the Hannibal, Minard’s depiction of the 1812-1813 invasion of Russia also tells an extraordinary story. It combines information about the size of the army, geography, sub-campaigns and temperature into a single narrative. This time, though, it is anything but subtle.

Note: Because of the elegant management of all this information – at any time showing the army’s direction, size, and loss relative to the start – some statisticians and data visualization experts (such as Edward Tufte, Etieene Jules Marey and Howard Wainer) have declared Minard’s depiction of Napoleon’s Invasion of Russia to be the greatest numerical graphic ever created.

From the very beginning, the army met with tremendous hardship. The cold temperatures and lack of available resources took an enormous toll. Napoleon had planned on scavenging for supplies to support his massive army, but the Russians had destroyed all of their crops and burned their villages so that Napoleon wouldn’t be able to make use of them. As a result, the army was heavily reliant on slow supply trains from Europe that were unable to keep pace with its quick advance. Hundreds of thousands died from starvation and exposure, greatly outnumbering those lost in combat.

Minard’s map shows this trend of devastating loss. Napoleon’s Grand Army starts out as a wide, brown line to the left of the map. As the army progresses across Russia, the number of men decreases dramatically. A few ribbons of men are dispatched to other targets, but by far the majority succumb to the elements.

Eventually, a substantially reduced force (100,000 men, about 20% of those who began the campaign) arrived in Moscow. What they found did little to improve their circumstances. The city had been abandoned and burned to the ground. No food was available. The only available option was to return to France.

Demoralized and broken, the Grand Army began a slow retreat to the west. Their progress is shown in the black. As they crossed the vast Russian wilderness, the temperature turned bitter cold (shown in the bottom third of the map). When the army arrived at the Niemen river, there were a scant 10,000 troops. More than 400,000 had died.

Such a devastating loss of life crippled Napoleon’s young empire and left him open to invasion by his enemies. He was defeated later in 1813 and exiled to the island of Elba.

Conclusion

Minard’s graphics are successful because they demonstrate powerful trends and place evidence within its proper context. In the case of Hannibal, it shows how he began in a position of strength and then undermined himself through a brilliant tactical innovation. In the case of Napoleon, it shows how distance and temperature ravaged an army and destroyed an emperor.

The clear evidence and thoughtful presentation define the story, they show where things are headed and help to foreshadow the inevitable conclusion. For this reason, they are powerful examples of clear communication.

____________________________________________________

Unrelated Thoughts: Re-Visioning Minard with Open Source Tools

Since first seeing these charts as part of an undergraduate statistics course, I’ve been obsessed with them. I even have a poster sized copy of the Russian invasion (available from Graphics Press) hanging in my office. Given this obsession, it should come as no surprise that I’ve wanted to include the maps in my book.

There is, however, just one problem; there is not anything new to say about them. Edward Tufte has held up Minard as a “Da Vinci of Data” for the better part of thirty years. Nearly every one of his books includes a copy of the maps (in the case of Beautiful Evidence, his most recent work, there is an entire chapter devoted to the graphics; including eight different reproductions). Other than, “They’re beautiful, study them carefully, read Tufte’s analysis,” there just isn’t that much to say.

That is, until I started looking for examples of scientific illustration that could be reproduced using the vector illustration program, Inkscape. Any such example should have simple lines and clean integration of text and images. I’m not sure that a better description of Minard’s work exists.

The two images above (and attendant PDFs) were my first pass at re-drawing the maps. And I have to say, I’m rather happy with the result. For the most part, I was able to get a very faithful reproduction (with the exception of the type face, which I deliberately changed because I hate script fonts), and only required a couple of hours worth of work.

I think that I may have found my book examples. It allows me to include one of the best statistical charts of all time, talk about flow maps, and even include some interesting how-tos which show how the graphics can be built in an automated fashion. Which, come to think of it, might just count as a new contribution after all.

Acknowledgements and Further Reading

More information on the book can be found at the publisher’s website.

The source images for the two maps included “Re-Visions of Minard” by Michael Friendly (1999) and the reproductions in “Beautiful Evidence” by Edward Tufte. Translation for the Hannibal map was adapted from those available on Edward Tufte’s website (Dawn Finley) and sources available at Re-Visions of Minard. Translation for the Russian campaign map were taken from the reproduction in Edward Tufte’s “Beautiful Evidence”, Wikimedia Commons and sources available from Re-Visions of Minard.

Additionally, the Re-Visions of Minard Website include a number of interesting adaptations of the Napoleon graphic that might be of interest, including both Matlab and GGPlot2 reconstructions.

Image Gallery: The Notebooks of Leonardo Da Vinci

Rob Oakes — Fri, 05 Feb 2010 04:31:37 +0000

When I graduated from college and had to choose between a career in industry or academics, I found it to be an easy decision: I stayed in academics. I like to have my head in the clouds and enjoy the intellectual lifestyle. (I actually consider the label of “absentminded” to be a compliment.)

It should come as no surprise, then, that one of the most enjoyable aspects of writing a book has been the opportunity to research my subject. My reading list has included books on analytic design, illustration, anatomy, typesetting, scientific communication, web technologies, LaTeX, the history of science, statistics and informational graphics. And as I worked my way through it, I took some extremely interesting side trips. One of the most intriguing, however, was an extended tangent through the notebooks of Leonardo Da Vinci.

Da Vinci died in the year 1519, nearly five hundred years ago. Yet, the modern world remains fascinated by him. His name adorns the side of best selling books and conspiracy fiction; and his drawings have become cultural icons. As an example of his popularity:

In October of 2009, Martin Kemp, a professor of art and history at the University of Oxford, found a portrait of an Italian girl. Up until Kemp took an interest, it was widely accepted that portrait had been painted sometime in the nineteenth century by an unknown artist. After a great deal of investigation and the use of a multispectral camera, however, Kemp discovered something startling. The painting had actually been done by Leonardo and nearly overnight, it went from a value of 19,000 British pounds to over 100 million.

I’m no different than the masses. Leonardo fascinates me. He had a very distinctive way of seeing the world and an engaging style. Maybe that’s why it’s so easy to lose yourself in the details of his work. Given my interest, a thorough study of Leonardo’s notebooks seemed only natural.

What I didn’t foresee, however, is that I would start to digitally collect his sketches; and in the past several months, I’ve put together a rather eclectic mix from across the internets. Earlier today, I realized that the images might be of interest to others as well; thus, I’ve created a special online gallery for them. It can be found under “Art and Photography” –> “The DaVinci Notebooks” To get there more quickly, you could also just click here.

Enjoy!

About This Book Thing

Rob Oakes — Thu, 28 Jan 2010 21:22:09 +0000

In the past few weeks, I’ve had several observant readers ask about one of my “secret” projects. They’ve wondered what I’m up to and why it’s detracting from other endeavors. After answering another query this morning, I decided that it’s probably time to speak openly about it. So, here’s my public confession: I’m writing a book.

It’s about scientific and professional writing and open source. Moreover, it will be interesting, intriguing and revolutionary. (Yes, I have an inflated sense of ego.)

Before really diving into the details, I’d like to give a bit of personal background. This might help you understand why I’m passionate about the subject.

Background

Ten years ago, had someone told me that I would end up a scientist and engineer, I would have laughed at them. At the time, I had just started at University and I was fully set on a career in either illustration, design or architecture. I was much too “visual” and “right-brained” to surround myself by geeks, freaks and nerds. It didn’t help that I spent a huge amount of time grooming myself to be an “artist”.

During high school, I had been cursed with moderate talent and highly indulgent instructors. They praised my artwork. They called it interesting and innovative. They encouraged me to refine my technique and to major in visual arts. So, I did.

But as time went on, I realized that I wasn’t very happy. I realized that I had other interests. I enjoyed art, I did well in it; but art classes weren’t my favorites. That honor, as it turned out, was reserved for mathematics and science.

There was also another problem, I found that I lacked the discipline required to systematically create an individual style and build a portfolio. I wanted create art for myself, not for other people; and that is a fatal flaw in an illustrator (the type of work that most interested me). Illustration, by definition, is work that has been requested for a particular use. I was more interested in my own whims than those of potential clients. Thus, not long after recognizing my problem, I decided to go a different direction and changed my major to engineering.

Analytic Design

But even though I condemned myself to a life of investigation and exploration, I remained very interested in art, photography, layout and design. This interest (and some substantial urging by the doctor I worked for at the time) resulted in a fascination with where science and art seemed to intersect.

And this is when I noticed, like many others, a strange trend. If you look at the work of those considered the greatest minds in history – Leonardo Da Vinci, Isaac Newton, Galileo Gallilei, Thomas Jefferson, Marie Curie, Albert Einstein, etc. – you find something intriguing. In nearly every case, those recognized as brilliant weren’t judged solely on the quality of their thinking, but also by how they presented and explained their insights. Indeed, the thing identified as “brilliance” was not merely exceptional thinking, but clear communication. After, all, clever thoughts and ingenious solutions mean nothing if they can’t be captured, understood and disseminated.

What is often overlooked, however, is just how much work is required to explain ideas which aren’t inherently simple, and that raises an important problem. Communication in general, and writing in particular, is very hard. To distill random and chaotic thought into a coherent and logical narrative requires energy, time and effort. This is especially true of scientific, technical or professional subjects.

Such documents contain an enormous amount of baggage: figures, theorems, mathematical equations, algorithms, cross references and indexes. And while such things simplify the life of a reader, they can positively drown a writer.

Tools and Technology

Luckily, however, there are tools and technologies that can simplify the organizational side of communication and allow for a writer to focus on the important parts: words, pictures and evidence. But finding these tools and making them work together can be a very daunting task. And that’s why I’m writing a book.

In its pages, I want to introduce principles and ideas that are characteristic of good communication. For example, why was Albert Einstein able to convince both scientists and the general public that E=MC^2? Why do we still read the scientific treatises of Isaac Newton or Charles Darwin – centuries after their ideas have been edited, amended, extended and restated? Why does scientific communication often fail and what can be done to prevent breakdowns or misunderstanding?

Then, I want to show how those principles can be applied. I’ll look at open source tools and how they work. What programs are available for capturing information and references, organizing thoughts, and eventually creating a cohesive manuscript? How can open source tools be used to create beautiful charts, maps, graphs and tables? How can the process of collaboration be simplified?

Finally, I want to show why open source is awesome. I want to demonstrate why it is functionally superior to proprietary alternatives; and I want to do this in a way that shows it to be accessible. (Far too often, I find that people are afraid of open source because they view at as “complicated”, which is very unfortunate.)

Conclusion

When I first switched from art to science, I found myself confused, lost and overwhelmed. Not only did I need to adjust to a completely new discipline and culture, but I also found that no one used the same computer programs! And while it took some time to understand why, once I did, I realized that I would never go back to the “old” way of doing things.

Thus, I’m writing the book I wish someone had given me as a new science student; a book that explained the functional bits of scientific and professional writing with lots of examples, code, and pretty pictures. This is what I’ve tried to put together, something that is part theoretical treatise but mostly instructional manual.

So, for those of you who have wondered where I’ve been and where I’ve wandered; that’s the answer. I’m writing book. The good news, however, is that it’s nearly finished. And it’s in a state where I can start to share thoughts and examples from its chapters. In the next few months, expect to see some of that information here.

But because I’m passionate about the subject, I would love to hear others have to say. What makes for a good piece of informative writing? What are examples of clear communication? What lessons might we in the sciences learn from others? And what tricks do you have for being more efficient?

Create a Unified Inbox in Gnome Evolution

Rob Oakes — Tue, 27 Oct 2009 07:43:00 +0000

I have a serious love-hate relationship with Linux. I love the fact that it’s free and open source. I love the fact that it can breathe new life into old hardware. I love the fact that it’s easy to extend. I love the fact that it has a vibrant and passionate user community.

What I do not love is that many open source programs are incomplete. They can do most everything that you need, but never get around to adding the one or two features that prevent them from being finished, polished and exceptional. I’ve ranted about this before, back when I was trying to find the perfect backup program.

Well … I’m at it again; except this time, I’m looking for the perfect email program.

For a long time, I thought that I had found it in Thunderbird 3. Thunderbird is powerful, highly configurable and easy to use. Moreover, alpha and beta builds hint at changes that will make it even better. The most obvious of these is a universal Inbox. This is one area where I should give Apple some credit. They figured out a long time ago that it’s better to have all your emails in the same place, regardless of what account they may belong to. It makes things easier if the communication from your your work account, home account, webmaster account, software development account, or hobby account all live in the same place. You just read the email and respond, the program makes sure that it gets routed along the proper pipes.

This makes it really easy to apply selective filters. On my home computer, I only subscribe to those email accounts for home related things. On the work computer, I subscribe to work related accounts. On my laptop, I have all of them.

But most people use their email client for much more than just sending messages. About ten years ago, Microsoft showed that calendar and tasks should also be part of the email application. It’s more convenient when the tools that you use to organize your life are in the same place as those you use to communicate. After all, who wants to use three or four apps when you can use one?

Yet, on both these fronts, open source has been pretty slow to catch up. The integrated inbox just appeared in Thunderbird. And while there is a calendar component, it’s pretty limited. It does alright with a local calendar, but a calendar chained to a single computer is mostly worthless. Thunderbird can’t connect to an Exchange server, where your business calendar is likely to live. It can’t talk to the family’s calendar, which might be kept on a consumer service like Yahoo or Google. And while it has experimental support for CalDav, that’s the problem, the support is still experimental. I’ve found that tasks or appointments don’t update reliably, and it only works when you are connected to an internet connection.

Perhaps worst of all, Thunderbird can’t talk to your cell phone. If you’re anything like me, my cell phone is my second memory. If I don’t have it written down and in front of me, I’m going to miss it or fail to do it. As a result, a robust cell phone sync is a must. Taken together, these limitations and rough edges are just too much. I need an alternative, even though I love what Thunderbird does for my email.

This is where the venerable Gnome Evolution steps in. In contrast to Thunderbird, which is promoted as an email application, Evolution is a collaboration app. That is to say, it takes calendar, contacts, tasks and notes seriously. CalDav? Got it covered. Google? Ditto. Yahoo calendar? Yep. Microsoft Exchange? You bet.

They have also given some serious thought to integration and user experience. Evolution is deeply integrated with the Gnome desktop. The desktop drop down calendar shows your appointments. Email and notifications are integrated into the dashboard. You don’t even need to have the application open!

But Evolution crashes and burns as an email client. It lacks all of the subtle tricks that Thunderbird, Mail.app and Microsoft Outlook use to keep organized. As just one example, the folder list is overwhelming and it’s hard to know when you’ve received new mail. Thunderbird and Mail.app solve these problems through their unified inbox. Microsoft Outlook solves it by having a “favorites” area where you can create shortcuts to folders that you ca re about. When you receive a new email message, this shortcut is bolded. It’s not quite as effective as the universal inbox, but at least it’s something. Evolution has nothing.

As a result, even though I’ve loved the calendaring and integration with the desktop, I was terribly afraid I would have to dump yet another application because it offered a substandard experience. Luckily, I found a workaround.

There is a pretty simple way to create a unified inbox in Evolution. It was so easy, in fact, that I was surprised that the Evolution developers haven’t taken the time to make it a standard part of the program.

The magic happens through the search folders. In a nutshell, you create a search folder that looks through all of your account inboxes for either read or unread email messages. Presto, a unified inbox!

Start by right clicking on the “Search Folders” entry in the folders list, then select “New Folder.” It will load a dialog similar to the one seen below. You will need to add two conditions. Set the first condition to “Status Is Read” and the second to “Status Is Not Read” as shown in the screenshot below. When done, press the “Ok” button.

The same thing can be done for your drafts, or outbox folders. Pretty slick, huh? What I find incredible is that such a simple hack makes such a huge difference in usability. Instead of separately monitoring six different email account settings, I can check just a single folder for new things. It transforms Evolution from a pain in the ass to a pretty decent little email client.

Do you have your own Evolution hacks? If so, let’s hear about them in the comments.

Install Time Drive On Ubuntu and Other Debian Linux Distributions

Rob Oakes — Mon, 05 Oct 2009 22:23:01 +0000

Since version 0.2.2 of Time Drive, installing Time Drive and keeping it up date has gotten a great deal easier. Instead of requiring that you install from source, we now offer Debian based packages through the Time Drive Personal Package Archvie (PPA). This article will briefly describe how to install Time Drive using that PPA.

Background

But before leaping headlong into the nitty gritty, let me provide a bit of background about packages and repositories. Unlike Windows or Mac OS X, software on Linux is organized into a container called a package. The package contains the files needed to run the program in addition to artwork and configuration information. Any given program may require many other programs to be installed before it can run. These other programs are referred to as dependencies. Ubuntu and uses a tool for managing packages and dependencies called Apt.

The benefits of using Apt over manually installing things is that Apt figures out all of the dependencies automatically. Further, it keeps everything up to date by periodically scanning the online repository (PPA) and downloading any updates. In practice, Apt works much like the Windows Update service, only better, since it monitors every piece of software installed on your computer.

Installation Instructions

To install Time Drive using the PPA, we need to do the following:

Add the Time-Drive-Devel and Duplicity PPA to your Ubuntu Software Sources.
Configure Ubuntu so that it will trust both Time-Drive-Devel and the Duplicity packages.
Install Duplicity and Time Drive

Step 1: Add the Package Repositories to the Ubuntu Software Sources

Open the Ubuntu Software Sources by going to the System –> Administration –> Software Sources. Then, select the “Third-Party” software tab of the application. “Software Sources” requires root access, therefore, you will be prompted to type in your administrator password.

We need to add two package repositories to the list, one for Duplicity and the other for Time Drive. Depending on the version of Ubuntu that you are using, copy and paste the following lines (one at a time) into the “Add Sources” dialog. After each line, press the “Add Source” button.

For Ubuntu Karmic Kaola (9.10):

deb http://ppa.launchpad.net/duplicity-team/ppa/ubuntu karmic main # Duplicity – Ubuntu 9.10

deb http://ppa.launchpad.net/time-drive-devel/stable/ubuntu karmic main # Time Drive – Ubuntu 9.10

For Ubuntu Jaunty Jackolope (9.04):

deb http://ppa.launchpad.net/duplicity-team/ppa/ubuntu jaunty main # Duplicity – Ubuntu 9.04

deb http://ppa.launchpad.net/time-drive-devel/stable/ubuntu jaunty main # Time-Drive – Ubuntu 9.04

For Ubuntu Intrepid Ibex (8.10):

deb http://ppa.launchpad.net/duplicity-team/ppa/ubuntu intrepid main # Duplicity – Ubuntu 8.10

deb http://ppa.launchpad.net/time-drive-devel/stable/ubuntu intrepid main # Time-Drive – Ubuntu 8.10

When finished, press the “Close” button. Ubuntu should then notify you that the software information is out of date and needs to be updated. Instead of clicking “Reload,” click the “Close” button. We will refresh Apt from the command line in Step 3.

Step 2: Configure Apt to Trust Time Drive and Duplicity Packages

After adding the repository information, you will then need to add the repository key to your system’s list of trusted keys. This is most easily done by running the update-launchpad shell script. (To download, right click on the link and select “Save As”.) When run, the script automatically downloads any needed signing keys and automatically add them to the “Trusted Sources” list.

To run the script, go to Accessories –> Terminal and navigate to where you saved the file, then type:

cd /path/to/folder
sudo bash update-launchpad.sh UbuntuVersion

Be sure to substitute the correct Ubuntu code name for UbuntuVersion in the second step – karmic for 9.10, jaunty for 9.04, or intrepid for 8.10 – otherwise, you will get an error. I happen to use Ubuntu 9.04, therefore, for my setup, I would type the following:

sudo bash update-launchpad.sh jaunty

Step 3: Install Duplicity and Time Drive

Finally, we are ready to install Time Drive and Duplicity. From the command line, refresh the package lists by typing:

sudo apt-get update

Then, to install time-drive and duplicity, type:

sudo apt-get install time-drive duplicity

Application Updates

As noted above, whenever a new version of Time Drive, Duplicity, or another dependency is added to the package repository, your system will automatically be updated. However, should you upgrade your system to a newer version of Ubuntu, you will need to return to this page to add the repository for that version.

__________________________________________________

Note: These instructions were modified from this tutorial on WineHQ.

Time Drive 0.1.5: Incremental Changes

Rob Oakes — Fri, 14 Aug 2009 15:19:10 +0000

It’s been an interesting couple of days. I was rather honored to see that Lifehacker did a short highlight of Time Drive, which I thought was pretty cool. It’s always been one of my goals to have something featured in Lifehacker or Gizmodo, and now I’m going to have to scratch that off the list of goals. But that’s okay, I’ve got other things to fill the void. Like … how exactly does one get invited to present at TED?

On another note … while I knew that I would see some kind of traffic bump due to the article in Lifehacker, I wasn’t necessarily prepared for the magnitude. In mathematics, there is this thing called a step function. It’s where you move from one value to another more or less instantaneously. It looks like a step, hence the name. Sure, It may not actually exist, since even very dramatic shifts still have a non vertical slope; but even so, the change in my traffic might as well be a step-function. Between yesterday and today, I’ve had more visits to this site than I’ve had in much of the rest of the year combined. I think that’s kind of cool, though it probably won’t last.

(This might be a good time to say that I am actually rather proud of my “lackluster” web traffic. Though it might not necessarily be that impressive, it is, nevertheless, mine. I’ve worked hard for it, and I revel in the fact that some 40 to 50 people each day find the unorganized garbage of my mind intoxicating. Some of them even come back!)

But as interesting as that might be, traffic stats is probably not why you’re here. Good thing, since I’ve got announcements.

Time Drive Update

As of this moment, Time Drive has been updated to version 0.1.5 and is available on the project page.

Let me start by saying what this release isn’t. I really wanted to write a blog post and say, “Time Drive is available, and now it’s really easy to install!” In the past few days, this has been by far the most requested “feature” and I’ve tried hard to deliver. (Honest!) Unfortunately, creating a binary installer for Windows and a self contained package for Mac has proven … illusive. I’m not really a programmer and the inner workings of py2app and py2exe are tremendously frustrating. I have made some serious headway, but stand-alone installers are still “forthcoming.”

But a lot of things are new, and some of them are even shiny! Moreover, the relatively huge interest has gotten me excited about the program, and I’ve spent some time refining things that didn’t quite work right. “So …” you ask, “What’s different?”

New Stuff

For starters, the Snapshot History on the main window actually works now. If you want to restore a previous version of a file, you can select it from the list and click the add button (+) to open the archive browser. It will automatically load the snapshot as it looked on the selected day and time.

If you can’t find your file there, don’t despair, from within the archive browser, you can easily navigate forward and backward between snapshots by using the arrow buttons or by dragging the slider to the appropriate place. Once you have it sitting on the snapshot of interest, hit the refresh button to reload the file list. This is starting to get closer to the overall vision of the program that I originally stole. Now, you can actually use Time Drive to keep tabs on how your files change.

I’ve also been trying to get things cleaned up so that the program runs better. I went back and modified the code so that all long running processes use their own threads. This should help the UI from freezing and lets you do several things at once.

I also cleaned up the way that Time Drive interfaces with cron. Now it works on Mac without requiring tweaks. In the relatively near future, I’ll write a proper backend for Windows so that is uses the Windows Vista task scheduler. Installing cron on Cygwin is just too much of a pain.

Looking into the future, I foresee support for Amazon S3, pruning and naming snapshots, and advanced folder options. There may be other changes as inspiration strikes. Expect additional announcement soon.

Conclusion

Well, that’s it. Why this may be a bit of a minor addition, there’s still some nice stuff. (I know, I’m biased.) Head over the download page and check it out.

Backup for Linux, Done Right – Part 2: Time Drive 0.1

Rob Oakes — Fri, 07 Aug 2009 17:39:48 +0000

In part 1 of this article, I shared a few of the frustrations and reasons why I decided to write my own backup utility rather than submit to the tyranny of currently available solutions. While some might find those ruminations interesting, the vast majority are probably far more interested in the end result. There is a reason why “Get to the point” is one of the most important sentences in the English language.

Here’s the short version: After becoming tremendously frustrated by the state of backup on Linux, I decided to take matters into my own hands and create my own tool.

And though I only want a few things, I want that tool to do each very well. First, I’m looking for a solution that can incrementally backup over the network and let me restore a file from an existing snapshot. Second, those snapshots should be compressed, encrypted and secure. Third, it should be easy to browse old backups for existing files and restoration should be a one-click affair. Fourth, I want a backup system that can protect me from disaster, carelessness and pathological stupidity.Time Drive is based on the excellent command line program, Duplicity though it was also inspired by another backup utility, Back In Time. Because it uses Duplicity for its backend, Time Drive plugs into the same industrial strength utilities used to backup and restore enormous sets of enterprise data. It just removes the industrial level complexity.

Like Back In Time for Linux and Time Machine for Mac OS X, Time Drive not only keeps a copy of the most recent file, but it remembers how that file looked at many points in the past. Put another way, it creates an incremental snapshot which lets you restore to a previous version should you so desire.

The Time Drive main window is designed to be simple and functional. Options related to backing up files are on the left, options related to restoring files are on the right. If you want to restore from a remote archive which is not currently specified in your settings, you can do so through the “Advanced Restore Options.”

Over the Network or Across the World

Backups can be made to remote servers and web services that support any one of the following protocols: file transfer protocol (ftp), secure shell (ssh), secure copy (scp), rsync, webdav (http), and secure webdav (https). Time Drive also supports backups to a locally attached drive or mounted volume. This includes shares that are mounted with virtual filesystems like sshfs, smbfs, and s3fs.

You can configure the remote backup by clicking on the general tab of the settings pane. Then, move the option to “Use a remote directory (SCP, WebDav or FTP).” From the drop down list, choose the communications protocol.

If making a backup to ftp or webdav (http), enter the name of the host and the path to your backup folder as a standard web url. It should look like:

computer.name.com/path/to/folder

For example, the backup path to my webdav share is:

backup.oak-tree.us/RobOakes/Backup

If backing up by scp or ssh, you will need to use a second slash after the server name, for example:

backup.oak-tree.us//RobOakes/Backup

Next, enter the username and password that you use to access the server and some sort of passkey in the “Encrypt Files” box. (While future versions of Time Drive will allow users to choose whether or not they wish to encrypt their files, this has not yet been implemented.) When finished, you can check the connection by clicking on the “Check Connection” button to the right of url text box. If Time Drive is able to connect to the store, a little green check mark will appear.

Restoring Files

Should you delete a file that you needed, you can quickly browse or search the archive to find it and restore it. Need to restore a bunch of files? You can add as many as needed to the queue and then restore them with a single click. The restore dialog is accessed by clicking on the “Add files” button of the main window. The restore dialog itself allows you to browse the folders available in your backup, and any subfolders of the archive.

But what if you need access to a file in your work archive and you happen to be at home? Time Drive makes it easy to brose and restore from any archive, anywhere. Simply type in the url and click “Add files” to enter the restore files dialog.

Set It, Then Forget It

While Time Drive can be configured to work many different ways, file backup should ultimately be a very simple affair. True, it’s nice to have more advanced tools available, but most things in life aren’t really all that complicated.

As a result, setting up Time Drive for the first time is designed to be as easy as possible. First, pick where you want to perform your backups: either to the local machine, or across the network. Next, enter in a passkey to encrypt your backup files and keep them safe. Finally, choose which folders you want to see included in the job. When finished, click “Ok.”

Because each folder is a run as a separate duplicity script, you can always re-add that folder at a later time and Time-Drive will update the existing snapshot with the most recent information. All of the existing history will be there should you ever need to access it.

Time Drive also makes it easy to schedule automated backups through the use of the powerful crontab utility available on most *nix machines. Because it uses cron, there is no need to have a dedicated background process. (Though cron must be running for scheduled backups to work.)

Automated backups are configured by opening the settings dialog and navigating to the “Options” tab. Then, just pick how frequently you would like the program to run: every hour, day, week or month. When done, select “Ok.”

Notifications and Feedback

When looking at other solutions, I quickly became frustrated by the way which they handled user notifications. In some cases, the program provided a level of detail that was simply overwhelming. In others, it didn’t provide any feedback whatsoever; even when there was a problem.

With Time Drive, I’ve tried to split the difference. It uses the notification system of Ubuntu 9.04 and will let a user know when new backup jobs start and it finishes, if there is a problem, it will let you know that too. But for those who would rather not be bothered with such trivial things, the notifications can be easily disabled through the settings tab.

Beta Software

Even though Time Drive is relatively full featured and offers a convenient backup package, it is still beta software. That means that it needs to be tested, beat upon and otherwise tortured. As you do so, I would ask that you be a bit patient and mind the potholes. If it doesn’t quite meet your needs, please leave a bug report or feature request on the bugtracker page. Be as specific as possible and I will try to incorporate your ideas into the next release.

Conclusion

With Time Drive, I finally have a backup utility for Linux that does things right. It can backup to a computer on the network or across the internet; keeps incremental snapshots of only the files that have changed; makes file restoration a one-click affair; and keeps my personal information safe. What’s better, no tweaks or hacks are required to make it work the way that I want. In short, I finally have something that “Just Works” without having to stick some stupid qualifier in front.
______________________________________________________________

If you are interested in downloading, testing or using Time Drive, you can find additional information on the project homepage. This includes installation instructions for Time Drive, Duplicity and PyQt in addition to the program documentation.

Backup for Linux, Done Right- Part 1: A Mini Melodrama

Rob Oakes — Fri, 07 Aug 2009 04:58:08 +0000

It is a terrible thing to realize that you are stuck in a rut. Being in a rut effectively means that you’ve stopped advancing and life has evolved to monotony. No one likes to be around people in ruts, but it’s even worse to discover that you are personally trapped in one. And, most unfortunately, I am in a rut.

Don’t believe me? Take a look at the home page of this blog. You will likely notice that a full six of the ten most recent posts have dealt with one subject: backing up your computer. That’s pretty conclusive evidence of a rut.

Now, backing up your computer is a very important thing to do; you should do it regularly and have a plan. But … well … it’s boring. Talking, thinking and writing about nothing but backup is dull. As one of the doctors I work with likes to say, “That isn’t sexy. If I’m going to spend any time with it – women, food, wine; it doesn’t matter – it should be sexy.”

He’s got a valid point, backup is not “sexy” and I’d like to write about things that are, at least for a while. This, therefore, will be my last post on backups, archives, or servers for the relatively foreseeable future (technology is just too cool to lay it aside for too long). But before doing that, I want summarize where I ended up in my quest for the ultimate backup system.

Backup on Mac is taken care of, I use Time Machine to a Samba share. More adventurous persons than I might even say that this arrangement approaches sexy. It’s convenient, fast, and robust. It even covers disaster recovery.

Backup on Windows is also covered. The built-in file backup is easy to use and works well. Moreover, setting up a disaster recovery system is relatively painless.

But the third major operating system, Linux, is a bit of the odd-man out. Certainly, you can find some excellent backup systems, Back In Time is one such example. With a bit of work, you can even tweak it so that it is almost perfect. But it’s the “almost perfect” and closely related cousins (“mostly useful” and “good enough”) that are the problem. They have those stupid qualifiers – almost, mostly, enough – bolted on.

Any time you hear a qualifier, you can rest assured that you aren’t going to like what follows. Consider the rather innocuous phrase, “that may be a problem.” Here, the term “may,” makes an already bad situation much worse. Instead of specifying some probability of problemhood, it all but guarantees it. Positive qualifiers are just as bad.

As a result, it angers me that nearly every backup program available for Linux requires some kind of qualifier. It shouldn’t be like this. Linux is a brilliant operating system in practically every way. It is highly integrated, wonderfully modular and tremendously easy to extend. So … after finding that nearly every backup utility in existence has failed to meet my needs, I found the situation intolerable and decided to do something about it.

I wrote my own.

Backgrounds and Backends

Fixation and ruts can make you do silly things like that. Instead of just accepting the limitations of an existing situation, a fixated person will demand that the world bend to their expectations. While this sometimes leads to great advances, more often it results in interpersonal disasters of epic scale. And the general rule is, the more menial the detail, the larger the scandal. (At this point, it might be argued that backup to an external hard drive versus across the network is a rather menial detail. To all such detractors, all I can say is: stuff it. This is my story.)

Basic Needs

Luckily, however, the story of my backup utility isn’t nearly so sordid. This is probably because my needs are actually pretty simple. I need a backup utility to do just a few things, but I require that it does them well:

It should create versioned snapshots of my drive. This lets me restore a file to any number of past states.
The utility should only transfer the portions of the file or directory that have changed. This makes the backup operation quick and efficient.
I should be able to backup over a network or across the internet. (Very important!)
Restoring a file should be quick, easy and painless. A backup is only as good as your ability to get at the information. And like it or not, information will be lost due to disaster, carelessness or miscellaneous stupidity.

Being a somewhat clever person, I didn’t to avoid duplicating as much work as possible. As a result, I started creating my program by reviewing the application that comes closest to doing what I want: Back In Time. The Back In Time user interface is simple and elegant. More importantly, though, the configuration options make sense. Whether you want to run a backup, give your snapshot a name, or just get rid of it; you don’t have to go hunting to find the controls. Additionally, Back In Time makes good use of existing open source programs to actually do the backing up. Though they might be a bit old, rsync and the Unix copy command, cp, are a formidable duo.

But I quickly discovered something unfortunate, Back In Time really couldn’t be tweaked to meet all of my requirements. You see, it utilizes a technique called hard-linking to create space efficient repositories. But hard links only work as long as they are on the same hard drive. Because I want to store my files on the network or across the internet, that isn’t good enough.

Duplicity

So, I started to look elsewhere for other options, which is when a colleague pointed me toward a command line script called Duplicity. In a striking bit of irony, while the GUI tools of Linux backup are all fundamentally flawed, the command line versions are the best in existence. They easily accomplish all of the requirements on my list, and do a great many things I’ve never even heard of. As a result, if you are willing to roll up your sleeves and hit the learning curve, there is nothing that you can’t accomplish. And, as you might deduce from my glowing introduction, Duplicity makes short work of my requirements list.

Can it do incremental backup? Yes. What about backup over the network? Ditto. In fact, it handles most protocols I’ve heard of, and quite a few that I haven’t. Should incremental through ssh, webdav, ftp, smb, or imap be insufficient, it also supports backup to Amazon S3. File restoration is also relatively painless, though you have to work through the command line interface to do so.

And that’s only for starters, Duplicity also supports encryption via passphrase or passkey, compression of the files into archives, and local caching of file signatures so that backup operations are lightning quick. I decided relatively quickly that Duplicity would be the ideal backend for my utility.

Frontend and Features

But while a solid foundation is a great starting point, it can only get you so far. After all, you don’t brag up the brickwork and reinforced concrete of your newly purchased home, you’re far more likely to show off the kitchen and home theater.

For a program, the points of interest are going to be the user interface. Which makes two lessons learned while working on LyX-Outline very important: 1) It’s really hard to put together an interface that is both intuitive and uncluttered. 2) I’m really bad at it. Both points prompted me to act on the advice of a famous painter:

Good artists copy; great artists steal.

Thus, I decided to steal the user interface from Back In Time. (It’s open source, which makes it okay.) And speaking from a practical standpoint, why wouldn’t I? The developer spent a great deal of time working out how he wanted his program to work. It would be silly to duplicate that effort. Moreover, the fact that I was able to rave about it for nearly 2800 words means that it wasn’t wasted effort. As you browse the menus and configuration panes of my derivative tool, you will probably notice that it bears a striking resemblance to other programs. A few things will be “innovatively different,” but not many. Just remember, I stole the interface. Wholesale. But I’m a discriminating thief, and only stole the best parts.

Which brings us to the real point of interest of this post and my program: the feature list. In part 2 of this article, we’ll take a look at my little creation, which I have unimaginatively dubbed “Time Drive.”

A Better Previous Versions: Time Traveler

Rob Oakes — Thu, 30 Jul 2009 18:16:22 +0000

The most recent versions of Microsoft Windows, Vista and Windows 7, include a wonderfully useful tool called Volume Shadow Copies. You can think of Volume Shadow Copies as insurance against momentary stupidity or negligence.

Consider that in any given day, the typical computer user (namely me) works with a lot of files. These include data, images, and text. As part of the workflow, I may be editing and combining changes to a document from many people. Over time, this can result in a great deal of cruft. Thus, while I’m working, I try and maintain some semblance of organization by applying edits to the most recent version and keeping a semi-automated log of the changes that have been made. Older versions of the document will typically be backed up in the subversion repository on my server, or in a dedicated archive.

However, in the process of shuffling and moving the digital detritus, occasionally I have accidentally deleted the wrong file. Which, invariably, happens while away from the backup server. Losing work is obnoxious, frustrating and embarrassing; thus, my healthy appreciation for Volume Shadow Copies.

On a regular schedule, Windows takes a picture of how your drive looks at that moment and saves it away. So, should you ever need to restore a lost file, you can use the Volume Shadow copies to do so. It’s even a relatively straightforward process. Simply right click on the folder you need to access and select “Properties.” Then, click on the “Previous Versions” tab and you will find a list of every snapshot that the computer has taken.

But while all versions of Windows have Volume Shadow Copies, Microsoft decided that only those who purchase the most expensive versions (Business, Enterprise and Ultimate) get the ability to use them. Home users are out of luck. The Shadow Copy service is still there, but you need a third party program to get at the stored information.

The open source Shadow Explorer is one such option. It’s free and gets the job done. But I recently stumbled upon a second alternative that is worth mentioning, Time Traveler, developed by Bears on the Loose Software.

While reading about Time Traveler, I learned something very interesting. Microsoft considers Volume Shadow Copies to be essential to the operation of Windows. More than a few of the internal services like System Restore and File Backup make extensive use of it. Support is even built-in to Windows Explorer (the file managing utility, not the internet browser). If you know the proper url, you can navigate to where the previous versions are stored on the hard drive and work with them like any other file.

But, bizarrely, Microsoft didn’t connect any of these technologies. They didn’t make it easy for the end-user to take advantage of their hard work. Sure, right clicking is easy enough … but … that’s the problem. It’s like saying that something is “good enough,” or “useful enough.” The “enough” is a qualifier, it implies that the solution is merely passable rather than excellent.

The problem with Microsoft’s implementation of Previous Versions is that you have to add the qualifier. Of course a good tool should be invisible, at least until you need it. But Microsoft’s Previous Versions is too invisible. It forgets that you should also be able to find said tool quickly and efficiently. To use Microsoft’s Previous Versions (assuming you even have the right version of Windows), you first have to load a context menu, then you have to go to the all-encompassing “Properties” option and find the right tab. Only after three unnecessary clicks can you actually review your shadow copies. And what happens if you don’t know where your lost file once lived? There’s no way to actually search through the archive.

You see, it’s “good enough.” And because there’s a gaping hole in the integration, others can make some money by patching it; enter Time Traveler. Bears on the Loose took the final step that Microsoft should have taken: they integrated Shadow Copies into the rest of the Windows.

The program does one thing, it points Windows explorer to the url where the relevant volume shadow copy lives. Simple, huh? Remember, Windows Explorer already has support for browsing the shadow copies built-Explorer Bar->Time Traveler. Alternatively, you can also it Ctrl+T." src="http://www.oak-tree.us/blog/wp-content/uploads/2009/07/image161.png" border="0" alt="Time Traveler installs itself as an add-on to Explorer. You can enable it by clicking on View->Explorer Bar->Time Traveler. Alternatively, you can also it Ctrl+T." width="300" height="310" align="left" />in. It just needs a little help getting there. And this works on every version of Windows: Basic, Home, Home Premium, Business, Enterprise and Ultimate.

Time Traveler represents “Previous Versions” as it should be. Invisible until needed, then easy to access and use. It’s loaded by clicking on View –> Explorer Bar -> Time Traveler. Alternatively, you can hit Control + T to toggle the slider on and off.

When loaded, it subtly clues you into what previous versions are available via a convenient timeline. You can open that point in time by moving the slider bar. Hence the name, you “travel back in time.” Windows Explorer does the rest of the work.

But even if Time Traveler makes the navigation of shadow copies easier, that one simple action doesn’t justify the $20 that Bears on the Loose charges for it. I would say that it’s the program’s “other” feature that makes it valuable. Namely, it makes it really easy to configure and manage the Volume Shadow Copy Service.

While I am aware that managing the Volume Shadow Copy Service is possible with the Control Panel, I’m not exactly sure how it’s done. Frankly, I’m not even sure where to begin. I once found a few options when I was looking for something else, but I can’t remember where I saw them. Time Traveller, however, takes all of those available settings and puts them on the same configuration page. Want to change how often the computer takes snaphosts? You can do that. Want do change how much hard drive space is used to store them? You can do that too. Want to manually delete or protect old snapshots? The settings for that are also available.

Sure, Time Traveler isn’t what anyone would call groundbreaking software. But that doesn’t mean that it isn’t valuable. It transforms Volume Shadow Copies from a system tool into a user tool, primarily by linking Microsoft’s own technologies together in a more cohesive manner. I hope that someone at Microsoft takes a good look at how Time Traveler works. It represents “Previous Versions” done right.

Back In Time (Part 2): Over the Network and Across the World

Rob Oakes — Tue, 21 Jul 2009 01:15:03 +0000

Despite all of its strengths, Back In Time has one major weakness: it doesn’t support backup over a network. But luckily, it runs on Linux,the single most customizable operating system in the world! As a result even though Back In Time might not officially support a backup to a remote computer, you can still make it work through a little bit of spit and grit.

The magical sauce is the Fuse SSH file system. This kernel module allows for Linux to mount a remote share as a local drive. And by doing so, Back In Time thinks that it is working with a local folder even though the data might be sent across the network or internet. To make it work, though, you’ll have to work at the command line and do a bit of programming. But, don’t panic because it isn’t that hard. It amounts to a few shell commands and about six lines of code. Below, I’ll show you how in seven simple steps.

Step 1: Prepare a Local Mount Point and Set Permissions

First, we first need to create a mount point. A mount point is a blank folder that the operating system uses to connect the local computer to the remote server. I typically like to have my mount points in the /media folder. If the mount point is in the /media folder, it will show a little icon on the desktop when active.

So, open up the terminal and type the following:

sudo mkdir /media/Backup

This command will create a new folder (Backup) in the media folder. As is, we could then mount the SSH file system to this mount point, but to do so would require that we run the command as the super user (root). It’s not necessarily good to run things as root. So, let’s change the folder’s owner and permissions so that we can mount it without requiring the super user:

sudo chown –R /media/Backup
sudo chmod a+x /media/Backup

In the first command above, replace with the username that you use to log-on to Linux. In my case, it would be: roakes. The second command properly sets the permissions so when it comes time to mount the remote share, the computer won’t prompt for the root password.

Step 2: Install the SSH File System (SSHFS)

Now that we have a valid mount point, let’s install the software packages for SSHFS. If using Ubuntu, type the following into your system terminal:

sudo apt-get install sshfs

Step 3: Get the IP Address or Url of the Remote Computer

With the mount point set and software installed, we need to do a bit of planning. “Proper preparation prevents pitifully poor performance” and all that. First, we need to know where the data will be stored. Will we be using a simple home server or will we be storing that data on a remote server over the internet? Also, will you be able to connect to the server using the secure shell (SSH) protocol? If so, what is the destination folder?

We will use this information to construct a connection string. It will have the form:

@Machine.Location:/path/to/backup/folder

SSH to a Machine on the Local Network

If we will be backing up to a computer on the local network, we need to know it’s IP address so that we can establish an SSH connection. To find this, go to the server and log-on. Then type ifconfig into the terminal window. This will return a block of text similar to that seen below, which was copied from my server. I have added a series of x to some of the information.

eth0 Link encap:Ethernet HWaddr 00:xx:xx:xx:e0:93
inet addr:xxx.xxx.1.106 Bcast:xxx.xxx.1.255 Mask:255.255.255.0
inet6 addr: fe80::xxx:xxxx:xxxx:xxxx/64 Scope:Link
UP BROADCAST RUNNING MULTICAST xxx:1500 Metric:1
RX packets:26460698 errors:23 dropped:145 overruns:17 frame:0
TX packets:14444204 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2320464437 (2.1 GB) TX bytes:2079312151 (1.9 GB)
Interrupt:3 Base address:0xd800

lo Link encap:Local Loopback
inet addr:127.0.0.x Mask:255.0.0.0
inet6 addr: ::x/xxx Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:9308 errors:0 dropped:0 overruns:0 frame:0
TX packets:9308 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4232264 (4.0 MB) TX bytes:4232264 (4.0 MB)

The IP address can be found on the second line below eth0, which starts : inet addr. In my case, the IP address is: xxx.xxx.1.106. The xxx.xxx will have numeric values that are specific to your network. To connect to the machine, I use a connection string that looks like:

@xxx.xxx.1.106:/path/to/backup/folder

SSH to a Machine on the Internet

If you will be connecting to another computer on the internet, then instead of using an IP address to connect, you will instead be using a url. You can get this information from the information page of your backup service, or from the network system administrator. To connect to the remote machine, you will use a connection string that looks like:

@machine.url:/path/to/backup/folder

To remotely access my outside of the network computer, I use:

roakes@remote.oak-tree.us:/media/Linux-Backup

Step 4: Test the Connection

Once I have the connection string, I can now test SSHFS and make sure that it can mount to my mount point (Step 1). To mount the remote drive, type:

sshfs @Machine.Location:/path/to/backup/folder /media/Backup

Be sure to replace the part in italics with your connection string. The first time that you connect, you will be prompted to accept the server’s key and be asked for your password. After you enter your user credentials, it should mount the file system to /media/Backup. Open up the location in finder and make sure that you can create folders and move files onto the remote machine.

You can unmount the volume by typing (no root required):

fusermount –u /media/Backup

Step 5: Use a Passkey Instead of a Password

While a password is acceptable when you want to connect manually, we want the backup script to run automatically. That means no password prompt.

Luckily, however, it’s easy to configure Linux so that it uses a passkey instead of a password. This will allow your computer to connect automatically instead of prompting you for your user credentials each time that you want to run a backup or restore a file. To do so:

Generate a key on your computer by typing ssh-keygen at the terminal. You will be prompted for the file where you want to save the key. Just use the default (/home/username/.ssh/id_rsa). It will then ask for a passphrase, leave this empty.
Next, append the public key that you just created to the authorized keys file on the remote server. To do so, type:

cat ~/.ssh/id_rsa.pub | ssh @Machine.Location ‘cat >> .ssh/authorized_keys’

Now when you use the command from Step 4 above, it will automatically mount without prompting for a password.

Step 6: Put It All Together With Custom Scripts

Now that we have the connection string and have configured a passkey instead of a password, we are ready put all the pieces together in two custom scripts. The first will be used for automating the backups and the second will be used for opening the GUI. There are three things in particular that need to happen:

The computer mounts the remote drive to the appropriate point in the file system.
Either the user interface or automated backup routine runs.
The remote store is unmounted when the program finishes and exits.

Creating an Automatic Backup

Below is a template for a script that can be used to start a backup job:

sshfs @Machine.Location:/path/to/backup/folder /media/Backup
backintime -b
fusermount -u /media/Backup

The first line mounts the remote folder, the second launches the command line version of backintime and tells it to make a new snapshot, and the last line then unmounts the remote drive after Back In Time has finished. Save this file to someplace in your home file as “Data-Backup.sh”. I like to keep all of my scripts together in a folder imaginatively called “Applications.”

Anytime that you want to run an automated backup, you can do so by typing:

bash /path/to/folder/Backup-Data.sh

Launching the GUI

Now that we have a script for running the backup to the remote location, we are half way done. But as a backup is only as good as ability to access the data, we also need a second script. This will be used for launching the program, retrieving files, or making changes to the configuration. Again, we follow the same steps as the automatic backup script. Below is the template:

sshfs @Machine.Location:/media/backup/Backup/Linux-Backup /media/Backup
backintime-gnome
fusermount -u /media/Backup

The first line mounts the remote share, the second line launches the GUI version of Back In time, and the third unmounts the volume after you exit. Save this as “GUI-Back-In-Time.sh” in your scripts folder.

Step 7: Easy Access and Automation

With your two custom scripts, you have all of the tools needed to backup and restore files to your remote computer. But launching the terminal every time you need to do one of these jobs can be a bit of a pain. And because we are backing up to a remote location, we can’t use the scheduler built in to Back In Time. So, to make things easier, let’s:

Create a custom menu launcher for easy access
Automate the backup process using Gnome Schedule and the user crontab file.

Create a Custom Menu Launcher for Easy Access

To make it easier to access our custom script, let’s create a custom launcher by using the Applications->System Tools->Main Menu utility.

After the utility launches, click on the “System Tools” sub-menu and then push the “New Item” button. In the pop-up, specify the type of launcher as “Application” and the name as “Back In Time.” For the command, use:

bash /path/to/scripts/folder/GUI-Back-In-Time.sh

Add a description to remind you what Back In Time does and, if you want, you can customize the icon by clicking on the picture at left. When finished, press “Close.” This will add your custom launcher to the main menu.

Automated Backup: Cron Jobs and Gnome Schedule

After all that preliminary work, we’re finally able to automate our backup. We’ll accomplish this through the use of the Linux cron daemon. Cron is a program that allows for tasks to be executed at a certain time. You can create cron tasks to update a website, or automatically download files, or perform system maintainence tasks.

Cron tasks are specified in a file called the “crontab.” Each user has a crontab file. There is also a system crontab that runs under the root user.

While editing the cron file by hand is easy enough, we will be using a special GUI called “Gnome Schedule.” To install it, type the following command in the terminal:

sudo apt-get install gnome-schedule

After the installation finishes, you can launch the utility by going to Applications->System Tools->Scheduled Tasks.

To create a new scheduled task, click on the “New” button and then choose “Recurrent Task” from the drop down list. A recurrent task is run at a specified interval: every second, minute, hour, month, or year. If you want to be fancy, you can specify which hour of the day that you would like it execute.

For the command, use the following:

bash /path/to/scripts/folder/Data-Backup.sh

When finished, press “Add.”

Conclusion

Congratulations! Through the use of the sshfs kernel module, you’ve now configured Back In Time to more or less duplicate all of the behavior of Time Machine. We’ve fooled it into thinking that it’s working on the local drive, when in reality that archive may be sitting across the network or across the world. And though the instructions provided here were specific to the SSH file system extension, extensions can also be found for Amazon Simple Storage Solutions (S3), WebDav, FTP and Samba. This makes it possible to backup your data to almost any service or destination imaginable. Even better, through the use of scripting, it is possible to automate the process so that it reaches “set it and forget it” simplicity.

Back In Time (Part 1): Linux Backup Made Easy

Rob Oakes — Tue, 21 Jul 2009 01:12:13 +0000

Linux is a bit of a funny creature. In many ways, it’s the most integrated and full featured operating system in existence. Take software development, for example. You couldn’t ask for a more integrated, well thought out solution. The tools are absolutely first rate, the system is stable, and because the platform is open source, most of the rough edges get worn off rather quickly.

But there’s also a downside to open source. You see, everyone tends to scratch their own itch first. Thus, while there are a bevy of tools that perfectly meet the needs of a few (most notably their authors), they often miss the needs of the many. That is a problem, because the masses don’t necessarily have the skill set required to adapt an already existing program so that it fits them. If anything, this is why Microsoft is valuable. They create software that more or less handles the needs of the masses.

And as might be expected, this is also the state of backup on Linux. It is no understatement to say that you can get your hands on the very best backup tools available, for free. All it requires is that you use some derivative of the following:

sudo apt-get install toolname

Unfortunately, while you can find the very best tools, that doesn’t necessarily mean that you will be able to use them. Rather that be written with the end-user in mind, they were created for programmers and system administrators. You know, the type of people who wonder why anyone would bother with a user interface. After all, it’s so much faster to do everything from the command line.

The result is that while the Linux backup tools may be powerful, they also have a learning curve steep enough to bang your head against. And that is an absolute shame. In general, I (and a great deal of humanity in general) am against activities that require me to learn new things or otherwise grow in what might end up being a painful direction. Even if it is for a good cause. (And I can think of few more important than making sure thatmy personal information is safe and secure.) Ideally, I want backup to be easy. I want to “Set it and forget it.” Time Machine for Mac OS X allows me to do this, as does the Windows Vista File Backup. But at first pass, such a solution for Linux is woefully absent.

That’s not to say that there aren’t contenders. In fact, a quick search on Google will reveal several: sbackup, Flyback, and TimeVault all look promising. But if you look at their project pages, you will notice something disconcerting: in each case, the programs are old and there isn’t much recent activity. This is typically a bad sign for an open source project. No activity usually means that the program is dead and unlikely to advance further. And that is really too bad, because sbackup, Flyback and TimeVault are good programs with potential. But, like many other open source options, they have a number of very rough edges.

Maybe that’s why I got so excited when I saw a new contender, called “Back In Time,” when it was recently featured on Lifehacker. At the time, I thought, “Have I finally found a backup program I can really use?” Now, after four months of using it full time, I can answer that question: Yes! Without a doubt!

Back In Time

Back in time is the full package. It includes a robust backend that makes incremental snapshots easy. You’ll also find a simple to use restoration system and a settings pane that actually make sense. What’s more, the defaults are actually useful! As a particularly nice touch, there are even separate packages for KDE and Gnome, so that the program will natively blend with whatever desktop you happen to have installed.

Installation

On Ubuntu, you can install Back In Time by downloading the packages for your Linux distribution. As noted above, you can find two different user interfaces available: one for KDE and another for Gnome. Don’t fret about downloading one over the other. They are identical in terms of functionality. (Note: The screenshots you see in this article were taken from the Gnome user interface.)

Configuration

Thanks to the well-thought out defaults, getting started with Back In Time is easy and straightforward. The first time you launch the program, you are prompted to configure the differenct backup options. (Should you ever want to change these, you can access the same pane by clicking on the “Settings” button.) In the configuration window, you will see several different tabs. Each tab corresponds to a different part of the backup job. There are the general settings which describe where you want your backups to be saved and how often you want the backup to run. And there are settings for what you want to include, exclude, and how many snapshots you want to keep. Back In Time can back up to any attached storage device, including a USB hard drive or flash drive. Unfortunately, however, it doesn’t work with network drives or with optical media.

The include tab lets you specify which folders you would like to have added to the backup. Since Back In Time runs in two different modes (user and root), I’ve been using each mode to accomplish a slightly different goal. The user mode is ideal for backing up files and settings, while the root mode is ideal for backing up the system itself (similar to the way in Time Machine works on Mac OS X).

By using the different modes in this way, I can keep user fuke backups are kept in one location and the system information in another. Thus, if I should lose my system, I can quickly and easily restore my system using the “System” backup and then copy back my files. I avoid duplicating data through the “Exclude” tab. In the root mode, I exclude the “home” directory, where all of the files and profiles are kept. But since I am backing up my home directory through the user mode, this doesn’t really matter.

Incremental backups are most valuable when run frequently. Time Machine creates snapshots every hour for more or less “maximum” protection. And as I learned recently (after accidently deleting my entire source code directory for LyX-Outline), hourly backups can be a tremendous lifesaver. After my ”accident,” I was able to restore the directory to the way it had looked just 15 or so minutes before. In doing so, I recovered a number of important changes that hadn’t yet been committed to Subversion. At that moment, I was tremendously grateful that Back In Time could be configured for hourly snapshots.

But over time, the side effect of hourly snapshots is a huge number of unnecessary backups clog the drive. After all, I don’t need to know what my drive looked like during every hour of 2009. It’s here that we see another fine example of Back In Time’s overall polish. The auto-remove settings provide a way to quickly and efficiently deal with too many snapshots. Back in time can be configured to prune the number of backups to something that is slightly more reasonable.

Restoring Files

File restoration is dead simple as well. You can quickly browse through backed up files and folders using the restore pane of the GUI. If you need a version from an older snapshot, simply select it from the “snapshots” menu on the left. Once you have selected the files you are interested in, click “Restore.” They will be copied to their old locations.

Limitations

But even though Back In Time is probably the best user-oriented backup solution I’ve found for Linux, it still isn’t perfect. (Though really, really close.) There is at least one major limitation: lack of a network based backup.

As I’ve said before, I prefer to keep my backups on my home server. There are a number of reasons for this, but the most important is that I don’t need to remember to turn my backup drive on. When it’s time to automatically back up, it just happens. Unfortunately, Back In Time does not natively support network or remote backups. (Fortunanately it runs on Linux, the most customizable operating system in the world, and there happens to be a workaround.)

Conclusion

Back In Time is a wonderful piece of software. In fact, it’s the only backup program on Linux I’ve found that really meets the dead simple needs of the typical computer user. It is built on the solid foundation of rsync, allows for incremental backups, has a well though out GUI and defaults, and generally “Just Works © .”

Even its greatest weakness (the lack of a network based backup option) is more of an inconvenience than a truly show stopping fault, since the same thing can readily be accomplished using a Linux kernel extension and a simple script. In Part 2 of this article, I will show you how.

Backup, Sync and Share – Part 4: Apple Time Machine and Samba

Rob Oakes — Wed, 10 Jun 2009 01:58:22 +0000

At it’s World Wide Developer’s conference in June of 2006, Apple released a product that changed the way that a great many people think about backup: Time Machine. While I cringe at the thought, I need to descend into the fawning adoration public relations speak that masquerades as critical coverage of Apple Products. (Actually, forget that, here’s how Apple describes their backup system.)

Time machine is a breakthrough automatic backup that’s built right into Mac OS X. It keeps an up-to-date copy of everything on your Mac – digital photos, music, movies, TV shows, and documents. Now, if you ever have the need, you can easily go back in time to recover anything … Time Machine takes care of it … Automatically, in the the background. You’ll never have to worry about backing up again.

General sarcasm and bitterness aside, Time Machine really is a spectacular piece of kit. Sure, you can very successfully imitate a Time Machine experience using the tools within Windows Vista or Linux (or via third party tools such as Norton Ghost). Even so, Time Machine is just just about the perfect combination of simple, powerful, and integrated. And like most Apple products, when used within the Apple eco-system is a lovely experience.

However, if you migrate too far out of the walled garden, Time Machine isn’t quite so nice to work with. Actually, it can be a bit demanding and temperamental. For example, it requires its own formatted hard drive or the ready availability of a specialized Apple router (called a TimeCapsule). Alternatively, it can be a bit flaky; when I was backing up to a local hard drive, it would often quit with an indecipherable error. Luckily, however, these limitations are pretty easy to overcome. In this article, I will look briefly at how to setup Time Machine so that it works with a simple home server running Samba and Subversion.

A Few Goals

Before jumping into how to change settings, it might be good to talk about why I’ve chosen the configuration options I have: it’s all about simplicity and convenience. As Apple (redundantly) points out, the more simple a task is, the more likely it is to happen. In terms of backup, this means that it should be automated. So, while having a disk connected to a computer is sufficient, backup over the network to a network attached storage (NAS) is much better. "Why," you ask? In the case of NAS, there is no worry about turning the dedicated backup disc on, letting Time Machine do its thing, and then shutting the disk off.

There is, however, one major drawback to backing up over the network. That first backup-up takes a REALLY long time. In my case, it took my MacBook Pro the better part of three straight days to finish. And should you ever need to restore your complete hard drive (due to failure or the purchase of a new computer), you can expect it to take the same amount of time. Once that first backup completes, however, this limitation no longer exists. Subsequent runs only include files that have been modified and go quite quickly.

But what about convenience? The point of a backup is knowing that your work/information is safe. For single files, restoring over the network is easy. But what happens should disaster strike and you lose a hard drive? No one wants to wait three full days before resuming work. Luckily, you can get the benefits of over the network back-up and the convenience of quick recovery by configuring your home back-up server to use USB hard-drives.

Should your hard-drive fail, you can detach the USB drive and then use TimeMachine to restore your data locally. I discovered this lovely side benefit recently when the hard-drive in my MacBook Pro decided to die and I refused to wait three days for it to restore over the network. I disconnected the network drive from the server and plugged it into the Mac. I then booted from the recovery CD and it magically found my Time Machine backup. Instead of three days, I was ready to go in slightly less than three hours. So, now that we have the goals (simplicity and convenience), let’s talk about how to reach them.

Network Backup to Samba

Out of the box, a default Leopard installation won’t work with a Samba share. You will need to follow a relatively simple three step procedure to have Samba supported Time Machine goodness: 1) Create a sparse bundle image on the local file system, 2) copy the sparsebundle to the root of the network share, 3) enable support for “unsupported” network volumes.

1) Create a Spase Bundle Image on the Local File System

Time machine saves backups in an extremely specific format, called a sparse bundle or image. A sparse bundle is a special file that can also behave like a physical hard drive. When you click on it, it will launch a new volume in the finder that then contains other files and folders. A unique characteristic of the sparse bundles included in Leopard is that they will automatically resize based on their contents.

When using Time Machine on Apple hardware, the process of creating a sparse bundle is automated. Unfortunately, this isn’t true on a generic Samba share. In order to make Time Machine work, we need to manually create a sparse bundle and copy it to the Samba share where we will be backing the computer up.

As we create our image, the file name of the image is extremely important. It has a general form of:

Computername_MACaddress.sparsebundle

Where Computername is the name of your computer and MACaddress is the MAC address for your ethernet card. Luckily, both of these pieces of information are relatively easy to find. The computer’s name is found by opening “System Preferences” and clicking on the “Sharing” link under “Internet and Network.”

For the Time Machine backup to work correctly, it is important that your computer’s name not have any spaces in it. Should you need to, this might be a good time to change it by clicking on the “Edit” button.

You can locate the MAC address by typing the following command into the Terminal:

ifconfig en0 | grep ether | sed s/://g | sed s/ether//

It will return a list of letters and numbers, for example: 001ec4b8f9b3.

Make note as this information is important in the next step. After you have edited your computer’s name (without symbols or spaces) and have your MAC address, you are ready to create the sparse bundle on your local computer. We will later copy it to the correct place on the NAS hard drive. To do, open the terminal and type the following:

sudo hdiutil create –size 250g –type SPARSEBUNDLE –nospotlight –volname “Backup of My Mac” –fs “HFS+J” –verbose ./Computername_MACaddress.sparsebundle

Be sure to replace everything in red with information that is pertinent to your computer (In my case, Computername_MACaddress is RobOakes-Mac_001ec4b8f9b3.sparsebundle). In this example, I have limited the size of the sparse bundle to 250g (through the –size flag). If you omit the "-size" flag, the sparse bundle will expand until it fills the entire available space of the drive.

Update (2009 07-02): The file system (fs) command line option above was changed from "Case Sensitive Journaled HFS+" to "HFS+J".

2) Copy the Sparse Bundle to the Appropriate Location

After you finish creating the sparse bundle, it needs to be copied to the root of the Samba share which you will be using for your backups. This can be done from the finder or from the command line. In my case, I keep my Time Machine backups on a Samba share called PC-Backup.

Update (2009 07-02): The first comment below explains how to copy the sparsebundle image using the command line.

3) Configure Time Machine to Support Samba Network Volumes

To enable Samba shares in Time Machine, type the following into the terminal:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

After you have changed this setting, you may need to restart the computer for it to take effect. (Alternatively, you can also log-out and then log back in.) At this point, everything is set-up for Time Machine to use your Samba share for back-ups.

Since Samba does not use Apple’s zero configuration network protocol (known as Bonjour), you will need to help Mac OS X find your backup-up volume the first time you use it. This can be done a variety of ways, you can either manually mount the volume by using the command line, or by navigating to the disk with the finder. (To check if the disk has been mounted, open up a finder window and see if a small eject icon appears next to the computer which hosts the share. Alternatively you can navigate to the /Volumes/ folder, where you will also see a list of currently mounted remote shares.)

Next, open the Time Machine configuration from the System Preferences and press the “Change Disk” button. This will give you a list of all the disks and samba shares that are currently available to OS X. Select which disk that you you would like to use and press the “Use for Backup” button. Since I placed my sparse bundle in the root of PC-Backup, that is the volume that I should select.

Time Machine will begin a full back-up that can be accessed either over the network, or by manually plugging the hard drive into your Mac.

I Killed Twitter

Rob Oakes — Thu, 14 May 2009 19:59:06 +0000

I just killed Twitter. The whole website is down, it can’t even be pinged. So … just in case I get asked about it, here’s my story. I was trying to post an update about the brand new, easy to install binaries of LyX-Outline and then … the little twitter widget (twidget, what a sickeningly sweet name) went black and crashed. Now, I can’t contact the website, even when I try to access it from another computer on the neighbor’s wireless. Who’s to say that little people don’t have any power?

Update: Okay, in the face of both reality and facts … it may be possible that I did not kill Twitter, per se. But my interaction most certainly had a negative consequence. That, at least, you have to give me.

Introducing LyX-Outline 0.1

Rob Oakes — Wed, 13 May 2009 22:16:12 +0000

Writing anything – whether it be a book, or a short story or an angry letter to your boss – is substantially more than starting from the first idea moment of inspiration and continuing to the final draft. Rather, writing involves a fair number of idea fragments, fleeting moments of inspiration, and a tremendous number of dead ends.

It is incredibly unfortunate that most writing software, however, is geared to organizing and structuring the document after most of the hard work has already happened. It simply assumed that most of the planning and layout has already happened and the author is ready to string words together. Unfortunately, this assumption overlooks one important truth: ideas are best defined as they are expressed. Thus, it’s usually about the time that the a writer sits down to compose the draft, that the document’s true structure becomes apparent. In my own case, this often leads to a flurry of reorganization. And it’s during the restructuring that the real battle begins.

When in full creative passion, I am typically working with three or four different programs all at the same time: OneNote is open so that I can access my ideas, Word is there to start collecting the somewhat finished text, and I’ll also probably be using a MindMapper so that I can see a visual representation of the document structure. The tools are separate and don’t communicate with one another. Thus, a change made in one place needs to be made everywhere. And all too often, that I end up fighting the word-processor and the notetaker, and the mind mapper. It is tremendously frustrating to battle the tools of your adopted trade. Luckily, I am not alone in my frustration.

In the past everal years, a number of programs have become available that leave the linear model of writing behind. On the Mac platform, one such tool looms above the others: Scrivener. Central to Scrivener’s function are two important metaphors: that of the outline and that of the corkboard. And it works really, really well. There are just a few minor problems. First: Scrivener is only available for Mac and Scrivener’s lead developer has made it clear that there won’t be versions for other platforms. Second: Scrivener was really designed with creative writing in mind. Thus, while it can be used for long and complicated documents, this is a slightly less than straightforward process. Last, to create said fancy documents, Scrivener requires the raw use of a markup language (and all of the associated headaches that come with it).

The document processor, LyX, however, excels in many areas where Scrivener falls short. It is built upon the mature and robust underpinnings of LaTeX, the typesetting language of choice in the science and engineering. And more importantly, it is easy to use (where LaTeX most decidedly is not). But it fails in the same way as Microsoft Word and other word processors, it is a linear writing tool and doesn’t offer a great deal of work-flow flexibility. That is where LyX-Outline comes in.

LyX-Outline is a marriage between Scrivener’s organizational tools and LyX’s typesetting tools.

Looking for Flexibility

While you can download and play with LyX-Outline, please reember that is still a forthcoming add-on to the LyX. And while you find it useful, this release is an alpha level technical preview. Right now, you can view the structure of your document at a glance in the corkboard, move things around, poke about in the outliner, and even jot your winded thoughts with the very basic plain text editor.

But even though some of the functionality is lacking, you can start to explore. And as you do so, one of the first things you might notice is that LyX-Outline has been designed with flexibility in mind. Nearly all of the tools can be docked, ripped away, or hidden if they are not needed. The work environment should adapt to your work habits, not the other way round. Whenever the writer needs to adapt to the environment, a moment of productivity and a spark of life has been lost.

A Quick Overview

The main interface consists of three main elements. This includes the document map, the editor pane, and the dock tools. As described above, the document map and the dock tools can be relocated, docked or left free floating (depending on your preference). And if you don’t find them useful, they can quickly be hidden.

The document map lets you see a hierarchical list of every item in your project. Selecting one of the items displays its content’s in the editor pane. The editor pane is the main text processing portion of LyX-Outlineand can be toggled between a text editor (plain text only at the moment) and a corkboard view.

In addition to the main editor and corkboard, there is an additional corkboard and outliner tool available as a dockable widgets. Just as in Scrivener, you can view the pieces of your documents in many way. Every item is a document, and an index card, and an outline point. They are intimately tied together and any change made to the order or content of one will be reflected everywhere. The document map is used to navigate the content of the main window.

The dock tools, however, can be controlled separately. You can view the entire document, or filter to a specific level in the hierarchy via a drop-down menu. In the future, filtering by search term, keyword, document structure, or tag will also be possible.

Item Summary and Organization

The health of any written work can also be greatly improved by connecting a simple synopsis to a larger chunk of text. This makes it much easier to check the flow of your work at a glance . A summary can easily be added to any individual item in the collection by double clicking on it’s index card in the corkboard or the synopsis column in the outliner. The summary text is independent from the full text and can include pertinent information that helps you structure the document. For now, the synopsis must be entered by hand; however, future versions will allow for automatic generation of the synopsis from the text.

Of Outlines and Index Cards

Scrivener (and LyX-Outline as it’s dedicated clone) utilizes multiple metaphors. The Corkboard may be useful in some instances, but will be less helpful in others. Ditto for the outliner. Either way, it is important to know that both the corkboard and the outliner display sub-items. If you are using the main corkboard, these sub items will be connected to the active item in the document map. If you are using the dock tool, they can be filtered via the drop-down menu. From the corkboard, you can then edit items by double clicking on either their title, or their summary. You can reorder them through drag and drop. From the outliner, you can add new items or sub items, or delete them from the document. You can also change their order (by using the up or down arrow buttons) or change their indentation level (with the left and right arrow buttons).

Minding the Potholes

And that is a very quick tour of the Outline modules for LyX. As noted above, this release of LyX-Outline is a prototype and it is still very far from its intended target. Thus, if it doesn’t meet your needs right out of the box, please be patient. The mantra of OpenSource is "release early, release often." In this case, I have erred on the side of early and as you might suspect, there are some dangerously rough edges. These include:

Unfinished interfaces. Most of your manipulation will probably need to happen in the outliner pane. This includes adding or removing new document nodes, as well as indenting or un-indenting the document selection. The final version will include ubiquitous drag and drop.
A crappy editor. The current editor is plain text is a placeholder only. Ultimately, LyX-Outline will be integrated with the LyX editor and have access to all of it’s very powerful features. It is important to keep that future in mind.
No export option. While you can save and open past documents, there is currently no way to actually get them out of the program.

In it’s current incarnation, this prototype is valuable for pecking out thoughts and playing with the organizational tools. I am releasing it so that people can see how the tools might look in their final form and provide feedback. Please do so. Download the source, use it as you can, and let me know what you think. What things work? What things don’t? What features would you as users like to see?

Update: You can download the source code and find installation instructions on the LyX-Outline project page.

Apolitically Incorrect » Cool Stuff

Temples and Timelines

Similar Posts:

Hannibal, Napoleon, and Joseph Charles Minard

Hannibal Invasion of Italy (218 BC, Second Punic War)

Napoleon’s Invasion of Russia (1812-1813)

Conclusion

Unrelated Thoughts: Re-Visioning Minard with Open Source Tools

Acknowledgements and Further Reading

Similar Posts:

Image Gallery: The Notebooks of Leonardo Da Vinci

Similar Posts:

About This Book Thing

Background

Analytic Design

Tools and Technology

Conclusion

Similar Posts:

Create a Unified Inbox in Gnome Evolution

Similar Posts:

Install Time Drive On Ubuntu and Other Debian Linux Distributions

Background

Installation Instructions

Step 1: Add the Package Repositories to the Ubuntu Software Sources

Step 2: Configure Apt to Trust Time Drive and Duplicity Packages

Step 3: Install Duplicity and Time Drive

Application Updates

Similar Posts:

Time Drive 0.1.5: Incremental Changes

Time Drive Update

New Stuff

Conclusion

Similar Posts:

Backup for Linux, Done Right – Part 2: Time Drive 0.1

Over the Network or Across the World

Restoring Files

Set It, Then Forget It

Notifications and Feedback

Beta Software

Conclusion

Similar Posts:

Backup for Linux, Done Right- Part 1: A Mini Melodrama

Backgrounds and Backends

Basic Needs

Duplicity

Frontend and Features

Similar Posts:

A Better Previous Versions: Time Traveler

Similar Posts:

Back In Time (Part 2): Over the Network and Across the World

Step 1: Prepare a Local Mount Point and Set Permissions

Step 2: Install the SSH File System (SSHFS)

Step 3: Get the IP Address or Url of the Remote Computer

SSH to a Machine on the Local Network

SSH to a Machine on the Internet

Step 4: Test the Connection

Step 5: Use a Passkey Instead of a Password

Step 6: Put It All Together With Custom Scripts

Creating an Automatic Backup

Launching the GUI

Step 7: Easy Access and Automation

Create a Custom Menu Launcher for Easy Access

Automated Backup: Cron Jobs and Gnome Schedule

Conclusion

Similar Posts:

Back In Time (Part 1): Linux Backup Made Easy

Back In Time

Installation

Configuration

Restoring Files

Limitations

Conclusion

Similar Posts:

Backup, Sync and Share – Part 4: Apple Time Machine and Samba

A Few Goals

Network Backup to Samba

1) Create a Spase Bundle Image on the Local File System

2) Copy the Sparse Bundle to the Appropriate Location

3) Configure Time Machine to Support Samba Network Volumes

Similar Posts: