Imagine for a minute that you’re writing a book or technical manual.  Let’s say it’s a book on technology, maybe the open source tools used for scientific writing (to randomly pick an example).  As you write this book, you realize that you need some way to cue the reader into different parts of the text.

For instance, you might want all definitions to appear in bolded text so that a reader pick out key terms quickly.  Or you might want code examples to appear in a different font than the regular text, again, so they’re easy to find.  What’s the best way to do this?

Sure, you could just bold the definitions, or manually change the font for the code examples.  But that’s painful!  Changing typeface and size every time that you have a section of code will eventually result in a lot of lost time.  Moreover, you might make a mistake, which destroys your consistency and makes your writing look unprofessional.  There must be a better way!

Thankfully, there is.  It’s through the consistent use of styles.

Styles Defined

A style is a collection of formatting commands with a meaningful name.  Examples in Microsoft Word include “Heading 1”, “Body”, and “Figure”.  When you use them consistently, styles allow for “logical markup” of a document.  That is to say, you tell the computer what the section, paragraph or text corresponds to and then it handles the formatting.  For this reason, styles form the basis of the “What You See Is What You Mean (WYSIWYM)” philosophy used by LyX and other “document processors”.

There are three major types of styles: section styles, paragraph styles and character styles.  You are likely familiar with section and paragraph styles, as they populate the main drop down box of LyX’s main toolbar. The most common section styles are: chapter, section, subsection.  Examples of paragraph styles include: standard text, quotations, and code.  But the third main type of style, “a character style” may be a stranger.  It isn’t as common as it’s two older brethren and easier to lose track of; which is a huge shame, because it is just as important.

Character styles, as the name implies, work on individual characters within the text of a paragraph.  To extend the examples given in the introduction, the most efficient way to bold a definition would be to use a pre-defined character style called “definition”.  Ditto for an in-paragraph sample of code.

“Now wait a minute,” you’re probably saying, “I’m not sure that I follow the difference between character styles and other text formatting commands like ‘bold’, ‘italic’, or ‘underline’!”

That is perfectly alright, the distinction can be a bit hazy.  Nonetheless, it is still key in understanding the difference between “logical markup” and formatting.  Let me see if I can clear it up.

As described above, a style is typically used to describe what a part of the document is, rather than how it should look.  Subsequently, you are unlikely to see a style called “bold”, “italic” or “center”.  Those terms describe how the text should appear; which is to say, they are formatting commands.  Bolded text uses a heavier line stroke and centered text appears in the middle of the page.  A style, on the other hand, is used to describe a “definition”.  You then associate that style with how you want “definition” to look.  It could be numbered, bolded and centered; or it might be larger than the surrounding text and italicized.  The appearance is determined by a “style sheet”.

Note: Since the use and customization of section and paragraph styles is well explained in the LyX help documentation, this article will focus on character styles.

Using Character Styles in LyX

LyX provides some common character styles through the use of its “Logical Markup” module.  These include styles for nouns, emphasizing text, and code – which may be useful for highlighting filenames or programming syntax within the body of the paragraph.

The character styles are not loaded by default, however.  Instead, they come as part of an optional module.  You can use them by going to the “Document->Settings->Modules” pane and adding “Logical Markup” to your document.

Once the “Logical Markup” module has been loaded, you  use of apply a particular character style through the “Edit->Text Styles” menu, or by first highlighting a block of text and selecting “Text Style” from the right click context menu.

Note: The emphasize style is distinct in that it can also be toggled on and off through the “Ctrl+E” keyboard shortcut.

After you’ve formatted a block of text with character style, it will appear differently from the surrounding text.  It will not only use a different font, but will also be surrounded by a set of hash marks.  The example below shows a document with two instances of the “Code” character style.  Optionally, character styles have a label which you be toggled on and off.  The screen shot below shows what this looks like.  The first instance, “TEXMFLOCAL”, has the label toggled off while the instance, “/var/lib/texmf/tex/latex”, shows the same character style with the label visible.

Customizing Character Styles

But what if you need access to other character styles?  Perhaps you’re writing a novel and would like to have a dedicated style for specifying when a character is talking to himself (called internal dialogue)?  Or you’d finally get back to working on your textbook with bolded definitions.  Such styles aren’t defined by the “Logical Markup” module, or any module at all for that matter.  Nor are they likely to be used so frequently that it is worth trying to modify the built-in modules or create a new one to support them.

Luckily, there is a solution to this problem.  In the first article in this series, I looked at how to create a template for an NIH grant proposal from an existing LaTeX document class.  In that article, the secret to using the “nih.cls” document class lay in its corresponding LyX layout file (“nih.layout”).  Also in that article, I explained how LyX layout files primarily contain information concerned with on-screen presentation.

Note: For more detail, see: Customizing LyX – Part 1, Understanding the Big Picture.

As a first approximation, what I said in that article is mostly true.  But, naturally, the reality is more complex.  LyX layout files can also contain information about character styles.  This includes both information about how they should be presented on-screen and how the underlying LaTeX typesetting language should process them.  Moreover, that information can either live in a global layout file (like nih.layout) or in the .lyx file you are working on (meaning that it can be conveniently added to a template).  In the remainder of this post, I will show how to create custom character styles by modifying the local layout.

Example: Creating a “User Interface” Character Style

As you might have guessed, the example introduced above isn’t so random.  For the past several weeks, I have been working on a book about scientific and technical writing.  The book will cover a number of programs and technologies including LyX, LaTeX, BibTeX, Mendeley, Scribus and Kile.

Since it is geared to a somewhat technical audience, it includes both instructions on how to use program’s UI and code samples.  For my code examples, I can use the internal LyX character style.  But I’ve found that I would also like to highlight references to the UI.  There isn’t a pre-defined UI character style.

Fortunately, I not the first person writing a book with UI references.  If you look at the LyX user manuals, you will see that the LyX documentation team created a character style specifically for highlighting LyX menu commands.  It uses a sans serif font and I think it is rather attractive.  But, how do you go about adding it to another document?

It’s actually pretty straightforward:

1. Since there isn’t a UI to modify the local layout in LyX, open your document in a text editor
2. Create a local_layout section
3. Define how you want your character style to appear (both on screen and in the LaTeX output)

Step 1: Open your document in a text editor

As explained above, custom character styles live in the local_layout section (enclosed with the “\begin_local_layout” and “\end_local_layout”) near the beginning of a LyX document.  Unfortunately, there is not currently a user interface pane within LyX to edit this section.  As a result, open the document in your favorite text editor.  The figure shows what this looks like for a file called “Book Chapter – Technical.lyx”.

Note: There is nothing to be afraid of, LyX’s file format is relatively easy to parse; even for human beings.

“Book Chapter – Technical.lyx” has three major sections:

• Document: enclosed by “\begin_document” and “\end_document” (off-screen)
• LaTeX Preamble: enclosed by “\begin_preamble” and “\end_preamble”
• Local Layout: enclosed by “\begin_local_layout” and “end_local_layout” (off-screen)

I have added some text to the LateX preamble in this example.  The command tells LyX to use the “Latin Modern” fonts instead of the LaTeX default fonts, which I find to be rather ugly:

% set fonts for nicer pdf view
\IfFileExists{\lmodern.sty}{\usepackage{lmodern}}{}

Step 2: Create a Local Layout Section

If it doesn’t already exist, insert a local_layout section.  For organizational purposes, I like for it to appear following the preamble.  Below is the full text of the layout section for “Book Chapter – Technical.lyx”:

\begin_local_layout
    Format 7
        InsetLayout    CharStyle:UserInterface
        LyxType               charstyle
        LabelString           userinterface
        LatexType             command
        LatexName             userinterface
        Font
            Family              Sans
        EndFont
        Preamble
            \newcommand*{\userinterface}[1]{{\sffamily #1}}
        EndPreamble
    End
\end_local_layout

In the example, there is some template code (also known as boilerplate), and values that you can modify in order to better customize the appearance of your character style.  The boilerplate includes “Format 7”, LyXType      charstyle”, and “LatexType    command”.  These values will be constant for all character styles, but everything else can be customized.

Step 3: Define how you want the character style to look

When customizing the appearance of your character style, the first thing to do is to decide what you want to call your style.  This is done by modifying the “InsetLayout” argument.  Since my book project covers many different programs, I’ve decided that I will use this character style for any reference to the ui.  That includes menus, button names, or dialog box text.  As a result, I’ve called it “CharStyle:UserInterface”.  This is what appears when I go to the “Edit->Text Style” menu and in the dialog box that appears when I right click.

Next, you need to decide what you would like the “label” to be.  The label appears when the “show label” option is toggled on.  In this example, it’s “userinterface”.

I also need to associate my character style with a LaTeX name.  Again, I’ve opted to call it “userinterface”.

The next block of options, “Font”, specify how the text will look on screen.  You could be extremely ornate specifying text size, typeface, and  font weight.  In this case, however, I want to go with something simple.  I’m just going to use the “Sans” font family for on-screen to display.  Boring, but functional.

Note: A complete list of available options can be found in the LyX help documentation: “Customizing LyX: Features for the Advanced User”.  See section 5.3.6.

The Preamble block tells LaTeX how it should process the character style.  Again, I don’t want to do anything fancy, so I’m just going to use the “Sans Serif” font family.  This is accomplished by the following LaTeX command:

\newcommand*{\userinterface}[1]{{\\sffamily #1}}

The “\newcommand” syntax tells LaTeX about my custom style, while everything after is specifying the text and the font family. “\ssfamily” refers to the sans serif font family.  Other options I could have used include the roman font family (“\rmfamily”), the typewriter family (“\ttfamily”), medium text (“\mdseries”), italic (“\itshape”), or small caps (“\scshape”).

Note: For a more complete list with sample output, see here (Table 4.5).

When finished, save your document and exit the text editor.  When you load the document back into LyX, your style should appear in the “Edit->Text Style” menu.  The screenshot below shows how the “User Interface” character style looks in use.

Miscellaneous Things and Conclusion

Like the example in the previous article, I like to save documents with custom character styles as templates.  This means that I can create additional documents very easily by going to the “File” menu and choosing the “New from template” option.  The new document will have all of the template’s settings, including the LaTeX preamble and the custom character styles.  It also means that I can keep separate templates for technical documents, scientific articles, book chapters and fiction.  Each has character styles defined that make sense for that type of publication.

Character styles make writing long documents much easier.  It’s so much better to label a piece of text as “code” or “userinterface” than it is to remember the exact formatting specifications.  Moreover, it makes the document more consistent and provides an easy path to change formatting at a later date.  As a first time author, I have no complaints about anything that makes my writing easier, more organized, and flexible.

Similar Posts:

• Typeset Your Curriculum Vitae – Part 2: Extending and Customizing an Existing Document Class
• Typeset Your Curriculum Vitae – Part 1: The xetexCV Document Class
• IronPython – Windows Presentation Foundation Tutorials
• A Better Previous Versions: Time Traveler
• Barn Architecture

Copywrite 2009: Rob Oakes. Apolitically Incorrect

Customizing LyX: Character Styles and the LyX Local Layout

LyX is a wonderful writing program.  It’s easy to use and produces beautifully typeset output.  More importantly, though, it lets an author focus on the content and structure of his writing; rather than the formatting.  It isn’t so easy to customize, though, which limits its usefulness in a big way.  What if you need to create a new layout or take advantage of one of the thousands of specialized  LaTeX styles?  How, exactly, do you go about doing that?

That’s why this article was written.  Recently, I was asked to help with a National Institutes of Health (NIH) R21 grant proposal.  After some talk amongst the different investigators, it was decided that we would use LaTeX and LyX to draft it.  Unfortunately, we hit a rather substantial hurdle early in the process:  LyX doesn’t have an NIH grant template.

After additional debate, we decided to proceed with LyX anyway.  But in the process, I found myself saddled with an additional job.  In addition to responsibilities as research flunky and copy editor, I was tasked with creating a LyX and LaTeX template for our NIH grant.  This article will summarize the steps I took and describe how to create a custom template using an available style on CTAN.

Note: All of the files in this tutorial can downloaded here (.zip).

Understanding the Big Picture

Before laying out the procedure, I will start with the theory. That includes understanding how LyX and LaTeX interact, as well as what files they use for their respective jobs.

An Introduction to LyX and LaTeX

As you probably already know, LyX is a document preparation system which supports LaTeX, DocBook, and other typesetting technologies.  Writing with LyX is easier than using regular LaTeX or DocBook because it abstracts away the markup language and most of the associated pain.  Want to insert a citation?  Just hit Insert –> Citation.  There’s no fussing with markup tags or macro commands.  The same goes for inserting figure legends, tags, labels, footnotes or any other relatively complex formatting.  LyX takes care of all the minutiae for you.

When you want to customize LyX, however, all of the ugliness and complexity gets shoved right back into your face.  Instead of just writing and letting the program sort out the details, all of a sudden you are transformed from a user into a collaborating developer.  You need to know how the underlying technologies work.  What’s the difference between a LaTeX class and a style sheet?  What does a LyX layout document do?  How does that differ from a LyX template document?  How do you install them and get the system to recognize that they’re present?

To get a handle on these questions, you need to understand that LyX and LaTeX accomplish two different (though complementary) goals.  LaTeX is largely in the job of creating printed documents (typesetting), whereas LyX is about helping you to write and organize the document (on-screen display).

I like to think of LyX as an editorial assistant while LaTeX is the production staff.  The editorial assistant organizes the various elements into a more logical layout, puts references and other material at your fingertips, and finally helps collate everything into a package that the production people can make sense of.  Once the package has been assembled, the production staff then goes about laying it all out on paper, making sure that references are all numbered and formatted correctly, and otherwise handling the business of making the document beautiful.

LaTeX Classes and Styles

To go about its work, LaTeX uses a couple of important files: document classes (.cls) and styles (.sty).  As mentioned, the entire purpose behind LaTeX is that you should be able to focus on the content of your writing without being distracted by its visual presentation.  Classes and styles are what make this possible.  As you are writing, you tell LyX (and by extension LaTeX) whether a block of text is a chapter, figure legend, table, or footnote.  When ready, you can then transform the markup (which makes sense to a computer) into something that makes sense to a human being.

LaTeX classes are the first step in the transformation.  They contain important information about what the structure of the document should be and where things should go.  You might even say that a class file tells a document “what” it is.  Subsequently, the most common types of document classes are articles, reports, and books.

A LaTeX style, in contrast, tells the file “how” it should look.  If a class says, “You’re a book,” then a style will say, “you should have one inch margins” and “the chapter heading should be in Arial 16 point font with 0.2 inches of space above and below them”.  Any given document will only have a single document class, but may use many different styles.

LyX Layout and Template Files

In contrast, a LyX layout file corresponds to a LaTeX class and tells LyX how to draw things on screen; and this is where things can get a little confusing.  Even though a LyX layout is supposed to look like the printed page, this isn’t always true.  For example, the final document may have two or three columns; but on-screen the LyX document will only have one.  What is true, however, is that a LyX layout file will contain information about what document elements are available.  If you’re writing a book, for example, the LyX layout will have an option called “Chapter”; even though the fonts and spacing used won’t always be exactly the same.

The final fie type that you need to know about is called a template.  Whereas a layout file largely contains information about the on-screen display, the template file has information about typesetting and presentation.  In many ways it inherits both the properties of the LaTeX .cls and .sty files and the LyX layout properties.  This means  that you can add both LaTeX and LyX information to it.

The figure below shows these relationships graphically.

Creating a Custom LyX Template – NIH Grant Proposal

With the definitions out of the way, here is the procedure for creating a custom document template.  There are four basic steps:

1. Download the necessary document classes and styles
2. Install the files into the proper folder of your LaTeX distribution
3. Create an appropriate layout file corresponding to the document class or style
4. Create a template file with additional details and information

Step 1: Download the necessary styles and templates

The very first step is to locate the document class that you need to use.  There are many good resources to do this.  The first is CTAN, which is probably the most comprehensive resource available.  (You can also find very good package lists on Wikipedia and through Google.)  Since I already know the name of the document class I need (NIH), I will download it from the associated CTAN page.

Step 2: Install the files into the proper folder for your  LaTeX distribution

Once you have saved the files onto your computer, you need to place them in one of several locations.  If all users of the computer need it, install it to the system wide LaTeX tree.  If, however, you are the only person who will be using the package, you can place it in your own “user” folder.

The location of the system tree varies depending on which version of LaTeX you happen to be running.  It is typically defined by the “TEXMFLOCAL” variable.  For individuals using Ubuntu Linux, you will find it at “/var/lib/texmf/tex/latex”.  For users of other LaTeX distributions or operating systems, you will need to consult the documentation files.  The “user” tree is located in your own user folder.  For users of Linux, this will be “/home/username/texmf/tex/latex”.  Note: Be sure to replace “username” with your appropriate login information.

To install the package, unpack the archive and copy all of the contents to the appropriate location.  After, you will need to update the LaTeX database by running texhash.

sudo texhash

Step 3: Create an appropriate style layout for LyX

Now that the package has been installed, we need to create a layout file so that LyX can make use of it.  This is a two step process:

1. Edit the document class and style so that it only contains general elements
2. Generate a new LyX layout file

Step 3.1: Edit the document class and style to only contain general elements

Working with LaTeX classes in LyX is slightly different than how you would work with them in a tool like Kile or TeXMaker.  As already mentioned, LyX abstracts a lot of the technical detail of LaTeX away.  However, when using a tool like Kile, you directly interact with the LaTeX code.  When working this way, it can be convenient to place commonly used elements into the document class or style.  The NIH files that we downloaded and installed in the steps above do this.  (The page headers and footers contain the name of the primary investigator and the reference number for the grant.)

As you might guess, however, this isn’t a good thing if you are trying to create a general template.  The document class and style files should only contain the elements that are going to applied to all of the documents.  This shouldn’t include dates specific to a given revision, the name of a principal investigator, or the title of the grant.  We want to install the LaTeX files and then forget about them, which is simply impossible if they contain such specific information.

As a result, before we go further, we need to redact this information from the nih.cls file.  But even though we are removing it, don’t throw it away! Stick it in a spare text file for the time being.  It is still needed so that the final document will match NIH style conventions.  We will add it back to the template created in Step 4.

In the original files from CTAN, open up nih.layout in your favorite text editor and cut out the following text:

% preamble stuff
\newcommand{\nih@PIname}{Oakes, Robert Stilman}
\newcommand{\piname}[1]{\renewcommand{\nih@PIname}{#1}}
\makeatletter

% constants
\newcommand{\nih@sillysize}{\scriptsize}

…

\newtheorem{proposition}{Proposition}
\newtheorem{lemma}{Lemma}

In addition to redacting the information above, you will also want to add the following line close to the end of nih.cls.  It will ensures that alphabetic section labels will be used instead of numeric labels:

\def\thesection{\Alph{section}}

Note: In the files for this tutorial, I have already redacted the unnecessary text from the nih.cls file and added it to the NIH Grant Proposal template.  I have also added the section label code.

Step 3.2: Generate a new LyX layout file

The easiest way to create a new layout file is to modify one of the existing examples.  If you look closely at the NIH document class (nih.cls), you will notice that it is based on the LaTeX “article” class.  As we try and create an appropriate nih.layout, let’s start by modifying article.layout.

When you first open the file, you will see the following:

# Do not delete the line below; configure depends on this
# \Declare LaTeXClass{article}
…
Format 11
Input stdclass.inc

SecNumDepth             3
TocDepth                3

NoStyle Chapter
NoStyle Chapter*

Style Part
    Align                 Left
    AlignPossible         Left
    TopSep                2
    BottomSep             1.5
    Font
      Size                Larger
    EndFont
End
…

As you study the layout, you should start to see a few general sections.  For example, the second line tells LyX that it is using the LaTeX class “article”.  “Input stdclass.inc” tells LyX to import the standard list of document elements.  Whenever you see a line that begins “NoStyle”, that is telling LyX to remove those particular section types from the available list of options since they aren’t supported by the document class.

Pay close attention to the section block that begin with “Style”.  This tells LyX how to display that particular document element on screen.  It includes information about how the text should be aligned (“Align”, “AlignPossible”) and what font should be used (“Font”, “Size”, “EndFont”).

To make article.layout work with nih.cls, we need to make two modifications.  First, we need to change the \DeclareLaTeXClass and then we need to add a new “Style” block.

First, let’s change the \DeclareLaTeXClass.  Add the following:

\DeclareLaTeXClass[nih]{proposal (NIH)}

In this example, nih is the name of the LaTeX document class (nih.cls) and the information in curly brackets is a description which will make it easier to find in the LyX document settings pane.

Next, we need to add a “Style” block that describes the “Section” element of the nih document class.  By default NIH sections are labeled alphabetically: “A Specific Aims”, “B Background”, “C Methods”, etc.  In contrast, the article labels its sections numerically: “1 Specific Aims”, “2 Background”, “3 Methods”, etc.  If we don’t modify the standard labeling, it will be displayed incorrectly on screen

Luckily, it’s an easy fix.  Add the following to article.layout:

Style Section
    LabelType             Counter
    LabelCounter          section
    LabelString           “\Alph{section}”
End

After you’ve made these changes, resave the file as “nih.layout”.  Then copy the file to either the main LyX directory layouts folder (typically LyXDir/layouts) or to your LyX user directory (typically found at .lyx/layouts/).  For LyX to see and use the new files, you will need to run Tools –> Reconfigure.

Note: A full version of the nih.layout file is available as part of the archive package for this tutorial.

Step 4: Create a LyX template file

With a fully functioning LyX layout (nih.layout), you can now create a template file that has all of the needed elements.  As mentioned above, a LyX template file is nothing more than a regular LyX file.  So, start by creating a new LyX document (File –> New).

After the new document is open, the very first thing that you should do is change the document class.  Go to Document –> Settings –> Document Class.  Then from the drop down menu, select “proposal (NIH)”.

Next, you need to add the information removed from nih.cls to the LaTeX Preamble.  Go to Document –> Settings –> LaTeX Preamble, and paste the text into the provided space.  While doing this, you may want to modify the name of the PI that will be shown in the document header.  You make this change by adding your name to the line that reads:

\newcommand{\nih@PIname}{Last Name, First Name Middle}

When done, press “Close”.  You now have a fully functional, NIH compliant grant proposal template.

But while that’s the end of the essential boilerplate, you may want to include a few additional goodies.  Wouldn’t it be nice, for example, if you had access to clickable hyperlinks?  Or if you were be able to use the bookmarks sidebar to navigate the document?  For these and other desirables, add the following to the preamble:

% Hyperlink Options
\usepackage{ifpdf} % part of the hyperref bundle
\ifpdf % if pdflatex is used

% Use True Type Fonts Instead of Older LaTeX Fonts
\IfFileExists{lmodern.sty}{\usepackage{lmodern}}{}

\fi % end if pdflatex is used

% for correct jump positions when clicking on a link to a float
\usepackage[figure]{hypcap}

This code enables the hyperref package.  Then, go to Document –> Settings –> PDF Properties and enable hyperref support in the options.  While changing the PDF properties, I personally like to customize the colors of the links.  I think that document links and cited references should be black; while urls and files should be blue.  To make this change, add the following to the “additional options” line:

linkcolor=black, citecolor=black, urlcolor=blue, filecolor=blue

Using the Template

If you’ve followed all of the steps in this tutorial, you will now have a fully functioning NIH grant template.  Now might be a good time to save it in the same location where you keep other templates.  (If you don’t have a templates folder, you might want to create one.)  I’ve saved mine as “NIH Grant Proposal.lyx”.  From here on out, whenever I want to create a new NIH document, I simply choose New from template from the File menu and load the appropriate version.

Now that my template is set up, I’m back to an automated wonderland.  LyX and LaTeX keep track of the references, labels, footnotes, formatting and the other minutiae that make complicated documents so painful.  I’m free to focus on the ideas,  voice, and message; which is exactly how it should be.

Similar Posts:

• Typeset Your Curriculum Vitae – Part 2: Extending and Customizing an Existing Document Class
• Customizing LyX: Character Styles and the LyX Local Layout
• Typeset Your Curriculum Vitae – Part 1: The xetexCV Document Class
• Creating the Perfect Writing Tool: A Proposal
• Introducing LyX-Outline 0.1

Copywrite 2009: Rob Oakes. Apolitically Incorrect

Customizing LyX: Create an NIH Grant Proposal Template

In addition to nouns, verbs, and adjectives; there are other tools which supplement the writer’s arsenal. some writers, this is a moleskin notebook and ballpoint pen. For others, writing is synonymous with the usage of computer word processors, such as Microsoft Word.  What is a given, however, is that the choice is intensely personal. If you ask ten different writers, "What is the perfect tool?" Expect ten different answers.

For those who create software, however, this is big problem. Programs can’t be customized to the individual whims of every writer. (A more diverse or idiosyncratic group does not exist.)  It is necessary to select a cross section of features that meet the needs of most people.  Unfortunately, however, this has resulted in some serious compromises and an emphasis on the technical process of writing, often to the exclusion of the creative process.

The Problem

Like most writers, I have some strong feelings about the tools I use; and like most writers, I have never found one that fits my needs.  But this is not for a lack of trying.  In fact, I’ve been searching “perfect tool” for a very long time, at least since I was a senior in high school.

At that time, I enjoyed writing things out in longhand, with separate drafts copied over from the first iteration. I fancied myself a budding novelist and fervently believed that such an exercise would stimulate my creativity.

A decade later, however, I have become impatient. Now, I prefer to peck out my thoughts at the keyboard.  Sometime in college, I realized that writing by hand is slow and while it can focus the mind on word choice, it’s a plodding and frustrating exercise. In contrast, typing at the computer is quick; and since I can type nearly as fast as I can think, it allows the text to keep pace with my thoughts.

Moreover, working on the computer is organic. There isn’t a discrete first, second, and final draft; instead, I can add, modify or change my writing as I form the ideas. If I want to jump back and change a word or sentence, I do so. Pen and paper isn’t nearly so adaptable and many ideas are lost before they can be captured and expanded.

The Limitations of Existing Tools

But even though I prefer to write at the computer, I haven’t been able to find a writing program that lets me creatively adapt the structure and flow of my document.  I have tried nearly everything available, but nothing fully meets my needs.

Consider Microsoft Word, for example.  Even with all of its power, there are a number of things that it doesn’t handle very well. Ever tried to work with a complex Word document, numbering into the hundreds or thousands of pages? If not, I don’t recommend that you do.  Large Word documents cause the program to slow down or crash and they have a tendency to become corrupted.  There is nothing more disconcerting than finding out one of your book chapters is no longer usable due to bit rot.

Introducing LyX: The Document Processor

Luckily, there may a potential solution.  The open source document processor, LyX, meets many of Word’s shortcomings.  LyX encourages people to approach writing based on the structure and content of the document, rather than the appearance. Thus, it isn’t a "What You See Is What You Get" so much as a "What You See Is What You Mean" (WYSIWYM).

When used right, LyX focuses the user on writing the content. The program handles the mechanics of layout and output. To do so, LyX leverages the LaTeX typesetting system, which is used heavily in the engineering and science disciplines to publish dissertations, grant proposals, journal articles, and books.

When I first found LyX three or four years ago, it fundamentally changed the way that I worked. Before, I manually formatted everything in a given document. While this approach is okay for simple letters or lists, it quickly breaks down in anything longer than a few pages.

In contrast, LyX makes extensive use of tags that tell the program what a given piece of text is. The title is enclosed by a "title" tag, book sections by the "section" tag, chapter headings by the "chapter" tag, block quotes by the "quote" tag, and paragraph text by the "body" tag. The program then applies formatting rules globally and in a consistent manner. Text within LyX has a meaning and will be processed in a specific way.

And thus we have the first important component of a perfect writing environment: the program is aware of what content is. Such awareness allows for extremely complicated written pieces to be assembled in a relatively easy fashion.

A Sloppy Process

Despite its strengths, though, LyX is an incomplete solution. It focuses on the mechanical aspects of writing and ignores what is arguably the more important component: the creative process. Using LyX, I can assemble a true monstrosity of a manuscript replete with footnotes, endnotes, references, tables, lists of figures, indices, appendices, and all of the other components of something unreadable. Yet such elements are mere after-thoughts rather than the heart and soul of the work. The center is all about ideas.

Which brings me to the serious limitations inherent in most word processing programs: joining ideas together is a sloppy process and they simply don’t facilitate the sloppiness.

Every writer is creative in a different way. I personally start by making a miscellaneous brain dump. Everything relevant to the topic is listed out and then I start to type additional thoughts which seem relevant. After a while, I run out of steam which prompts research, reference collection, reference reading and chicken scratch notes.  Then I groan, swear a little, go for a walk, and kick ideas around in my head.

When I come back to the computer, I move things around even further and excise any bits that now seem irrelevant (carefully saving them for later). Usually by this point, I have probably started over a few times and hopefully clarified my goals.  If the process works, I get a slightly better fix on direction I intend to take and, eventually, a first draft magically seems to appear.  That is when the real fun starts.

Unfortunately, LyX was designed to work linearly. You start at the beginning and continue to the end. All of the words and ideas are supposed to come marching out like perfect little soldiers on parade day. Using such a tool can be extremely frustrating to someone who doesn’t think in a linear fashion: I want the freedom to be intuitive!

And in the name of freedom, I will probably use of four or five programs spanning two operating systems: OneNote for data collection, KDissert or FreeMind for MindMapping, and Word or LyX for the actual writing. Then I go back and update my files in the other programs.

There should be a better way! I want to use the same tool from beginning to end, not continually move between the word processor, the outliner, the mind mapper, and the note taker.

A Potential Solution: Scrivener

There just happens to be a program on Mac OS X that was apparently made for scattered people: Scrivener. Scrivener employs a number of simple (yet profoundly powerful) metaphors that let you manage the anarchy of creativity. These include:

1. An intuitive and simple map of the project (similar to Word’s Outline View).
2. A corkboard that lets you create any number of "index cards" with a title and short summary of the card’s contents.
3. An outline view where the entire structure of your document can be seen and manipulated through drag and drop.

Scrivener leverages its corkboard metaphor and outliner by linking the "index card" to a real piece of text. Thus, the index card actually represents the physical chunk of text in the underlying draft. The index cards can be moved, nested and set into folders and any changes made to the index cards are then instantly applied to the overall structure of the document. Pretty cool, huh? A given index card can represent a section, paragraph, chapter, or sentence. Even better is that cards can be split or joined together easily.

The practical result is that the entire draft is instantly fluid. You can experiment with new ideas, split away sections of text for later use, and re-arrange things as inspiration strikes you. It substantially enhances the organic process already provided by the computer. I know a surprising number of people who have moved to the Mac platform just so that they can use Scrivener.

While other programs (like Semantik, formally known as KDissert) use similar metaphors, the use of a "summary" field is what makes Scrivener stand apart. The summary begins life as an outline point and later becomes the synopsis of the section. It can keep notes, goals, or thoughts in addition to a regurgitation of what is in the text.  And no other program uses anything like it. Therefore, while you can replicate many of Scrivener’s best features through Word or LyX’s outline view, you are stuck manipulating the entire text (which becomes impractical and unwieldy very quickly).

The Outliner. Scrivener includes a high quality outliner that allows individual elements to be repositioned via drag and drop. Each outline item is connected to the text in the actual draft; therefore, changes in the outline view are immediately reflected in the structure of the document. Scrivener’s outliner is particularly effective via the inclusion of a "synopsis" field that contains a summary of the text. This summary field isn’t part of the full draft, and as a result, can begin life as an outline point that reflects the goal of the section. In contrast, the outline view in Microsoft Word uses the text of the actual document. In long or complicated documents, it is a slightly less than effective way to add new ideas.

The Corkboard. As an alternative to the outliner, Scrivener also provides a "corkboard" where parts of the document can be moved around on index cards. Each card has a title and summary that is is linked to a block of text in the document. Unlike the outliner, however, the corkboard only deals with one navigational element of the document. When at the top level, the cards might represent the parts or chapters of a book. When working on the corkboard of a book chapter, they might represent sections or individual scenes. There is no limit to the number of cards or nesting that can be done.

Scrivener’s Singular Strength

Though Scrivener handles the creative process beautifully, it isn’t a serious word or document processor. On the program’s website, the primary developer states that Scrivener is a "personal writing tool." As a result, it seems uniquely geared to creative writing; that is to say, novels and screenplays. Not many novels or screenplays have a need for a bibliography, index, appendix, footnotes, or endnotes. As a result, Scrivener cannot handle the minutiae of complicated documents.

Just forget support for hyperlinks, cross-links, cross-references, figures, legends or graphics. They’re outside the developer’s stated need and are unlikely to appear. That’s too bad, because it makes Scrivener worthless to those who need support for these features. Microsoft Word has them and LyX has them, OpenOffice pretends that it has them. However, Word, LyX, OpenOffice and other word processors completely lack the creative tools that make Scrivener a joy to use.

A Glorified Text Editor. The weakest component of Scrivener is the actual word processor. Advanced (and even not so advanced) features like footnotes, endnotes, cross references of links are completely unsupported. It is simply unsuited to writing a complicated document like an article or cross-referenced book.

A Combined Approach

The simplicity of Scrivener’s word processor causes a semi-crisis. If given my choice, I would compose nearly everything in it for the convenience of its outliner and index cards. However, it isn’t up to supporting the type of heavy lifting I am often asked to do. While I could use it as my creative crutch, that worsens the overall problem. Instead of four programs across two platforms (Window/Linux), I would be using five programs across three platforms; not what I’m aiming for. I want to simplify my work process, not make it more complicated.

I’ve tried to find what I want in the software of others and determined that it doesn’t exist. The fallback is to develop it myself by turning to opensource solutions. Since LyX already meets my needs as a document processor, it is a great place to start. As a front-end to LaTeX, it automatically supports the advanced features I need. It is cross-platform, which means that the writing environment would be the same regardless of whether I am working on Windows, Mac OS X, or Linux. Additionally, it has deep integration with revision control systems, which greatly simplify collaboration. Last, it is available under the GNU public license, which means that I can access the source code. LyX combined with the same creative tools found in Scrivener would be nearly perfect.

Almost There: LyX’s Existing Features

After a careful review of LyX, it becomes obvious that much of the functionality required for a robust outliner and corkboard is already present. At left is the outline panel in LyX. The outline is automatically generated from the heading information in a given document and any child documents. The particular example shown here is series of three essays kept in four different files: a master document and three child documents.

LyX already recognizes blocks of text and understands their contextual meaning. In fact, you can already specify the order of any element and what position it holds. Further, individual text elements can be moved from one child document to another. In other words, nearly all of the background elements for a Scrivener-like outliner or corkboard are already present.

Form and Function

What is missing is the ability to assign a summary or "synopsis" to a given chunk of text. This would ideally exist as associated meta-data and affiliated with a particular tag. Additionally, it might prove convenient to associate other meta-data in addition to the summary.

To add the "creative elements,"  most development could focus on creating a user interface that  blends the existing outline view with the functional equivalents of the corkboard and outliner from Scrivener. Convenient drag and drop of document sections from within the document map, outliner, or corkboard would allow text to be moved and positioned. The outliner would show the complete document with the option to expand or collapse a given section. The corkboard, in contrast, would show only the elements contained in one particular outline node.

The figures below show an example of what such a user-interfaces might look like. The first figure shows the outliner and the second shows the corkboard. (Though the index cards and outline points of the example correspond to the chapters and sections of the document map, this should not necessarily be required in an ideal implementation. Instead, the user would be able to determine which index cards correspond to sections and which correspond to a specific piece of evidence, argument, or narrative. That is to say, the user should be allowed the freedom to create the outline structure, rather than have it imposed.)

Proposal

I propose to create an add-on for LyX which would combine the best features of Scrivener with the already powerful capabilities of LyX. As described above, this would consist of two inter-related tools:

1) The Outliner. The outliner would be an expandable panel where the structure and rough content of a document could be laid out prior to beginning the first draft. Specific features of the Outliner would include:

a. Each outline point would have room for a title, synopsis and type of content (drawn from the list provided by the LaTeX class).

b. Individual nests in the outline can be expanded or collapsed.

c. Elements can be dragged and dropped from one point to another. The underlying document will be updated to reflect any changes.

d. Though most points in the outline might correspond to headings (e.g. book parts, chapters, sections, or subsections), this would not be required. Instead, any block of text could be assigned to a point in the outline. It would behave in precisely the same manner as other outline elements.

2) The Corkboard. Housed in the same panel as the Outliner would also be a digital Corkboard which would display a small "Index card" with the title of an outline element in addition to the synopsis. However, in contrast to the outline, only the individual elements for a selected point in the document map would be shown. In the example below, the sections of a specific chapter, "Eragon Shadeslayer: Sociopath" are shown with their corresponding summaries. Changes in the order of the cards would be reflected in the underlying draft.

Conclusion

The proposed additions to LyX would further expand its other excellent features, making it the only document/word processor with a powerful set of creative tools. By associating blocks of text with additional meta-data (like a synopsis), working with the document structure would become a visual and interactive process, thereby allowing for much greater flexibility. Indeed, such changes would bring LyX one step closer to writing perfection.

Note: A PDF copy of this article can be found here.

Similar Posts:

• Introducing LyX-Outline 0.1
• Typeset Your Curriculum Vitae – Part 2: Extending and Customizing an Existing Document Class
• On Writing – Software that cuts out distraction and clutter
• WPF – SVG Graphics and XAML – Part 2
• Customizing LyX: Create an NIH Grant Proposal Template

Copywrite 2009: Rob Oakes. Apolitically Incorrect

Creating the Perfect Writing Tool: A Proposal