| March 4, 2009 10:32 pm

In addition to nouns, verbs, and adjectives; there are other tools which supplement the writer’s arsenal. some writers, this is a moleskin notebook and ballpoint pen. For others, writing is synonymous with the usage of computer word processors, such as Microsoft Word.  What is a given, however, is that the choice is intensely personal. If you ask ten different writers, "What is the perfect tool?" Expect ten different answers.

For those who create software, however, this is big problem. Programs can’t be customized to the individual whims of every writer. (A more diverse or idiosyncratic group does not exist.)  It is necessary to select a cross section of features that meet the needs of most people.  Unfortunately, however, this has resulted in some serious compromises and an emphasis on the technical process of writing, often to the exclusion of the creative process.

The Problem

Like most writers, I have some strong feelings about the tools I use; and like most writers, I have never found one that fits my needs.  But this is not for a lack of trying.  In fact, I’ve been searching “perfect tool” for a very long time, at least since I was a senior in high school.

At that time, I enjoyed writing things out in longhand, with separate drafts copied over from the first iteration. I fancied myself a budding novelist and fervently believed that such an exercise would stimulate my creativity.

A decade later, however, I have become impatient. Now, I prefer to peck out my thoughts at the keyboard.  Sometime in college, I realized that writing by hand is slow and while it can focus the mind on word choice, it’s a plodding and frustrating exercise. In contrast, typing at the computer is quick; and since I can type nearly as fast as I can think, it allows the text to keep pace with my thoughts.

Moreover, working on the computer is organic. There isn’t a discrete first, second, and final draft; instead, I can add, modify or change my writing as I form the ideas. If I want to jump back and change a word or sentence, I do so. Pen and paper isn’t nearly so adaptable and many ideas are lost before they can be captured and expanded.

The Limitations of Existing Tools

But even though I prefer to write at the computer, I haven’t been able to find a writing program that lets me creatively adapt the structure and flow of my document.  I have tried nearly everything available, but nothing fully meets my needs.

Consider Microsoft Word, for example.  Even with all of its power, there are a number of things that it doesn’t handle very well. Ever tried to work with a complex Word document, numbering into the hundreds or thousands of pages? If not, I don’t recommend that you do.  Large Word documents cause the program to slow down or crash and they have a tendency to become corrupted.  There is nothing more disconcerting than finding out one of your book chapters is no longer usable due to bit rot.

Introducing LyX: The Document Processor

Luckily, there may a potential solution.  The open source document processor, LyX, meets many of Word’s shortcomings.  LyX encourages people to approach writing based on the structure and content of the document, rather than the appearance. Thus, it isn’t a "What You See Is What You Get" so much as a "What You See Is What You Mean" (WYSIWYM).

When used right, LyX focuses the user on writing the content. The program handles the mechanics of layout and output. To do so, LyX leverages the LaTeX typesetting system, which is used heavily in the engineering and science disciplines to publish dissertations, grant proposals, journal articles, and books.

When I first found LyX three or four years ago, it fundamentally changed the way that I worked. Before, I manually formatted everything in a given document. While this approach is okay for simple letters or lists, it quickly breaks down in anything longer than a few pages.

In contrast, LyX makes extensive use of tags that tell the program what a given piece of text is. The title is enclosed by a "title" tag, book sections by the "section" tag, chapter headings by the "chapter" tag, block quotes by the "quote" tag, and paragraph text by the "body" tag. The program then applies formatting rules globally and in a consistent manner. Text within LyX has a meaning and will be processed in a specific way.

And thus we have the first important component of a perfect writing environment: the program is aware of what content is. Such awareness allows for extremely complicated written pieces to be assembled in a relatively easy fashion.

A Sloppy Process

Despite its strengths, though, LyX is an incomplete solution. It focuses on the mechanical aspects of writing and ignores what is arguably the more important component: the creative process. Using LyX, I can assemble a true monstrosity of a manuscript replete with footnotes, endnotes, references, tables, lists of figures, indices, appendices, and all of the other components of something unreadable. Yet such elements are mere after-thoughts rather than the heart and soul of the work. The center is all about ideas.

Which brings me to the serious limitations inherent in most word processing programs: joining ideas together is a sloppy process and they simply don’t facilitate the sloppiness.

Every writer is creative in a different way. I personally start by making a miscellaneous brain dump. Everything relevant to the topic is listed out and then I start to type additional thoughts which seem relevant. After a while, I run out of steam which prompts research, reference collection, reference reading and chicken scratch notes.  Then I groan, swear a little, go for a walk, and kick ideas around in my head.

When I come back to the computer, I move things around even further and excise any bits that now seem irrelevant (carefully saving them for later). Usually by this point, I have probably started over a few times and hopefully clarified my goals.  If the process works, I get a slightly better fix on direction I intend to take and, eventually, a first draft magically seems to appear.  That is when the real fun starts.

Unfortunately, LyX was designed to work linearly. You start at the beginning and continue to the end. All of the words and ideas are supposed to come marching out like perfect little soldiers on parade day. Using such a tool can be extremely frustrating to someone who doesn’t think in a linear fashion: I want the freedom to be intuitive!

And in the name of freedom, I will probably use of four or five programs spanning two operating systems: OneNote for data collection, KDissert or FreeMind for MindMapping, and Word or LyX for the actual writing. Then I go back and update my files in the other programs.

There should be a better way! I want to use the same tool from beginning to end, not continually move between the word processor, the outliner, the mind mapper, and the note taker.

A Potential Solution: Scrivener

There just happens to be a program on Mac OS X that was apparently made for scattered people: Scrivener. Scrivener employs a number of simple (yet profoundly powerful) metaphors that let you manage the anarchy of creativity. These include:

  1. An intuitive and simple map of the project (similar to Word’s Outline View).
  2. A corkboard that lets you create any number of "index cards" with a title and short summary of the card’s contents.
  3. An outline view where the entire structure of your document can be seen and manipulated through drag and drop.

Scrivener leverages its corkboard metaphor and outliner by linking the "index card" to a real piece of text. Thus, the index card actually represents the physical chunk of text in the underlying draft. The index cards can be moved, nested and set into folders and any changes made to the index cards are then instantly applied to the overall structure of the document. Pretty cool, huh? A given index card can represent a section, paragraph, chapter, or sentence. Even better is that cards can be split or joined together easily.

The practical result is that the entire draft is instantly fluid. You can experiment with new ideas, split away sections of text for later use, and re-arrange things as inspiration strikes you. It substantially enhances the organic process already provided by the computer. I know a surprising number of people who have moved to the Mac platform just so that they can use Scrivener.

While other programs (like Semantik, formally known as KDissert) use similar metaphors, the use of a "summary" field is what makes Scrivener stand apart. The summary begins life as an outline point and later becomes the synopsis of the section. It can keep notes, goals, or thoughts in addition to a regurgitation of what is in the text.  And no other program uses anything like it. Therefore, while you can replicate many of Scrivener’s best features through Word or LyX’s outline view, you are stuck manipulating the entire text (which becomes impractical and unwieldy very quickly).

The Outliner. Scrivener includes a high quality outliner that allows individual elements to be repositioned via drag and drop. Each outline item is connected to the text in the actual draft; therefore, changes in the outline view are immediately reflected in the structure of the document. Scrivener’s outliner is particularly effective via the inclusion of a "synopsis" field that contains a summary of the text. This summary field isn’t part of the full draft, and as a result, can begin life as an outline point that reflects the goal of the section. In contrast, the outline view in Microsoft Word uses the text of the actual document. In long or complicated documents, it is a slightly less than effective way to add new ideas.

The Corkboard. As an alternative to the outliner, Scrivener also provides a "corkboard" where parts of the document can be moved around on index cards. Each card has a title and summary that is is linked to a block of text in the document. Unlike the outliner, however, the corkboard only deals with one navigational element of the document. When at the top level, the cards might represent the parts or chapters of a book. When working on the corkboard of a book chapter, they might represent sections or individual scenes. There is no limit to the number of cards or nesting that can be done.

Scrivener’s Singular Strength

Though Scrivener handles the creative process beautifully, it isn’t a serious word or document processor. On the program’s website, the primary developer states that Scrivener is a "personal writing tool." As a result, it seems uniquely geared to creative writing; that is to say, novels and screenplays. Not many novels or screenplays have a need for a bibliography, index, appendix, footnotes, or endnotes. As a result, Scrivener cannot handle the minutiae of complicated documents.

Just forget support for hyperlinks, cross-links, cross-references, figures, legends or graphics. They’re outside the developer’s stated need and are unlikely to appear. That’s too bad, because it makes Scrivener worthless to those who need support for these features. Microsoft Word has them and LyX has them, OpenOffice pretends that it has them. However, Word, LyX, OpenOffice and other word processors completely lack the creative tools that make Scrivener a joy to use.

A Glorified Text Editor. The weakest component of Scrivener is the actual word processor. Advanced (and even not so advanced) features like footnotes, endnotes, cross references of links are completely unsupported. It is simply unsuited to writing a complicated document like an article or cross-referenced book.

A Combined Approach

The simplicity of Scrivener’s word processor causes a semi-crisis. If given my choice, I would compose nearly everything in it for the convenience of its outliner and index cards. However, it isn’t up to supporting the type of heavy lifting I am often asked to do. While I could use it as my creative crutch, that worsens the overall problem. Instead of four programs across two platforms (Window/Linux), I would be using five programs across three platforms; not what I’m aiming for. I want to simplify my work process, not make it more complicated.

I’ve tried to find what I want in the software of others and determined that it doesn’t exist. The fallback is to develop it myself by turning to opensource solutions. Since LyX already meets my needs as a document processor, it is a great place to start. As a front-end to LaTeX, it automatically supports the advanced features I need. It is cross-platform, which means that the writing environment would be the same regardless of whether I am working on Windows, Mac OS X, or Linux. Additionally, it has deep integration with revision control systems, which greatly simplify collaboration. Last, it is available under the GNU public license, which means that I can access the source code. LyX combined with the same creative tools found in Scrivener would be nearly perfect.

Almost There: LyX’s Existing Features

After a careful review of LyX, it becomes obvious that much of the functionality required for a robust outliner and corkboard is already present. At left is the outline panel in LyX. The outline is automatically generated from the heading information in a given document and any child documents. The particular example shown here is series of three essays kept in four different files: a master document and three child documents.

LyX already recognizes blocks of text and understands their contextual meaning. In fact, you can already specify the order of any element and what position it holds. Further, individual text elements can be moved from one child document to another. In other words, nearly all of the background elements for a Scrivener-like outliner or corkboard are already present.

Form and Function

What is missing is the ability to assign a summary or "synopsis" to a given chunk of text. This would ideally exist as associated meta-data and affiliated with a particular tag. Additionally, it might prove convenient to associate other meta-data in addition to the summary.

To add the "creative elements,"  most development could focus on creating a user interface that  blends the existing outline view with the functional equivalents of the corkboar
d and outliner from Scrivener. Convenient drag and drop of document sections from within the document map, outliner, or corkboard would allow text to be moved and positioned. The outliner would show the complete document with the option to expand or collapse a given section. The corkboard, in contrast, would show only the elements contained in one particular outline node.

The figures below show an example of what such a user-interfaces might look like. The first figure shows the outliner and the second shows the corkboard. (Though the index cards and outline points of the example correspond to the chapters and sections of the document map, this should not necessarily be required in an ideal implementation. Instead, the user would be able to determine which index cards correspond to sections and which correspond to a specific piece of evidence, argument, or narrative. That is to say, the user should be allowed the freedom to create the outline structure, rather than have it imposed.)

Proposal

I propose to create an add-on for LyX which would combine the best features of Scrivener with the already powerful capabilities of LyX. As described above, this would consist of two inter-related tools:

1) The Outliner. The outliner would be an expandable panel where the structure and rough content of a document could be laid out prior to beginning the first draft. Specific features of the Outliner would include:

a. Each outline point would have room for a title, synopsis and type of content (drawn from the list provided by the LaTeX class).

b. Individual nests in the outline can be expanded or collapsed.

c. Elements can be dragged and dropped from one point to another. The underlying document will be updated to reflect any changes.

d. Though most points in the outline might correspond to headings (e.g. book parts, chapters, sections, or subsections), this would not be required. Instead, any block of text could be assigned to a point in the outline. It would behave in precisely the same manner as other outline elements.

2) The Corkboard. Housed in the same panel as the Outliner would also be a digital Corkboard which would display a small "Index card" with the title of an outline element in addition to the synopsis. However, in contrast to the outline, only the individual elements for a selected point in the document map would be shown. In the example below, the sections of a specific chapter, "Eragon Shadeslayer: Sociopath" are shown with their corresponding summaries. Changes in the order of the cards would be reflected in the underlying draft.

Conclusion

The proposed additions to LyX would further expand its other excellent features, making it the only document/word processor with a powerful set of creative tools. By associating blocks of text with additional meta-data (like a synopsis), working with the document structure would become a visual and interactive process, thereby allowing for much greater flexibility. Indeed, such changes would bring LyX one step closer to writing perfection.

Note: A PDF copy of this article can be found here.

Comments

12 Responses to “Creating the Perfect Writing Tool: A Proposal”

George wrote a comment on March 23, 2009

What you are proposing is exactly what I’m looking for! I have an old mac just for Scrivener. Its corkboard metaphor is perfect for managing large and complex documents. But there’s nothing like it for Windows.

LyX is fine (a bit more complicated than I need), but horrible at managing the structure of a document like Scrivener.

If you build this plugin, I’ll use it!

Amber Vaesca wrote a comment on March 25, 2009

Well written article, but I feel compelled to correct a number of errors here. Most of the complaints levelled at Scrivener were regarding features that Scrivener has support for, and this becomes even more true when you consider that the author is proposing an addition to LyX, which primarily functions in the LaTeX working space. If the author were addressing Scrivener from the viewpoint of a WYSIWYG philosophy and comparing it to Word, then the situation would be different. Instead, we can analyse the comments in the framework of Scrivener’s more advanced LaTeX export. Scrivener has a very accomplished export workflow utilising the MultiMarkdown syntax framework on the front end, and a set of Perl and XSL scripts for document conversion on the back end. In its default configuration, a well formed LaTeX document can be produced just as easily as an RTF. No more complex than LyX. Advanced customisation of the back-end scripts can produce more exotic results. For example, I have a Scrivener project for which I have coded a export engines for both BBCode and MediaWiki. I can, from a single source and with no tweaks, publish to either if those, or PDF, or LaTeX, or XHTML, in a few seconds. Onward.

First, the features it supports in both semantic (LaTeX-space) and rich-text modes of operation, erroneously marked as being unsupported in the article:

* Footnotes: Supported in both semantic and rich-text modes. Footnotes are added in-line, as with LyX, making them more contextual and easy to maintain. They can be exported as either footnotes or endnotes depending upon user preference in the document compile option. In semantic mode, footnotes will be converted to LaTeX syntax, or look-alike XHTML cross-linking.
* Figures: Handled easy via drag and drop in rich-text mode. Place the image resource unit the document flow and it will be exported as RTF image and can be imported into compatible word processors. In semantic mode, usage of MMD syntax is required, and images will be collected and organised during Compile for the user.
* Legends: See above (not sure why identical technical features are listed separately in the article)
* Graphics: See above.
* Commentation: In semantic mode, Scrivener annotations will be marked as such by classed span elements which can be handled programmatically, or inserted as marginalia in LaTeX. In rich-text mode they can be formatted as Word pop-up comments, or embedded red text.

Now for features supported in semantic mode:

* Hyperlinks: Simple Markdown syntax both inline and cross-referenced.
* Cross-links: Use of anchor style ID linking the header titles, and header labelling is supported. By default it produces automatic labels based on camelCasedContent, but manual labels can be supplied and linked to from anywhere in the document. Arbitrary point-to-point cross referencing is not possible by default, though adding support for it would be fairly trivial.
* Cross-references: See above.

Front matter:
* Table of contents can be generated via LaTeX, naturally, and there is also a built-in script for adding an anchored ToC to XHTML documents.
* Abstract & Synopsis: Built-in support.

Back matter:

* Bibliography: Full BibTeX support is built-in.
* Glossary: Has basic support built-in.
* Appendix: Has basic support built-in.

Revision control:

While not built-in, Scrivener has full a SVN compatibility mode. If used with a front-end which can handle auto-add commits (Scrivener generates a lot if files based on the user’s usage of summary and notation features), full version control and collaboration is possible.

In short, the article is well written. I was a huge fan of LyX for many years, and I still do use it as a post-processor for Scrivener (the workflow is practically seamless since semantically defined data in Scrivener turns into LaTeX marked files which LyX can then convert and utilise). I have nothing against LyX, but I feel the author of the article failed to dig deeply enough into Scrivener’s more advanced features and a capabilities.

Yes, on the surface it supports a simple model, and for most authors this is plenty enough as their editors and publishers handle all the nitty gritty. For academic usage, there is enough support in rich-text mode for many, and those that require more power and are willing to dip into MultiMarkdown and LaTeX or some other format, the power is nearly limitless.

There are futher interface niceties in Scrivener that this plug-in proposal would not address. Its full-screen mode for distractionless writing is very elegant and completely “widgetless”, and the Edit Scrivenings mode which allows the user to selectively designate edit focus (as opposed to always being confronted with the entire text as in LyX), can be great for the creative mind which suffers under clutter and distraction.

Emre Ayca wrote a comment on April 19, 2009

This is just what I was looking for ages! I am very much fond of LyX and using it consistently for some time now (after a period of hesitation, which proved to be unnecessary in the end). On the other hand, I “found” Scrivener randomly surfing on the internet a while ago, and have been “stalking” it since then (I think it has almost been 2 years now:), with a hope that if there could be some add-on, plug-in whatever for lyx to have such features like the corkboard metaphor (to buy a mac for scrivener is not an option just only due to financial constraints:-). I don’t know anything about coding, so I am sorry for not being able offer any help in this regard. But I would be glad to help in other things, as much as I can (documentation, graphics, whatever I can).

I have also came up with your post on LyX-devel, (in fact, I was googling with the exact terms “a corkboard implementation for lyx” so I found your both posts here and in lyx-devel). I scanned through both now, saving to read thoroughly for some time later as I am very much “loaded” now with papers to be submitted, etc. But please keep up the idea and know that you have supporters here:)

Best wishes

[…] first started experimenting with the Qt framework about two months ago, after deciding to tackle a major add-on to one of my favorite writing programs, LyX.

JT wrote a comment on January 25, 2010

This looks really really good. It won’t be easy to do something as good as Scrivener but this one will be open source and multi-plattform, so… thumbs up!

I can’t do any programming but I’m willing to help in anything I can, by giving suggestions, ideas, helping with design issues, translating it to spanish or even considering making a donation 😉 I’ve downloaded and tested your first release, and just added some bugs reports and feature requests to the tasklist. Keep this development going!

Jose wrote a comment on February 21, 2010

Love the project. I hope you keep pushing it. It shows that you know your tools, and have an exquisite sense of design. I used to love oneNote for this, but on linux I’m using LyX.

[…] little more than a year ago, I wrote an essay about writing.  I talked about how creative pursuits (at least for me) aren’t linear processes.  I explained […]

[…] to I believe. Rob Oakes is a writer and programmer who is currently redesigning Lyx to facilitate the ultimate tool – Lyx-Outline. I will not test it yet as I will not risk wasting lots of time on software […]

Lara wrote a comment on February 24, 2012

Hi Rob,
I am absolutely in love with your idea! I first came across your software a year ago and since then I’m coming by every couple of weeks if you finally have a testable version of the Lyx-Outline.
I think if I actually had the software it would be the piece of software that I would spend most of my next years working with! It actually seems to be the perfect writing tool!
I also downloaded your sourcecode and the programming software and spend two long nights following your instructions to compile it by myself, but I could not get it work..
It would be fantastic if you go on with this project and if you would compile the software so that it would work on Mac OS Lion.
Take care and thank you very much for your work,
Lara

Rob Oakes wrote a comment on February 27, 2012

Hi Lara,

Sorry for the long wait. I’ve been completely swamped trying to finish up a book. All I can offer is, please be patient. It is coming.

Cheers,

Rob

[…] Rob Oakes – Oak-Tree Blog » Creating the Perfect Writing Tool: A Proposal- An excellent, thorough and well thought out proposal for adding outline tools to LyX. The intention being to match the Mac OS X app Scrivener which is well liked for enabling writers to easily (drag-and-drop) re-organise sections of the document, get high-level overviews of the structure and general write in a more organic way.Ample code has been written but it’s not quite ready yet: […]

H. Vosberg wrote a comment on March 16, 2012

Dear Rob,
I think the above proposal sounds fantastic!!! I am looking very forward to use the software you are working on! (I tried to compile the source code myself – as you suggested – but I didn’t menage, unfortunately.)
I (and also my colleagues here at the research institute) would be thrilled if it was possible to try out the software ourselves. Wouldn’t it be possible that you precompile it for download for OSX Lion??
We’d appreciate it! Thank you for your interesting website!!
Cheers, Harald

Care to comment?