| May 13, 2009 6:16 pm

Writing anything – whether it be a book, or a short story or an angry letter to your boss – is substantially more than starting from the first idea moment of inspiration and continuing to the final draft. Rather, writing involves a fair number of idea fragments, fleeting moments of inspiration, and a tremendous number of dead ends.

It is incredibly unfortunate that most writing software, however, is geared to organizing and structuring the document after most of the hard work has already happened.  It simply assumed that most of the planning and layout has already happened and the author is ready to string words together.  Unfortunately, this assumption overlooks one important truth: ideas are best defined as they are expressed.  Thus, it’s usually about the time that the a writer sits down to compose the draft, that the document’s true structure becomes apparent.  In my own case, this often leads to a flurry of reorganization.  And it’s during the restructuring that the real battle begins.

When in full creative passion,  I am typically working with three or four different programs all at the same time: OneNote is open so that I can access my ideas, Word is there to start collecting the somewhat finished text, and I’ll also probably be using a MindMapper so that I can see a visual representation of the document structure.  The tools are separate and don’t communicate with one another.  Thus, a change made in one place needs to be made everywhere.  And all too often, that I end up fighting the word-processor and the notetaker, and the mind mapper.  It is tremendously frustrating to battle the tools of your adopted trade.  Luckily, I am not alone in my frustration.

In the past everal years, a number of programs have become available that leave the linear model of writing behind.  On the Mac platform, one such tool looms above the others: Scrivener.  Central to Scrivener’s function are two important metaphors: that of the outline and that of the corkboard.  And it works really, really well.  There are just a few minor problems.  First: Scrivener is only available for Mac and Scrivener’s lead developer has made it clear that there won’t be versions for other platforms.  Second: Scrivener was really designed with creative writing in mind.  Thus, while it can be used for long and complicated documents, this is a slightly less than straightforward process.  Last, to create said fancy documents, Scrivener requires the raw use of a markup language (and all of the associated headaches that come with it).

The document processor, LyX, however, excels in many areas where Scrivener falls short.  It is built upon the mature and robust underpinnings of LaTeX, the typesetting language of choice in the science and engineering.  And more importantly, it is easy to use (where LaTeX most decidedly is not).  But it fails in the same way as Microsoft Word and other word processors, it is a linear writing tool and doesn’t offer a great deal of work-flow flexibility.  That is where LyX-Outline comes in.

LyX-Outline is a marriage between Scrivener’s organizational tools and LyX’s typesetting tools.


 LyX-Outline Main Window (Mac OS X)

Looking for Flexibility

While you can download and play with LyX-Outline, please reember that is still a forthcoming add-on to the LyX.  And while you find it useful, this release is an alpha level technical preview.  Right now, you can view the structure of your document at a glance in the corkboard, move things around, poke about in the outliner, and even jot your winded thoughts with the very basic plain text editor.

But even though some of the functionality is lacking, you can start to explore.  And as you do so, one of the first things you might notice is that LyX-Outline has been designed with flexibility in mind.  Nearly all of the tools can be docked, ripped away, or hidden if they are not needed.  The work environment should adapt to your work habits, not the other way round.  Whenever the writer needs to adapt to the environment, a moment of productivity and a spark of life has been lost.

All of the tools in LyX-Outline can be docked, or float freely (Mac OS X).

A Quick Overview

The main interface consists of three main elements.  This includes the document map, the editor pane, and the dock tools.  As described above, the document map and the dock tools can be relocated, docked or left free floating (depending on your preference).  And if you don’t find them useful, they can quickly be hidden.

The document map lets you see a hierarchical list of every item in your project.  Selecting one of the items displays its content’s in the editor pane.  The editor pane is the main text processing portion of LyX-Outlineand can be toggled between a text editor (plain text only at the moment) and a corkboard view.

In addition to the main editor and corkboard, there is an additional corkboard and outliner tool available as a dockable widgets.  Just as in Scrivener, you can view the pieces of LyX-Outline Corkboards (Linux)your documents in many way.  Every item is a document, and an index card, and an outline point.  They are intimately tied together and any change made to the order or content of one will be reflected everywhere.  The document map is used to navigate the content of the main window.

The dock tools, however, can be controlled separately.  You can view the entire document, or filter to a specific level in the hierarchy via a drop-down menu.  In the future, filtering by search term, keyword, document structure, or tag will also be possible.

Item Summary and Organization

The health of any written work can also be greatly improved by connecting a simple synopsis to a larger chunk of text.  This makes it much easier to check the flow of your work at a glance .  A summary can easily be added to any individual item in the collection by double clicking on it’s index card in the corkboard or the synopsis column in the outliner.  The summary text is independent from the full text and can include pertinent information that helps you structure the document.  For now, the synopsis must be entered by hand; however, future versions will allow for automatic generation of the synopsis from the text.

The outline pane can be filtered using the drop down menu.  Choose to see the structure of the entire documen t, or just a single section (Linux).

Of Outlines and Index Cards

Scrivener (and LyX-Outline as it’s dedicated clone) utilizes multiple metaphors.  The Corkboard may be useful in some instances, but will be less helpful in others.  Ditto for the outliner.   Either way, it is important to know that both the corkboard and the outliner display sub-items.  If you are using the main corkboard, these sub items will be connected to the active item in the document map.  If you are using the dock tool, they can be filtered via the drop-down menu.  From the corkboard, you can then edit items by double clicking on either their title, or their summary.  You can reorder them through drag and drop.  From the outliner, you can add new items or sub items, or delete them from the document.  You can also change their order (by using the up or down arrow buttons) or change their indentation level (with the left and right arrow buttons).

LyX-MainWindow-Mac

Minding the Potholes

And that is a very quick tour of the Outline modules for LyX.  As noted above, this release of LyX-Outline is a prototype and it is still very far from its intended target.  Thus, if it doesn’t meet your needs right out of the box, please be patient.  The mantra of OpenSource is “release early, release often.”  In this case, I have erred on the side of early and as you might suspect, there are some dangerously rough edges.  These include:

  • Unfinished interfaces.  Most of your manipulation will probably need to happen in the outliner pane.  This includes adding or removing new document nodes, as well as indenting or un-indenting the document selection.  The final version will include ubiquitous drag and drop.
  • A crappy editor.  The current editor is plain text is a placeholder only.  Ultimately, LyX-Outline will be integrated with the LyX editor and have access to all of it’s very powerful features.  It is important to keep that future in mind.
  • No export option.  While you can save and open past documents, there is currently no way to actually get them out of the program.

In it’s current incarnation, this prototype is valuable for pecking out thoughts and playing with the organizational tools.  I am releasing it so that people can see how the tools might look in their final form and provide feedback.  Please do so.  Download the source, use it as you can, and let me know what you think.  What things work?  What things don’t?  What features would you as users like to see?

Update: You can download the source code and find installation instructions on the LyX-Outline project page.

Comments

24 Responses to “Introducing LyX-Outline 0.1”

Emre Ayca wrote a comment on May 14, 2009

Very promising start, congratulations.. I installed it on my netbook (aspire one/ win XP) and despite limited hardware resources it worked flawlessly without putting too much burden on the system.

For your info:

- The save command works like “save as” : everytime it asks for a new name and location to save
- the corkboard: doesn’t display the long-waited corkboard background:(
- the index cards work well
- full screen doesn’t work yet

Thanks a million for this add-on + please keep up the good work

Anonymous wrote a comment on May 16, 2009

>Ultimately, LyX-Outline will be integrated with the LyX editor and have access to all of it’s very powerful features.

I see you use PyQt. I dont understand how is this going to be integrated with C++ LyX? (Not question only for me — but for you too. If this is not somehow solved in advance you maybe just waste your time…)

Rob Oakes wrote a comment on May 16, 2009

@Anon. You ask an excellent question. When I started this project, I also had to surmount an additional hurdle: my own ignorance. I haven’t done much with the Qt framework. Thus, I decided to start with the Python code as a proof of concept as a way to learn the classes. By building this prototype I was able to solve two problems at once: 1) learn the Qt classes, 2) figure out both how I wanted things to work prior to committing myself to C++.

This was a very good decision because it allowed me to experiment. Utlimately, I discovered the extremely flexible Qt view/model framework and was able to implement everything on top of a QStandardItemModel. This kept the overall code-base relatively compact (a few hundred lines of code only) and straightforward. To re-implement things in C++ (started at the beginning of this month) hasn’t been hard at all. The more substantial challenge has been integrating with the existing LyX code-base. Even this has gone well and I hope to release another version integrated with LyX in early June.

jjjjjj wrote a comment on May 21, 2009

I really love this idea!!! This looks like exactly what I would want. Please hurry in finishing it ;-)

Somethings did not work so well for me…as I was sort of expecting that as it is early release, but the save and open seemed broken for me and did not do anything. Did not open or save. (a Windows XP user)

*If* it was just for me, I would not work much on complete integration on lyx interface integration. at first. From my perspective, it is better to complete core functions, as a standalone program. I really want to use this! As long as it did not do damage to the lyx files. I think that it would be incredibly useful as separate program.

For my work flow, I suspect, it would be enough to use lyx-outline in the earlier stages of drafting. Then when the overall content and core outline is complete, then save it and open in lyx to add the other stuff. It even, for me, might be an advantage. When trying to quickly draft something, it is an advantage for me to not get bogged down in tweaking details, citations. or similar as you can in lyx. (I sometimes use stripped down editors like Blackroom, for that very reason. ) After first drafts are done and need to add details, then usually for me there are only minor tweaks in outline that lyx can handle.

I suppose that the possibility of two files being edited at same time by different systems is slightly worrisome. Several possibilities, that I am sure you are aware of…also have to assume some care on part of users. Also if you had a good backup copies system that would be great. (e.g. save last X copies with time stamp or backup number, or something.) For quick documents, svn etc. seem overkill and I am not always at a computer that has been set up. When drafting early versions, I often do not save enough versions, so maybe even an option of autosaving X backups, where x is configurable?

In any event, here are some thoughts of one likely user.

Thanks for this!

jjjjjj wrote a comment on May 21, 2009

(sorry if I am being verbose, but your idea is so close to what I need, that I had to write about it without enough editing!)

The more I think of it, I would really like it if a simple-appearing editor, as an optional mode, was possible, at least for initiating new files/drafts. Whether that would be true for you or others is, of course, less clear.

There might be something to be said for a limited editor that only allowed you to change the wording and overall sections/subsections/etc. When I come to a block in writing, I spend too much time, too early tweaking the details in lyx or trying new packages to do “fun” things …

A distraction-free editor for first drafts that can spit out basic lyx files, but ignores all but text and simple hierarchy would be really useful to me. (given number of simple editors with deliberately limited options that have sprung up, that seems true of some others, as well… I like those editors, but miss outline hierarchies that help drafting ideas.)

A major complication for export to lyx files would be in whether first level is Part, Chapter, or Section, for example, in export.

And what about second level, etc. One cannot predict that under “Part level” is chapter level, as in Beamer slides one has Part and, then, Section with Chapter not even an option). Perhaps allow users to map the levels, upon export to lyx format? (e.g. for beamer in a multi-lecture situation, first might be Part, 2nd might be Section, 3 might be Subsection, 4th might be Frame. 5th could be Standard environment. or could, maybe, even do a 5th, as a list item. For this level, some things would need to be tweaked, back in the lyx editor (e.g. some might be better as numbered lists, other as bullets, others as a sort of algorithms or box. However, for first 4 points the mapping could be exact.

The second complication are “round-trips” and imports of existing lyx files.

I could imagine that it would be hard to handle the diversity of environments, How should one handle the rarer environments? The complexity of handling environments becomes too much.

But is that needed? I would be happy, if one were opening a pre-existing lyx, that it just opened in the current lyx editor with current options. Assume that after a complete first draft, that the up and down arrows in lower right panel of current lyx outline view would suffice for many. When I have a complete first draft, I seldom make *many* changes in outline.
Is that true for you?

Rob Oakes wrote a comment on May 21, 2009

@jjjjjj : I agree, I am also excited. I am spending a sizable chunk of my free time trying to finish the integration with the existing LyX codebase. Here’s a brief status update: I have finished porting the widgets from python to C++. I am now modifying a few of the LyX classes so that they will support the additional features. This is going to take quite some time since I do not want this add-in to negatively impact the experience of users who have no interest in it. Unfortunately, integrating code is always substantially more time consuming than starting from scratch. You first have to understand the logic of the other programmers (which can hard) and then you need to understand how your own ideas can fit into the larger picture of their work (which is really hard).

My additions have been added to the work environment (though they currently don’t do much) and will correctly show the existing document outline. However, the synopsis, keywords and other goodies I have planned are not yet implemented within the document structure, and this is going to take quite a bit of time to do correctly.

I’ve had several people ask, so I’ll provide you with a general time table:

1.) ~July 1, 2009. Complete integration with the LyX code-base. Submit for approval to LyX-Developers group. If accepted, they will likely be included for the upcoming 2.0 release. (Sorry, I don’t have any information on the timetable for this release. You might try checking the main LyX site).
2.) July – August. Spend time in closed alpha testing finding and squashing bugs related to the newly introduced code.
3.) ~August 1, 2008. Release for closed beta testing of LyX-Outline for Linux, Mac OS X and Windows. The beta will focus only on the LyX-Outline features, though the binaries will be development snap-shots of the upcoming LyX 2.0 release.

To specifically address your ideas, I also like the idea of a simple editor and I have considered a separate, stand-alone program with the outliner and corkboard (similar to the version released as an alpha prototype). This version would hook into the LyX libraries and be able to export to native LaTeX, LyX, OpenOffice, Word or any other document type currently supported by the LyX backend. That, however, is a secondary pursuit. As said though, I am most concerned about getting all of the add-ins ported and integrated with LyX. But I am finding the LyX codebase to be extremely modular and it shouldn’t be very hard to use it for export functionality.

Thinking about how the given levels of the document should be treated in export is something that would probably need to happen in the future though. I have my hands full trying to learn LyX’s codebase, C++ and proper development. But stretching oneself is supposed to be a good thing, it causes us to grow.

jjjjjj wrote a comment on May 22, 2009

Thanks for update on timeline.

I manage some software for biology research, so know how hard it is to add to some other programmers work!!!

I am finishing a book on a scientific topic, and was interested to see if I could use it this summer. Looks not, but this looks like it could be good for future. For purely selfish reasons, was hoping to see a simple standalone editor like this one for organizing some new content.

Always, thought that Scrivener seems like such a good idea, I am surprised there is not a Windows or Linux clone that is good (that I know of), even one that does not export to lyx, though that is where the content is really going, eventually.

Good luck, your plans look like this will really add a lot to lyx.

pavel wrote a comment on May 23, 2009

i see you mean this work seriously ;) Rob, i recommend you to send even uncomplete but somehow working prototype patch to lyx devel. it is very usual that third party patches wont be taken exactly as they are and remarks to both general design and implementation details are given. it is good to know especially general remarks before because it saves you from 1. spending too much time on a (wrong) coding details 2. frustration when the philosophies about the matter differ substantially and the code is already implemented… ;)

jre wrote a comment on June 4, 2009

I, too, am very excited about this project. I have a proposal due in a month, and need to restructure a large LyX document for it. So, when you get LyX-Outline finished, if you could just go to Schedule->Discontinuity->Pipeline and ship a copy back to me, it would be much appreciated. Oh, rats. They haven’t released that module yet, either.

Orkhan wrote a comment on July 25, 2009

Hi, any news on the work?

Rob Oakes wrote a comment on July 25, 2009

Hi Orkhan,

Things are going well. I’ve spent much of the past six weeks working on the code that coordinates what the computer sees versus what the humans see (it’s called a model). This will make it possible for edits in the corkboard and Outliner to change the underlying document. At present, it only shows what the underlying document looks like.

I’m not the world’s best C++ programmer, so it has been very slow going. But, once it’s finished, things will go very quickly since I’ve already got the corkboard up and working (with drag and drop!). Then I’ll just need to write the outliner.

Once I have a barebones version, I am going to start making custom builds available for testing purposes. I hope to have that done by late August. Any interest in being a tester?

Cheers,

Rob

Rob Oakes wrote a comment on July 25, 2009

If you are interested in the development of LyX-Outline, you can find more information at http://www.oak-tree.us/bugtracker/ If you are currently a tester, you can open bug reports. If you are a future user and there is something that you would like to see, please put in a feature request. Thanks!

altoid wrote a comment on August 15, 2009

Hi Rob,

Is there a way to download latest snapshots of LyX-Outline, or at least to know current dev/release roadmap? That would help a lot to ease a Scrivener-related frustration of a certain Windows user.

Rob Oakes wrote a comment on August 15, 2009

Hi altoid, I am still in the process of merging the LyX-Outline prototype (from which the article above was taken) with the real LyX source code. Doing this without breaking anything has been very, very hard. Worse, it has involved translating the code from Python (which is lovely, brilliant and fantastic) to C++ (which is hard, slow and painful). I’ve been stuck on a particularly tough nut for about a month, though I am working with the central LyX developers to work my way through it.

Here’s the current state of development: all of the special views (corkboard, outliner, etc.) have been ported to C++ and work with the existing LyX document system. However, editing only happens one way (from the main document to the views). You can move things around and change the level of outline items (which is two way), but you can’t make an edit in the special view and have it updated in the document. To support two way interchange of information requires that I completely rewrite the LyX model system. But since so many things rely on that model, I have to be very careful in doing so. The idea is to make LyX better, not break it completely ;0)

I’ve been keeping a list of people who are interested in updates and in serving as beta testers when LyX-Outline does become available. If you want to shoot me an e-mail at LyX-Devel at oak-tree.us with your contact information, I would be happy to add you to the list.

Sybill wrote a comment on November 2, 2009

Hi Rob,
how is your project going on? I check in every once in a while to see how its progress is, since I really like the idea of it. So please keep us up to date. :-)

Cheers,
Sybill

Jyri wrote a comment on January 30, 2010

Hello Rob,

fascinating. Thank you. A question: for the likes of me – a person who is already in the early middle parts of a Scrivener based project – any ideas, as your project matures, on getting partial or completed manuscripts out Scrivener and into Lyx-Outline? Would that work?

How might it work, would it be an easy or a difficult proposition? Any ideas? Could one instruct Lyx-Outline to simply import in some sort of format? Or would one have to do it by hand – copy-pasting and using the tools of Lyx-Outline segment by segment as one goes? Is this territory you have entertained?

I hope the question is not naive. Writing of software and scripts is rocket science for me. But the thought of having a document processor available in a Scrivener-similar context sounds like heaven. Good luck.

Jyri

Rob Oakes wrote a comment on January 30, 2010

Hi Jyri,

Thank you very much for the interest. I’m glad that it looks useful.

Like you say, this version of LyX-Outline is largely a proof of concept. I wanted to show myself and others that a Scrivener type program could be put together with Qt. Currently, I’m in the process porting over the foundation code from the python into the real version of LyX. Since last week, I’ve made some promising (if slow) progress. In order to support two-way editing of documents, I’ve needed to make some extensive changes to the underlying model used for the table of contents in LyX. This has resulted in instability and I need to work the bugs out before proceeding.

However, that doesn’t mean that you can’t take advantage of LaTeX/LyX to write in Scrivener. The secret is to use of a syntax called multi-markdown that can be used to export to XHTML or LaTeX:

http://fletcherpenney.net/multimarkdown/using_multimarkdown_with_scriv/

You write your document in Scrivener using the multi-markdown syntax. When complete, you export to LaTeX. LyX has an excellent LaTeX import function. You then finalize your draft in LyX and create your final PDF. For creative things, I’ve found it to be a dream. Even better, it doesn’t require any scripts or computer black magic.

It doesn’t work so well for academic type documents though. They come with too many riders. (Footnotes, endnotes, citations, tables, figures, etc.) For academic or professional writing, I’m much more productive using LyX from the outset.

Hope this is of some help.

Rob

gravityman wrote a comment on February 14, 2010

Wow. This sure does look great – I will be staying tuned. For me the ultimate writing tool is scrivener cork-board + tree outlining + tagging + Liquid Story Binder’s writing pleasing, colorful, environment. As it stands right now, I have use four or five applications to pull that off. evernote for my “cards”; wikipad for outlining and tagging; mendeley for document management/full text search; IRL (in real life) note cards; and, finally, at the drafting stage, Liquid. It’s getting to be a pain. :)

Orkhan wrote a comment on February 22, 2010

Hello there,

How are you? And how is the project? I might be missing the news, but it seems there aren’t any.

Orkhan wrote a comment on February 22, 2010

Sorry, if I sounded blatant :)

Orkhan wrote a comment on February 23, 2010

Hi Rob,

I thought it might be interesting to know that there is another project underway marrying Scrivener with Word.

http://writingoutliner.com/

Rob Oakes wrote a comment on February 24, 2010

@Orkhan. Thank you for the link. I was unaware of this project. It looks wonderful. Even though I’m pretty invested in Open Source tools (like LyX and LaTeX), I quite like Microsoft Word. It’s an excellent general purpose tool. This add-on would make it even more powerful.

However, I’m not sure that i would be a very good fit for the type of writing that I typically do (long, boring academic stuff). For that, I tend to prefer LaTeX and LyX, which is why I’m still slowly working on LyX-Outline.

To answer your previous question, I’m still working on it, but it is progressing slowly. My highest priority at the moment is to finish a book that I’m writing. As soon as that is finished, I have to wrap-up a study for publication. Once that is out of the way, then I intend to dive into LyX-Outline with a passion.

[...] I was excited and motivated, and had the time necessary to make headway.  Within two months, I released a prototype and worked out most of the major design issues.  I even ported much of the prototype code from [...]

Dave Garbutt wrote a comment on April 1, 2012

WOW! I am so happy I found this (via your XetexCV work).
I have started using LyX for all my own writing the last couple of years, Embedded Falsh movie in PDF? Sorted) mainly beamer and the Tufte classes. And I was thinking of moving to Scrivener just get corkboarding and the compile to concept.

Now I can safely leave it :-)

Care to comment?