To Vim

I see that Apple has updated its word-processor Pages again. The new version has some sweet features, but if you open a document created in the previous version, you will be asked if you wish to upgrade the file format. On clicking ‘yes’, the previous version of Pages will never again be able to open the file.

That, in itself, is not a problem. But, let’s say, you don’t upgrade your version of Pages immediately. Let’s say you wait until the version after that. Will that version open the files you have right now? Possibly not.

When a file format is updated, you get new features. I understand that and I applaud. But there are disadvantages. Once you’ve been writing for a few years, and you look back for your floppy, your ClarisWorks files, or even your Kindwords files, you realise that file format change is the kryptonite of longetivity.

Check out this article by Charles Stross on Microsoft Word, entitled ‘Why Microsoft Word Must Die’. Now, we all hate Word, don’t we? Come on. You do.

I hate Word from a position of some expertise, because, back in 2003-2005, I used it to write my PhD thesis. That was a single document containing multiple contents tables (some for chapters, sure, but others for psycholinguistic examples), cross-references, a bibliography, and a great deal besides. I learned the hell out of that program. Thus did I learn to hate it. It is buggy, poorly designed, and over-featured.

Going back to the point made by Charles Stross, it is a real shame that the publishing industry relies on Word as its workhorse.

Stross mentioned another program that he sometimes uses. It is called vim. I also use it.

Format Wars: A New Hope

Back in 1976, the year I was born, Bill Joy wrote a text editor for UNIX. That editor was called vi. It was designed to work over a computer terminal (i.e. a text-based interactive interface). It had two modes. In the first mode, whatever the user typed would be entered as text in the current document. In the second mode, the keyboard became a way of navigating around the document. You can read more about the program over at Wikipedia.

The program was updated by Bram Moolenaar for the Commodore Amiga, a computer I used as a kid. Moolenaar called his program vim. This stood for ‘Vi improved’. The year was 1991.

What’s It Like Using Vim?

Where I grew up, we often bought fruit from the village shop. The apples didn’t come from China or South America. They tasted good, but were a bit small and occasionally bruised. Later, we bought fruit from supermarkets. They were never bruised and they all looked the same. Didn’t taste as good, but by then I’d forgotten what non-supermarket apples tasted like. Nowadays I eat ponsy ‘organic’ apples, and they tend to come from Kent, where I live. They’re smaller, more bruised, but the taste real.

Where am I going with this? Is Vim some kind of home-grown product? No, it’s American.

Is it tastier than Microsoft Word or Apple Pages?

Well.

It’s like this. When you write in Vim, it doesn’t pretend that you’re looking at a book. It’s text. The notion of ‘presentation’ is off the table. Layout can take a running jump.

Vim presents you with the text at a much simpler level.

If you—by which I mean ‘me’—write a story in Word, or Pages, and print that bad boy out, the product you hold in your hand is somewhat disingenuous. It masquerades as a finished product. The imperfections and shortfalls of your prose are very slightly obscured by the layout and presentation, both of which are telling you, unconsciously, that the work is already like the work you see in books.

To Vim (verb): To remove superficial presentation in order to reveal substance.

Example: ‘We were larging it at the Time Piece last week until the last tune, Get Lucky. Then the main lights came on, the music turned off, and the bouncers moved in. The place was totally vimmed.’

Every imperfection jumps out. It’s just you, your eyeballs, and your text.

Stop Being So Arty-Farty. What Is It Actually Like To Use?

Vim is Fast

When you’re typing text into the command line, the computer is not rendering graphical gubbins. Letters appear slightly faster. Not so much faster that you notice it in Vim, but fast enough to notice that text rendering is slower in most other places, notably Word and Pages.

Vim as a Learning Curve

Vim has keyboard shortcuts for:

  • Deleting sentences
  • Moving the caret to the start/beginning of a sentence
  • Moving x words forward or back
  • Jumping to the top of the document, to the bottom, to the middle, and so on

These take time to learn. I’m still learning them. But, even after a few minutes, it becomes much faster to navigate a document using the keyboard than using the mouse.

Rock-solid

Vim is used by a lot of geeks (mostly for programming). I’ve never encountered a bug or had it crash.

How Do I Start?

Every journey starts with a single jump, grasshopper. From Engadget, VIM 101: a quick-and-dirty guide to our favorite free file editor.

Happy vimming.

vim

Writing a Novel Using Markdown: Part Two

In my previous post, I spoke a little about my Markdown workflow. I ended it prematurely because I didn’t have a prepared version of my book, Red Star Falling, that I could share with you.

Now I do.

You can download Multimarkdown / Markdown1 source of the complete book here. This ZIP archive contains the text of the book (cheekily, I’ve redacted most of the story), the Writer as a Stranger Cascading Style Sheet, the cover image, and a little Möbius vector graphic I’m using as a section break.

As I mentioned in my previous post, once you have your book in Markdown (i.e. a plain text file), you can use whatever tools you wish to produce your ebook–in whatever format(s) that might be.

I use a Mac OS X feature called a service. This does nothing more than run a bash script, which is a series of commands telling the computer what to do with the Markdown files. You can download the service here. It works on OS X Mountain Lion, but it will probably work on earlier versions of the Mac. The raw source for the bash script (i.e. just the series of commands) is available here, but I’ve also appended it to this post. All computers, including those running Windows, can execute bash scripts–but they might need some tweaking.

Here are the files that make up the text of the book (i.e. those in the ZIP):

Screen Shot 2013 06 23 at 20 33 23

Note that the files are numbered to preserve their order. Each Markdown file ends with a few blank lines to pad them out when they’re combined into a single file 2.

All the important information about the book–who the author is, the title, revision number, where the cover file is located–all of that is declared in the file ‘1_meta_data.mmd’. This means that, if you wish to use this method to produce your own book, the script will work as long as you’ve updated the metadata in this file.

What extra stuff you need

It would be nice if OS X came with ebook-publication tools built-in. Alas, it doesn’t. My script uses two powerful programs: ebook-convert and multimarkdown. You need to install these locally first. Note that my script tells the computer exactly where the multimarkdown command lives (it’s in /usr/local/bin/multimarkdown). I don’t know why I had to give this path to make the script work, but I did.

Using the OS X Service

This is really straightforward. On your Mac, put the service into Home > Library > Services. If your Library folder is hidden, you’ll need to Option-click the ‘Go’ menu in Finder and select it. Logout, login, and then you should see service (called • [Toy] Create MOBI-EPUB-PDF from multiple MMD, using Metadata copy) in the contextual menu when any Finder items are highlighted.

Download my example archive, highlight all the Markdown files, then right-click and select the service. It will then create .mobi, .epub and .pdf versions of Red Star Falling.

These files will be placed in the sub-folder called Publication_ready. A log file in that folder, called Log.txt, will be updated with each new conversion; look there if you want to find out why things have gone wrong. Intermediate .html files will be deleted.

Because I publish (for now) only to the Kindle, I’ve made sure that the .mobi file works perfectly on the Kindle for long-form prose3. The .epub is serviceable but not pretty. The .pdf likewise. One advantage of scripting the process is that you can tweak the settings for .epub output until you’re happy and then apply those to all your books using the same script.

Cool things you can now do

  • Because this is a script, you can tweak it to:

    • Pass the .epub file to a validation tool. I’d recommend epubcheck. Once this is in the script, it’ll happen every time.

    • Push the finalised files to a central folder where you keep all your ‘publication ready’ books.

  • You can use a single style sheet for all your books. The format–.epub, .mobi, or whatever–should not change the look and feel of your final books. Once set, you don’t need to worry about this again.

  • Creating a new book is as simple as duplicating the directory and changing the metadata. The main text of your story is simply a Markdown file: human-readable, portable, and small.

  • Use aliases. In this book, I have a section–i.e. a Markdown file–that tells the reader about my other books. I don’t want to replicate this for every new book, and I want any future books to compile with an update list of my works. So I use an alias to point to a central Markdown file that contains this information. When I update that file, any book will compile with an updated list of my works.

  • Set a watch on the folder. Perhaps trigger the script to run (suitably modified) when the main story file is updated. That way, all the publication-ready files will self-update.

What are the implications of this workflow for the writing process?

None. The act of writing itself should be completely independent of all the complexities that happen to the Markdown file later, when it’s processed. It’s more convenient in several ways. You’ll not have to worry about smart quotes, or font sizes, or anything like that. All you have to concentrate on is writing the story in this format:

## Red Star Falling

In the moment before Saskia Brandt awoke, she had a vision of red chrysanthemums falling. The flowers looked unreal. Their stems were too straight and their falls too slow. Their *Gestalt* was artful sadness.

Then the sky beyond them wintered and the dream faded.

For that, I’d recommend any number of cool programs, including ByWord and iAWriter.

Coda: The script itself

In case you’re here just for this script and want to have a look at it, here it is.

## MULTIMARKDOWN FILES > MOBI, PDF AND EPUB

## IH, 23 JUNE 2013

## This script takes a group of files in Multimarkdown,
## combines them into a single file, and then produces
## MOBI, EPUB and PDF. Also runs an epub check program.

## Expects metadata in a file called 1_meta_data.md

## Expects to have a directory called Publication_Ready.
## Will pipe command output to Publication_Ready/Log.txt

## Unusual commands used (i.e. you'll need to install these):
## ebook-convert [<http://manual.calibre-ebook.com/cli/ebook-convert.html>]
## multimarkdown [<http://fletcherpenney.net/multimarkdown/use/>]

###########################################

# Find out the directory that this script is working within
DIR=`dirname "$1"`

# Go there
cd "$DIR"

# Extract meta data from the file called '1_meta_data.mmd'
title=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="Title"`
revision=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="Revision"`
short_title=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="ShortTitle"`
author=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="Author"`
series=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="Series"`
series_index=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="SeriesIndex"`
tags=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="Tags"`
cover=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="Cover"`

# LOG Date
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo `date` $revision >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt

# Concatenate all the files passed to this script
cat "$@" > $short_title.mmd 

# LOG
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo ******* Compiling HTML from Multimarkdown... >> Publication_Ready/Log.txt

# Run 'multimarkdown' to produce HTML, smart mode, appending result to log
/usr/local/bin/multimarkdown $short_title.mmd --output=$short_title.html --smart >> Publication_Ready/Log.txt

# LOG
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo ******* Compiling MOBI… >> Publication_Ready/Log.txt

# Make MOBI
ebook-convert $short_title.html Publication_Ready/$short_title[$revision].mobi --authors="$author" --series="$series" --series-index=$series_index --title="$title" --tags="$tags" --output-profile=kindle >> Publication_Ready/Log.txt

# LOG
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo ******* Compiling PDF… >> Publication_Ready/Log.txt

# Make PDF
ebook-convert $short_title.html Publication_Ready/$short_title[$revision].pdf --authors="$author" --series="$series" --series-index=$series_index --title="$title" --tags="$tags" >> Publication_Ready/Log.txt

# LOG
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo ******* Compiling EPUB… >> Publication_Ready/Log.txt

# Make EPUB
ebook-convert $short_title.html Publication_Ready/$short_title[$revision].epub --remove-first-image --authors="$author" --series="$series" --series-index=$series_index --title="$title" --tags="$tags" --output-profile=ipad >> Publication_Ready/Log.txt 

# LOG
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo Cleaning up temporary files... >> Publication_Ready/Log.txt

# Clean up the temporary HTML and mmd files
rm $short_title.html $short_title.mmd >> Publication_Ready/Log.txt

# LOG
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo Job done bish bash `date` >> Publication_Ready/Log.txt


  1. Feel free to use this CSS with your own books. Think of it as the Writer as a Stranger ‘house style’. It will do things like indent all paragraphs unless they follow a chapter heading or section break and deal with blockquotes appropriately. I’ve iterated it a few times over the past few weeks to get it looking how I like it. ↩

  2. I’m aware that I can do this programmatically. (And, interestingly, the Multimarkdown command line tool has an option to concatenate multiple input files; however, it didn’t work for me and I couldn’t figure out why. Hence the old-school solution.) ↩

  3. You should always check your .mobi file using the Kindle Previewer application. ↩

Writing a Novel Using Markdown: Part One

This story begins with a person like me. When I publish ebooks, I want them to turn out exactly as I intend. That means taking the text of the story, sandwiching it with front matter (like copyright) and more text at the end (like acknowledgments).

Until recently, I wrote my books using the word processor Apple Pages. I would export this to a Microsoft Word document for my editors and proofreaders. Then, when the book was ready to be converted into an ebook, I would do a lot of the work by hand.

And therein lies the problem. There’s a lot of manual effort involved when you take a Word document and produce something that Amazon or iBooks will accept as an ebook.

In this post I’m going to talk about using plain text files to consistently and painlessly produce ebooks. I won’t assume much knowledge.

Styling Text

There should be a separation between style and content. Style is font size, paragraph spacing, and all that jazz. The content is the text itself. Word processors like Pages or Microsoft Word combine style and content into a single file. These files are complex, contain ambiguities, and are difficult to read without a working copy of the program that produced them.

When you separate style and content, many advantages result. The content file doesn’t need to be in a proprietary format. It can be plain text. The content file will be easily read by a computer, which means the computer can automate certain things for you: create an ebook, a nice-looking PDF, a PowerPoint presentation, a web page…all from the same content file.

But how do you make the content file?

Use Markdown

Markdown was created by John Gruber as a way of writing for the web without including the distraction of mark-up tags (more below). It has been extended by subsequent programmers.

The best way to describe Markdown is to show you an example. Here is the opening of my novella, Red Star Falling:

## Red Star Falling

In the moment before Saskia Brandt awoke, she had a vision of red chrysanthemums falling. The flowers looked unreal. Their stems were too straight and their falls too slow. Their *Gestalt* was artful sadness.

What you’re looking at above is a plain text snippet. The # character means that the text Red Star Falling should be treated as a heading (just like a heading in Word). Two ## characters means that this is a heading level two. Immediately, you can see that this is a foolproof way of telling a computer that the text is heading level two; at the same time, it’s easy for a human to read.

After

## Red Star Falling

there is a blank line. This tells the computer to treat the next text it encounters as the beginning of a new paragraph. Notice that, in this new paragraph, I’ve used a

*

character to enclose the German word Gestalt. This tells the computer to emphasise the word. In this case, it makes it italic.

OK. So I’ve written the first paragraph of my glorious story. What do I do with it?

Well, the Amazon Kindle format is based on HTML, which is the simple tagging format that most webpages use. So I want to produce HTML from the Markdown. There are many tools that will help you do this. I’ll talk about some of them later. When I convert the above text to HTML I get the following:

<h2>Red Star Falling</h2>

<p>In the moment before Saskia Brandt awoke, she had a vision of red chrysanthemums falling. The flowers looked unreal. Their stems were too straight and their falls too slow. Their <em>Gestalt</em> was artful sadness.</p>

If you’re not familiar with HTML, this will look a bit weird. The important thing to note is that the original Markdown format text will reliably produce the HTML.

Why this is exciting

How much do I love Markdown? Let me count the ways.

  1. The thing that I, as a writer, look at, is a plain text file written in Markdown. It doesn’t get much simpler. I don’t need to worry about smart quotes and dash sizes–all this is done when creating the HTML.

  2. Because it’s a plain text file, I can work on it from a computer of virtually any type (desktop, laptop, mobile), any age (even ancient PCs can read text files) and any platform (plain text files are very portable).

  3. Helper programs can take this Markdown put it into any form I like: PDF, .mobi, epub, and so on. It can take some time to set up these helper programs, but you only have to do it once.

  4. In ten years’ time, the file will be super easy to open. It’s plain text, not a proprietary word-processing format.

My Workflow

Most of my writing takes place on two devices: A MacBook Air and an iPad. My workflow is therefore Mac-centric. However, the approach should work on any system.

Here is the point where I realise that my own workflow files are hopelessly hard-coded to my own system, and providing you with a working example of Red Star Falling that you can download and play with is impossible without editing my files. So, instead of giving you these files, I’ll explain the process in plain English.

  • My book is a series of plain text files, all of which are in Markdown format. Each file represents a section of the book that I’ve isolated for convenience. (Note that file number six is actually an alias to a file held in a more central central location, which is the same across all my books.)

image

I’ve now written a follow-up to this post that includes a detailed example.

Useful resources

A Markdown cheatsheet.

My Writer as a Stranger CSS (Cascading Style Sheet). Use this in combination with a Markdown document to produce a book with the same final formatting as Red Star Falling.

Markdown tools from Brett Terpstra.