To Vim

I see that Apple has updated its word-pro­cessor Pages again. The new ver­sion has some sweet fea­tures, but if you open a doc­u­ment cre­ated in the pre­vi­ous ver­sion, you will be asked if you wish to upgrade the file format. On click­ing ‘yes’, the pre­vi­ous ver­sion of Pages will nev­er again be able to open the file.

That, in itself, is not a prob­lem. But, let’s say, you don’t upgrade your ver­sion of Pages imme­di­ately. Let’s say you wait until the ver­sion after that. Will that ver­sion open the files you have right now? Possibly not.

When a file format is updated, you get new fea­tures. I under­stand that and I applaud. But there are dis­ad­vant­ages. Once you’ve been writ­ing for a few years, and you look back for your floppy, your ClarisWorks files, or even your Kindwords files, you real­ise that file format change is the krypton­ite of lon­get­iv­ity.

Check out this art­icle by Charles Stross on Microsoft Word, entitled ‘Why Microsoft Word Must Die’. Now, we all hate Word, don’t we? Come on. You do.

I hate Word from a pos­i­tion of some expert­ise, because, back in 2003–2005, I used it to write my PhD thes­is. That was a single doc­u­ment con­tain­ing mul­tiple con­tents tables (some for chapters, sure, but oth­ers for psy­cho­lin­guist­ic examples), cross-ref­er­ences, a bib­li­o­graphy, and a great deal besides. I learned the hell out of that pro­gram. Thus did I learn to hate it. It is buggy, poorly designed, and over-fea­tured.

Going back to the point made by Charles Stross, it is a real shame that the pub­lish­ing industry relies on Word as its work­horse.

Stross men­tioned anoth­er pro­gram that he some­times uses. It is called vim. I also use it.

Format Wars: A New Hope

Back in 1976, the year I was born, Bill Joy wrote a text edit­or for UNIX. That edit­or was called vi. It was designed to work over a com­puter ter­min­al (i.e. a text-based inter­act­ive inter­face). It had two modes. In the first mode, whatever the user typed would be entered as text in the cur­rent doc­u­ment. In the second mode, the key­board became a way of nav­ig­at­ing around the doc­u­ment. You can read more about the pro­gram over at Wikipedia.

The pro­gram was updated by Bram Moolenaar for the Commodore Amiga, a com­puter I used as a kid. Moolenaar called his pro­gram vim. This stood for ‘Vi improved’. The year was 1991.

What’s It Like Using Vim?

Where I grew up, we often bought fruit from the vil­lage shop. The apples didn’t come from China or South America. They tasted good, but were a bit small and occa­sion­ally bruised. Later, we bought fruit from super­mar­kets. They were nev­er bruised and they all looked the same. Didn’t taste as good, but by then I’d for­got­ten what non-super­mar­ket apples tasted like. Nowadays I eat ponsy ‘organ­ic’ apples, and they tend to come from Kent, where I live. They’re smal­ler, more bruised, but the taste real.

Where am I going with this? Is Vim some kind of home-grown product? No, it’s American.

Is it tasti­er than Microsoft Word or Apple Pages?

Well.

It’s like this. When you write in Vim, it doesn’t pre­tend that you’re look­ing at a book. It’s text. The notion of ‘present­a­tion’ is off the table. Layout can take a run­ning jump.

Vim presents you with the text at a much sim­pler level.

If you—by which I mean ‘me’—write a story in Word, or Pages, and print that bad boy out, the product you hold in your hand is some­what disin­genu­ous. It mas­quer­ades as a fin­ished product. The imper­fec­tions and short­falls of your prose are very slightly obscured by the lay­out and present­a­tion, both of which are telling you, uncon­sciously, that the work is already like the work you see in books.

To Vim (verb): To remove super­fi­cial present­a­tion in order to reveal sub­stance.

Example: ‘We were lar­ging it at the Time Piece last week until the last tune, Get Lucky. Then the main lights came on, the music turned off, and the boun­cers moved in. The place was totally vimmed.’

Every imper­fec­tion jumps out. It’s just you, your eye­balls, and your text.

Stop Being So Arty-Farty. What Is It Actually Like To Use?

Vim is Fast

When you’re typ­ing text into the com­mand line, the com­puter is not ren­der­ing graph­ic­al gub­bins. Letters appear slightly faster. Not so much faster that you notice it in Vim, but fast enough to notice that text ren­der­ing is slower in most oth­er places, not­ably Word and Pages.

Vim as a Learning Curve

Vim has key­board short­cuts for:

  • Deleting sen­tences
  • Moving the caret to the start/beginning of a sen­tence
  • Moving x words for­ward or back
  • Jumping to the top of the doc­u­ment, to the bot­tom, to the middle, and so on

These take time to learn. I’m still learn­ing them. But, even after a few minutes, it becomes much faster to nav­ig­ate a doc­u­ment using the key­board than using the mouse.

Rock-solid

Vim is used by a lot of geeks (mostly for pro­gram­ming). I’ve nev­er encountered a bug or had it crash.

How Do I Start?

Every jour­ney starts with a single jump, grasshop­per. From Engadget, VIM 101: a quick-and-dirty guide to our favor­ite free file edit­or.

Happy vimming.

vim

Writing a Novel Using Markdown: Part Two

In my pre­vi­ous post, I spoke a little about my Markdown work­flow. I ended it pre­ma­turely because I didn’t have a pre­pared ver­sion of my book, Red Star Falling, that I could share with you.

Now I do.

You can down­load Multimarkdown / Markdown1 source of the com­plete book here. This ZIP archive con­tains the text of the book (cheekily, I’ve redac­ted most of the story), the Writer as a Stranger Cascading Style Sheet, the cov­er image, and a little Möbius vec­tor graph­ic I’m using as a sec­tion break.

As I men­tioned in my pre­vi­ous post, once you have your book in Markdown (i.e. a plain text file), you can use whatever tools you wish to pro­duce your ebook–in whatever format(s) that might be.

I use a Mac OS X fea­ture called a ser­vice. This does noth­ing more than run a bash script, which is a series of com­mands telling the com­puter what to do with the Markdown files. You can down­load the ser­vice here. It works on OS X Mountain Lion, but it will prob­ably work on earli­er ver­sions of the Mac. The raw source for the bash script (i.e. just the series of com­mands) is avail­able here, but I’ve also appen­ded it to this post. All com­puters, includ­ing those run­ning Windows, can execute bash scripts–but they might need some tweak­ing.

Here are the files that make up the text of the book (i.e. those in the ZIP):

Screen Shot 2013 06 23 at 20 33 23

Note that the files are numbered to pre­serve their order. Each Markdown file ends with a few blank lines to pad them out when they’re com­bined into a single file 2.

All the import­ant inform­a­tion about the book–who the author is, the title, revi­sion num­ber, where the cov­er file is located–all of that is declared in the file ‘1_meta_data.mmd’. This means that, if you wish to use this meth­od to pro­duce your own book, the script will work as long as you’ve updated the metadata in this file.

What extra stuff you need

It would be nice if OS X came with ebook-pub­lic­a­tion tools built-in. Alas, it doesn’t. My script uses two power­ful pro­grams: ebook-con­vert and mul­ti­mark­down. You need to install these loc­ally first. Note that my script tells the com­puter exactly where the mul­ti­mark­down com­mand lives (it’s in /usr/local/bin/multimarkdown). I don’t know why I had to give this path to make the script work, but I did.

Using the OS X Service

This is really straight­for­ward. On your Mac, put the ser­vice into Home > Library > Services. If your Library folder is hid­den, you’ll need to Option-click the ‘Go’ menu in Finder and select it. Logout, login, and then you should see ser­vice (called • [Toy] Create MOBI-EPUB-PDF from mul­tiple MMD, using Metadata copy) in the con­tex­tu­al menu when any Finder items are high­lighted.

Download my example archive, high­light all the Markdown files, then right-click and select the ser­vice. It will then cre­ate .mobi, .epub and .pdf ver­sions of Red Star Falling.

These files will be placed in the sub-folder called Publication_ready. A log file in that folder, called Log.txt, will be updated with each new con­ver­sion; look there if you want to find out why things have gone wrong. Intermediate .html files will be deleted.

Because I pub­lish (for now) only to the Kindle, I’ve made sure that the .mobi file works per­fectly on the Kindle for long-form prose3. The .epub is ser­vice­able but not pretty. The .pdf like­wise. One advant­age of script­ing the pro­cess is that you can tweak the set­tings for .epub out­put until you’re happy and then apply those to all your books using the same script.

Cool things you can now do

  • Because this is a script, you can tweak it to:

    • Pass the .epub file to a val­id­a­tion tool. I’d recom­mend epubcheck. Once this is in the script, it’ll hap­pen every time.

    • Push the final­ised files to a cent­ral folder where you keep all your ‘pub­lic­a­tion ready’ books.

  • You can use a single style sheet for all your books. The format–.epub, .mobi, or whatever–should not change the look and feel of your final books. Once set, you don’t need to worry about this again.

  • Creating a new book is as simple as duplic­at­ing the dir­ect­ory and chan­ging the metadata. The main text of your story is simply a Markdown file: human-read­able, port­able, and small.

  • Use ali­ases. In this book, I have a section–i.e. a Markdown file–that tells the read­er about my oth­er books. I don’t want to rep­lic­ate this for every new book, and I want any future books to com­pile with an update list of my works. So I use an ali­as to point to a cent­ral Markdown file that con­tains this inform­a­tion. When I update that file, any book will com­pile with an updated list of my works.

  • Set a watch on the folder. Perhaps trig­ger the script to run (suit­ably mod­i­fied) when the main story file is updated. That way, all the pub­lic­a­tion-ready files will self-update.

What are the implications of this workflow for the writing process?

None. The act of writ­ing itself should be com­pletely inde­pend­ent of all the com­plex­it­ies that hap­pen to the Markdown file later, when it’s pro­cessed. It’s more con­veni­ent in sev­er­al ways. You’ll not have to worry about smart quotes, or font sizes, or any­thing like that. All you have to con­cen­trate on is writ­ing the story in this format:

## Red Star Falling

In the moment before Saskia Brandt awoke, she had a vision of red chrysanthemums falling. The flowers looked unreal. Their stems were too straight and their falls too slow. Their *Gestalt* was artful sadness.

Then the sky beyond them wintered and the dream faded.

For that, I’d recom­mend any num­ber of cool pro­grams, includ­ing ByWord and iAWriter.

Coda: The script itself

In case you’re here just for this script and want to have a look at it, here it is.

[code language=“bash”]## MULTIMARKDOWN FILES > MOBI, PDF AND EPUB

## IH, 23 JUNE 2013

## This script takes a group of files in Multimarkdown,
## com­bines them into a single file, and then pro­duces
## MOBI, EPUB and PDF. Also runs an epub check pro­gram.

## Expects metadata in a file called 1_meta_data.md

## Expects to have a dir­ect­ory called Publication_Ready.
## Will pipe com­mand out­put to Publication_Ready/Log.txt

## Unusual com­mands used (i.e. you’ll need to install these):
## ebook-con­vert [<http://manual.calibre-ebook.com/cli/ebook-convert.html>]
## mul­ti­mark­down [<http://fletcherpenney.net/multimarkdown/use/>]

###########################################

# Find out the dir­ect­ory that this script is work­ing with­in
DIR=‘dirname “$1”‘

# Go there
cd “$DIR”

# Extract meta data from the file called ‘1_meta_data.mmd’
title=‘/usr/local/bin/multimarkdown 1_meta_data.mmd –extract=“Title“‘
revision=‘/usr/local/bin/multimarkdown 1_meta_data.mmd –extract=“Revision“‘
short_title=‘/usr/local/bin/multimarkdown 1_meta_data.mmd –extract=“ShortTitle“‘
author=‘/usr/local/bin/multimarkdown 1_meta_data.mmd –extract=“Author“‘
series=‘/usr/local/bin/multimarkdown 1_meta_data.mmd –extract=“Series“‘
series_index=‘/usr/local/bin/multimarkdown 1_meta_data.mmd –extract=“SeriesIndex“‘
tags=‘/usr/local/bin/multimarkdown 1_meta_data.mmd –extract=“Tags“‘
cover=‘/usr/local/bin/multimarkdown 1_meta_data.mmd –extract=“Cover“‘

# LOG Date
echo “” » Publication_Ready/Log.txt
echo “” » Publication_Ready/Log.txt
echo “” » Publication_Ready/Log.txt
echo ‘date‘ $revi­sion » Publication_Ready/Log.txt
echo “” » Publication_Ready/Log.txt

# Concatenate all the files passed to this script
cat “$@” > $short_title.mmd

# LOG
echo “” » Publication_Ready/Log.txt
echo “” » Publication_Ready/Log.txt
echo ******* Compiling HTML from Multimarkdown… » Publication_Ready/Log.txt

# Run ‘mul­ti­mark­down’ to pro­duce HTML, smart mode, append­ing res­ult to log
/usr/local/bin/multimarkdown $short_title.mmd –output=$short_title.html –smart » Publication_Ready/Log.txt

# LOG
echo “” » Publication_Ready/Log.txt
echo “” » Publication_Ready/Log.txt
echo ******* Compiling MOBI… » Publication_Ready/Log.txt

# Make MOBI
ebook-con­vert $short_title.html Publication_Ready/$short_title[$revision].mobi –authors=”$author” –series=”$series” –series-index=$series_index –title=”$title” –tags=”$tags” –output-profile=kindle » Publication_Ready/Log.txt

# LOG
echo “” » Publication_Ready/Log.txt
echo “” » Publication_Ready/Log.txt
echo ******* Compiling PDF… » Publication_Ready/Log.txt

# Make PDF
ebook-con­vert $short_title.html Publication_Ready/$short_title[$revision].pdf –authors=”$author” –series=”$series” –series-index=$series_index –title=”$title” –tags=”$tags” » Publication_Ready/Log.txt

# LOG
echo “” » Publication_Ready/Log.txt
echo “” » Publication_Ready/Log.txt
echo ******* Compiling EPUB… » Publication_Ready/Log.txt

# Make EPUB
ebook-con­vert $short_title.html Publication_Ready/$short_title[$revision].epub –remove-first-image –authors=”$author” –series=”$series” –series-index=$series_index –title=”$title” –tags=”$tags” –output-profile=ipad » Publication_Ready/Log.txt

# LOG
echo “” » Publication_Ready/Log.txt
echo “” » Publication_Ready/Log.txt
echo Cleaning up tem­por­ary files… » Publication_Ready/Log.txt

# Clean up the tem­por­ary HTML and mmd files
rm $short_title.html $short_title.mmd » Publication_Ready/Log.txt

# LOG
echo “” » Publication_Ready/Log.txt
echo “” » Publication_Ready/Log.txt
echo Job done bish bash ‘date‘ » Publication_Ready/Log.txt

[/code]


  1. Feel free to use this CSS with your own books. Think of it as the Writer as a Stranger ‘house style’. It will do things like indent all para­graphs unless they fol­low a chapter head­ing or sec­tion break and deal with block­quotes appro­pri­ately. I’ve iter­ated it a few times over the past few weeks to get it look­ing how I like it. ↩

  2. I’m aware that I can do this pro­gram­mat­ic­ally. (And, inter­est­ingly, the Multimarkdown com­mand line tool has an option to con­cat­en­ate mul­tiple input files; how­ever, it didn’t work for me and I couldn’t fig­ure out why. Hence the old-school solu­tion.) ↩

  3. You should always check your .mobi file using the Kindle Previewer applic­a­tion. ↩

Writing a Novel Using Markdown: Part One

This story begins with a per­son like me. When I pub­lish ebooks, I want them to turn out exactly as I intend. That means tak­ing the text of the story, sand­wich­ing it with front mat­ter (like copy­right) and more text at the end (like acknow­ledg­ments).

Until recently, I wrote my books using the word pro­cessor Apple Pages. I would export this to a Microsoft Word doc­u­ment for my edit­ors and proofread­ers. Then, when the book was ready to be con­ver­ted into an ebook, I would do a lot of the work by hand.

And therein lies the prob­lem. There’s a lot of manu­al effort involved when you take a Word doc­u­ment and pro­duce some­thing that Amazon or iBooks will accept as an ebook.

In this post I’m going to talk about using plain text files to con­sist­ently and pain­lessly pro­duce ebooks. I won’t assume much know­ledge.

Styling Text

There should be a sep­ar­a­tion between style and con­tent. Style is font size, para­graph spa­cing, and all that jazz. The con­tent is the text itself. Word pro­cessors like Pages or Microsoft Word com­bine style and con­tent into a single file. These files are com­plex, con­tain ambi­gu­ities, and are dif­fi­cult to read without a work­ing copy of the pro­gram that pro­duced them.

When you sep­ar­ate style and con­tent, many advant­ages res­ult. The con­tent file doesn’t need to be in a pro­pri­et­ary format. It can be plain text. The con­tent file will be eas­ily read by a com­puter, which means the com­puter can auto­mate cer­tain things for you: cre­ate an ebook, a nice-look­ing PDF, a PowerPoint present­a­tion, a web page…all from the same con­tent file.

But how do you make the con­tent file?

Use Markdown

Markdown was cre­ated by John Gruber as a way of writ­ing for the web without includ­ing the dis­trac­tion of mark-up tags (more below). It has been exten­ded by sub­sequent pro­gram­mers.

The best way to describe Markdown is to show you an example. Here is the open­ing of my novella, Red Star Falling:

## Red Star Falling

In the moment before Saskia Brandt awoke, she had a vis­ion of red chrys­an­them­ums fall­ing. The flowers looked unreal. Their stems were too straight and their falls too slow. Their *Gestalt* was art­ful sad­ness.

What you’re look­ing at above is a plain text snip­pet. The # char­ac­ter means that the text Red Star Falling should be treated as a head­ing (just like a head­ing in Word). Two ## char­ac­ters means that this is a head­ing level two. Immediately, you can see that this is a fool­proof way of telling a com­puter that the text is head­ing level two; at the same time, it’s easy for a human to read.

After

## Red Star Falling

there is a blank line. This tells the com­puter to treat the next text it encoun­ters as the begin­ning of a new para­graph. Notice that, in this new para­graph, I’ve used a

*

char­ac­ter to enclose the German word Gestalt. This tells the com­puter to emphas­ise the word. In this case, it makes it ital­ic.

OK. So I’ve writ­ten the first para­graph of my glor­i­ous story. What do I do with it?

Well, the Amazon Kindle format is based on HTML, which is the simple tag­ging format that most webpages use. So I want to pro­duce HTML from the Markdown. There are many tools that will help you do this. I’ll talk about some of them later. When I con­vert the above text to HTML I get the fol­low­ing:

Red Star Falling

In the moment before Saskia Brandt awoke, she had a vision of red chrysanthemums falling. The flowers looked unreal. Their stems were too straight and their falls too slow. Their Gestalt was artful sadness.

If you’re not famil­i­ar with HTML, this will look a bit weird. The import­ant thing to note is that the ori­gin­al Markdown format text will reli­ably pro­duce the HTML.

Why this is exciting

How much do I love Markdown? Let me count the ways.

  1. The thing that I, as a writer, look at, is a plain text file writ­ten in Markdown. It doesn’t get much sim­pler. I don’t need to worry about smart quotes and dash sizes–all this is done when cre­at­ing the HTML.

  2. Because it’s a plain text file, I can work on it from a com­puter of vir­tu­ally any type (desktop, laptop, mobile), any age (even ancient PCs can read text files) and any plat­form (plain text files are very port­able).

  3. Helper pro­grams can take this Markdown put it into any form I like: PDF, .mobi, epub, and so on. It can take some time to set up these help­er pro­grams, but you only have to do it once.

  4. In ten years’ time, the file will be super easy to open. It’s plain text, not a pro­pri­et­ary word-pro­cessing format.

My Workflow

Most of my writ­ing takes place on two devices: A MacBook Air and an iPad. My work­flow is there­fore Mac-cent­ric. However, the approach should work on any sys­tem.

Here is the point where I real­ise that my own work­flow files are hope­lessly hard-coded to my own sys­tem, and provid­ing you with a work­ing example of Red Star Falling that you can down­load and play with is impossible without edit­ing my files. So, instead of giv­ing you these files, I’ll explain the pro­cess in plain English.

  • My book is a series of plain text files, all of which are in Markdown format. Each file rep­res­ents a sec­tion of the book that I’ve isol­ated for con­veni­ence. (Note that file num­ber six is actu­ally an ali­as to a file held in a more cent­ral cent­ral loc­a­tion, which is the same across all my books.)

image

I’ve now writ­ten a fol­low-up to this post that includes a detailed example.

Useful resources

A Markdown cheat­sheet.

My Writer as a Stranger CSS (Cascading Style Sheet). Use this in com­bin­a­tion with a Markdown doc­u­ment to pro­duce a book with the same final format­ting as Red Star Falling.

Markdown tools from Brett Terpstra.