Writing a Novel Using Markdown: Part Two

In my previous post, I spoke a little about my Markdown workflow. I ended it prematurely because I didn’t have a prepared version of my book, Red Star Falling, that I could share with you.

Now I do.

You can download Multimarkdown / Markdown1 source of the complete book here. This ZIP archive contains the text of the book (cheekily, I’ve redacted most of the story), the Writer as a Stranger Cascading Style Sheet, the cover image, and a little Möbius vector graphic I’m using as a section break.

As I mentioned in my previous post, once you have your book in Markdown (i.e. a plain text file), you can use whatever tools you wish to produce your ebook–in whatever format(s) that might be.

I use a Mac OS X feature called a service. This does nothing more than run a bash script, which is a series of commands telling the computer what to do with the Markdown files. You can download the service here. It works on OS X Mountain Lion, but it will probably work on earlier versions of the Mac. The raw source for the bash script (i.e. just the series of commands) is available here, but I’ve also appended it to this post. All computers, including those running Windows, can execute bash scripts–but they might need some tweaking.

Here are the files that make up the text of the book (i.e. those in the ZIP):

Screen Shot 2013 06 23 at 20 33 23

Note that the files are numbered to preserve their order. Each Markdown file ends with a few blank lines to pad them out when they’re combined into a single file 2.

All the important information about the book–who the author is, the title, revision number, where the cover file is located–all of that is declared in the file ‘1_meta_data.mmd’. This means that, if you wish to use this method to produce your own book, the script will work as long as you’ve updated the metadata in this file.

What extra stuff you need

It would be nice if OS X came with ebook-publication tools built-in. Alas, it doesn’t. My script uses two powerful programs: ebook-convert and multimarkdown. You need to install these locally first. Note that my script tells the computer exactly where the multimarkdown command lives (it’s in /usr/local/bin/multimarkdown). I don’t know why I had to give this path to make the script work, but I did.

Using the OS X Service

This is really straightforward. On your Mac, put the service into Home > Library > Services. If your Library folder is hidden, you’ll need to Option-click the ‘Go’ menu in Finder and select it. Logout, login, and then you should see service (called • [Toy] Create MOBI-EPUB-PDF from multiple MMD, using Metadata copy) in the contextual menu when any Finder items are highlighted.

Download my example archive, highlight all the Markdown files, then right-click and select the service. It will then create .mobi, .epub and .pdf versions of Red Star Falling.

These files will be placed in the sub-folder called Publication_ready. A log file in that folder, called Log.txt, will be updated with each new conversion; look there if you want to find out why things have gone wrong. Intermediate .html files will be deleted.

Because I publish (for now) only to the Kindle, I’ve made sure that the .mobi file works perfectly on the Kindle for long-form prose3. The .epub is serviceable but not pretty. The .pdf likewise. One advantage of scripting the process is that you can tweak the settings for .epub output until you’re happy and then apply those to all your books using the same script.

Cool things you can now do

  • Because this is a script, you can tweak it to:

    • Pass the .epub file to a validation tool. I’d recommend epubcheck. Once this is in the script, it’ll happen every time.

    • Push the finalised files to a central folder where you keep all your ‘publication ready’ books.

  • You can use a single style sheet for all your books. The format–.epub, .mobi, or whatever–should not change the look and feel of your final books. Once set, you don’t need to worry about this again.

  • Creating a new book is as simple as duplicating the directory and changing the metadata. The main text of your story is simply a Markdown file: human-readable, portable, and small.

  • Use aliases. In this book, I have a section–i.e. a Markdown file–that tells the reader about my other books. I don’t want to replicate this for every new book, and I want any future books to compile with an update list of my works. So I use an alias to point to a central Markdown file that contains this information. When I update that file, any book will compile with an updated list of my works.

  • Set a watch on the folder. Perhaps trigger the script to run (suitably modified) when the main story file is updated. That way, all the publication-ready files will self-update.

What are the implications of this workflow for the writing process?

None. The act of writing itself should be completely independent of all the complexities that happen to the Markdown file later, when it’s processed. It’s more convenient in several ways. You’ll not have to worry about smart quotes, or font sizes, or anything like that. All you have to concentrate on is writing the story in this format:

## Red Star Falling

In the moment before Saskia Brandt awoke, she had a vision of red chrysanthemums falling. The flowers looked unreal. Their stems were too straight and their falls too slow. Their *Gestalt* was artful sadness.

Then the sky beyond them wintered and the dream faded.

For that, I’d recommend any number of cool programs, including ByWord and iAWriter.

Coda: The script itself

In case you’re here just for this script and want to have a look at it, here it is.

## MULTIMARKDOWN FILES > MOBI, PDF AND EPUB

## IH, 23 JUNE 2013

## This script takes a group of files in Multimarkdown,
## combines them into a single file, and then produces
## MOBI, EPUB and PDF. Also runs an epub check program.

## Expects metadata in a file called 1_meta_data.md

## Expects to have a directory called Publication_Ready.
## Will pipe command output to Publication_Ready/Log.txt

## Unusual commands used (i.e. you'll need to install these):
## ebook-convert [<http://manual.calibre-ebook.com/cli/ebook-convert.html>]
## multimarkdown [<http://fletcherpenney.net/multimarkdown/use/>]

###########################################

# Find out the directory that this script is working within
DIR=`dirname "$1"`

# Go there
cd "$DIR"

# Extract meta data from the file called '1_meta_data.mmd'
title=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="Title"`
revision=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="Revision"`
short_title=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="ShortTitle"`
author=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="Author"`
series=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="Series"`
series_index=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="SeriesIndex"`
tags=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="Tags"`
cover=`/usr/local/bin/multimarkdown 1_meta_data.mmd --extract="Cover"`

# LOG Date
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo `date` $revision >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt

# Concatenate all the files passed to this script
cat "$@" > $short_title.mmd 

# LOG
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo ******* Compiling HTML from Multimarkdown... >> Publication_Ready/Log.txt

# Run 'multimarkdown' to produce HTML, smart mode, appending result to log
/usr/local/bin/multimarkdown $short_title.mmd --output=$short_title.html --smart >> Publication_Ready/Log.txt

# LOG
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo ******* Compiling MOBI… >> Publication_Ready/Log.txt

# Make MOBI
ebook-convert $short_title.html Publication_Ready/$short_title[$revision].mobi --authors="$author" --series="$series" --series-index=$series_index --title="$title" --tags="$tags" --output-profile=kindle >> Publication_Ready/Log.txt

# LOG
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo ******* Compiling PDF… >> Publication_Ready/Log.txt

# Make PDF
ebook-convert $short_title.html Publication_Ready/$short_title[$revision].pdf --authors="$author" --series="$series" --series-index=$series_index --title="$title" --tags="$tags" >> Publication_Ready/Log.txt

# LOG
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo ******* Compiling EPUB… >> Publication_Ready/Log.txt

# Make EPUB
ebook-convert $short_title.html Publication_Ready/$short_title[$revision].epub --remove-first-image --authors="$author" --series="$series" --series-index=$series_index --title="$title" --tags="$tags" --output-profile=ipad >> Publication_Ready/Log.txt 

# LOG
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo Cleaning up temporary files... >> Publication_Ready/Log.txt

# Clean up the temporary HTML and mmd files
rm $short_title.html $short_title.mmd >> Publication_Ready/Log.txt

# LOG
echo "" >> Publication_Ready/Log.txt
echo "" >> Publication_Ready/Log.txt
echo Job done bish bash `date` >> Publication_Ready/Log.txt


  1. Feel free to use this CSS with your own books. Think of it as the Writer as a Stranger ‘house style’. It will do things like indent all paragraphs unless they follow a chapter heading or section break and deal with blockquotes appropriately. I’ve iterated it a few times over the past few weeks to get it looking how I like it. ?

  2. I’m aware that I can do this programmatically. (And, interestingly, the Multimarkdown command line tool has an option to concatenate multiple input files; however, it didn’t work for me and I couldn’t figure out why. Hence the old-school solution.) ?

  3. You should always check your .mobi file using the Kindle Previewer application. ?

Writing a Novel Using Markdown: Part One

This story begins with a person like me. When I publish ebooks, I want them to turn out exactly as I intend. That means taking the text of the story, sandwiching it with front matter (like copyright) and more text at the end (like acknowledgments).

Until recently, I wrote my books using the word processor Apple Pages. I would export this to a Microsoft Word document for my editors and proofreaders. Then, when the book was ready to be converted into an ebook, I would do a lot of the work by hand.

And therein lies the problem. There’s a lot of manual effort involved when you take a Word document and produce something that Amazon or iBooks will accept as an ebook.

In this post I’m going to talk about using plain text files to consistently and painlessly produce ebooks. I won’t assume much knowledge.

Styling Text

There should be a separation between style and content. Style is font size, paragraph spacing, and all that jazz. The content is the text itself. Word processors like Pages or Microsoft Word combine style and content into a single file. These files are complex, contain ambiguities, and are difficult to read without a working copy of the program that produced them.

When you separate style and content, many advantages result. The content file doesn’t need to be in a proprietary format. It can be plain text. The content file will be easily read by a computer, which means the computer can automate certain things for you: create an ebook, a nice-looking PDF, a PowerPoint presentation, a web page…all from the same content file.

But how do you make the content file?

Use Markdown

Markdown was created by John Gruber as a way of writing for the web without including the distraction of mark-up tags (more below). It has been extended by subsequent programmers.

The best way to describe Markdown is to show you an example. Here is the opening of my novella, Red Star Falling:

## Red Star Falling

In the moment before Saskia Brandt awoke, she had a vision of red chrysanthemums falling. The flowers looked unreal. Their stems were too straight and their falls too slow. Their *Gestalt* was artful sadness.

What you’re looking at above is a plain text snippet. The # character means that the text Red Star Falling should be treated as a heading (just like a heading in Word). Two ## characters means that this is a heading level two. Immediately, you can see that this is a foolproof way of telling a computer that the text is heading level two; at the same time, it’s easy for a human to read.

After

## Red Star Falling

there is a blank line. This tells the computer to treat the next text it encounters as the beginning of a new paragraph. Notice that, in this new paragraph, I’ve used a

*

character to enclose the German word Gestalt. This tells the computer to emphasise the word. In this case, it makes it italic.

OK. So I’ve written the first paragraph of my glorious story. What do I do with it?

Well, the Amazon Kindle format is based on HTML, which is the simple tagging format that most webpages use. So I want to produce HTML from the Markdown. There are many tools that will help you do this. I’ll talk about some of them later. When I convert the above text to HTML I get the following:

<h2>Red Star Falling</h2>

<p>In the moment before Saskia Brandt awoke, she had a vision of red chrysanthemums falling. The flowers looked unreal. Their stems were too straight and their falls too slow. Their <em>Gestalt</em> was artful sadness.</p>

If you’re not familiar with HTML, this will look a bit weird. The important thing to note is that the original Markdown format text will reliably produce the HTML.

Why this is exciting

How much do I love Markdown? Let me count the ways.

  1. The thing that I, as a writer, look at, is a plain text file written in Markdown. It doesn’t get much simpler. I don’t need to worry about smart quotes and dash sizes–all this is done when creating the HTML.

  2. Because it’s a plain text file, I can work on it from a computer of virtually any type (desktop, laptop, mobile), any age (even ancient PCs can read text files) and any platform (plain text files are very portable).

  3. Helper programs can take this Markdown put it into any form I like: PDF, .mobi, epub, and so on. It can take some time to set up these helper programs, but you only have to do it once.

  4. In ten years’ time, the file will be super easy to open. It’s plain text, not a proprietary word-processing format.

My Workflow

Most of my writing takes place on two devices: A MacBook Air and an iPad. My workflow is therefore Mac-centric. However, the approach should work on any system.

Here is the point where I realise that my own workflow files are hopelessly hard-coded to my own system, and providing you with a working example of Red Star Falling that you can download and play with is impossible without editing my files. So, instead of giving you these files, I’ll explain the process in plain English.

  • My book is a series of plain text files, all of which are in Markdown format. Each file represents a section of the book that I’ve isolated for convenience. (Note that file number six is actually an alias to a file held in a more central central location, which is the same across all my books.)

image

I’ve now written a follow-up to this post that includes a detailed example.

Useful resources

A Markdown cheatsheet.

My Writer as a Stranger CSS (Cascading Style Sheet). Use this in combination with a Markdown document to produce a book with the same final formatting as Red Star Falling.

Markdown tools from Brett Terpstra.

My Desktop

[See the larger picture]

I’m fascinated by the series My Desktop, which is a Guardian column by my friend and writer-abite-tine Ben Johncock. The series features a snapshot of a writer’s computer screen, which allows the word-botherer in question to riff on their workflow and habit; something all writers itch to know about, I would suggest.

Here’s mine. You’ll notice that I use a Mac. This is OS X Lion running on my dad’s old Mac Mini. My writing is done within a separate user account. (I have an exact duplicate of this account on my MacBook Air.) This means that (i) I can set it up in a way particular to my wordly habits (it does not, for instance, have a configured email client) and (ii) it provides an uncluttered environment that I can rely upon, no matter how untidy my main user account gets. The background was a monochrome grey until last year, when I changed it to a more friendly, physical grey. The text ‘Be original’ is a reminder that I would like to write books that stand apart from others. The image itself comes from InterfaceLift.

At the bottom of the screen, in the OS X dock, there are no ‘pinned’ applications. Minimally does it. The blue compass is the Mac browser, Safari. I keep this running for quick checks of Google maps, Wikipedia, and various pages relevant to the current work. I try to observe Stephen King’s excellent rule that you write with the door closed and edit with the door open, so Safari gets an outing only when I’m editing, as I am now. Otherwise, I enter [‘TC’] in the text if I feel that I need to insert something that I can’t at that moment. ‘TC’ means ‘to come’. For instance, I can never remember the full name of one of my characters, so I often write ‘[TC Pasha’s name’], which I’ll replace with Pavel Eduardovitch Nakhimov later.

The pen-and-ink icon is Pages, where I keep the text of the work. I’ve tried writing in applications like Byword (where I’m writing this post) but I prefer the feel of Pages. To the right of this is a strange, double-paned icon: OmniFocus.

OmniFocus is a task-manager application that I find useful for maintaining lists of edits. Although I haven’t opened it for a few weeks, it was great for working through the edits suggested by my agent. I categorised them as major or minor and kept a record of what I’d done.

On the left of the desktop, in the window called ‘Amber Rooms 101’, you’ll see three files. One of these is my current draft. Another is the previous draft, which I’m ransacking. The third is a text file containing notes about what I want to do with the current draft. This folder is actually a search. It’s looking for ‘AR101’ in the title of the file. When I move onto another draft, I’ll create new files with ‘AR102’.

In the Finder bar at the top, you’ll see a small blue box icon. This is DropBox. I use it to synchronise files between my machines. I have a Writing folder accessible to this user account, but my other DropBox files are hidden.

Music? Never listen to it during writing or editing. The only other things on my physical desk, at which I stand rather than sit, are late-nineteenth-century maps of St Petersburg.