Eureka! I Have a Workflow

Warning sign WARNING: Geeky HTML and Makefile stuff in this post will render your eyeballs glassy if you’re not into that sort of thing.

I have been trying for weeks to create a workflow that allows me to edit my manuscript in Markdown format and then, with a simple command, generates all of the document formats I need to publish my book. That way I can create review copies and send them out, get feedback and make the changes I want, and then do it again — all in a precisely correct, reproducible way. Yesterday, in a long session, I managed to get the tools to do what I want. Maybe sharing this will help someone someday?

The requirements I had were:

  1. No manual steps at all. Just edit the manuscript and type make to get all of the required output files, ready to print or upload to Amazon, Google, Apple, my own Kindle, or whatever.

  2. Allow me to tag text with classes in my manuscript so I can include things like a real title page in a title-page-y font, a haiku, emails or text messages formatted so they look like they would on a computer screen, or even a THE END at the end of my story that is more than just some bold text on a line by itself. I wanted control over my inline and block element classes so I could do custom CSS for these special things.

  3. Let me include fonts in the output files, minimized to just the glyphs I use in my manuscript and ready to be used by the eBook reader if the user wishes it.

I use Python Markdown to generate HTML from the Markdown formatted manuscript and Calibre‘s ebook-convert command to do the multi-format conversion, taking my HTML, CSS, and font files as input and generating EPUB, AZW3, PDF, and maybe MOBI format.

Even though I could have Calibre invoke it for me, I use the Python Markdown tool directly. I was having trouble getting Calibre to turn on the Markdown extensions I needed, and I just decided to Hell with it. Don’t get me wrong — Calibre is a wonderful tool, and I thank the obviously brilliant Kovid Goyal for creating it for us all to use. By all means, if you use Calibre, please consider donating some money or some code. The Markdown extensions I ended up needing for Dying to Live Forever were attr_list, smarty, fenced_code, and sane_lists. The output of the Python Markdown step is an HTML file with inline and block element classes I can style using my CSS to make everything look just the way I want it.

Some notes about Calibre, in case anyone wonders why I did things the way I did.

First, I couldn’t find a way to get Calibre to substitute my scene breaks with what I wanted. The HTML file I gave Calibre used simple <hr />, but nothing I did — and I tried for quite a while — seemed to be able to trick Calibre into substituting what I wanted, which was <div class="hr">&larr;&nbsp;&rarr;</div>. So I just brute-forced this using a global search-and-replace operation on the HTML before I gave it Calibre to handle the rest.

Second, as I mentioned before, I couldn’t get Calibre to use the extensions I wanted to pass to markdown_py, so I just brute forced that too.

After the Markdown conversion, I run ebook-convert for each file format I need. Today I have EPUB and AZW3 formats, but I expect to add PDF soon so I can give a dead trees copy of the book to some folks who don’t use eBook readers. Along the way, this dead trees format was very nice for marked up review copies with scribbles everywhere for comments and corrections.

This ZIP file contains the Makefile and CSS I used. I used Calibre version 2.4 and Python Markdown version 2.4 (coincidence?).

Slogging and blogging

Gawd this is a slog! I’m at the 63% point in my manuscript, according to my text editor. This is going slowly, but I feel good about it. The revisions are valuable — I’m making this much better. And I’m finding various small (usually) errors and inconsistencies in the story telling. I needed to do this.

Some people might be interested in the tool chain I’m using to finish this book. Most of the book has been written using these tools, although I started out — years ago, now — using what used to be called StarOffice back in the day.

I have always used Linux from the beginning. I hate Windows, and I didn’t get a Macbook until recently, and that is the property of my employer anyway. So, Linux it is. I use KDE on Ubuntu, and I’m pretty happy with it. I have eschewed fancy and distracting WYSIWYG word processors for something simpler and more contained.

The GNU Emacs logo I use GNU Emacs for editing the text itself, a GNU Makefile to drive the tools to transform this into the finished PDF and EBOOK formats, and various other tools as they’re needed. I couldn’t live without Emacs’ ispell package, and I have made good use of ediff to merge versions of the manuscript when I screwed up and edited them on two different computers and had to recover from that.

I have an ARM based Samsung Chromebook for when I’m working on the train or out and about. ChromeOS is wonderful for many things, but I wanted my Linux tools, so I installed crouton to get emacs and the other Linux goodies. I use a desktop PC (built it myself!) at home for the really heavy editing when I’m there.

After editing the text form, which is in Markdown format by the way, I pass the whole thing through pandoc to get HTML for one version and EPUB for another. The HTML goes through LibreOffice using an unfortunately still mostly manual process to create a PDF for someone who needs dead-trees copy to write on.

The EPUB version goes through Calibre to be transformed into the various eBook formats required by Google, Amazon, Apple, and SmashWords. These conversions are still somewhat buggy. Before I publish this thing I’m going to have to make them bullet-proof. I already have a very nice book design and cover, but getting the tools to generate a form that each of these eBook vendors can gobble up and feed to the various eBook readers while retaining some semblance of the original formatting is hard.

I’m learning by doing. I have new respect for the publishing houses and the companies that do all of these things as a service. When I’m rich someday I might ask someone else to do these jobs for me. For now, my time is free.