Descriptions of the Nondescript

Eureka! I Have a Workflow

Warning sign
Warning sign

WARNING: Geeky HTML and Makefile stuff in this post will render your eyeballs glassy if you're not into that sort of thing.

I have been trying for weeks to create a workflow that allows me to edit my manuscript in Markdown format and then, with a simple command, generates all of the document formats I need to publish my book. That way I can create review copies and send them out, get feedback and make the changes I want, and then do it again --- all in a precisely correct, reproducible way. Yesterday, in a long session, I managed to get the tools to do what I want. Maybe sharing this will help someone someday?

The requirements I had were:

  1. No manual steps at all. Just edit the manuscript and type make to get all of the required output files, ready to print or upload to Amazon, Google, Apple, my own Kindle, or whatever.

  2. Allow me to tag text with classes in my manuscript so I can include things like a real title page in a title-page-y font, a haiku, emails or text messages formatted so they look like they would on a computer screen, or even a THE END at the end of my story that is more than just some bold text on a line by itself. I wanted control over my inline and block element classes so I could do custom CSS for these special things.

  3. Let me include fonts in the output files, minimized to just the glyphs I use in my manuscript and ready to be used by the eBook reader if the user wishes it.

I use Python Markdown to generate HTML from the Markdown formatted manuscript and Calibre's ebook-convert command to do the multi-format conversion, taking my HTML, CSS, and font files as input and generating EPUB, AZW3, PDF, and maybe MOBI format.

Even though I could have Calibre invoke it for me, I use the Python Markdown tool directly. I was having trouble getting Calibre to turn on the Markdown extensions I needed, and I just decided to Hell with it. Don't get me wrong --- Calibre is a wonderful tool, and I thank the obviously brilliant Kovid Goyal for creating it for us all to use. By all means, if you use Calibre, please consider donating some money or some code. The Markdown extensions I ended up needing for Dying to Live Forever were attr_list, smarty, fenced_code, and sane_lists. The output of the Python Markdown step is an HTML file with inline and block element classes I can style using my CSS to make everything look just the way I want it.

Some notes about Calibre, in case anyone wonders why I did things the way I did.

First, I couldn't find a way to get Calibre to substitute my scene breaks with what I wanted. The HTML file I gave Calibre used simple <hr />, but nothing I did --- and I tried for quite a while --- seemed to be able to trick Calibre into substituting what I wanted, which was <div class="hr">&amp;larr;&amp;nbsp;&amp;rarr;</div>. So I just brute-forced this using a global search-and-replace operation on the HTML before I gave it Calibre to handle the rest.

Second, as I mentioned before, I couldn't get Calibre to use the extensions I wanted to pass to markdown_py, so I just brute forced that too.

After the Markdown conversion, I run ebook-convert for each file format I need. Today I have EPUB and AZW3 formats, but I expect to add PDF soon so I can give a dead trees copy of the book to some folks who don't use eBook readers. Along the way, this dead trees format was very nice for marked up review copies with scribbles everywhere for comments and corrections.

This ZIP file contains the Makefile and CSS I used. I used Calibre version 2.4 and Python Markdown version 2.4 (coincidence?).

Journalistic Brilliance

My eldest daughter, Sarah, is a journalist at the National Journal in Washington, DC. I guess her editors figured she was perfect to write this piece for the Journal on Cathy McMorris Rodgers, one of the most powerful political figures in the country who is also a woman and is from Spokane. After, Sarah grew up there. And they were right.

I'm proud of my brilliant daughter's accomplishments. She put many hours into this piece, including a trip to Spokane, and it has taken her years to get to a position to write it and the pieces of comparable excellence that I'm sure she will be creating in the future.

I hope you find her article, "A Ceiling of Her Making", as interesting as I do. I'm certain there are many such to come, and that I'll have many opportunities to say, proudly, that this deep thinking and talented journalist is my daughter.

Tools for the Writer

I devoted the entirety of my work time yesterday to trying to solve my workflow problems. I failed.

I want a completely reproducible process to go from edits to my Markdown formatted manuscript to eBook formats for Kindle, Google, iBook, Nook, Smashwords, and possibly others. I had been using pandoc to transform the Markdown into the various forms, using a bizarre sequence of complicated command lines in a Makefile. I still want to do this, but each of the processing steps seemed to me to be full of fatal flaws, fatal limitations, or fatal pain-in-the-ass. The key word here is of course "fatal."

As wonderful as Kovid's product is, and usable for many things, I found bugs in the latest (2.3) release of Calibre to be too much for me. Then I tried using Calibre's command line stuff, which worked best with my Makefile scheme anyway. I failed --- mostly because the processing steps in Calibre's tools take any HTML class I specify and replace it either with Calibre's own class name (e.g., "calibre42") which is useless for me to build CSS rules for, or it would replace the class name with the class I specified in the first HTML div I put in my document --- like I wanted all of my divs to be of that class. Even if I specified a class on the div it was overridden by this bug. Of course I could go debug Calibre (I have spent many weeks lately debugging Python code at work trying to get OpenStack to work for me, for example).

Instead, I decided to try something else. I tried using Python Markdown directly, instead of using Calibre to call it. This produces HTML as its output, which has nearly all of what I need. Putting the resulting HTML through Calibre's converter to get EPUB or AZW3 suffered from the same bugs. But I need a way to take the correctly translated HTML from Python Markdown and split it into separate documents at chapter boundaries, to write the TOC and contents.opf files, etc.

So I downloaded and built Sigil, which I gather is the spiritual predecessor of Calibre in some way? It is written in C++ and is, therefore, a very long and painful build process. But Sigil works --- for the things it does --- very well. It just doesn't do all of the things I need, and it's not a command line thing, so I can't use Makefiles to automate the process like I wanted.

I'm about to go write some Node.js code to do this. Javascript is a language I'm familiar with, and I have wanted a good project using it anyway.

But first, I need to manually build my eBook and get this fucking book published. Then I can build tools to replicate that process for revisions and future books. I think I'll open source the result.

Bug icon made by Dave Gandy from Flaticon which is licensed by Creative Commons BY 3.0

Yes, As a Matter of Fact, Kindle Quality DOES Matter


I finished my book. I don't mean it's ready to publish, but I have it done enough that I have made a PDF and corresponding dead trees version for my wife to proofread and an eBook copy for my own reading process for when I'm commuting. I was excited to be on the home stretch, to be honest. I was excited to pick up my little Kindle Fire tablet and start reading my own book this morning on the train at 6:30am.

And read, I did. I read, highlighted, made correction comments, and I loved doing it. It was a good mix of easy to hold for reading and trivial to comment and highlight for noting changes to be made. And there were quite a few changes to make.

This afternoon on the train, about fifteen minutes into my last leg of commuting for the day, simply turning the page made the screen of my Kindle go dark and the damned thing rebooted. When it finished its "oopsie" process, I found two things that I really wish I hadn't. First, my book was not present on the Kindle. Second, all of my roughly two hours of reading, highlighting, and correcting was irretrievably lost. Since I have the cheapo Kindle with no cellular network connection, and since I hadn't bothered to connect it to WiFi at work during the day, there was no crucial "sync."

Goddammit to hell! I am so pissed at whatever or whoever is responsible for my loss of data.

At least now I can go more quickly through the same parts over again --- having some memory of what I did should make it easier to do again. But shit. What a waste of time.

Collective Nouns and Plurals and Data - Oh My!

(Blogging using a phone while on a moving bus can be challenging. The first version of this post was gibberish.)

Who decided that collective nouns should be treated as singular in context by Americans and as plural by the English? To illustrate, an American would say "The army is invading," while a Brit would choose "The army are invading." This is a perfectly ordinary discrepancy in language like leaving out extra vowels in the spelling of "color." I think these sorts of differences came about when communication between Americans and real English speakers came with months of delay because a sea voyage was needed to carry people or letters between the two pools of usage. That's a natural quirk in language as a result of its history.

But I'm here to talk about this because it is also schizophrenic in some cases.

Take the noun "data" for instance. Is it plural? Most people use it as a collective noun, so it's singular in context but it refers to a number of data items. But stuffy old Scientific American, for example, uses "data" as if they were British: they say things like "The data show" rather than "The data shows."

What do UKians do with this word? I think they use it like Americans for some reason!

Language is weird. Naturally.

Neil Gaiman's Eight Rules of Writing

Neil Gaiman]( Neil Gaiman's 8 rules of writing.

Wil Wheaton often surprises me. He's profane, silly, stupid, and often very wise. I find he suffers from Sturgeon's Law, but the ten percent that isn't crap is sublime. Wheaton finds things for me that I wouldn't find any other way. It was Wheaton's blog post that led me to Gaiman's eight writing tips, even if I choose to link to the original post instead of Wil's graphical form.

These "rules" are right on. I find that I have accidentally managed to follow them in my writing over the years I have been learning how by doing. Gaiman, as usual, is bloody well right. If John Scalzi is living my image of The Perfect Life, Neil Gaiman is living the one I would live if I weren't an old, stodgy person - he's married to Amanda Palmer, for Christ's sake! That fascinatingly wild woman is someone I couldn't survive living with, I'm certain; my wonderful wife is a perfect match for me. In fact, as my mother would tell you if she were still living, I was born stodgy and conservative. To corrupt an Indiana Jones quote, apparently it's not the years, honey, it's the genes.

Each of these rules of writing is important in its own right. The one about putting your manuscript away for a while before re-reading it has been vital for me. I found a lot of bugs in the storytelling from doing this, but also my perspective had changed over time. That might have something to do with the amount of time --- I put it away for nearly a decade while I worked for a startup that demanded all of my energy, and later got a divorce. Sometimes life gets in the way of art.

Listening to your pre-readers is hard. Not in the sense of ego being crushed --- I expected them to say things that crushed my baby's soul. But knowing what to do about the issues they found, where they arguably had a point of view more relevant to my book than I did, now that was difficult. I had two people read my book that weren't science fiction fans of any stripe. In fact, they were both English majors. They found a lot in my book to like, which helped me feel better about it. But they also saw things that I could never perceive because I was standing in the middle of the stream that had flowed through my own experience reading science fiction for fifty years.

Gaiman and Scalzi are geniuses in their field. I'm only striving to be above average at this point.

Words and Language and Analyzing the Human Condition

I'm a writer of words --- an artist who constructs pictures in people's minds by lying to them with words and hoping they get it.

I have loved words my whole life. While I was trying to earn an undergraduate degree simultaneously in two different branches of engineering at University while working two nearly full-time jobs to pay for it, I also did a minor in Linguistics out of love for words. Linguistics isn't about a language, it's about languages. It's literally the study of how languages work internally, how they evolve, and their relationships to each other. I loved linguistics because it was all about words and meaning and pronunciation and symbols to represent them. (I have a thing for symbols and alphabets and fonts and typography too.) I think that study of linguistics was the perfect complement to my computer science and electrical engineering --- both almost entirely analytical disciplines. Language is about humans, conversation, emotion, and communication. Linguistics is the intersection between the humanity of language and the analytical science of grammar --- the math or logical structure of how those things work at a nuts-and-bolts level.

One of my early fascinations with language as a child came from the realization that there were actual rules that made languages work, and yet nobody who participated in the creation of the language, building it, mixing in words from other languages, adding to it, morphing it over time --- none of these people actually knew a damned thing about the rules they were following all the while! Natural languages are always structured like the Winchester Mystery House, but there are, in fact, actual consistent grammatical rules underlying all of it. People follow the rules intuitively, by rote. Those that don't understand and don't follow the rules are guilty of "bad grammar" --- which, if it goes on long enough, results in new rules of grammar, rather than in those people learning to do language "right."

One of the hardest things for me, as an engineer and as someone who tries to do things "right" is to accept the fact that the definition of "right" for language is not whether it follows rules properly. Not really. The rules are a side-effect. Language is actually defined by how people use the it --- for real, in actual communication. For example, if you think ending an English sentence with a preposition is wrong, I have news for you. That old "rule" was abolished in conversational language a hundred years ago, and in print several decades ago. It's dead. Get over it. This is a situation up with which I simply cannot put. Or something.

Oh, and did you notice my mention of the crazy university workload I was carrying? As I'm sure you have already guessed, I collapsed under that load. I'm a college dropout who has never looked back. The only time that has ever been even slightly an issue was when I was interviewing to work for a company founded by two dropouts who became billionaires: Apple Computer. I got the job anyway. Go figure.

Are versus Can Be

The difference between what we are in the moment, and what we can be is the source our of striving and inspiration. Those things and, more importantly, those who drive us to be more than we are --- they build our art and our selves from the questions they create.

Some create their questions by simply asking, directly. From others, questions condense into being through their very existence or way of life. And it is our perception of these questions, as artists, our response, that is the seed from which inspired art grows.

My wife and I just finished the highly enjoyable movie "Words and Pictures." We both would recommend it highly to anyone who loves words, pictures, or the human condition. The movie is all about the difference between what the artist is and what the artist can be.

Early retirement of my young friend Mike

A middle aged couple exits a convenience store with LOTTO signs in its windows. The woman says, One of my favorite humans and coworkers, my friend Mike, has just announced he is retiring at age thirty-something. He has worked in high tech since he was 15, never finishing high school or going to college, never marrying or having children, never living in an extravagant fashion, saving his resources all along to achieve this result. He's adamant he isn't moving on to another company, that he is retiring, and I believe him. Mike has the courage to get off the treadmill, to choose to live without need for the trappings and traps and pitfalls of the life most of us live by default.

This unique young man has been an encouragement, an inspiration, and a friend to me during the years I have known him. I will miss chances to share his brilliant technological insights. Hopefully, there will be times in the future when we can share a meal or chat.

Happy retirement, Mike. I'm sorry to see you go, but I know you will go far and wide and well.