Wes Morgan

CTO of Democracy Works. Coder, brewer, mountain explorer, dad.

Read this first

RabbitMQ cluster on kubernetes with StatefulSets

UPDATE 3/18/2017: Improved the postStart command based on further testing and refinement.

Since I couldn’t find a blog post showing how to do this, I told myself I should write one up once I figured it out. So here it is.

Kubernetes has a relatively new feature called StatefulSets that is designed to make it easier to run containerized services that are inherently stateful. And RabbitMQ is nothing if not inherently stateful. RabbitMQ clusters double down on the whole statefulness thing. After a few years of experience trying to force RabbitMQ clusters into stateless container infrastructure, I was curious to try embracing the statefulness instead. Paint with the grain and all that.

The first step is you need a kubernetes (k8s) cluster. If you don’t already have one, minikube is a good option to get started on your local dev machine.

Once you have a working k8s cluster and can talk to

Continue reading →


You don’t (always) get to decide when you pay down technical debt

“[I]f you cut corners in the short term, you’re borrowing against the future maintainability of the software and that at some point, not unlike a credit card, you’ll pay down that debt by going back and fixing the hacky solutions that were put in place.” - Ben Lakey in this Programmers StackExchange answer.

The quote above is one of the best explanations for the term “technical debt” that I’ve run across. I like that it’s succinct, and I like it’s credit card analogy. Like credit card debt’s increasing minimum payments, loading up on technical debt will represent an ever-increasing drag on your productivity. This plays out in the “How does this work again? I wish we had docs,” and “Uh oh, we broke feature X when we added feature Y. I wish we had tests,” type of problems that you routinely run into if you are cutting corners.

But even worse than credit card debt (whose collectors will

Continue reading →


Deploying Clojure apps with Docker and Immutant 2

While Docker is a very cool technology that holds a lot of promise for the future of devops, it’s not always the best fit for JVM-based apps. The standard approach of baking every runtime dependency into the image and running each container as a self-contained system gets out of hand rather quickly from a resource utilization perspective. How many JVMs can your server fit into memory at once?

At Democracy Works we wanted to find a way to share JVMs, for this and other reasons1. We use Immutant as our production app server for this. Our first attempt to combine Docker and Immutant 1 worked but it was a bit clunky. It involved building .ima (Immutant archive) files in the CMD’s of the app containers and then doing a –volumes-from for each (now-stopped) app container when we ran the Immutant container. Especially in dev, it was hard to get all the pieces to line up just so. We no longer

Continue reading →


CoreOS, Docker, and batteries

When the CoreOS folks announced Rocket, it was with much concern over the “new direction” of the Docker project, as they saw it. They felt compelled to launch a competitor to Docker because Docker was straying from its roots as being merely a container standard and runtime.

Something rang a bit hollow as I read that original announcement, and it was not the theory that CoreOS was really responding to Docker encroaching on the market they were trying to capture. Others have speculated on this at length, and there probably is something to it. But no, I think the thing that rang hollow to me was CoreOS’ face-value, negative reaction to Docker’s “batteries included, but removable” philosophy. That finally came into focus today as Ubuntu announced Snappy Ubuntu Core (or is it “Ubuntu Snappy” or “Ubuntu Core Snappy”?).

Snappy Ubuntu Core is positioning itself in the same space as CoreOS (and

Continue reading →


Recruiting Software Developers: Language Matters

When I first proposed that we start doing all of our new development at Democracy Works in Clojure instead of Ruby, one of the first concerns brought up by my colleagues was developer recruitment. The assumption–which I agreed with at the time–was that pulling from a much smaller pool of talent would hamper our ability to find people with experience in our language-of-choice (let’s leave aside for now the debate over whether that should be a priority in the first place). It was probably the biggest concern we had with moving to the new language.

Then a funny thing happened1. As soon as we became “a Clojure shop,” smart, experienced developers wanted to work with us. Rather than being yet-another-Rails-shop and having to convince highly-sought-after Ruby developers why they should work here instead of there, we were one of the few places offering developers who had discovered this new

Continue reading →