Google has published an explanation of the widespread GMail outage from September 1st. In this explanation, they trace the root cause to a layer of “request routers”: …a few of the request routers became overloaded and in effect told the rest of the system “stop sending us traffic, we’re too slow!”. This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded.

I’ve been doing some work with Hadoop lately, and I just ran into an interesting problem with networking. This isn’t a bug, per se, but a conflict in my configuration. I’m running on a laptop, using a pseudo-distributed cluster. That means all the different processes are running, but they’re all running on one box. That makes it possible to test jobs with full network communication, but without deploying to a production cluster.

Spiros Tzavellas pointed me to his implementation of Circuit Breaker. His approach uses AspectJ and can be applied using a bytecode weaver or AspectJ compiler. He's also got unit tests with 85% coverage. Spiros' project page is here, and the code is (where else?) on GitHub. He appears to be quite actively developing the project.

The excellent Will Sargent has created a Circuit Breaker gem that's quite nice. You can read the docs at rdoc.info. He's released the code (under LGPL) on GitHub. The other one has actually been out for a couple of months now, but I forgot to blog about it. Scott Vlamnick created a Grails plugin that uses AOP to weave Circuit Breaker functionality as "around" advice. This one can also report its state via JMX.

We’ve all heard the old saw, “It’s a poor workman that blames his tools.” Let’s think about that for a minute. Does it actual mean that a skilled craftsman can do great work with shoddy implements? Well, can a chef make a souffle with a skillet? Can a cabinetmaker round an edge with dull router bits? I’m not going to rule it out. Perhaps there’s a brilliant chef who—at this very moment—is preparing to introduce the world to the “skiffle.

As you can probably tell from my recent posts, I’ve been learning Scala. I recently dug into another Scala book, Beginning Scala by David Pollak. Beginning Scala is a nice, gentle introduction to this language. It takes a gradual, example driven approach that emphasizes running code early. This makes it a good intro for people who want to use the language for applications first, then worry about creating frameworks later.

Failure to understand or represent units has caused several major disasters, including the costly Ariane 5 disaster in 1996. This is one of those things that DSLs often get right, but mainstream programming languages just ignore. Or, worse, they implement a clunky unit of measure library that ensures you can never again write a sensible arithmetic expression. While I was at JAOO Australia this week, Amanda Laucher showed some F# code for a recipe that caught my attention.

It’s been a while since I blogged anything, mainly because most of my work lately has either been mind-numbing corporate stuff, or so highly contextualized that it wouldn’t be productive to write about. Something came up last week, though, that just blew me away. For various reasons, I’ve engaged Relevance to do a project for me. (Actually, the first results were so good that I’ve now got at least three more projects lined up.

The Australian JAOO conferences are now just one month away. I’ve wanted to get to Australia for at least ten years now, so I am thrilled to finally get there. I’ll be delivering a tutorial on production ready software in both the Brisbane and Sydney conferences. This tutorial was a hit at QCon London, where I first delivered it. The Australian version will be further improved. During the main conference, I’ll be delivering a two-part talk on common failure modes of distributed systems break and how to recover from such breakage.

Backups are the only macroscopic system we commonly deal with that exhibits quantum mechanical effects. This is odd enough that I’ve spent some time getting tangled up in these observations. Until you attempt a restore, a backup set is neither good nor bad, but a superposition of both. This is the superposition principle. The peculiarity of the superposition principle is dramatically illustrated with the experiment of Schrödinger’s backup. This is when you attempt to restore Schrödinger’s pictures of his cat, and discover that the cat is not there.