Back in the Before Times, I went to a Haskell-flavored FP conference, where one of the speakers said something that blew my mind. Sadly, it seems that I didn’t write this up at the time (although I swear I wrote it somewhere… maybe in an internal company memo) and I’ve lost the details of who said it. If by some quirk of odds, it was you, dear reader, please let me know!
Continue Reading »Blogs
-
Constrain the Provider to Liberate Callers -
Rule of Eights The “rule of eights” is a handy way to think about feedback cycle time and the effect it has on human attention spans. This is something I heard–and am probably misremembering to some extent–at an agile conference back in the day. I can’t take credit for this but I also can’t remember who I heard it from. If you know who came up with this, please contact me so I can properly attribute this.
Continue Reading » -
The Bad Idea Game About ten years ago, I was introduced to something called “The Bad Idea Game” by Danvers Fleury. We were doing a company strategy retreat. Fortunately we did not spend it all on wordcrafting mission and values statements, and we actually engaged in some good strategy. The bad idea game was a fun exercise that didn’t seem to produce any directly useful results. At first I thought it had been a waste of a precious hour from our limited supply.
Continue Reading » -
Everything We Build Has a Future Cost Suppose we build a road. If we build it road and walk away, it will decay into a hazard before long. It will be scoured by wind, rain, and sand. Ultraviolet rays from the sun will break down its molecular structure. The shifting earth beneath will crack and buckle it. We must maintain what we build, and that requires expense. Suppose we decide that the road is no longer needed or that it costs more to maintain than it is worth.
Continue Reading » -
Four Meanings of Priorities When trying to communicate, we sometimes use the same word thinking that it means the same thing to everyone. But words are slippery, multivalent things. I can speak a word with one meaning and you might hear it with another. The result is the illusion of communication. As a leader you must be aware that your words can be taken in different ways. In one kind of culture, people might look for the most sinister possible interpretation and assume that’s what you must have meant.
Continue Reading » -
Transactions Aren't Everything When building an application, we tend to select a database technology based on its transactional characteristics. We consider raw performance, API style, consistency model, data model, and deployment architecture. That’s about as much as your service cares about: can it meet the functional and non-functional requirements for the production behavior of the service? Even in a microservice architecture where no other application is allowed to access the service’s database, that database probably has a bunch of other clients.
Continue Reading » -
Counterfactuals are not Causality Suppose we’ve had a recent error with a Kubernetes cluster. As often happens with a problem in our systems, we noticed it first in terms of the visible error, which we could state as “Builds did not complete.” Now we want to trace backwards to figure out what happened. A common technique is the “Five Whys” popularized by Lean thinking. So we ask “Why did builds not complete” and we find “Kubernetes could not start the pod, and the operation timed out after 1 hour.
Continue Reading » -
"Manual" and "Automated" are just words Driving down a shady road, windows down, listening to the frogs and crickets, my family was in the car talking about various stuff and things. This summer evening we happened to talk about the invention and emergence of the word “yeet.” I observed that it was kind of cool to have a word with a known origin and etymology, even if that was only because it was a made-up word. My daughter instantly responded that “all words were made up by someone.
Continue Reading » -
Blocker? Pre-requisite. In discussions about change in a complex system I commonly hear people object, “We can’t do that because X.” (That statement often follows a passive-aggressive prelude such as “That’s all well and good” or “being tactical for a moment.” Depending on your organizational culture you may also hear “That’s great in theory…” Or if your company is more aggressive-aggressive, “Get real!") My advice is to reformulate that statement. Treat the blocker as a missing prerequisite: “In order to do that, X must be true.
Continue Reading » -
Delay Induces Lamination I’ve seen a repeated pattern that plays out in many companies. Delay, or more accurately, the perception of delay induces the creation of “extra” layers in the architecture. The pattern goes like this: A component or subsystem needs to add a capability to serve some end-user need. It will take “too long” to implement that capability in the component. (This is where the perception part really steps in.) Maybe the team is stretched too thinly.
Continue Reading » -
Complexity Collapse There’s a pattern I’ve observed a few times through scientific and computing history. I think of it as “complexity collapse”. It’s probably related to Kuhn’s paradigm shift. The pattern starts with an approach that worked in the past. Gaps in the approach lead to accretions and additions. These restore the approach to functionality, but at the expense of added complexity. That added complexity at first appears preferable to rebuilding the approach from the ground up.
Continue Reading » -
Staggering Skeleton We’ve talked before about a walking skeleton. That is a fully connected, but not very functional, system that includes all the major integrations. It serves to demonstrate that anything at all can run in the expected topology. But some languages and frameworks ask you to get more correct to form a walking skeleton. Strongly typed languages, frameworks that require you to run from a non “-SNAPSHOT” library version, deployment tools that only fetch from official repositories, etc.
Continue Reading » -
Weakness Invites Competition Today, nobody wants to start up a competitor to Amazon. New ecommerce retailers aim at niche markets because Amazon is such a juggernaut and fierce competitor that it would be foolhardy to go against them. Those niche retailers look at Amazon as an exit strategy more than a competitor. Like Microsoft in the 90’s, Amazon isn’t the competition, they’re the environment that any entrant deals with. Competitors emerge when they sense an opportunity to take away market share from a weakened incumbent.
Continue Reading » -
Scaffold or Straightjacket? Scaffold or Straightjacket? Douglas Adams' classic sci-fi comedy novel “The Hitchhiker’s Guide to the Galaxy” opens with a bulldozer approach Arthur Dent’s house. Since Arthur is still inside the house, he is naturally concerned. When Arthur confronts the foreman of the demolition crew, he is informed that his house is to be destroyed to make way for a highway bypass. When discussing the public notice that the local planning office had posted, they have this conversation:
Continue Reading » -
Deleting From Databases is Not Cleanup Creating thousands or millions of entities and then deleting them does not return your database to its initial state. Queries won’t show the deleted entities, but operational results can. For example, a table in an RDBMS may have extra storage segments allocated to it. These can generate higher I/O times until someone runs an analyze job to reset the table stats for the query planner. Some databases treat “DELETE FROM USER” very differently from “TRUNCATE USER”.
Continue Reading » -
Narrow but Deep? In “A Philosophy of Software Design,” (ISBN-13: 978-1732102200) John Ousterhout describes the ideal functional interface as “narrow but deep.” That is, it should not expose many methods or functions, but the ones it does expose should be powerful. I have mixed reactions to this principle, so I’d like to explore some examples that support it and others that argue against it. Throughout this section, my lens is malleability. First, imagine a somewhat typical Java domain object with a “broad but shallow” interface.
Continue Reading » -
Consequences are not Pros or Cons I’ve noticed a pattern in much business writing, including technical writing. People feel compelled to label every effect as “pro” or “con”. I think this springs from our primary-school training in persuasive writing. As a result, what should be an engineering analysis often reads like marketing copy. (A related effect, when writing persuasively, people tend to minimize or discount the effects they don’t like. Richard Feymann advised students to be their own harshest critics, to find ways to poke holes in their own arguments.
Continue Reading » -
Why did we stop at 2? In the dim reaches of Unix history, the first shell was written. It attached file descriptor 0 as a pipe from the TTY device to a process. That became “stdin”. File descriptor 1 is a pipe from the process out to the TTY. That’s “stdout”. I don’t know when FD 2 became “stderr” but it was early. When you write a Unix program, you don’t have to open these file descriptors.
Continue Reading » -
Time Emerges From Events Without an event, no time passes. This may seem like an odd assertion. You may say, “I can see time passing all around me!” But how do you see it? Do you look at the ticking hands of a clock? In a mechanical clock, each tick is an event: when the tension on an escapement exceeds the friction between its prong and the gear, and the escapement knocks over to the other side with the familiar “tick.
Continue Reading » -
Reading List Architecture & Development Required Reading Architecture Decision Records C4 Model (Note: we will only use the first 3 C's.) Accelerate Wardley Maps Failure Modes and Continuous Resilience Recommended Reading The Principles of Product Development Flow Software Architecture in Practice Domain-Driven Design Data and Reality, 2ed (Note, the 3rd edition is not as good. Best to stick with 2nd edition.) The Phoenix Project - A novel about IT transformation with a devops flavor.
Continue Reading » -
Shared Mutable Team State Shared State When programming distributed systems, the hardest kind of data to manage is shared mutable state. It requires some kind of synchronization between writers to avoid missed updates. And, after changes, it requires some kind of mechanism to restore coherence between readers. I previously wrote about that idea of a coherence penalty as it applies to humans. Following those lines, we might regard the system of development teams in an organization as its own distributed system.
Continue Reading » -
My Favorite Bit of Language Design An elegant design conserves mechanisms. It combines a small number of primitives in various ways. When I first learned about this elegant bit of design in Smalltalk-80, I laughed with delight. In Smalltalk, the primitives are “object” and “message”. That’s basically it – except for blocks, which we will see a little later. Behavior arises via objects sending messages to each other. In fact, Smalltalk doesn’t even need control structures in the language grammar.
Continue Reading » -
Networking Topics Another quick post based on a Twitter exchange. (Maybe this will help save content from the ephemera of Tweets.) A short, incomplete list topics in networking that programmers should know about: ICMP messages Frame size and fragmentation Socket options Listen queue and behavior when full. All the timeouts and why they exist. When read, write, and connect calls block and why. When memory buffers are copied and how to avoid.
Continue Reading » -
Joyful Isolation Way back in January, Sam Newman tweeted this (perhaps rhetorical) question: I was in the middle of creating this slide (wrt patch hygiene) and had to stop half-way through and ask myself - aren’t we all just making this worse? pic.twitter.com/fCTAYDc3Pn — Sam Newman (@samnewman) January 14, 2018 It got a handful of retweets recently, and I responded with: I've said it before, but each of these layers is another attempt to achieve isolation between apps.
Continue Reading » -
Evolving Away From Entities Hat tip to Stuart Halloway… once again a 10 minute conversation with Stu grew into a combination of code and writing that helped me clarify my thoughts. I've been working on new content for my Monolith to Microservices workshop. As the name implies, we start with a monolith. Everyone gets their own fully operational production environment, running a fork of the code for Lobsters. It's a link sharing site with a small but active group of users.
Continue Reading » -
Data is the New Oil The other day I tweeted that "Data is the New Oil." A lot of people retweeted, but a quite a few asked what I meant by that. I'll amplify a bit to explain the analogy. This ended up being a lot to unpack from a quick tweet! For quite a few years now, I've used Twitter as a way to scratch the itch of personal expression. A quick sound bite there, highly compressed and idiosyncratic was just enough to relieve the mental pressure.
Continue Reading » -
Coherence Penalty for Humans This is a brief aside from my ongoing series about avoiding entity services. An interesting dinner conversation led to thoughts that I needed to write down. Amdahl's Law In 1967, Gene Amdahl presented a case against multiprocessing computers. He argued that the maximum speed increase for a task would be limited because only a portion of the task could be split up and parallelized. This portion, the "
Continue Reading » -
Services By Lifecycle This post took a lot longer to pull together than I expected. Not because it was hard to write, but because it was too easy to write too much. Like a pre-bonsai tree, it would grow out of control and get pruned back over and over. In the meantime, I delivered a workshop and spent some lovely holiday time with my family. But it’s a new year now, and January is devoid of holidays so it’s high time I got back to business.
Continue Reading » -
The Entity Service Antipattern In my last post I talked about the need to keep things separated once they've been decoupled. Let's look at one of the ways this breaks down: entity services. If a pattern is a solution to a problem in a context, what is an antipattern? An antipattern is a commonly-rediscovered solution to a problem in a context, that inadvertently creates a resulting context we like less than the original context.
Continue Reading » -
Keep 'Em Separated Software doesn't have any natural boundaries. There are no rivers, mountains, or deserts to separate different pieces of software. If two services interact, then they have a sort of "attractive force" that makes them grow towards each other. The interface between them becomes more specific. Semantic coupling sneaks in. At a certain point, they might as well be one module running in one process. If you're building microservices, you need to make sure they don't grow together into an impenetrable bramble.
Continue Reading » -
Root Cause Analysis as Storytelling Humans are great storytellers and even better story-listeners. We love to hear stories so much that when there aren't any available, we make them up on our own. From an early age, children grasp the idea of narrative. Even if they don't understand the forms of storytelling so much, you can hear a four-year-old weave a linked list of events from her day. We look for stories behind everything.
Continue Reading » -
Release It Second Edition in Beta I’m excited to announce the beta of Release It! Second edition. It’s been ten years since the first edition was released. Many of the lessons in that book hold strong. Some are even more relevant today than in 2007. But a few things have changed. For one thing, capacity management is much less of an issue today. The rise of the cloud means that developers are more exposed to networks than ever.
Continue Reading » -
Spectrum of Change I’ve come to believe that every system implicitly defines a spectrum of changes, ordered by their likelihood. As designers and developers, we make decisions about what to embody as architecture, code, and data based on known requirements and our experience and intuition. We pick some kinds of changes and say they are so likely that we should represent the current choice as data in the system. For instance, who are the users?
Continue Reading » -
Queuing for QA Queues are the enemy of high-velocity flow. When we see them in our software, we know they will be a performance limiter. We should look at them in our processes the same way. I've seen meeting rooms full of development managers with a chart of the year, trying to allocate which week each dev project will enter the QA environment. Any project that gets done too early just has to wait its turn in QA.
Continue Reading » -
Availability and Stability Last post covered technical definitions of fault, error, and failure. In this post we will apply these definitions in a system. Our context is a long-running service or server. It handles requests from many different consumers. Consumers may be human users, as in the case of a web site, or they may be other programs. Engineering literature has many definitions of "availability." For our purpose we will use observed availability.
Continue Reading » -
Fault, Error, Failure Our systems suffer many insults when they contact the real world. Flaky inputs, unreliable networks, and misbehaving users, to name just a few. As we design our components and systems to thrive in the only environment that matters, it pays to have mental schema and language to discuss the issues. A fault is an incorrect internal state in your software. Faults are often introduced at component, module, or subsystem boundaries.
Continue Reading » -
Power Systems This is an excerpt from something I'm working on this Labor Day holiday: – Large scale power outages act a lot like software failures. It starts with a small event, like a power line grounding out on a tree. Ordinarily that would be no big deal but under high-stress conditions it can turn into a cascading failure that affects millions of people. We can also learn from how power gets restored after an outage.
Continue Reading » -
Remember DAT? Do you remember Digital Audio Tape? DAT was supposed to have all the advantages of digital audio—high fidelity and perfect reproduction—plus the "advantages" of tape. (Presumably those advantages did not include melting on the dashboard of your Chevy Chevelle or spontaneously turning into The Best of Queen after a fortnight.) In hindsight, we can see that DAT was a twilight product. As the sun set on the cassette era, DAT was an attempt to bridge the discontinuous technology change to digital music production.
Continue Reading » -
QA Instability Implies Production Instability Many companies that have trouble delivering software on time exhibit a common pathology. Developers working on the next release are frequently interrupted for production support issues with the current release. These interrupts never appear in project schedules but can take up half of the developers' hours. When you include the cost of task-switching, this means less than half of their available time is spent on the new feature work.
Continue Reading » -
Wittgenstein and Design What does a philosopher born in the 19th Century have to say about software design? More than you might think, particularly his ideas about family resemblance. Wittgenstein used the subject of "games" to illustrate an idea. We'll start with a counter-example. Suppose we operate with the then-prevailing notion that words are defined like sets in axiomatic set theory. Then there is a decision procedure that will let us decide whether something is a member of the set "
Continue Reading » -
In Love With Your Warts If you help other people solve problems, you may have run into this phenomenon: a person gleefully tells you how messed up their environment is. Processes make no sense, roadblocks are everywhere, and all previous attempts to make things better have failed. As you explore the details, you talk about directions to try. But every suggestion is met with an explanation of why it won't work. I say that these folks are "
Continue Reading » -
Some Useful Techniques From Bygone Eras CRC I find the old object-oriented design technique of CRC Cards to be useful when defining service designs. CRC is short for "Class, Responsibilities, Collaborators." It's a way to define what behavior belongs inside a service and what behavior it should delegate to other services. Simulating a system via CRC is a good exercise for a group. Each person takes a CRC card and plays the role of that service.
Continue Reading » -
Generalized Minimalism My daily language is Clojure. One of the joys of working in Clojure is its great core library. The core library has a wealth of functions that apply broadly across data structures. A typical function looks like this: (defn nthnext "Returns the nth next of coll, (seq coll) when n is 0." {:added "1.0" :static true} [coll n] (loop [n n xs (seq coll)] (if (and xs (pos? n)) (recur (dec n) (next xs)) xs))) I want to call your attention to two specific forms.
Continue Reading » -
Redeeming the Original Sin While reading Bryan Cantrill's slides from Papers We Love NYC, I was struck by something. One of the very first slides says: The traditional UNIX security model is simple but inexpressive. The papers go on to describe a progression of techniques to isolate processes from the host environment to greater and greater degrees. It began with the ancient precursor 'chroot', through Jails, and Zones. Each builds upon the previous work to improve the degree of isolation.
Continue Reading » -
What's Lost With a DevOps Team Please understand, dear Reader, that I write this with positive intention. I'm not here to impugn any person or organization. I want to talk about some decisions and their natural consequences. These consequences seem negative to me and after reading this post you may agree. When an established company faced a technology innovation, they often create a new team to adopt and exploit that innovation. During my career, I've seen this pattern play out with microcomputers, client/server architecture, open systems, web development, agile development, cloud architecture, NoSQL, and DevOps.
Continue Reading » -
Give Them The Button! Here's a syllogism for you: Every technical review process is a queue Queues are evil Therefore, every review process is evil Nobody likes a review process. Teams who have to go through the review look for any way to dodge it. The reviewers inevitably delegate the task downward and downward. The only reason we ever create a review process is because we think someone else is going to feed us a bunch of garbage.
Continue Reading » -
C9D9 on Architecture for Continuous Delivery Every single person I've heard talk about Continuous Delivery says you have to change your system's architecture to succeed with it. Despite that, we keep seeing "lift and shift" efforts. So I was happy to be invited to join a panel to discuss architecture for Continuous Delivery. We had an online discussion last Tuesday on the C9D9 series, hosted by Electric Cloud. They made the recording available immediately after the panel, along with a shiny new embed code.
Continue Reading » -
Software Eats the World During this morning's drive, I crossed several small overpasses. It reminded me that the American Society of Civil Engineers rated more than 20% of our bridges as structurally deficient or functionally obsolete. That got me to thinking about how we even know how many bridges there are in a country as large as the U.S. Some time in the past, it would require an army of people to go survey all the roads, looking for bridges and adding them to a ledger.
Continue Reading » -
Microservices versus Lean Back in April, I had the good fortune to speak at Craft Conf in lovely Budapest. It's a fantastic conference that I would recommend. During that conference, Randy Shoup talked about his experience migrating from monoliths to microservices at EBay and Google. David, one of the audience members asked an interesting question at the end of Randy's talk. (I'm sorry that I didn't get the full name of the questioner… if you are reading this, please leave a comment to let me know who you are.
Continue Reading » -
Development is Production When I was at Totality, we treated an outage in our customers' content management system as a Sev 2 issue. It ranked right behind "Revenue Stopped" in priority. Content management is critical to the merchants, copy writers, and editors. Without it, they cannot do their jobs. For some reason, we always treated dev environment or QA environment issues as a Sev 3 or 4, with the "when I get around to it"
Continue Reading » -
The Fear Cycle Once you begin to fear your technology, you will shortly have cause to fear it even more. The Fear Cycle goes like this: Small changes have unpredictable, scary, or costly results. We begin to fear making changes. We try to make every change as small and local as possible. The code base accumulates warts, knobs, and special cases. Fear intensifies. Fear starts when an innocuous change goes badly.
Continue Reading » -
Components and Glue There's a well-known architectural style in desktop applications called "Components and Glue". The central idea is that independent components are composed together by a scripting layer. The glue is often implemented in a different or more dynamic language than the components. The C2 wiki's page on ComponentGlue has been stable since 2004, so obviously this is not a new idea. Emacs is one example of this approach.
Continue Reading » -
Faceted Identities I have a rich and multidimensional relationship with Amazon. It started back in 1996 or 1997, when it became the main supplier for my book addiction. As the years went by, I became an "Amazon Affiliate" in a futile attempt to balance out my cash flow with the company. Later, I started using AWS for cloud computing. I also claimed my author page. Let's contemplate the data architecture needed to maintain such a set of relationships.
Continue Reading » -
Inverted Ownership, Part 2 My last post on the subject of inverted ownership felt a bit abstract, so I thought I might illustrate it with a typical scenario. In this first figure, we see a newly-extracted Catalog service, freshly factored out of the old monolithic application. It's part of the company's effort to become more maneuverable. We don't know, or particularly care, what storage model it uses internally. From the outside, it presents an interface that looks like "
Continue Reading » -
Inverted Ownership One of the sources of semantic coupling has to do with identifiers, and especially with synthetic identifiers. Most identifiers are just alphanumeric strings. Systems share those identifiers to ensure that both sides of an interface agree on the entity or entities they are manipulating. In the move to services, there is an unfortunate tendency to build in a dependency on an ambient space of identifiers. This limits your organization's maneuverability.
Continue Reading » -
The Perils of Semantic Coupling On the subject of maneuverability, many organizations run into trouble when they try to enter new lines of business, create a partnership, or merge with another company. Updating enterprise systems becomes a large cost factor in these business initiatives, sometimes large enough to outweigh the benefits case. This is a terrible irony: our automation provides efficiency, but removes flexibility. If you break down the cost of such changes, you'll find it comes in equal parts from changes to individual systesm and changes to integrations across systems.
Continue Reading » -
Maneuverability Agile development works best at team scale. When a team can self-organize, refine their methods, build the toolchain, and modify adapt it to their needs, they will execute effectively. We should be happy to achieve that! I worry when we try to force-fit the same techniques at larger scales. At the scale of a whole organization, we need to look at the qualities we want to have. (We can't necessarily produce those qualities directly, but we can create the conditions that allow them to emerge.
Continue Reading » -
Bad Layering If I had to guess, I would say that "Layers" is probably the most commonly applied architecture pattern. And why not? Parfaits have layers, and who doesn't like a parfait? So layers must be good. Like everything else, though, there's a good way and a bad way. The usual Neapolitan stack looks like this: On one of my favorite projects of all, we used more layers because we wanted to further isolate different behaviors.
Continue Reading » -
People Don't Belong to Organizations One company that gets this right is Github. I exist as my own person there. I'm affiliated with my employer as well as other organizations. We are long past the days of "the company man," when a person's identity was solely bound to their employer. That relationship is much more fluid now. A company that gets it wrong is Atlassian. I've left behind a trail of accounts in various Jirae and Confluences.
Continue Reading » -
Glue Fleet and Compojure Together Using Protocols Inspired by Glenn Vanderburg's article on Clojure templating frameworks, I decided to try using Fleet for my latest pet project. Fleet has a very nice interface. I can call a single function to create new Clojure functions for every template in a directory. That really makes the templates feel like part of the language. Unfortunately, Glenn's otherwise excellent article didn't talk about how to connect Fleet into Compojure or Ring. I chose to interpret that as a compliment, springing from his high esteem of our abilities.
Continue Reading » -
Metaphoric Problems in REST Systems I used to think that metaphor was just a literary technique, that it was something you could use to dress up some piece of creative writing. Reading George Lakoff’s Metaphors We Live By, though has changed my mind about that. I now see that metaphor is not just something we use in writing; it’s actually a powerful technique for structuring thought. We use metaphor when we are creating designs. We say that a class is like a factory, that an object is a kind of a thing.
Continue Reading » -
Metaphoric Problems in REST Systems (audio) Metaphoric problems in rest systems by mtnygard
-
Time motivates architecture Let’s engage in a thought experiment for a moment. Suppose that software was trivial to create and only ever needed to be used once. Completely disposable. So, somebody comes to you and says, “I have a problem and I need you to solve it. I need a tool that will do blah-de-blah for a little while.” You could think of the software the way that a carpenter thinks of a jig for cutting a piece of wood on a table saw, or a metalworker thinks of creating a jig to drill a hole at the right angle and depth.
Continue Reading » -
Circuit Breaker in Scala FaKod (I think that translates as "The Fatalistic Coder"?) has written a nice Scala implementation of the Circuit Breaker pattern, and even better, has made it available on GitHub. Check out http://github.com/FaKod/Circuit-Breaker-for-Scala for the code. The Circuit Breaker can be mixed in to any type. See http://wiki.github.com/FaKod/Circuit-Breaker-for-Scala/ for an example of usage.
-
The Future of Software Development I’ve been asked to sit on a panel regarding the future of software development. This is always risky and makes me nervous, for two reasons. First, prediction is a notoriously low success-rate activity. Second, the people you always see making predictions like this are usually well past their “use by” date. Nevertheless, here are a collection of barely-related thoughts I have on that subject. Two obvious trends are cloud computing and mobile access.
Continue Reading » -
Failover: Messy Realities People who don't live in operations can carry some funny misconceptions in their heads. Some of my personal faves: Just add some servers! I want a report of every configuration setting that's different between production and QA! We're going to make sure this (outage) never happens again! I've recently been reminded of this during some discussions about disaster recovery. This topic seems to breed misconceptions.
Continue Reading » -
Life's Little Frustrations A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable. -Leslie Lamport On my way to QCon Tokyo and QCon China, I had some time to kill so I headed over to Delta's Skyclub lounge. I've been a member for a few years now. And why not? I mean, who could pass up tepid coffee, stale party snacks, and a TV permanently locked to CNN?
Continue Reading » -
Topics in Architecture I’m working on a syllabus for an extensive course on web architecture. This will be for experienced programmers looking to become architects. Like all of my work about architecture, this covers technology, business, and strategic aspects, so there’s an emphasis on creating high-velocity, competitive organizations. In general, I’m aiming for a mark that’s just behind the bleeding edge. So, I’m including several of the NoSQL persistence technologies, for example, but not including Erjang because it’s too early.
Continue Reading » -
"If the last one goes, we'll be up here all night!" There’s an old joke about a couple of folks on a plane who hear the captain successively announce that they’ve lost one, two, then three engines. Each time, he reassures the passengers that they’re OK, but will be progressively later to land. After the losing the third engine, one passenger tells the other, “If the last one goes, we’ll be up here all night!” It’s a remarkable aircraft that can fly on just one out of four engines.
Continue Reading » -
Coupling and Coevolution The mighty Mississippi River starts in Minnesota, at Lake Itasca. Every kid in Minnesota has to make the ritual pilgrimage to Itasca State Park at some point, where wading across North America’s longest river is a rite of passage. One of the very interesting things in Itasca State Park is a section of forest that is fenced off so that deer cannot enter it. It’s part of a decades-long experiment to see how forests are affected by browsing herbivores.
Continue Reading » -
GMail Outage Was a Chain Reaction Google has published an explanation of the widespread GMail outage from September 1st. In this explanation, they trace the root cause to a layer of “request routers”: …a few of the request routers became overloaded and in effect told the rest of the system “stop sending us traffic, we’re too slow!”. This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded.
Continue Reading » -
Hadoop versus VPN I’ve been doing some work with Hadoop lately, and I just ran into an interesting problem with networking. This isn’t a bug, per se, but a conflict in my configuration. I’m running on a laptop, using a pseudo-distributed cluster. That means all the different processes are running, but they’re all running on one box. That makes it possible to test jobs with full network communication, but without deploying to a production cluster.
Continue Reading » -
An AspectJ Circuit Breaker Spiros Tzavellas pointed me to his implementation of Circuit Breaker. His approach uses AspectJ and can be applied using a bytecode weaver or AspectJ compiler. He's also got unit tests with 85% coverage. Spiros' project page is here, and the code is (where else?) on GitHub. He appears to be quite actively developing the project.
-
Two New Circuit Breaker Implementations The excellent Will Sargent has created a Circuit Breaker gem that's quite nice. You can read the docs at rdoc.info. He's released the code (under LGPL) on GitHub. The other one has actually been out for a couple of months now, but I forgot to blog about it. Scott Vlamnick created a Grails plugin that uses AOP to weave Circuit Breaker functionality as "around" advice. This one can also report its state via JMX.
Continue Reading » -
Workmen, tools, etc. We’ve all heard the old saw, “It’s a poor workman that blames his tools.” Let’s think about that for a minute. Does it actual mean that a skilled craftsman can do great work with shoddy implements? Well, can a chef make a souffle with a skillet? Can a cabinetmaker round an edge with dull router bits? I’m not going to rule it out. Perhaps there’s a brilliant chef who—at this very moment—is preparing to introduce the world to the “skiffle.
Continue Reading » -
Minireview: Beginning Scala As you can probably tell from my recent posts, I’ve been learning Scala. I recently dug into another Scala book, Beginning Scala by David Pollak. Beginning Scala is a nice, gentle introduction to this language. It takes a gradual, example driven approach that emphasizes running code early. This makes it a good intro for people who want to use the language for applications first, then worry about creating frameworks later.
Continue Reading » -
Units of Measure in Scala Failure to understand or represent units has caused several major disasters, including the costly Ariane 5 disaster in 1996. This is one of those things that DSLs often get right, but mainstream programming languages just ignore. Or, worse, they implement a clunky unit of measure library that ensures you can never again write a sensible arithmetic expression. While I was at JAOO Australia this week, Amanda Laucher showed some F# code for a recipe that caught my attention.
Continue Reading » -
Kudos to Relevance and Clojure It’s been a while since I blogged anything, mainly because most of my work lately has either been mind-numbing corporate stuff, or so highly contextualized that it wouldn’t be productive to write about. Something came up last week, though, that just blew me away. For various reasons, I’ve engaged Relevance to do a project for me. (Actually, the first results were so good that I’ve now got at least three more projects lined up.
Continue Reading » -
JAOO Australia in 1 Month The Australian JAOO conferences are now just one month away. I’ve wanted to get to Australia for at least ten years now, so I am thrilled to finally get there. I’ll be delivering a tutorial on production ready software in both the Brisbane and Sydney conferences. This tutorial was a hit at QCon London, where I first delivered it. The Australian version will be further improved. During the main conference, I’ll be delivering a two-part talk on common failure modes of distributed systems break and how to recover from such breakage.
Continue Reading » -
Quantum Backups Backups are the only macroscopic system we commonly deal with that exhibits quantum mechanical effects. This is odd enough that I’ve spent some time getting tangled up in these observations. Until you attempt a restore, a backup set is neither good nor bad, but a superposition of both. This is the superposition principle. The peculiarity of the superposition principle is dramatically illustrated with the experiment of Schrödinger’s backup. This is when you attempt to restore Schrödinger’s pictures of his cat, and discover that the cat is not there.
Continue Reading » -
Update: Sun Cloud API Not the Same as Amazon It looks like the early reports that Sun’s cloud API would be compatible with AWS resulted from the reporters' exuberance (or mere confusion.) It’s actually nicer than Amazon’s. It is based on the REST architectural style, with representations in JSON. In fact, I might start using it as the best embodiment of REST principles. You start with an HTTP GET of “/”. In this repsonse to this and every other request, it is the hyperlinks in the response that indicate what actions are allowed.
Continue Reading » -
Can you make that meeting? I’m convinced that the next great productivity revolution will be de-matrixing the organizations we’ve just spent ten years slicing and dicing. Yesterday, I ran into a case in point: What are the odds that three people can schedule a meeting this week versus having to push it into next week? Turns out that if they’re each 75% utilized, then there’s only a 15% chance they can schedule a one hour meeting this week.
Continue Reading » -
Amazon as the new Intel Update: Please read this update. The information underlying this post was based on early, somewhat garbled, reports. A brief digression from the unpleasantness of reliability. This morning, Sun announced their re-entry into the cloud computing market. After withdrawing Network.com from the marketplace a few months ago, we were all wondering what Sun’s approach would be. No hardware vendor can afford to ignore the cloud computing trend… it’s going to change how customers view their own data centers and hardware purchases.
Continue Reading » -
Getting Real About Reliability In my last post, I user some back-of-the-envelope reliability calculations, with just one interesting bit, to estimate the availability of a single-stacked web application, shown again here. I cautioned that there were a lot of unfounded assumptions baked in. Now it's time to start removing those assumptions, though I reserve the right to introduce a few new ones. Is it there when I want it? First, lets talk about the hardware itself.
Continue Reading » -
Reliability Math Suppose you build a web site out of a single stack of one web, app, and database server. What sort of availability SLA should you be willing to support for this site? We'll approach this in a few steps. For the first cut, you'd say that the appropriate SLA is just the expected availability of the site. Availability is defined in different ways depending on when and how you expect to measure it, but for the time being, we'll say that availability is the probability of getting an HTTP response when you submit a request.
Continue Reading » -
2009 Calendar as OmniGraffle Stencil I had need of a stencil that would let me drop monthly calendars on a number of pages. I found it useful, and someone else might, too. Download the stencil.
-
Fast Iteration versus Elegant Design I love the way that proggit bubbles stuff around. Today, for a while at least, the top link is to a story from Salon in May of 2000 about Bill and Lynne Jolitz, the creators of 386BSD. [An aside: I'm not sure exactly when I became enough of a graybeard to remember as current events things which are now discussed as history. It's really disturbing that an article from almost a decade ago talks about events seven years earlier than that, and I remember them happening!
Continue Reading » -
Why Do Enterprise Applications Suck? What is it about enterprise applications that makes them suck? I mean, have you ever seen someone write 1,500 words about how much they love their corporate expense reporting system? Or spend their free time mashing up the job posting system together with Google maps? Of course not. But why not? There’s a quality about some software that inspires love in their users, and it’s totally devoid in enterprise software. The best you can ever say about enterprise software is when it doesn’t get in the way of the business.
Continue Reading » -
Tracking and Trouble Pick something in your world and start measuring it. Your measurements will surely change a little from day to day. Track those changes over a few months, and you might have a chart something like this. Now that you've got some data assembled, you can start analyzing it. The average over this sample is 59.5. It's got a variance of 17, which is about 28% of the mean. You can look for trends.
Continue Reading » -
Booklist I made a LibraryThing list of books relevant to the stuff that’s banging around in my head now. These are in no particular order or organization. In fact, this is a live widget, so it might change as I think of other things that should be on the list. The key themes here are time, complexity, uncertainty, and constraints. If you’ve got recommendations along these lines, please send them my way.
Continue Reading » -
Cold Turkey -
Subtle Interactions, Non-local Problems Alex Miller has a really interesting blog post up today. In LBQ + GC = slow, he shows how LinkedBlockingQueue can leave a chain of references from tenured dead objects to live young objects. That sounds really dirty, but it actually means something to Java programmers. Something bad. The effect here is a subtle interaction between the code and the mostly hidden, yet omnipresent, garbage collector. This interaction just happens to hit a known sore spot for the generational garbage collector.
Continue Reading » -
Combining here docs and blocks in Ruby Like a geocache, this is another post meant to help somebody who stumbles across it in a future Google search. (Or as an external reminder for me, when I forget how I did this six months from now.) I've liked here-documents since the days of shell programming. Ruby has good support for here docs with variable interpolation. For example, if I want to construct a SQL query, I can do this:
Continue Reading » -
Beautiful Architecture O'Reilly has released "Beautiful Architecture," a compilation of essays by software and system architects. I'm happy to announce that I have a chapter in this book. The finished book is shipping now, and available through Safari. I think the whole thing has turned out amazingly well, both instructive and interesting. One of the editors, Diomidas Spinellis, has posted an excellent description and summary.
-
Another Cause of TNS-12541 -
Using a custom WindowProc from Ruby This is off the beaten path today, maybe even off the whole reservation. Still, I searched for some code to do this, and couldn't find it. Maybe this will help somebody else trying to do the same thing. I'm currently prototyping a desktop utility using Ruby and wxRuby. The combination actually makes Windows desktop programming palatable, which is a very pleasant surprise. Part of what I'm doing involves showing messages with Snarl.
Continue Reading » -
OTUG Tonight -
Attack of Self-Denial, 2008 Style -
(Human | Pattern) Languages, part 2 At the conclusion of the modulating bridge, we expect to be in the contrasting key of C minor. Instead, the bridge concludes in the distantly related key of F sharp major... Instead of resolving to the tonic, the cadence concludes with two isolated E pitches. They are completely ambiguous. They could belong to E minor, the tonic for this movement. They could be part of E major, which we've just heard peeking out from behind the minor mode curtains.
Continue Reading » -
(Human | Pattern) Languages We missed the point when we adopted "patterns" in the software world. Instead of an organic whole, we got a bag of tricks. The commonly accepted definition of a pattern is "a solution to a problem in a context." This is true, but limiting. This definition loses an essential characteristic of patterns: Patterns relate to other patterns. We talk about the context of a problem. "Context" is a mental shorthand. If we unpack the context it means many things: constraints, capabilities, style, requirements, and so on.
Continue Reading » -
Connection Pools and Engset In my last post, I talked about using Erlang models to size the front end of a system. By using some fundamental capacity models that are almost a century old, you can estimate the number of request handling threads you need for a given traffic load and request duration. Inside the Box It gets tricky, though, when you start to consider what happens inside the server itself. Processing the request usually involves some kind of database interaction with a connection pool.
Continue Reading » -
Thread Pools and Erlang Models Sizing, Danish Style Folks in telecommunications and operations research have used Erlang models for almost a century. A. K. Erlang, a Danish telephone engineer, developed these models to help plan the capacity of the phone network and predict the grade of service that could be guaranteed, given some basic metrics about call volume and duration. Telephone networks are expensive to deploy, particularly when upgrading your trunk lines involves digging up large portions of rocky Danish ground or running cables under the North Sea.
Continue Reading » -
Constraint, Chaos, Collapse -
Licensing for Windows on EC2 One thing I noticed when I fired up my first Windows instances on EC2 was that Windows never asked me for a license key. From examining the registry, it appears that a valid license key is installed at boot time. On two instances of image ami-b53cd8dc (ec2-public-windows-images/Server2003r2-i386-anon-v1.01 for i386) I got exactly the same key. Likewise, on two different instances of ami-7b2bcf12 (ec2-public-windows-images/Server2003r2-x86_64-anon-v1.00 or x64), I got the same license key--though not the same key as the i386 image.
Continue Reading » -
Windows on EC2, from a Mac It may be a bit perverse, but I wanted to hit a Windows EC2 instance from my Mac. After a little hitch getting started, I got it to work. There are a few quirks about accessing Windows instances, though. First off, SSH is not enabled by default. You'll need to use remote desktop to access your instance. Remote desktop uses port 3389, so the first step is to create a new security group for Windows desktop access
Continue Reading » -
Don't Break My Heart, EC2! I'm a huge booster of AWS and EC2. I have two talks about cloud computing, and one that's pretty specific to AWS, on the No Fluff, Just Stuff traveling symposium. With today's announcement about EC2 coming out of beta, and about Windows support, I wanted to try out a Windows server on EC2. Heartbreak! ec2-describe-images -a | grep windows IMAGE ami-782bcf11 ec2-public-windows-images/Server2003r2-i386-anon-v1.00.manifest.xml amazon available public i386 machine IMAGE ami-792bcf10 ec2-public-windows-images/Server2003r2-i386-EntAuth-v1.00.manifest.xml amazon available public i386 machine IMAGE ami-7b2bcf12 ec2-public-windows-images/Server2003r2-x86_64-anon-v1.
Continue Reading » -
Perfection is Not Always Required In my series on dirty data, I made the argument that sometimes incomplete, inaccurate, or inconsistent data was OK. In fact, not only is it OK, but it can be an advantage. There's a really slick Ruby library called WhatLanguage that illustrates this beautifully. The author also wrote a nice article introducing the library. WhatLanguage automatically determines the language that a piece of text is written in. For example (from the article)
Continue Reading » -
Arrival at JAOO Considering that it's 7:30 AM local time---where "local" means Aarhus, Denmark---and I'm awake and online, it looks like I've successfully reset my internal clock. Of course, my approach consisted of staying awake for 28 hours continuously then having three excellent beers with dinner. There are probably easier ways, and there may be repercussions later. I've always heard good things about JAOO, so it was an honor and a delight to be invited.
Continue Reading » -
The Infamous Seinfeld-Gates Ad -
In Korean "Release It" has now been translated into Korean. I just received three copies of a work that's hauntingly familiar, but totally opaque to me. I kind of wonder how the pop-culture jokes came through. I bet C3PO and R2D2 made it OK, but I wonder whether "dodge, duck, dip, dive, and dodge" made it past the Korean copy editor. (For that matter, I'm faintly surprised it made it past the English copy editor.
Continue Reading » -
ReadWriteWeb on Dirty Data A short while back, I did a brief series on the value of "dirty data"---copious amounts of unstructured, non-relational data created by the many interactions user have with your site and each other. ReadWriteWeb has a post up about Four Ad-Free Ways that Mined Data Can Make Money, along very similar lines. Well worth a read.
-
97 Things Every Software Architect Should Know O'Reilly is creating a new line of "community-authored" books. One of them is called "97 Thing Every Software Architect Should Know". All of the "97 Things" books will be created by wiki, with the best entries being selected from all the wiki contributions. I've contributed several axioms that have been selected for the book: Talk about the arch, but see the scaffolding beneath itYou're negotiating more often than you thinkSoftware architecture has ethical consequencesEverything will ultimately fail Engineer in the white spaces Long-time readers of this blog may recognize some of these themes.
Continue Reading » -
How Buildings Learn Stewart Brand's famous book How Buildings Learnhas been on my reading queue for a while, possibly a few years. Now that I've begun reading it, I wish I had gotten it sooner. Listen to this: The finished-looking model and visually obsessive renderings dominate the let's-do-it meeting, so that shallow guesses are frozen as deep decisions. All the design intelligence gets forced to the earliest part of the building process, when everyone knows the least about what is really needed.
Continue Reading » -
Dan Pritchett on Availability Dan Pritchett is a man after my own heart. His latest post talks about the path to availability enlightenment. The obvious path--reliable components and vendor-supported commercial software--leads only to tears. You can begin on the path to enlightenment when you set aside dreams of perfect software running on perfect hardware, talking over perfect networks. Instead, embrace the reality of fallible components. Don't design around them, design for them. How do you design for failure-prone components?
Continue Reading » -
Agile Tool Vendors There seems to be something inherently contradictory about "Enterprise" agile tool vendors. There's never been a tool invented that's as flexible in use or process as the 3x5 card. No matter what, any tool must embed some notion of a process, or at least a meta-process. I've looked at several of the "agile lifecycle management" and "agile project management" tools this week. To me, they all look exactly like regular project management tools.
Continue Reading » -
Beyond the Village As an organization scales up, it must navigate several transitions. If it fails to make these transitions well, it will stall out or disappear. One of them happens when the company grows larger than "village-sized". In a village of about 150 people or less, it's possible for you to know everyone else. Larger than that, and you need some kind of secondary structures, because personal relationships don't reach from every person to every other person.
Continue Reading » -
S3 Outage Report and Perspective Amazon has issued a more detailed statement explaining the S3 outage from June 20, 2008. In my company, we'd call this a "Post Incident Report" or PIR. It has all the necessary sections: Observed behaviorRoot cause analysisFollowup actions: corrective and operationalThis is exactly what I'd expect from any mature service provider. There are a few interesting bits from the report. First, the condition seems to have arisen from an unexpected failure mode in the platform's self-management protocol.
Continue Reading » -
Article on Building Robust Messaging Applications I've talked before about adopting a failure-oriented mindset. That means you should expect every component of your system or application to someday fail. In fact, they'll usually fail at the worst possible times. When a component does fail, whatever unit of work it's processing at the time will most likely be lost. If that unit of work is backed up by a transactional database, well, you're in luck. The database will do it's Omega-13 bit on the transaction and it'll be like nothing ever happened.
Continue Reading » -
Kingpins of Filthy Data If large amounts of dirty data are actually valuable, how do you go about collecting it? Who's in the best position to amass huge piles? One strategy is to scavenge publicly visible data. Go screen-scrape whatever you can from web sites. That's Google's approach, along with one camp of the Semantic Web tribe. Another approach is to give something away in exchange for that data. Position yourself as a connector or hub.
Continue Reading » -
Inverting the Clickstream Continuing my theme of filthy data. A few years ago, there was a lot of excitement around clickstream analysis. This was the idea that, by watching a user's clicks around a website, you could predict things about that user. What a backwards idea. For any given user, you can imagine an huge number of plausible explanations for any given browsing session. You'll never enumerate all the use cases that motivate someone to spend ten minutes on seven pages of your web site.
Continue Reading » -
Mounds of Filthy Data Data is the future. The barriers to entering online business are pretty low, these days. You can do it with zero infrastructure, which means no capital spent on depreciating assets like servers and switches. Open source operating systems, databases, servers, middleware, libraries, and development tools mean that you don't spend money on software licenses or maintenance contracts. All you need is an idea, followed by a SMOP. With both the cost side trending toward zero, how can there be any barrier to entry?
Continue Reading » -
Hard Problems in Architecture Many of the really hard problems in web architecture today exist outside the application server. Here are three problems that haven't been solved. Partial solutions exist today, but nothing comprehensive. Uncontrolled demandUsers tend to arrive at web sites in huge gobs, all at once. As the population of the Net continues to grow, and the need for content aggregators/filters grows, the "front page" effect will get worse. One flavor of this is the "Attack of Self-Denial", an email, radio, or TV campaign that drives enough traffic to crash the site.
Continue Reading » -
Creeping Fees A couple of years ago, the Minneapolis-St. Paul airport introduced self-pay parking gates. Scan a credit card on the way in and on the way out, and it just debits the card. This obviously saves money on parking attendants, and it's pretty convenient for parkers. At first, to encourage adoption, they offered a discount of $2 per day. Every time you'd approach the entry, a friendly voice from a Douglas Adams novel would ask, "Would you like to save $2 per day on parking?
Continue Reading » -
Word Cloud Bandwagon Wordle has been meming it's way around the 'Net lately. Figured I'd join the crowd by doing a word cloud for Release It. This is from the preface. Considering that this is just from fairly simple text analysis, I'm surprised at how accurately it represents the key concerns. "Software" and "money" have roughly equal prominence. "Life" appears near the middle, along with "excitement", "revenue", "production" and "systems". Not bad for an algortihm.
Continue Reading » -
Webber and Fowler on SOA Man-Boobs InfoQ posted a video of Jim Webber and Martin Fowler doing a keynote speech at QCon London this Spring. It's a brilliant deconstruction of the concept of the Enterprise Service Bus. I can attest that they're both funny and articulate (whether on the stage or off.) Along the way, they talk about building services incrementally, delivering value at every step along the way. They advocate decentralized control and direct alignment between services and the business units that own them.
Continue Reading » -
Coincidence or Back-end Problem? An odd thing happened to me today. Actually, an odd thing happened yesterday, but it's having the same odd thing happen today that really makes it odd. With me so far? Yesterday, while I was shopping at Amazon, Amazon told me that my American Express card had expired. While it is set for a May expiration, it's several years in the future. I didn't think too much of it, because when I re-entered the same information, Amazon accepted it.
Continue Reading » -
Social Factors I mentioned Tom DeMarco just a couple of days ago. I'm re-reading his great book, Why Does Software Cost So Much?for the first time in about ten years. Personally, I credit Tom as one of the unsung progenitors of the agile movement. Long before we had "Agile" or even "lightweight methods", Tom was talking about the psycho-social nature of software development. For instance, here's an excerpt from essay 8, "Nontechnological Issues in Software Engineering":
Continue Reading » -
Plurk. A friend invited me to Plurk. So far, I've resisted Twitter for no good reason (other than a vague sense of social insecurity.) I figure I'll dip my toe into Plurk, though. This link is an open invite to Plurk. It'll let anyone join. Fair warning, it's also a "friend" link.
-
Six Word Methods In his great collection of essays Why Does Software Cost So Much?, Tom DeMarco makes the interesting point that the software industry had grown from zero to $300 billion dollars (in 1993). This indicates that the market had at least $300B worth of demand for software, even while complaining continuously about the cost and quality of the very same software. It seems to me that the demand for software production, together with the time and cost pressures, has only increased dramatically since then.
Continue Reading » -
New Article: S2AP + Eclipse + Maven walkthrough See Getting Started With SpringSource Application Platform, Eclipse, and Maven. Most of the information out there about programming in S2AP is in blogs or references to really old OSGi tutorials. It took me long enough to configure some basic Eclipse project support that I figured it was worth writing down. All of the frameworks and tool sets are very flexible, which means you have more choices to deal with when setting up a project.
Continue Reading » -
Canadian Privacy Commissioner Highlights Cloud Privacy Concerns A little while ago, I wrote a piece about the conflict between "clouds" and the hard boundaries of the political sphere. There's no physical place called "cyberspace", and any cloud computing infrastructure has to actually exist somewhere. Like many U.S. citizens, I really hate the idea that facts about me become somebody else's copyrighted property just because they get stored in a database. Canada has a justifiably good reputation for protecting its citizens' privacy.
Continue Reading » -
Quickie: GAE is GA According to eWeek, Google will make GAE open to public use on May 28th. Which would be today. The original GAE site isn't updated at this point, but you can get started anyway. I just set up my account and registered an app. (I predict tens of thousands of empty apps. Long-tail distribution here, just like SourceForge: an overwhelming majority of empty projects, with a vanishingly tiny minority that have 99% of the traffic.
Continue Reading » -
Wii Wescue So, I got a Wii for Father's Day last year. It's been a lot of fun to play together with my kids, my wife, and even my parents and in-laws. It's fantastic to have a game system that we can all play together and be reasonably competitive. My six-year old can hold her own in Wii bowling, but she cries a lot when we play Halo. (I'm just kidding...)
Continue Reading » -
Opening Up SpringSource AP Just now getting my hands on the SpringSource Application Platform. It's deceptive, because there's very little functionality exposed when you run it. It starts up with less ceremony than Apache or Tomcat. (Which is kind of funny, when you consider that it includes Tomcat.) When you look at the bundle repository, though, it's clear that a lot of stuff is packaged in here. In a way, that's like the Spring framework itself.
Continue Reading » -
JavaOne is a Hot Zone Apparently, there's a virus attack. Not a computer virus. A real virus. Hot zone instead of a hot spot. From my inbox this morning: The JavaOne conference team has been notified by the San Francisco Department of Public Health about an identified outbreak of a virus in the San Francisco area. Testing is still underway to identify the specific virus in question, but they believe it to be the Norovirus, a common cause of the "stomach flu", which can cause temporary flu-like symptoms for up to 48 hours.
Continue Reading » -
Grab Bag of Demos Sun opened the final day of JavaOne with a general session called "Extreme Innovation". This was a showcase for novel, interesting, and out-of-this-world uses of Java based technology. VisualVM VisualVM works with local or remote applications, using JMX over RMI to connect to remote apps. While you have to run VisualVM itself under JDK 1.6, it can connect to any version of JVm from 1.4.2 through 1.7. Local apps are automatically detected and offered in the UI for debugging.
Continue Reading » -
SOA: Time For a Rethink The notion of a service-oriented architecture is real, and it can deliver. The term "SOA", however, has been entirely hijacked by a band of dangerous Taj Mahal architects. They seem innocuous, it part because they'll lull you to sleep with endless protocol diagrams. Behind the soporific technology discussion lies a grave threat to your business. "SOA" has come to mean top-down, up-front, strong-governance, all-or-nothing process (moving at glacial speed) implemented by an ill-conceived stack of technologies.
Continue Reading » -
The JVM is Great, But Much of the interest in dynamic languages like Groovy, JRuby, and Scala comes from running on the JVM. That lets them leverage the tremendous R&D that’s gone into JVM performance and stability. It also opens up the universe of Java libraries and frameworks. And yet, much of my work deals with the 80% of cost that comes after the development project is done. I deal with the rest of the software’s lifetime.
Continue Reading » -
SAP's SOA ESR SAP has been talking up their suite of SOA tools. The names all run together a bit, since they each contain some permutation of "enterprise" and "builder", but it's a very impressive set of tools. Everything SAP does comes off of an enterprise service repository (ESR). This includes a UDDI registry, and it supports service discovery and lookup. Development tools allow developers to search and discover services through their "ES Workspace".
Continue Reading » -
Type Inference Without Gagging I am not a language designer, nor a researcher in type systems. My concerns are purely pragmatic. I want usable languages, ones where doing the easy thing also happens to be doing the right thing. Even today, I see a lot of code that handles exceptions poorly (whether checked or unchecked!). Even after 13 years and some trillion words of books, most Java developers barely understand when to synchronize code. (And, by the way, I now believe that there's only one book on concurrency you actually need.
Continue Reading » -
When Should You Jump? JSR 308. That's When. One of the frequently asked questions at the No Fluff, Just Stuff expert panels boils down to, "When should I get off the Java train?" There may be good money out there for the last living COBOL programmer, but most of the Java developers we see still have a lot of years left in their careers, too many to plan on riding Java off into it's sunset. Most of the panelists talk about the long future ahead of Java the Platform, no matter what happens with Java the Language.
Continue Reading » -
SOA at 3.5 Million Transactions Per Hour Matthias Schorer talked about FIDUCIA IT AG and their service-oriented architecture. This financial services provider works with 780 banks in Europe, processing 35,000,000 transactions during the banking day. That works out to a little over 3.5 million transactions per hour. Matthias described this as a service-oriented architecture, and it is. Be warned, however, that SOA does not imply or require web services. The services here exist in the middle tier. Instead of speaking XML, they mainly use serialized Java objects.
Continue Reading » -
Project Hydrazine Part of Sun's push behind JavaFX will be called "Project Hydrazine". (Hydrazine is a toxic and volatile rocket fuel.) This is still a bit fuzzy, and they only left the boxes-and-arrows slide up for a few seconds, but here's what I was able to glean. Hydrazine includes common federated services for discovery, personalization, deployment, location, and development. There's a "cloud" component to it, which wasn't entirely clear from their presentation. Overall, the goal appears to be an easier model for creating end-user applications based on a service component architecture.
Continue Reading » -
JavaOne: After the Revolution What happens to the revolutionaries, once they've won? It's been about ten years since I last made the pilgramage to JavaOne, back when Java was still being called an "emerging technology". Many things have changed since then. Java is now so mainstream that the early adopters are getting itchy feet and looking hard for the next big thing. (The current favorite is some flavor of dynamic language running on the JVM: Groovy, Scala, JRuby, Jython, etc.
Continue Reading » -
Who Ordered That? Yesterday, I let myself get optimistic about what Jonathan Schwartz coyly hinted about over the weekend. The actual announcement came today. OpenSolaris will be available on EC2. Honestly, I'm not sure how relevant that is. Are people actually demanding Solaris before they'll support EC2? There is a message here for Microsoft, though. The only sensible license cost for a cloud-based platform is $0.00 per instance. Addendum I said that OpenSolaris would be available on EC2.
Continue Reading » -
Sun to Emerge from Behind in the Clouds? Nobody can miss the dramatic proliferation of cloud computing platforms and initiatives over the last couple of years. All through the last year, Sun has remained oddly silent on the whole thing. There is a clear, natural synergy between Linux, commodity x86 hardware, and cloud computing. Sun is conspicuously absent from all of those markets. Sun clearly needs to regain relevance in this space. On the one hand, Project Caroline now has its own website.
Continue Reading » -
Agile IT! Experience On June 26-28, 2008, I'll be speaking at the inagural Agile IT! Experience symposium in Reston, VA. Agile ITX is about consistently delivering better software. It's for development teams and management, working and learning together. It's a production of the No Fluff, Just Stuff symposium series. Like all NFJS events, attendance is capped, so be sure to register early. From the announcement email: The central theme of the Agile ITX conference (www.
Continue Reading » -
Amazon Blows Away Objections Amazon must have been burning more midnight oil than usual lately. Within the last two weeks, they've announced three new features that basically eliminate any remaining objections to their AWS computing platform. Elastic IP Addresses Elastic IP addresses solve a major problem on the front end. When an EC2 instance boots up, the "cloud" assigns it a random IP address. (Technically, it assigns two: one external and one internal. For now, I'm only talking about the external IP.
Continue Reading » -
Geography Imposes Itself On the Clouds In a comment to my last post, gvwilson asks, "Are you aware that the PATRIOT Act means it's illegal for companies based in Ontario, BC, most European jurisdictions, and many other countries to use S3 and similar services?" This is another interesting case of the non-local networked world intersecting with real geography. Not surprisingly, it quickly becomes complex. I have heard some of the discussion about S3 and the interaction between the U.
Continue Reading » -
Suggestions for a 90-minute app Some of you know my obsession with Lean, Agile, and ToC. Ideas are everywhere. Idea is nothing. Execution is everything. In that vein, one of my No Fluff, Just Stuff talks is called "The 90 Minute Startup". In it, I build a real, live dotcom site during the session. You can't get a much shorter time-to-market than 90 minutes, and I really like that. In case you're curious, I do it through the use of Amazon's EC2 and S3 services.
Continue Reading » -
Google's AppEngine Appears, Disappoints Google finally got into the cloud infrastructure game, announcing their Google AppEngine. As rumored, AppEngine opens parts of Google's legendary scalable infrastructure for hosted applications. AppEngine is in beta, with only 10,000 accounts available. They're already long gone, but you can download the SDK and run a local container. Here are some quick pros and cons: Pro Dynamically scalableGood lifecycle managementQuota-based management for cost containmentCon Python apps onlyYou deploy code, not virtual machinesWeb apps onlyAt this point, I'm a bit underwhelmed.
Continue Reading » -
Reality -
OmniFocus Coming to the iPhone Over the last six months, I've grown thoroughly dependent on OmniFocus. It's a "Getting Things Done" application that lets me juggle more projects, personal and professional, than I ever thought I could. Now, Omni says they're going to bring OmniFocus to the iPhone. So far, the iPhone hasn't compelled me, but I think that will be the trigger.
-
Release It has won a Jolt Productivity award It's an honor and a thrill for me to report that Release It received a Jolt Productivity award!
-
Steve Jobs Made Me Miss My Flight Or: On my way to San Jose. On waking, I reach for my blackberry. It tells me what city I'm in; the hotel rooms offer no clues. Every Courtyard by Marriott is interchangeable. Many doors into the same house. From the size of my suitcase, I can recall the length of my stay: one or two days, the small bag. Three or four, the large. Two bags means more than a week.
Continue Reading » -
The Granularity Problem I spend most of my time dealing with large sites. They're always hungry for more horsepower, especially if they can serve more visitors with the same power draw. Power goes up much faster with more chassis than with more CPU core. Not to mention, administrative overhead tends to scale with the number of hosts, not the number of cores. For them, multicore is a dream come true. I ran into an interesting situation the other day, on the other end of the spectrum.
Continue Reading » -
Sun Joining the Cloud Crowd As I was writing my last post, I somehow missed the news that Sun is building their own cloud platform, called Project Caroline. There's a PDF about it. It appears to be a presentation for JavaOne. It may be locked down at any minute, so the link might not work by the time you read this. Caroline looks a lot like Amazon EC2, but with some very nice control over VLANs (I suppose they would be Virtual VLANs?
Continue Reading » -
A Cloud For Everyone The trajectory of many high-tech products looks like this: Very expensive. Only a few exist in the world. They are heavily time-shared, and usually oversubscribed.Within the reach of institutions and corporations, but not individuals. The organization wants to maximize utilization. Corporations own many, as productivity enhancers, some wealthy or forward-looking individuals own one. Families time share theirs. Virtually everyone has one. To lack one is to fall behind. No longer a competitive advantage, the lack of the technology puts one at a disadvantage.
Continue Reading » -
Outrunning Your Headlights Agile developers measure their velocity. Most teams define velocity as the number of story points delivered per iteration. Since the size of a "story point" and the length of an iteration vary from team to team, there's not much use in comparing velocity from one team to the next. Instead, the team tracks its own velocity from iteration to iteration. Tracking velocity has two purposes. The first is estimation. If you know how many story points are left for this release, and you know how many points you complete per iteration, then you know how long it will be until you can release.
Continue Reading » -
Software Failure Takes Down Blackberry Services Anyone who's addicted to a Blackberry already knows about Monday's four-hour outage. For some of us, the Blackberry isn't just an electronic leash, it's part of our business operations. Like cell phones, Blackberries have a huge, hidden infrastructure behind them. Corporate Blackberry Event Servers (BES) relay email, calendar, and contact information through RIM's infrastructure, out through the wireless carriers. It was RIM's own infrastructure that suffered from intermittent failures during the outage.
Continue Reading » -
Tim Ross' C# Circuit Breaker Tim Ross has published his implementation of the Circuit Breaker pattern from Release It, complete with unit tests. I barely speak C#, so I'm not in any position to review his implementation, but I'm delighted to see it!
-
The Pragmatic Architect on Security Catching up on some reading, I finally got a chance to read Ted Neward's article "Pragmatic Architecture: Security". It's very good. (Actually, the whole series is pretty good, and I recommend them all. At least as of February 2008... I make no assertions about future quality!) Ted nails it. I agree with all of the principles he identifies, and I particularly like his advice to "fail securely". I would add one more, though: Be visible.
Continue Reading » -
Two Books That Belong In Your Library I seldom plug books---other than my own, that is. I've just read two important books, however, that really deserve your attention. Concurrency, Everybody's Doing It The first is "Java Concurrency in Practice by Brian Goetz, Tim Peierls, Joshua Bloch, Joseph Bowbeer, David Holmes, and Doug Lea. I've been doing Java development for close to thirteen years now, and I learned an enormous amount from this fantastic book. For example, I knew what the textbook definition of a volatile variable was, but I never knew why I would actually want to use one.
Continue Reading » -
Well Begun Is Half Done How long is your checklist for setting up a new development environment? It might seem like a trivial thing, but setup costs are part of the overall friction in your project. I've seen three page checklists that required multiple downloads, logging in as several users (root and non-root), and hand-typing SQL strings to set up the local database server. I think the paragon of environment setup is the ubiquitous GNU autoconf system.
Continue Reading » -
"Release It" is a Jolt Award Finalist The Jolt Awards have been described as "the Oscar's of our industry". (Really. It's on the front page of the site.) The list of past book winners reads like an essential library for the software practitioner. Even the finalists and runners-up are essential reading. Release It has now joined the company of finalists. The competition is very tough... I've read "Beautiful Code" and "Manage It!", and both are excellent. I'll be on pins and needles until the awards ceremony on March 5th.
Continue Reading » -
Should Email Errors Keep Customers From Buying? Somewhere inside every commerce site, there's a bit of code sending emails out to customers. Email campaigning might have been in the requirements and that email code stands tall at the brightly-lit service counter. On the other hand, it might have been added as an afterthought, languishing in some dark corner with the "lost and found" department. Either way, there's a good chance it's putting your site at risk. The simplest way to code an email sending routine looks something like this:
Continue Reading » -
Two Sites, One Antipattern This week, I had Groundhog Day in December. I was visiting two different clients, but they each told the same tale of woe. At my first stop, the director of IT told me about a problem they had recently found and eliminated. They're a retailer. Like many retailers, they try to increase sales through "upselling" and "cross-selling". So, when you go to check out, they show you some other products that you might want to buy.
Continue Reading » -
Read-write splitting with Oracle Speaking of databases and read/write splitting, Oracle had a session at OpenWorld about it. Building a read pool of database replicas isn't something I usually think of doing with Oracle, mainly due to their non-zero license fees. It changes the scaling equation. Still, if you are on Oracle and the fees work for you, consider Active Data Guard. Some key facts from the slides: Average latency for replication was 1 secondThe maximum latency spike they observed was 10 seconds.
Continue Reading » -
Budgetecture and it's ugly cousins It's the time of year for family gatherings, so here's a repulsive group portrait of some nearly universal pathologies. Try not to read this while you're eating. Budgetecture We've all been hit with budgetecture. That's when sound technology choices go out the window in favor of cost-cutting. The conversation goes something like this. "Do we really need X?" asks the project sponsor. (A.k.a. the gold owner.) For "X", you can substitute nearly anything that's vitally necessary to make the system run: software licenses, redundant servers, offsite backups, or power supplies.
Continue Reading » -
Releasing a free SingleLineFormatter A number of readers have asked me for reference implementations of the stability and capacity patterns. I've begun to create some free implementations to go along with Release It. As of today, it just includes a drop-in formatter that you can use in place of the java.util.logging default (which is horrible). This formatter keeps all the fields lined up in columns, including truncating the logger name and method name if necessary.
Continue Reading » -
A Dozen Levels of Done What does "done" mean to you? I find that my definition of "done" continues to expand. When I was still pretty green, I would say "It's done" when I had finished coding. (Later, a wiser and more cynical colleague taught me that "done" meant that you had not only finished the work, but made sure to tell your manager you had finished the work.) The next meaning of "done" that I learned had to do with version control.
Continue Reading » -
Postmodern Programming It's taken me a while to get to this talk. Not because it was uninteresting, just because it sent my mind in so many directions that I needed time to collect my scattered thoughts. Objects and Lego Blocks On Thursday, James Noble delivered a Keynote about "The Lego Hypothesis". As you might guess, he was talking about the dream of building software as easily as a child assembles a house from Lego bricks.
Continue Reading » -
Conference: "Velocity" O'Reilly has announced an upcoming conference called Velocity. From the announcement: Web companies, big and small, face many of the same challenges: sites must be faster, infrastructure needs to scale, and everything must be available to customers at all times, no matter what. Velocity is the place to obtain the crucial skills and knowledge to build successful web sites that are fast, scalable, resilient, and highly available. Unfortunately, there are few opportunities to learn from peers, exchange ideas with experts, and share best practices and lessons learned.
Continue Reading » -
Putting My Mind Online Along with the longer analysis pieces, I've decided to post the entirety of my notes from QCon San Francisco. A few of my friends and colleagues are fellow mind-mappers, so this is for them. Nygard's Mind Map from QCon This file works with FreeMind, an fast, fluid, and free mind mapping tool.
-
Two Ways To Boost Your Flagging Web Site Being fast doesn't make you scalable. But it does mean you can handle more capacity with your current infrastructure. Take a look at this diagram of request handlers. You can see that it takes 13 request handling threads to process this amount of load. In the next diagram, the requests arrive at the same rate, but in this picture it takes just 200 milliseconds to answer each one. Same load, but only 3 request handlers are needed at a time.
Continue Reading » -
Two Quick Observations Several of the speakers here have echoed two themes about databases. 1. MySQL is in production in a lot of places. I think the high cost of commercial databases (read: Oracle) leads to a kind of budgetechture that concentrates all data in a single massive database. If you remove that cost from the equation, the idea of either functionally partitioning your data stores or creating multiple shards becomes much more palatable.
Continue Reading » -
Three Vendors Worth Evaluating Several vendors are sponsoring QCon. (One can only wonder what the registration fees would be if they didn't.) Of these, I think three have products worth immediate evaluation. SemmleIn the category of "really cool, but would I pay for it?" is Semmle. Their flagship product, SemmleCode, lets you treat your codebase as a database against which you can run queries. SemmleCode groks the static structure of your code, including relationships and dependencies.
Continue Reading » -
Catching up through the day One of the great things about virtual infrastructure is that you can treat it as a service. I use Yahoo's shared hosting service for this blog. That gives me benefits: low cost and very quick setup. On the down side, I can't log in as root. So when Yahoo has a problem, I have a problem. Yesterday, there was something wrong with Yahoo's install of Movable Type. As a result, I couldn't post my "five things".
Continue Reading » -
Architecting for Latency Dan Pritchett, Technical Fellow at eBay, spoke about "Architecting for Latency". His aim was not to talk about minimizing latency, as you might expect, but rather to architect as though you believe latency is unavoidable and real. We all know the effect latency can have on performance. That's the first-level effect. If you consider synchronous systems---such as SOAs or replicated DR systems---then latency has a dramatic effect on scalability as well.
Continue Reading » -
SOA Without the Edifice Sometimes the best interactions at a conference aren't the talks, they're shouting. An crowded bar with an over-amped DJ may seem like an unlikely place for a discussion on SOA. Even so, when it's Jim Webber, ThoughtWorks' SOA practice lead doing the shouting, it works. Given that Jim's topic is "Guerilla SOA", shouting is probably more appropriate than the hushed and reverential cathedral whispers that usually attend SOA discussions. Jim's attitude is that SOA projects tend to attract two things: Taj Mahal architects and parasitic vendors.
Continue Reading » -
Cameron Purdy: 10 Ways to Botch Enterprise Java Scalability and Reliability Here at QCon, well-known Java developer Cameron Purdy gave a fun talk called "10 Ways to Botch Enterprise Java Scalability and Reliability". (He also gave this talk at JavaOne.) While I could quibble with Cameron's counting---there were actually more like 16 points thanks to some numerical overloading---I liked his content. He echoes many of the antipatterns from Release It. In particular, he talks about the problem I call "Unbounded Result Sets".
Continue Reading » -
Three Programming Language Problems Solved Forever It's often been the case that a difficult problem can be made easier by transforming it into a different representation. Nowhere is that more true than in mathematics and the pseudo-mathematical realm of programming languages. For example, LISP, Python, and Ruby all offer beautiful and concise constructs for operating on lists of things. In each of them, you can make a function which iterates across a list, performing some operation on each element, and returning the resulting list.
Continue Reading » -
Eric Evans: Strategic Design Eric Evans, author of Domain-Driven Design and founder of Domain Language, embodies the philosophical side of programming. He gave a wonderful talk on "Strategic Design". During this talk, he stated a number of maxims that are worth pondering. "Not all of a large system will be well designed." "There are always multiple models." "The diagram is not the model, but it is an expression of part of the model." These are not principles to be followed, Evans says.
Continue Reading » -
Kent Beck's Keynote: "Trends in Agile Development" Kent Beck spoke with his characteristic mix of humor, intelligence, and empathy. Throughout his career, Kent has brought a consistently humanistic view of development. That is, software is written by humans--emotional, fallible, creative, and messy--for other humans. Any attempt to treat development as robotic will end in tears. During his keynote, Kent talked about engaging people through appreciative inquiry. This is a learnable technique, based in human psychology, that helps focus on positive attributes.
Continue Reading » -
From QCon San Francisco I'm at QCon San Francisco this week. (An aside: after being a speaker at No Fluff, Just Stuff, it's interesting to be the audience again. As usual, on returning from travels in a different domain, one has a new perspective on familiar scenes.) This conference targets senior developers, architects, and project managers. One of the very appealing things is the track on "Architectures you've always wondered about". This coveres high-volume architectures for sites such as LinkedIn and eBay as well as other networked applications like Second Life.
Continue Reading » -
Pragmatic Podcast Has anyone ever been happy to listen to their own voice? Probably not. The Pragmatic Podcast is up and running on the redesigned Pragmatic Programmers site. In the first episode, Daniel Steinberg interviews me about the book. Also available on iTunes.
-
Make Time a Weapon Here's an list of books about putting time to work as your own weapon, instead of being victimized by it: The Mind of War The Art of Maneuver Lean Thinking Birth of the Chaordic Age Agile Software Development Lean Software Development Software by Numbers
-
Normal Accidents While I was writing Release It!, I was influenced by James R. Chile's book Inviting Disaster. One of Chile's sources is Normal Accidents, by Charles Perrow. I've just started reading, and even the first two pages offer great insight. Normal Accidents describes systems that are inherently unstable, to the point that system failures are inevitable and should be expected. These "normal" accidents result from systems that exhibit the characteristics of high "interactive complexity" and "tight coupling".
Continue Reading » -
You Keep Using That Word. I Do Not Think It Means What You Think It Means. "Scalable" is a tricky word. We use it like there's one single definition. We speak as if it's binary: this architecture is scalable, that one isn't. The first really tough thing about scalability is finding a useful definition. Here's the one I use: Marginal revenue / transaction > Marginal cost / transaction The cost per transaction has to account for all cost factors: bandwidth, server capacity, physical infrastructure, administration, operations, backups, and the cost of capital.
Continue Reading » -
Engineering in the White Space "Is software Engineering, or is it Art?" Debate between the Artisans and the Engineers has simmered, and occasionally boiled, since the very introduction of the phrase "Software Engineering". I won't restate all the points on both sides here, since I would surely forget someone's pet argument, and also because I see no need to be redundant. Deep in my heart, I believe that building programs is art and architecture, but not engineering.
Continue Reading » -
On the Widespread Abuse of SLAs Technical terminology sneaks into common use. Terms such as "bandwidth" and "offline" get used and abused, slowly losing touch with their original meaning. ("Bandwidth" has suffered multiple drifts. It started out in radio, not computer networking, let alone the idea of "personal attention space".) It is the nature of language to evolve, so I would have no problem with this linguistic drift, if it were not for the way that the mediocre and the clueless clutch to these seemingly meaningful phrases.
Continue Reading » -
Y B Slow? I've long been a fan of the Firebug extension for Firefox. It gives you great visibility into the ebb and flow of browser traffic. It sure beats rolling your own SOCKS proxy to stick between your browser and the destination site. Now, I have to also endorse YSlow from Yahoo. YSlow adds interpretation and recommendations to Firebug's raw data. For example, when I point YSlow at www.google.com, here's how it "grades" Google's performance:
Continue Reading » -
The 5 A.M. Production Problem I've got a new piece up at InfoQ.com, discussing the limits of unit and functional testing: "Functional testing falls short, however, when you want to build software to survive the real world. Functional testing can only tell you what happens when all parts of the system are behaving within specification. True, you can coerce a system or subsystem into returning an error response, but that error will still be within the protocol!
Continue Reading » -
ITIL and Extreme Programming Esther Schindler asked if I'd be willing to post my earlier article on staying agile in the face of ITIL at CIO.com. How could I say no? The piece is here.
-
ITIL and XP The Agile Manifesto is explicit about it. "We value individuals and interactions over processes and tools." How should an Agile team---more specifically, an XP team---respond to the IT Infrastructure Library (ITIL), then? After all, ITIL takes seven books just to define the customizable framework for the actual practices. An IT organization usually takes at least seven more binders to define its actual processes. Can XP and ITIL coexist in the same building, or is XP just incompatible with ITIL?
Continue Reading » -
Release It holding strong at Amazon Well, Release It continues to hold the #1 spot in Amazon's "Hot New Releases" list for Design Tools and Techniques. I've even got a couple of five-star reviews... and they weren't written by friends or family.
-
Heads down I've been quiet lately for a couple of reasons. First, I'm thrilled to say that I'm joining the No Fluff, Just Stuff stable of speakers. It's an honor and a pleasure to be invited to keep such company. The flip side is, I'm spending a lot of my free time polishing up my inventory of presentations. More frankly, I'm rebuilding them all with Keynote. (Brief aside, I'm coming to love Keynote.
Continue Reading » -
Coach and Team From Same Firm Is it an antipattern to have a consulting firm provide both the coach and developers? By providing the developers, the firm is motivated to deliver on the project, with coaching as an adjunct. If, instead, the firm provides just the coach, it will be judged by how well the client adopts the process. These two motives can easily conflict. Case in point: at a previous client of mine, my employer was charged with completing the project, using a 50-50 mix of contractors and client developers.
Continue Reading » -
Moving Your Home Directory on Leopard Since NetInfo Manager is going away under Leopard, we've got a gap in capability. How do you relocate your home directory without the GUI? There are a few reasons you might want to move your home directory to another volume. For example, you might reinstall your OS frequently. Or, perhaps you just want to keep your data on a bigger disk than the one that came in the machine. In my case, both.
Continue Reading » -
What makes a POJO so great, anyway? My friend David Hussman once said to me, "The next person that says the word 'POJO' to me is going to get stabbed in the eye with a pen." At the time, I just commiserated about people who follow crowds rather than making their own decisions. David's not a violent person. He's not prone to fits of violence or even hyperbole. What made this otherwise level-headed coach and guru resort to non-approved uses of a Bic?
Continue Reading » -
Release It! is shipping Release It is now shipping! People who ordered directly from The Pragmatic Programmers are receiving their hardcopies now. It will take Amazon and Barnes and Noble a few days or a week to work the inventory through their supply chain, but they should be shipping soon, too!
-
Flash Mobs and TCP/IP Connections In Release It, I talk about users and the harm they do to our systems. One of the toughest types of user to deal with is the flash mob. A flash mob often results from Attacks of Self-Denial, like when you suddenly offer a $3000 laptop for $300 by mistake. When a flash mob starts to arrive, you will suddenly see a surge of TCP/IP connection requests at your load-distribution layer.
Continue Reading » -
Release It! is released! "Release It!" has been officially announced in this press release. Andy Hunt, my editor, also posted announcements to several mailing lists. It's been a long road, so I'm thrilled to see this release. When you release a new software system, that's not the end of the process, but just the beginning of the system's life. It is the same thing here. Though it's taken me two years to get this book done and on the market, this is not the end of the book's creation, but the beginning of it's life.
Continue Reading » -
Self-Inflicted Wounds My friend and colleague Paul Lord said, "Good marketing can kill you at any time." He was describing a failure mode that I discuss in "Release It!: Design and Deploy Production-Ready Software" as "Attacks of Self-Denial". These have all the characteristics of a distributed denial-of-service attack (DDoS), except that a company asks for it. No, I'm not blaming the victim for electronic vandalism... I mean, they actually ask for the attack.
Continue Reading » -
Design Patterns in Real Life I've seen walking cliches before. There was this one time in the Skyway that I actually saw a guy with a white cane being led by a woman with huge dark sunglasses and a guide dog. Today, though, I realized I was watching a design pattern played out with people instead of objects. I've used the Reactor pattern in my software before. It's particularly helpful when you combine it with non-blocking multiplexed I/O, such as Java's NIO package.
Continue Reading » -
Another Path to a Killer Product Give individuals powers once reserved for masses Here's a common trajectory: 1. Something is so expensive that groups (or even an entire government) have to share them. Think about mainframe computers in the Sixties. 2. The price comes down until a committed individual can own one. Think homebrew computers in the Seventies. The "average" person wouldn't own one, but the dedicated geek-hobbyist would. 3. The price comes down until the average individual can own one.
Continue Reading » -
Quantum Manipulations I work in information technology, but my first love is science. Particularly the hard sciences of physics and cosmology. There've been a series of experiments over the last few years that have demonstrated quantum manipulations of light and matter that approach the macroscopic realm. A recent result from Harvard (HT to Dion Stewart for the link) has gotten a lot of (incorrect) play. It involves absorbing photons with a Bose-Einstein condensate, then reproducing identical photons at some distance in time and space.
Continue Reading » -
A path to a product Here's a "can't lose" way to identify a new product: Enable people to plan ahead less. Take cell phones. In the old days, you had to know where you were going before you left. You had to make reservations from home. You had to arrange a time and place to meet your kids at Disney World. Now, you can call "information" to get the number of a restaurant, so you don't have to decide where you're going until the last possible minute.
Continue Reading » -
How to become an "architect" Over at The Server Side, there's a discussion about how to become an "architect". Though TSS comments often turn into a cesspool, I couldn't resist adding my own two cents. I should also add that the title "architect" is vastly overused. It's tossed around like a job grade on the technical ladder: associate developer, developer, senior developer, architect. If you talk to a consulting firm, it goes more like: senior consultant (1 - 2 years experience), architect (3 - 5 years experience), senior technical architect (5+ years experience).
Continue Reading » -
Planning to Support Operations In 2005, I was on a team doing application development for a system that would be deployed to 600 locations. About half of those locations would not have network connections. We knew right away that deploying our application would be key, particularly since it is a "rich-client" application. (What we used to call a "fat client", before they became cool again.) Deployment had to be done by store associates, not IT.
Continue Reading » -
"Us" and "Them" As a consultant, I've joined a lot of projects, usually not right when the team is forming. Over the years, I've developed a few heuristics that tell me a lot about the psychological health of the team. Who lunches together? When someone says "whole team meeting," who is invited? Listen for the "us and them" language. How inclusive is the "us" and who is relegated to "them?" These simple observations speak volumes about the perception of the development team.
Continue Reading » -
Inviting Domestic Disaster We had a minor domestic disaster this morning. It's not unusual. With four children, there's always some kind of crisis. Today, I followed a trail of water along the floor to my youngest daughter. She was shaking her "sippy cup" upside down, depositing a full cup of water on the carpet... and on my new digital grand piano. Since the entire purpose of the "sippy cup" is to contain the water, not to spread it around this house, this was perplexing.
Continue Reading » -
Book Completed I'm thrilled to report that my book is now out of my hands and into the hands of copy editors and layout artists. It's been a long trip. At the beginning, I had no idea just how much work was needed to write an entire book. I started this project 18 months ago, with a sample chapter, a table of contents, and a proposal. That was a few hundred pages, three titles, and a thousand hours ago.
Continue Reading » -
Reflexivity and Introspection A fascinating niche of programming languages consists of those languages which are constructed in themselves. For instance, Squeak is a Smalltalk whose interpreter is written in Squeak. Likewise, the best language for writing a LISP interpreter turns out to be LISP itself. (That one is more like nesting than bootstrapping, but it's closely related.) I think Ruby has enough introspection to be built the same way. Recently, a friend clued me in to PyPy, a Python interpreter written in Python.
Continue Reading » -
Education as mental immune system Education and intelligence act like a memetic immune system. For instance, anyone with knowledge of chemistry understands that "binary liquid explosives" are a movie plot, not a security threat. On the other hand, lacking education, TSA officials told a woman in front of me to throw away her Dairy Queen ice cream cones before she could board the plane. Ice cream. How in the hell is anyone supposed to blow up a plane with ice cream?
Continue Reading » -
Expressiveness, revisited I previously mused about the expressiveness of Ruby compared to Java. Dion Stewart pointed me toward F-Script, an interpreted, Smalltalk-like scripting language for Mac OS X and Cocoa. In F-Script, invoking a method on every object in an array is built-in syntax. Assuming that updates is an array containing objects that understand the preProcess and postProcess messages. updates preProcess updates postProcess That's it. Iterating over the elements of the collection is automatic.
Continue Reading » -
Inviting Disaster I'm reading a fabulous book called "Inviting Disaster", by James R. Chiles. He discusses hundreds of engineering and mechanical disasters. Most of them caused serious loss of life. There are several common themes: 1. Enormously complex systems that react in sometimes unpredictable ways 2. Inadequate testing, training, or preparedness for failures -- particularly for multiple concurrent failures 3. A chain of events leading to the "system fracture".
Continue Reading » -
New Interview Question So many frameworks... so much alphabet soup on the resumes. Anyone that ever reads The Server Side or Monster.com knows exactly which boxes to hit when they're writing a resume. The recruiters telegraph their needs a mile away. (Usually because they couldn't care less about the differences or similarities between Struts, JSF, WebWork, etc.) As long as the candidate knows how to spell Spring and Hibernate, they'll get submitted to the "preferred vendor" system.
Continue Reading » -
JAI 1.1.3 in beta I've been using JAI 1.1.2 for the past year. It's an incredibly powerful tool, though I will confess that the API is more than a bit quirky. Early this year, Sun made JAI an open-source project available at java.net. That project has been working on the 1.1.3 release for most of the year. It's now in beta, with a few enhancements and a lot of bug fixes. The most significant enhancement is that JAI can now be used with Java WebStart.
Continue Reading » -
Ruby expressiveness and repeating yourself Just this week, I was reminded again of how Java forces you to repeat yourself. I had an object that contains a sequence of "things to be processed". The sequence has to be traversed twice, once before an extended process runs and once afterwards. The usual Java idiom looks like this: public void preProcess(ActionContext context) { for (Iterator iter = updates.iterator(); iter.hasNext(); ) { TwoPhaseUpdate update = (TwoPhaseUpdate) iter.
Continue Reading » -
MySQL 5.0 Stored Procedures The MySQL 5.0 release is finally adding stored procedures, triggers, and views. This is a welcome addition. With the strong storage management features, clustering, and replication from the 4.x releases, MySQL now has all the capabilities of an "enterprise" database. (Of course, the lack of these features didn't stop thousands of users from deploying earlier versions in enterprises, even for "mission-critical" applications.)* Here's a fairly trivial example: create procedure count_table_rows () reads sql data begin select table_name, table_rows from information_schema.
Continue Reading » -
The dumbest thing I've seen today I generally like Swing, but I just found something in the Metal L&F for JSlider that strikes me as a big WTF. The BasicSliderUI allows you to click in the "track" of the slider to scroll by a block. That's either 10% of the span of the slider, or a minimum of 1 unit. The MetalSliderUI overrides that sensible behavior with a method that scrolls by just one unit. Period.
Continue Reading » -
Programmer productivity measurements don't work. Programmer productivity measurements don't work. The most common metric was discredited decades ago, but continues to be used: KLOC. Only slightly better is function points. At least it's tied to some deliverable value. Still, the best function point is the one you don't have to develop. Likewise, the best line of code is the one you don't need to write. In fact, sometimes my most productive days are the ones in which I delete the most code.
Continue Reading » -
More Beanshell Goodness Thanks to the clean layered architecture in our application, we've got a very clear interface between the user interface (just Swing widgets) and the "UI Model". In the canonical MVC mode, our UI Model is part controller and part model. It isn't the domain model, however. It's a model of the user interface. It has concepts like "form" and "command". A "form" is mainly a collection of property objects that are named and typed.
Continue Reading » -
Smalltalk style prototyping for Java? I've been eyeing Beanshell for some time now. It's a very straightforward scripting language for Java. Its syntax is about what you would expect if I said, "Java with optional types and no need to pre-declare variables." So, a Java programmer probably needs all of about thirty seconds to understand the language. What I didn't expect was how quickly I could integrate it into my applications. Here's an example.
Continue Reading » -
One of the most fun features of my current project One of the most fun features of my current project is our "extreme feedback monitor". We're using CruiseControl to build our entire codebase, including unit tests, acceptance tests, and quality metrics, every five minutes. To make a broken build painfully obvious, we've got a stoplight hanging on one wall of the room. (I may post some pictures later, if there's interest.) Kyle Larson found the stoplight itself in a gift shop (Spencer's, maybe, I can't remember.
Continue Reading » -
I forgot to mention that I will be speaking at OTUG I forgot to mention that I will be speaking at OTUG on April 19th! I will be speaking on "Living With Systems in Production: Avoiding Heartbreak in Long-Term Relationships With Your Code" From the summary of the talk: Everything changes after Release 1.0. One batch of consultants leave, key developers jockey to get themselves reassigned, and the free-wheeling development environment is replaced by the painful rigor of operations. Or, at least, it should be.
Continue Reading » -
Leaving AntHill for CruiseControl We've been using AntHill to do continuous builds. It has served us well, but we're now moving away from it and towards CruiseControl. There are a few main reasons for this. First and foremost, AntHill runs inside of Tomcat. This is billed as a feature, but for us, it was a big problem. There are two layers of Java containers between your OS and your build. Trying to get environment variables (like "DISPLAY=localhost:99.
Continue Reading » -
The Veteran and the Master The aged veteran said to the master, "See how many programs I have written in my labors. All of these works I have created needed no more than a text editor and a compiler." The master said, "I do have an editor; indeed, I have also a compiler." Said the aged one, "Yet you shackle them within an 'environment'. Why must your environment be integrated? My environment has never been integrated, yet I am a mighty programmer.
Continue Reading » -
On Relativism and Social Constructions The key operative precept of post-modernism is that all reality is a social construct. Since no institution or normative behavior stems from natural cause, and there is no objective, external reality, then all institutions and attitudes are just social constructs. They exist only through the agreement of the participants. Nothing can be sacred, since sanctification comes from outside, by definition. If nothing is sacred, and institutions have no more reality than a children's amorphous game of ball, they deduce that any construct can be reconstructed through willful choice.
Continue Reading » -
An IKEA Weekend I've been building a new office in my downstairs space for quite a while now. It's a "weekends" project for someone who doesn't have very many weekends. In early December, I broke down and hired a contractor to install the laminate ("cardboard") flooring, which was the penultimate step in the master plan. Last comes furniture, then moving in. (Which starts the chain of dominoes, as my eldest gets the bedroom which used to be my office, then my youngest takes her spot, which makes room for the new baby.
Continue Reading » -
Uniting Reason and Passion Reason and Passion need not conflict. Reason without passion is dusty, dry, and dead. Reason without passion leads to moral relativity. If nothing moves the thinker to passion, then all subjects are equal and without distinction. As well to discuss the economic benefits of the euthanasia of infants as the artistic merits of urinals. Passion without reason brings the indiscriminate energy of a summer's thunderstorm. Too much energy unbound, without direction, it's fury as constant as the winds of the air.
Continue Reading » -
More Wiki My personal favorite is TWiki. It has some nice features like file attachments, a great search interface, high configurability, and a rich set of available plugins (including an XP tracker plugin.) One cool thing about TWiki: configuration settings are accomplished through text on particular topics. For example, each "web" (set of interrelated topics) has a topic called "WebPreferences". The text on the WebPreferences topics actually controls the variables. Likewise, if you want to set personal preferences, you set them as variables--in text--on your personal topic.
Continue Reading » -
Wiki Proliferation Wikis have been thoroughly mainstreamed now. You know how I can tell? Spammers are targeting them. Any wiki without access control is going to get steamrolled by a bunch of Russian computers that are editing wiki pages. They replace all the legitimate content with links to porn sites, warez, viagra, get rich now, and the usual panoply of digital plaque. The purpose does not appear to be driving traffic directly to those sites from the wikis.
Continue Reading » -
Moving on The latest in my not-exactly-daily news and commentary... As of December 10th, I will be leaving Totality Corporation. It has been a challenge and an education. It has also been an interesting time, as we uncovered the hidden linkages from daily activities to ultimate profitability. The managed service provider space is still new enough that the business models are not all so well-defined and understood as in consulting. I earnestly hope that I am leaving Totality in a much better place than it was when I joined.
Continue Reading » -
Too Much Abstraction The more I deal with infrastructure architecture, the more I think that somewhere along the way, we have overspecialized. There are too many architects that have never lived with a system in production, or spent time on an operations team. Likewise, there are a lot of operations people that insulate themselves from the specification and development of systems for which they will ultimately take responsibility. The net result is suboptimization in the hardware/software fit.
Continue Reading » -
The Lights Are On, Is Anybody Home? We pay a lot of attention to stakeholders when we create systems. The end users get a say, as do the Gold Owners. Analysts put their imprimatur on the requirements. In better cases, operations and administration adds their own spin. It seems like the only group that doesn't have any input during requirements gathering is the development team itself. That is truly unfortunate. Not even the users will have to live with the system more than the developers will.
Continue Reading » -
Don't Build Systems That Boink Note: This piece originally appeared in the "Marbles Monthly" newsletter in April 2003 I caught an incredibly entertaining special on The Learning Channel last week. A bunch of academics decided that they were going to build an authentic Roman-style catapult, based on some ancient descriptions. They had great plans, engineering expertise, and some really dedicated and creative builders. The plan was to hurl a 57 pound stone 400 yards, with a machine that weighed 30 tons.
Continue Reading » -
Plugging the Marbles Newsletter Not too much going on here lately. Most of my waking hours have been billable for the past few months. That's good and bad, in so many different ways. Most of my recent writing has been for the Marbles, Inc. monthly newsletter. Dec 2006 Edit: Marbles IT has not been a going concern for some time. My articles for the Marbles Monthly newsletter are now available under the Marbles category of this blog.
Continue Reading » -
Multiplier Effects Here's one way to think about the ethics of software, in terms of multipliers. Think back to the last major email virus, or when the movie "The Two Towers" was released. No doubt, you heard or read a story about how much lost productivity this bane would cause. There is always some analyst willing to publish some outrageous estimate of damages due to these intrusions into the work life. I remember hearing about the millions of dollars supposedly lost to the economy when Star Wars Episode I was released.
Continue Reading » -
Keep Your Secrets Here's a system I call "KeepYourSecrets.org". Recall a film noir detective telling the criminal mastermind that unless he drops a postcard in the mail in the next three days, all the details will go straight to the newspaper. You can upload any kind of file -- it's all treated like binary. You can set some parameters like a distribution list and a checkin frequency. The system uses an IRC-like network to split your file in n parts, of which some k parts are needed to re-create the original.
Continue Reading » -
The Paradox of Honor You can use a person's honor against him only if he values honor. Only the honest man is threatened by the pointed finger. The liar is unaffected by that kind of accusation. I think it is because there is no such thing as "dishonesty". There is only honesty or it's lack. Not a thing and it's opposite, but a thing and it's absence. One or zero, not one or minus-one. One who is lacking a thing cannot be threatened at the prospect of its loss.
Continue Reading » -
I think I'd like to I think I'd like to do some Smalltalk (or Squeak) development sometime. Just for myself. It would be good for me -- like an artist going to a retreat and setting aside all notions of practicality. I know I'll never work in Squeak professionally. That's why it would be like saying to yourself, "In this now, purity of expression is all that matters. Tomorrow, I will worry about making something I can sell.
Continue Reading » -
Nostalgia This kind of thing makes me wish I were back at Caltech.
-
Bill Joy Knocks the Open Source Business Model Bill Joy had some doubts to voice about Linux. Of course, like so many others he immediately jumps to the wrong conclusion. "The open-source business model hasn't worked very well," he says. Tough nuts. Here's the point that seems to get missed over and over again. There is no "open source business model". There never was, and I doubt there ever will be. It doesn't exist. It's a contradiction in terms.
Continue Reading » -
Needles, Haystacks So, this may seem a little off-topic, but it comes round in the end. Really, it does. I've been aggravated with the way members of the fourth estate have been treating the supposed "information" that various TLAs had before the September 11 attacks. (That used to be my birthday, by the way. I've since decided to change it.) We hear that four of five good bits of information scattered across the hundreds of FBI, CIA, NSA, NRO, IRS, DEA, INS, or IMF offices "clearly indicate" that terrorists were planning to fly planes into buildings.
Continue Reading » -
MLP Here's a good roundup of recent traffic regarding REST.
-
Here's my number one frustration Here's my number one frustration with the state of the industry today. I am a professional. I regard my work as a craft to be studied and learned. Yet, in most domains, there is no benefit to developing a high level of skill. You end up surrounded by people who don't understand a word you say, can't work at that level, and don't really give a damn. They'll get the same rewards and go home happy at 5:00 every day.
Continue Reading » -
Multiplier Effects Here's another way to think about the ethics of software, in terms of multipliers. Think back to the last major virus scare, or when Star Wars Episode II was released. Some "analyst"--who probably found his certificate in a box of Cracker Jack--publishing some ridiculous estimate of damages. BTW, I have to take a minute to disassemble this kind of analysis. Stick with me, it won't take long. If you take 1.
Continue Reading » -
REST and Change in APIs In case it didn't come through, I'm intrigued by REST, because it seems more fluid than the WS-* specifications. I can do an HTTP request in about 5 lines of socket code in any modern language, from any client device. The WS-splat crowd seem to be building YABS (yet another brittle standard). Riddle me this: what use is a service description in a standardized form if there is only one implementor of that service?
Continue Reading » -
Here's another excellent discussion about Here's another excellent discussion about REST for web services.
-
Debating "Web Services" There is a huge and contentious debate under way right now related to "Web services". A sizable contingent of the W3C and various XML pioneers are challenging the value of SOAP, WSDL, and other "Web service" technology. This is a nuanced discussion with many different positions being taken by the opponents. Some are critical of the W3C's participation in something viewed as a "pay to play" maneuver from Microsoft and IBM.
Continue Reading » -
Decoupling For the ultimate in temporal, architectural, language and spatial decoupling, try two of my favorite fluid technologies: publish-subscribe messaging and tuple-spaces.
-
Prison of our own Making -
Ethical decisions in software development Ethical decisions in software development do not only arise when we are talking about malware or copyright infringement. If my programs are successful, then they impact the lives of thousands or millions of people. That impact can be positive or negative. The program can make their lives better or worse–even if just in minute proportions. Every time I make a decision about how a program behaves, I am really deciding what my users can and cannot do.
Continue Reading » -
Designing for Emergent Behavior Lately, I’ve been grooving on emergent behavior. This fuzzy term comes from the equally fuzzy field of complexity studies. Mix complex rules together with non-linear effects (like humans) and you are likely to observe emergent behavior. Recent example: web browser security holes. Any program inherently constitutes a complex system. Add in some dynamic reprogramming, downloadable code, system-level scripting, and millions upon millions of users and you’ve got a perfect petri dish.
Continue Reading » -
Names have Power Names have power. Shamanic primitives guard their true names -- give me your name and you give me power over you. In the ether, your name is your only identity. Give away your name and you give away yourself. No cause, issue, or crusade has a follower until it has a name. A good name evokes images, emotions. A well-named issue becomes uncontestable. (Who is really opposed to "family values", anyway?
Continue Reading » -
Lately, I have been struggling Lately, I have been struggling to find the meaning in my work. I suppose that’s not surprising. I am a human being–a mortal creature. My age will soon flip a decimal digit. (I decline to specify which.) These can certainly cause one to spend time reflecting on one’s legacy. They can also cause one to buy a flaming red sports car. I may explore that option later. I also work in a field of incredible transience.
Continue Reading »