There’s a pattern I’ve observed a few times through scientific and computing history. I think of it as “complexity collapse”. It’s probably related to Kuhn’s paradigm shift.

The pattern starts with an approach that worked in the past. Gaps in the approach lead to accretions and additions. These restore the approach to functionality, but at the expense of added complexity.

That added complexity at first appears preferable to rebuilding the approach from the ground up. Eventually, however, the tower of complexity becomes impossible to extend further. At this point, the field is ripe for a complexity collapse and replacement with a fundamentally different approach.

In the realm of science, this complexity collapse has led to the most famous reformulations in history:

• Ancient astronomers assumed that the heavens were perfect. The stars were permanently fixed to a sphere, except for the “wanderers.” Planets and our moon, being heavenly bodies, must move in circles. The fly in the ointment was that circles alone could not explain apparent retrograde motion. Hipparchus and Ptolemy believed the explanation to be epicycles – circles superimposed on circles. Eventually, Copernicus showed that the number of epicycles needed would be drastically reduced with a heliocentric model. However, further improvements in optics and measurements caused the epicycles to proliferate again.

• Kepler swept the epicycles away with a clean, simple explanation: orbits are ellipses. His three laws, derived with the aid of Tycho Brahe’s incredibly accurate observations, described the motion of all the wanderers with a single explanation. Newton later showed the inverse square law of gravity would produce those ellipses. Newton could not have discovered the universal law of gravitation in the paradigm of epicycles.

• Near the end of the nineteenth century, physicists faced a similar tower of complexity when it came to explaning black-body radiation spectra. All the existing models for light, heat, and emission predicted much higher energy radiation at high frequencies than was actually observed. This would imply something called the “ultraviolet catastrophe” (which should absolutely be your next band name). It meant the night sky should be blazing with hard ultraviolet light. Not just that, but the farther you looked to the high frequency end of the spectrum, the higher the energy you would find. In other words, a black-body radiator could produce nearly infinite energy by just sitting there.

• As with the epicycles, the first response was to add adjustments to fix the ultraviolet catastrophe within existing equations of classical mechanics. Many such models were created by theorists whose names are only known to students of science history today. They all focused on adding corrective terms–based on unknown mechanisms–to the high end of the spectrum.

• Max Planck showed instead that the entire observed spectrum could be explained with one simple law. It only required that light came in particles (later called “photons”) whose energy depended on their wavelength rather than their mass times velocity. Planck swept away the complexity of the old model, replaced it with a simple set of equations, and laid the foundation for quantum mechanics.

• In the Java programming world, the challenge of building data-based shared systems with HTML front ends led a collection of vendors (virtually all gone now, absorbed into either Oracle or IBM) to create the “Java Enterprise Edition” specifications including the notorious Enterprise JavaBeans (EJB). This built on a tower of complex specifications for remote invocation and activation, interface descriptions, several roles that didn’t exist before (or since.) This stack could indeed allow programmers to create HTML based applications with a database hiding in the shadows.

• The Spring framework emerged as an alternative that focused instead on “plain old Java objects” (POJOs). It replaced complex interactions across development time, configuration time, deployment time, and run time with a simple model: objects that could be “injected”. Then it offered a collection of libraries that included classes useful for building the kind of applications developers needed to build.

Admittedly, the examples from astronomy and quantum physics are more fundamental to our understanding of the universe than XML-based dependency injection. But these examples all illustrate a similar dynamic. Complexity accumulates, a new theory replaces the old one, leading to complexity collapse.

All those examples include a common coda as well: complexity grows again!

• The elliptical orbits of Kepler and Newton work when bodies are far from very massive objects. There is a “correction” needed. That correction was predicted in 1915 and observed in 1919 in what may be the only planetary occultation to reach the front page of the New York Times. The corrected theory neatly explained the same simple elliptical orbits. It also predicted that Mercury, being very close to our Sun, would exhibit orbital precession because it made a simple ellipse on curved space (a geodesic ellipse.) That new theory explained the earlier results along with some new ones… but we did have to fundamentally change our view of space and time, and open the door to black holes, the twins paradox, and a host of other counterintuitive (but verified!) phenomena. Einstein’s beautiful equation fits on a single line of one page… but it takes a stack of books on one side of you to understand the equation, and a taller stack of books on the other side to explore the incredible implications.

• Planck’s law is simple, but nobody would make the same claim about the quantum mechanics it ushered in. The more we pursued Planck’s implications, the weirder our universe got.

• Of Spring, many people have quoted Harvey Dent from “The Dark Knight”… “You either die a hero or live long enough to see yourself become the villian.”

Today, the contradictions between quantum mechanics and general relativity lead many physicists to look for a new model. Not adjustments but a new paradigm to sweep away and unify the towers of complexity in both fields.

In the realm of data-based applications with web-ish interfaces, complexity collapse led many to embrace Ruby on Rails or Node.js. Both ecosystems have had mini-collapses but no complete replacement, yet.