The Decoherence Trap: When AI slop drowns out the signal in an endless sea of noise

TL;DR This essay introduces the concept of decoherence: the accumulation of locally valid but mutually incompatible or redundant models within a system, such that the cost of change is dominated by translating among those models and preventing unintended side-effects. In other words, it's what happens when slop drowns out the signal in an endless sea of noise. Decoherence grows relative to the rate of change of a system, and it increases faster the more decoherent a system is, resulting in runaway exponential growth. It is not a new phenomenon, and humans have always countered it through carefully finding ways to do more with less, unifying disparate patterns and models into a cohesive whole. However, the widespread use of AI is currently producing it at a rate that is unsustainable. The good news is that this rules out the specter of runaway recursive self-improvement, the intelligence explosion that goes parabolic before anyone can react, as decoherence both prevents AI from improving in a bubble and limits the rate at which it can improve outside of it. The bad news is that we are barreling toward a different kind of disaster: Decoherence Collapse. Imagine a building, let's say a small office tower. When this building was built, they used a type of pipe that will degrade over time and eventually need to be replaced. This is standard infrastructural debt; basic wear and tear, as they say. Then, let's say that when it comes time to replace the pipes, you need to change them over to a new standard. This could be because of new building code requirements, because the old pipes simply aren't made anymore, or because the new pipes are much better and easier to maintain. This is standard technical advancement debt, the need to upgrade and transition due to the forward progress of technology over time. There is a third form of debt that can arise based on how the transition from old pipes to new pipes is done. Consider the following options: Shut down the building and replace all of the pipes at once. This is a very disruptive and expensive method, since you can't use the building while the process is underway and you have to pay for the replacement up front. But the process is complete by the end of it. Shut down each floor, one at a time, and replace the pipes on that floor. This is less disruptive overall, but somewhat more expensive because you still have to pay up front, and you have to pay a premium for the extra labor required to keep things running smoothly as the work goes on. Replace the pipes as needed, when they begin to fail. This is the least disruptive strategy, and it allows you to amortize the cost of replacing all of the pipes over time, with one small repair after another. For most people, option 3 is very appealing. It allows them to keep things running smoothly and to spread out the cost of maintenance. And if we can just make like the Ship of Theseus[1] and replace the pipes one at a time, why shouldn't we? Well, for one thing, it creates decoherence. And decoherence is, by far, the worst form of tech debt. Evolution The Ship of Theseus actually gives us a good lens for understanding the true scope of decoherence. We know how to build ships made of wood, and we know how to build ships made of steel. You can repair a ship one piece at a time with the same material, as the original thought experiment goes, and the ship remains functional, with all of the original designs and structures that went into it intact. But do we know how to build every iteration of a ship in between wood and steel? Could you, say, slowly replace every piece of a man-of-war and incrementally turn it into a modern destroyer? And have it functional (for some definition of functional) at every step in the process? 1234 It's not that the answer is an absolute "no". If we are creative enough, if we had the best minds working on it, then I'm sure there would be some path to doing this. But it would require ingenuity and skill and craftsmanship. It would require attention to detail. And, crucially, you would really want to complete the process, because the half-man-of-war, half-destroyer vessel is likely not going to be very good in modern combat. Returning to our plumbing example, we can see how the same strategy plays out regularly in the world of construction. Every time a pipe breaks, the plumber is called, walls are opened, the repair is hand-crafted for that specific area, and then everything is closed back up. If you're lucky, the plumber will leave detailed notes, and if you're not, then years later some poor sod is going to open that wall back up and say to themselves, "What the hell is going on here?" Now, multiply that process over time as we slowly replace the pipes in our building, and the end result is a patchwork of old and new pipe that has much more complexity than either system on its own. Every joint, every hand-crafted patch, is its own little bundle of unique forces and properties that are understudied, and thus requires an expert in both systems to fully understand and work with. On the surface, this is fine. The pipes work, why does it matter how they work? As long as it's a good patch, who cares? Well, what happens if we, say, want to build an annex? For the purposes of this thought experiment, we'll say that the water for the annex needs to be routed through at least part of our existing building, using the existing pipes. And, crucially, we need to increase the overall water pressure in the system. Now, the question is: Is it safe to increase the pressure? If we had all type A or all type B pipes, even ones that had been patched and repaired over time, then this would be a straightforward question. Both systems are well understood, both have tools and processes for calculating their capacity, and people run these calculations all of the time. But, if we have an unknown mixture of both systems, suddenly we have to do a lot more analysis to answer that question. Every patch could change the answer, even ones at the other end of the system, because they're all tied together. And this is where the true complexity of decoherence becomes clear. When we try to make changes to decoherent systems, it requires far more expertise and skill than working with either system alone. The engineer working on the system needs to trace and understand it much more thoroughly and carefully, because their changes may have unintended consequences. And yes, I count plumbers as engineers, at least as much as software devs. Have you seen them think through pressure forces in drain pipes in real time? It's remarkable. That is no small task, and it is best if it's not rushed. But for the outside observer, without an understanding of the depth of the problem, this can be frustrating. Normally this part is easy. Why is it taking so long this time? In this situation, management will oftentimes apply pressure to get things done faster. When pressure is applied, the expert doesn't do the full analysis faster; they just stop earlier and do something they're pretty sure will work. A sloppy join, an unusual part, maybe some extra safeguards, just in case. Not completing the task thoroughly, however, creates a knock-on effect: this particular change has lower quality, and it makes the system more fragile and complex. This means the next change will take even more thought and care, and if the pressure on the engineer is the same, then they will do an even worse job. That is a pretty messy situation as it stands, and you can already see a sort of exponential growth of complexity taking place here. But wait, it gets worse! Let's consider the next level of decoherence: Starting a second transition before finishing the first. The Maze The tricky part about software, as compared to a physical building or other types of real-world systems, is that it's all sort of invisible. Coders walk through the halls of codebases every day. We navigate the labyrinth successfully so often that we become familiar with every corner and every turn, and when our walks become longer, with more corridors and dead ends and trap doors, we feel it much more immediately and intuitively. But this is difficult to relate to people who can't even see the maze, who don't walk it with you every day. And when it comes time to clean up some of your debt, to make the maze shorter and less winding, your proposals are hard to sell up the chain, even to those who used to walk mazes a long time ago. In the end, none of them know your maze. They don't know the trials you face each and every time you enter it. By comparison, our plumber friend in the real world can show the building owner the absurdity of their pipes much more directly. Courtesy of r/plumbing[2] When we start to mix more than two systems together, with multiple patches over time in a haphazard way, the complexity of the maze explodes. It is now combinatorial: you need to think of A -> B, A -> C, B -> C, A -> B -> C, C -> B -> A, and so on. We can of course insulate engineers from this somewhat. We can come up with best practices, we can use tools and frameworks to isolate part of the system from another part (for instance, by using a pressure-limiting valve in our pipes). Inevitably, though, it gets to a point where even the best engineers, the smartest people, can no longer safely navigate the maze. They may be able to do it 80 or 90% of the time, but they will eventually take a wrong turnforget to leave a trail of string and find themselves stuckwithin the immensityof the unending growththat outpaces their every stepinto eternity. This is the Decoherence Trap. The maze grows onward and onward until it swallows you whole and leaves you lost in the dark. Until it becomes so massive that it would take lifetimes to navigate. Well, lifetimes for humans anyway. But what about AI? Simulation Humans get tired and reach physical limits, but autonomous agent swarms can keep going around the clock, collectively and