Neural Networks Are Already Clean

The Short VersionI think I have good news about the nature of neural networks.After many months studying neurons and circuits in image and language models, I’ve come to believe a network’s own mechanisms are its legible building blocks, and that the outlook for mechanistic interpretability may be far better than I’d feared. The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread. What looked like arbitrary packing was mostly a motif I’ll call recursive generalization, which is much less concerning. This means we can study networks directly, instead of first untangling them. Neural networks are already clean.There are at least two ways to approach mechanistic interpretability. The first is rapid normal science on the network itself: look at a network’s neurons and circuits, study how they relate, watch data flow through them, and run experiments at agentic scale to test explanations. The second starts from a list of hypothesized features and tries to extract them all, designing increasingly sophisticated reconstruction objectives and usually ignoring the original.The field began with the first approach and fell back to the second because neural networks seemed to be tangled knots. We had canonical examples of polysemantic neurons, the broad name for a neuron that appears to have multiple unrelated jobs or that looks impossible to understand. The most famous one, in an image model, responded to both cat faces and the fronts of cars. And the last layer of the early field’s model organism, InceptionV1, seemed so full of arbitrary mixtures it wasn’t clear how to study it: almost none of the neurons were easily interpretable.The leading explanation for polysemanticity was superposition. This proposed a new learning principle: when a neural network runs out of neurons, it stops grouping related things together and instead begins folding unrelated things together, simply because they don’t interfere. The result is a kind of knot: the network could have folded in many almost equally good ways, so which things a neuron groups together tells you almost nothing. You have to unfold the knot before you can study anything. In practice that means training a second model – a decomposition such as a sparse autoencoder – to approximate the network as it would look unfolded and studying that instead. If the network really is knotted, there may be no alternative.One main result is that superposition was the wrong explanation for InceptionV1. Architectural problems made later layers overflow. Weak interpretability tools amplified these problems, making the late layers seem incoherent. Since then, architectural changes designed to help deep layers learn better have had the unintended effect of making them more interpretable. When we first studied InceptionV1, we found almost no interpretable neurons in its last layer. But with these ordinary improvements, and no change to model size or training data, 99-100% of neurons in that last layer are easily interpretable and the model is broadly legible the whole way through. It’s the first non-trivial neural network I feel I mostly understand; you can browse it here. I trained versions of it across a 300× range in size, and it’s broadly legible at every scale. With less room, the network doesn’t fold unrelated things together; its neurons just cover coarser-grained concepts, like one car neuron covering many kinds of car. With more room, that neuron differentiates into many specific cars. So the problem wasn’t the presence of a second learning principle, it was just an ordinary learning failure of the first one, and it’s easy to fix.The canonical vision examples of polysemanticity and superposition were errors too. In 2019, Chris Olah and I found the circuit that was originally introduced as the superposition phenomenon in Zoom InZoom In: An Introduction to CircuitsOlah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., 2020. Distill. References ↓. It showed a car feature appearing to spread into a set of unrelated dog neurons. It was the only one we showed, and the only one I ever saw. We read it as the network placing the car in a high-dimensional hiding place, for downstream neurons to read back out. But nothing reads that combination. It wasn’t hiding anything; it was ordinary generalization and composition. To my knowledge, no one has published a second circuit with the right shape.Modern language models are also far more interpretable than generally believed. Anthropic’s Towards MonosemanticityTowards Monosemanticity: Decomposing Language Models With Dictionary LearningBricken, T., Templeton, A., Batson, J., et al., 2023. Transformer Circuits Thread. References ↓ (2023), studying a small one-layer model, gave the median neuron a 0 on its rubric out of a possible 14, meaning the rater couldn’t form any hypothesis. The automated interpretability projectLanguage Models Can Explain Neurons in Language ModelsBills, S., Cammarata, N., Mossing, D., et al., 2023. OpenAI. References ↓ I started at OpenAI (also 2023) found about 1.7% of neurons interpretable by our most generous measure.With better tools and modern models, the situation looks quite different. In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understoodA note on this number: the share of neurons with one job varies depending on how you set the threshold for counting weak activations, depth, density of sub-circuits, and what you view as “one thing”.How far to go into weak activations is a big factor here. There’s a section describing weak activations below. It tends to increase how many “related things” there are, more than whether the neuron is a judgement of something legible. For instance, the Trump neuron’s weak activations bring in MAGA hats and people related to him. The Titanic neuron’s weak activations bring in stories of more general voyages, or steam engines and other technologies of the era.There’s also the question of whether to care about activation percentiles, or fraction of the strongest activations. In general I think what the neuron learns should be trusted, its natural sparsity and activation ranges and so on. Which I used to mostly not believe! I’m somewhat against trying to standardize things across all neurons. Percentiles are more common today, I think because they’re easier to think about.It goes the other way too. For instance, when you look at the Titanic neuron’s “unrelated jobs,” it’s often detecting that “boat” comes next. The tool missed this because it only looked backward when judging context. When we look at these “unrelated jobs” in depth, it’s almost always this kind of thing. It’s easy to build an imperfect tool.Overall, I don’t think coming to a single score is actually that important. But when it was taken out, early reviewers felt unanchored without even a rough number. In the sets I’ve looked at more closely, considering strong activations, it’s closer to 97% non-arbitrary mixtures, but these are relatively small samples.Overall distilling everything into one “cleanliness” score should be taken as a broad pointer to the regime a neural network is in, rather than something precise to either optimize directly or hold too tightly. A lot of this essay is for being less dimensionality reduced, treating each neuron and circuit differently, according to their motifs, depth, role etc. Despite this, a rough number can be useful.. Dense and Mixture of Experts models seem roughly similar. We found similar results in models up to trillions of parameters, like Kimi K3, which the companion paper will cover in more detail. These percentages compress a high-dimensional picture into a few numbers, and the precise number depends on where you set thresholds for what counts as “one thing” – like whether you lump together closely related idioms in two languages. I don’t think much in interpretability hangs on that question. What matters a lot more is why things share a neuron in the first place.Most of the remaining apparent polysemanticity seems to be a motif I call recursive generalization. It’s simple: as a neuron grows, it sprouts offshoots the way a tree branch sprouts sub-branches. For instance, one neuron that detected “iy” grew to detect “diy,” and then “do it yourself,” which doesn’t contain “iy” at all. This idea first appeared in a forgotten subsection of a paper I worked on called Multimodal NeuronsMultimodal Neurons in Artificial Neural NetworksGoh, G., Cammarata, N., Voss, C., et al., 2021. Distill. References ↓ (2021). At the time, I hadn’t watched neurons grow over training. Now it’s easier, and it holds up everywhere I’ve seen. Even the original InceptionV1 wasn’t knotted. When I retrain its original architecture, which still produces messy deep layers, and follow the least legible neurons I can find, they turn out to be overgrowing locally into reasonable, adjacent territory.So there were two explanations for polysemanticity, and I helped propose both. In some ways, they’re opposites: generalization groups similar things together and superposition groups unrelated things together. Superposition is the one the field bet on and, until recently, so did I. But I now think recursive generalization was basically right. That leaves one learning principle after all. It’s still generalization, but it turns out generalization is fractal. That’s far less concerning: there’s something to learn from the way things were grouped, and nothing to learn from two items in superposition. And since the generalization itself is fractal, our explanations can be too: start with a basic explanation, then zoom in when you want to understand more. In that sense, almost the entire network is interpretable if we look closely enough, especially if we look