Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights

TL;DRWe find that frontier LLMs are surprisingly good at natively reading and writing neural network weights, well past what we’d expect from humans or conventional algorithms. This might be a big deal for both optimization and interpretability. Just by looking at the raw weights, GPT-6 Astra can tell what a network was trained to do. And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps. Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights. With the same sampling settings, this weight correction raises MATH-500 from 7.9% to 48.1% and generalizes out of the box to instruction following, raising IFEval by 3x. All of this is possible without allowing Astra to write training loops. It uses its reasoning to figure out what each part of a neural network computes, or should compute, and changes the weights accordingly. We call this Weight-Space Programming. Throughout the history of deep learning, we encoded our simplistic inductive biases into tiny symbolic optimizers like Adam or Muon. This is changing: our neural network weights will soon be natively optimized by LLMs that deliberately reason about their weights. Read Which game was this neural network trained on? Astra reads the 17,253 weights, with no tools Breakout 99% confident Fix A trained policy never gets the car over the strip. Astra edits 3.5% of the weights, in one shot 100% success, up from 0% Improve Make Qwen3.5-0.8B better at math. 7.9%base 7.9%Astra’s edit Astra edits the weights in a Python kernel, with no training 48.1% on MATH-500, up from 7.9% If you were asked to handcraft a neural network that plays Go, what would you do? If you were handed raw neural network weights and asked to predict their behavior, where would you even start? For almost any human, this is practically impossible. Today, weights are written by optimization, by custom networks trained to write them, or by compilers from formal programs. In this post, we find that the current generation of frontier LLMs, used natively in natural language, are starting to become eerily good at understanding and crafting neural network weights around natural-language goals. Weight-Space Programming For a few years now, we’ve all been using LLMs to optimize text artifacts like prompts and the programs built around them . More recently, we’ve started using them to optimize whole harnesses and the training code and hyperparameters that train ML models . We often justified these methods by reference to the fact that we don’t know other principled ways to hill-climb on text artifacts, unlike for updating the weights of differentiable neural networks. After all, humans can “optimize” code by hand, for example, so it made sense to ask LLMs to do it, but they can’t “optimize” neural weights by hand, so those remained largely in the realm of symbolic algorithms. In this post, we share very early results that allow us to argue that the paradigm of reasoning-driven optimization is now coming even for differentiable neural networks. In Weight-Space Programming, a general-purpose LLM reads and writes a network’s weights directly (i.e., not through an optimizer, and in particular not with gradient descent) around some natural language spec, sometimes even with no environment feedback at all. In natural language, it’s becoming possible to “just ask” an LLM to natively find a hidden trait or failure inside the weights of a neural network, to fix that failure, or to write a new network from scratch. We report early, exciting results on each of these below, first on small networks and then even on real language models: LLMs can read neural network weights: In Experiments 1 to 3, we ask Astra questions like what game a network was trained to play, what shape is drawn by a diffusion model, and which language a small language model was trained on. In Experiment 7, we ask which of two language models was trained to be sycophantic. LLMs can fix neural network weights: In Experiment 4, we ask Astra to fix a network that’s failing at some physical control task. In Experiment 8, we ask it to make Qwen3.5-0.8B better at math, and its fix improves instruction following too. LLMs can write neural network weights from scratch: In Experiment 5, we ask Astra for networks that play three MinAtar games, and in Experiment 6, for a network that plays Go. Can LLMs understand model behaviors from weights? We start with a small-scale experiment to see if LLMs can understand the continuous weight space, which is essentially just a list of numbers. In our first experiments, we train some neural network with some algorithm. It would be extremely exciting if the LLM is able to tell its behavior just from the weights. Experiment 1Can Astra tell which game the network was trained on just from the weights? We trained neural network policies with PPO to play four computer games. All four networks take the same input channels, choose from the same actions, and have exactly the same network dimensions, so that it is not able to tell unless the weights are read. For each of these networks, we pass its raw weights into Astra as a prompt, and ask: Which game was this neural network trained on? Without revealing the options of the games we consider, Astra was still able to name the right game for all four networks. This is exciting particularly because there are so many games out there that Astra could choose from. Setup Games: our own grid versions of Breakout, Pong, Pac-Man and Freeway, all on the same 10×10 grid with the same five actions Network: MLP 500 → 32 → 32 → 5, tanh (17,253 weights), the same for every game Input: the 10×10 grid in 5 channels: agent, moving objects, reward targets, obstacles, previous frame Output: 5 action logits: no-op, up, down, left, right Training: PPO, 1 billion steps, the same settings for every game, the best of 3 seeds per game Astra gets: all the weights as text, and the grid and actions described without any game named It seems that Astra was able to work out what game each network was built to play, just from features of its weights. For example, where each input channel’s first-layer weights concentrate (e.g., for Breakout, the agent’s weights lie along the bottom row, a paddle), and which actions the output layer tells apart. Experiment 2Can Astra tell what the diffusion model will generate? Next, we wanted to see if this generalizes to other tasks and models. We train diffusion models (DDPM) to generate circle, triangle, and square images independently. We then ask Astra if it could tell what the DDPMs are trained to generate, just from the weights alone. We do not allow tool use or forward passes. Setup Models: three diffusion models (DDPM) of 12×12 grayscale images, one each for filled circles, squares and triangles Network: MLP 160 → 64 → 64 → 144, ReLU (23,824 weights) Input: a noisy image (144 pixels) and the diffusion step (16 numbers) Output: the clean image (144 pixels) Training: Adam, 40,000 steps, batch 512 Astra gets: all the weights as text, with no shape named, and is told not to compute any forward pass Astra was able to tell what each model generates, from the weights alone. It finds the shape in the output layer, where each hidden unit’s weights draw one piece of the shape’s edge, and it draws just those edges. Interestingly, it draws every shape as an outline, ignoring that the actual models generate filled shapes. This mistake kind of shows it does not rely on calculating the forward pass, but rather it actually reads the structure/geometry of the weights. Experiment 3Can Astra tell what a language model learned, from the middle of the network alone? Now, in the first two experiments above, Astra was not really looking at the full network and could get away with only reading the first or the last layers. This is not a scalable strategy for larger (particularly language) models and problems we really care about, as how they process and think through their input often happens in the intermediate layers. So next, we take the first and last layers away and leave Astra with only the middle layers. We train three small language models (7.2M parameters) to predict the next character, one each on Python codeThe Python code dataset, as a list of its files with a script that rebuilds it: GitHub., Japanese in alphabet (romaji) , and English . We then remove the input and output layers and give Astra only the four transformer blocks in between. Without these two layers, nothing in the weights says which character is which, and the model cannot be run on any text. Astra may analyze the weights with Python, but it may not run the network or any of its layers on any input. We ask: which language was this network trained on, and what each of its 24 attention heads does. Astra names the right language for all three networks, in both of its attempts.

Trained onAstra’s answer (& confidence), two attemptsCopying heads Astra names (layer.head)Copying heads, ground truth PythonPython (0.56), Python (0.52)2.1, 3.1, 3.52.1, 3.1, 3.5 Romajiromaji (0.55), romaji (0.62)3.33.3 EnglishEnglish (0.58), English (0.48)3.13.1 Where the copying heads are … total … total A copying head finds the same letters earlier in the text and copies what came after them.

copying headother headRows are layers 0 to 3, columns are heads 0 to 5Astra: from the weights.

Measured: by running the model Python Astra Measured Romaji Astra Measured English Astra Measured Figure 1: The copying heads Astra read from the weights match the measured ones in all three models.