{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Synaptic Tagging for Agent Memory" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Sriniketh Ponnada*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Your brain does not record everything that happens to you. Most of today will be gone within a week, while a few moments will stay for years, and you did not choose which. Some of that selection happens at individual synapses, through a mechanism neuroscientists call **synaptic tagging and capture**.\n", "\n", "The idea is that a memory is kept only when two things coincide. The first is a cheap, local mark, left wherever activity happened, that says nothing about whether the activity mattered. The second is a separate, global signal that fires when something surprising happens and decides which of those marks are made permanent. Marks that never meet the signal fade.\n", "\n", "I wanted to know whether a transformer has equivalents of both. If it does, a model's own internal activity, recorded while it works, might flag which moments of a long agent transcript matter, with no second model reading the transcript afterwards to judge it. I tested one candidate for each half. For the local mark, I used how concentrated the model's attention is at each token. For the surprise signal, I used how sharply a learned estimate of \"is this attempt going to succeed?\" changes from one step to the next. I instrumented an open-weight model running as a coding agent and checked both against turn-by-turn importance ratings." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## How a synapse decides to remember" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When two neurons are active together, the connection between them strengthens. This is *long-term potentiation* (LTP), and it comes in two phases. **Early LTP** sets in within minutes and needs nothing new from the cell. It works by modifying proteins already present at the synapse, and it fades within a few hours. **Late LTP** lasts for days or longer, and it requires the cell to manufacture new proteins. Brief, weak activity produces only the early phase. Strong or repeated activity triggers the late phase.\n", "\n", "That raises a puzzle. The new proteins are made centrally in the cell, yet a neuron has thousands of synapses, and only the ones that were active should be strengthened. How do the proteins end up in the right place?\n", "\n", "In 1997, Uwe Frey and Richard Morris proposed an answer, based on experiments in slices of rat hippocampus. They stimulated two separate pathways onto the same population of neurons. One pathway received weak stimulation, enough for early LTP only, which normally fades. The other received strong stimulation, which triggers protein synthesis. When the two happened within a window of a couple of hours of each other, the weak pathway's potentiation *also* became long-lasting. It did so even when protein synthesis was blocked at the moment of the weak stimulation, as long as the strong stimulation elsewhere had already produced the proteins.\n", "\n", "Their interpretation has two parts:\n", "\n", "1) **The tag.** Activity at a synapse sets a local, temporary mark. Setting it is cheap, and even weak activity is enough. The tag carries no judgment about whether the activity mattered. It only marks the synapse as eligible.\n", "2) **The capture.** Strong activity anywhere in the cell triggers the production of plasticity-related proteins, which spread broadly. Any synapse that is tagged while they are available captures them and is stabilized. Tags that never meet those proteins decay, and the change they marked is lost.\n", "\n", "Neither part is enough on its own. Memory forms where a tag and a capture coincide." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The same logic shows up in behaviour. In *behavioural tagging* experiments, an animal given a training session too weak to leave a lasting memory will remember it anyway if it explores a novel environment within roughly an hour before or after. The novel experience supplies the proteins the weak memory needed. Blocking dopamine receptors removes the effect, and dopamine is the strongest candidate for the signal that says \"something significant just happened, build the proteins now\".\n", "\n", "Dopamine has a precise computational interpretation of its own. In the 1990s, Wolfram Schultz and colleagues showed that dopamine neurons do not simply fire at rewards. They fire at *unexpected* rewards, stay silent at rewards that were predicted, and dip when a predicted reward fails to arrive. Their activity tracks the gap between what was expected and what happened: a reward-prediction error. That is the same quantity as the temporal-difference (TD) error used in reinforcement learning, which updates an estimate of future reward by how much it was wrong at each step.\n", "\n", "So the biological recipe is a cheap, local tag that records engagement, combined with a sparse, global surprise signal that decides which tags are kept." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The parallel in a transformer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I suspected that both halves had counterparts in a language model, ones that could be computed as a side effect of generation, with no extra model call.\n", "\n", "**My guess for the tag: attention entropy.** At every step, each attention head distributes a total weight of one across all earlier tokens. If the weight is concentrated on a few tokens, the distribution has low entropy, and the model is leaning hard on something specific. That seemed to me like the analogue of a synapse firing strongly along one pathway. Like a synaptic tag, it is local, it is computed per token, and it carries no judgment about whether the moment matters. My starting hypothesis was that low-entropy tokens should be tagged." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "For an attention head with weights $p_1, \\dots, p_n$ over the preceding tokens, the entropy is\n", "\n", "$$ H = -\\sum_{j=1}^{n} p_j \\log p_j $$\n", "\n", "which is zero when all the weight sits on one token and largest when it is spread evenly. I averaged it over heads within a layer.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**My guess for the capture: TD error.** If dopamine reports surprise about reward, the closest thing I could think of for an agent was a value function: an estimate, at each step, of how likely the current attempt is to succeed. The TD error is the step-to-step change in that estimate, and I suspected a large change would mark the moments where the model, taken as a whole, had just encountered something it did not expect.\n", "\n", "If both guesses were right, a token would be worth keeping when it was tagged and sat near a capture event.\n", "\n", "To test this, I ran Qwen3.6-27B, an open-weight model, as a coding agent on real bugs from my own projects' commit history, checked out just before each fix. Reading attention weights requires running the model yourself, so the experiment was limited to an open-weight model. Recording the signals per token is straightforward with forward hooks. One complication: only 16 of this model's 64 layers use standard softmax attention. The rest are linear-attention layers that never form a distribution to take the entropy of, so the attention signals come from those 16 layers." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Attention entropy worked, backwards" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I tagged the lowest-entropy quarter of tokens and read through the results. The most heavily tagged passage in the first run was about forty near-identical copies of the sentence \"Let me list the repository root directory.\" The turns where the model actually diagnosed the bug were tagged far less often. Across all the transcripts, roughly 80–90% of what the signal flagged was filler, restated intentions, or repetition.\n", "\n", "That run had a weakness. I had written my own minimal agent loop, and the model handled it badly: it produced malformed tool calls and never solved a task. A confused agent repeats itself, so the harness could explain the result. I re-ran the tasks in opencode, a real coding agent, and then fed the recorded session back through the instrumented model, token by token, to measure it. One session ran to completion, 39 turns ending in a correct fix, and it showed the same pattern." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "

What the low-entropy tag marks

\n", "
\n", "
\n", "

Turn 0 · opening boilerplate

\n", "

\"I'll start by exploring the repository structure and understanding the codebase, then find all the paths where…\"

\n", "

30 of 31 tokens tagged

\n", "
\n", "
\n", "

Turn 36 · the model catches its own bug

\n", "

\"This is backwards. The health check should show a toast when the engine is unhealthy, not when it's healthy.\"

\n", "

2 of 20 tokens tagged

\n", "
\n", "
\n", "
Two moments from the completed session. The opening announcement of a plan is tagged almost completely; the sentence where the model notices its own logic is inverted, the clearest insight in the session, barely registers.
\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In hindsight the reason is simple. Low attention entropy means the model knows exactly which earlier text it is continuing from, and the most predictable thing a model writes is a repetition of something it has just said. The tag was measuring predictability, which is close to the opposite of what a memory should keep.\n", "\n", "So I turned the question around and asked whether *high* entropy marks importance. To test that, each of the session's 39 turns was rated from 1 (routine) to 5 (the pivotal moment). The ratings were done by Claude, reading the full transcript once, not by people. Against those ratings, attention entropy in the model's later layers correlates positively with importance: Spearman $\\rho = +0.34$ across turns. The turns where the model's attention was spread across many competing parts of the context were the turns the rater judged important. The biological analogy predicted the opposite sign. The model's version of \"something important is happening here\" looks less like a focused synapse and more like a model weighing several alternatives at once." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The TD head, and how I trained it wrong" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The capture half needed a value head: a small model on top of Qwen's hidden states that predicts, at each step, whether the current attempt will succeed. No such head exists for this model, so I trained one.\n", "\n", "The training data was 30 attempts at real GitHub issues from a public dataset of agent trajectories (`nebius/SWE-agent-trajectories`), each labelled by whether the repository's tests actually passed: 15 successes, 15 failures, 18 distinct issues. I replayed each attempt through Qwen to collect its hidden states and trained the head to predict the eventual outcome, discounted so that steps closer to the end carry more of the credit ($\\gamma = 0.97$). The head itself was deliberately simple: PCA to 16 components, then ridge regression. I evaluated it with cross-validation grouped by issue, so that two attempts at the same issue could never end up on both sides of a split.\n", "\n", "On issues it had not seen, the head learned something real: its predictions correlated with eventual success at about $+0.27$. On my own session, its predictions, their step-to-step changes (the TD error), and the size of those changes all correlated with importance at roughly zero ($-0.006$, $-0.009$, and $-0.034$). Adding the TD error to entropy made entropy worse, lowering its correlation from $+0.34$ to $+0.23$.\n", "\n", "Looking back, the head was set up to fail. It was trained on other models' actions, not Qwen's own. Those trajectories came from a narrow kind of task with a clean pass-or-fail result, and it was then asked about an ordinary session with no defined finish line. And 30 trajectories is very little data for any value function. Next time, I would train it on Qwen's own trajectories, on the same kind of task it will be scored on, with an outcome defined for that task." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Every other signal, and why none of them worked" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If my two chosen signals were the wrong ones, perhaps another internal quantity was the right one. So in the replay of the completed session, I recorded everything I reasonably could, 259 signals per token in total:\n", "\n", "- attention entropy (mean, maximum, and minimum over heads) and peak attention weight, at each of the 16 softmax layers;\n", "- the size of the hidden state, and how far it jumped from its recent average (raw and as a z-score), at all 64 layers, including the linear-attention layers;\n", "- the entropy of the output distribution, the probability of the top token, and how much the output distribution shifted from one step to the next.\n", "\n", "Then I correlated every one of them with the turn ratings. At first glance this looked promising: 68 of the 259 matched or beat attention entropy's $\\rho = 0.34$, and a handful of hidden-state signals reached about $\\pm 0.5$.\n", "\n", "Two checks took that apart. First, testing 259 signals guarantees some high correlations by chance. To find how high, I shuffled the ratings 5,000 times and recorded the best correlation any of the 259 signals achieved against each shuffle. Chance alone produced a best correlation above $0.52$ in 5% of shuffles. Only one real signal cleared that bar, a hidden-state norm in layer 2, and only barely ($p = 0.046$).\n", "\n", "Second, most of the high scorers were partly measuring turn length. Long turns tend to be the important ones, and many of these signals change systematically with length. The output-distribution signals were the clearest case: they reached $\\pm 0.33$ to $0.43$, and once length is accounted for, all three fall to zero. The hidden-state signals that reached $\\pm 0.5$ keep between $0$ and $0.35$. What held up best was attention: after removing length, the ten strongest signals are all attention measures, led by mean entropy in a late layer at $+0.44$. None of them, adjusted or not, comes anywhere near turn length itself.\n", "\n", "Every signal and every number is listed below." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "

All 259 internal signals against the importance ratings

\n", "
\n", "Turn length, for comparison\n", "ρ = +0.79\n", "
\n", "
Attention entropy · mean over heads · 16 signals · best per turn +0.36 · best with length removed +0.44
Signalρ per turnρ with lengthρ, length removedp (all 259)
Layer 3+0.24+0.02+0.360.99
Layer 7+0.21+0.04+0.291.00
Layer 11+0.12-0.05+0.271.00
Layer 15+0.16+0.02+0.251.00
Layer 19+0.02-0.19+0.301.00
Layer 23+0.17+0.07+0.201.00
Layer 27+0.08-0.04+0.181.00
Layer 31+0.08+0.00+0.141.00
Layer 35+0.09-0.01+0.151.00
Layer 39+0.11+0.01+0.161.00
Layer 43+0.22+0.11+0.211.00
Layer 47+0.08-0.13+0.301.00
Layer 51+0.06-0.20+0.361.00
Layer 55+0.24+0.02+0.380.99
Layer 59+0.34+0.09+0.440.72
Layer 63+0.36+0.18+0.360.61
Attention entropy · max over heads · 16 signals · best per turn +0.51 · best with length removed +0.38
Signalρ per turnρ with lengthρ, length removedp (all 259)
Layer 3+0.48+0.33+0.380.11
Layer 7+0.48+0.34+0.370.11
Layer 11+0.51+0.38+0.360.07
Layer 15+0.29+0.16+0.260.91
Layer 19+0.27+0.16+0.220.95
Layer 23+0.24+0.14+0.220.99
Layer 27+0.23+0.13+0.201.00
Layer 31+0.17+0.09+0.151.00
Layer 35+0.21+0.11+0.211.00
Layer 39+0.26+0.15+0.230.97
Layer 43+0.25+0.17+0.190.98
Layer 47+0.27+0.09+0.320.94
Layer 51+0.21+0.00+0.351.00
Layer 55+0.29+0.14+0.290.90
Layer 59+0.37+0.23+0.320.57
Layer 63+0.38+0.26+0.300.48
Attention entropy · min over heads · 16 signals · best per turn +0.44 · best with length removed +0.39
Signalρ per turnρ with lengthρ, length removedp (all 259)
Layer 3+0.12-0.10+0.331.00
Layer 7-0.16-0.36+0.211.00
Layer 11+0.02-0.10+0.161.00
Layer 15+0.04+0.09-0.051.00
Layer 19+0.14+0.28-0.141.00
Layer 23+0.27+0.14+0.270.95
Layer 27-0.04-0.24+0.251.00
Layer 31-0.00-0.08+0.101.00
Layer 35+0.03-0.08+0.151.00
Layer 39+0.02-0.07+0.131.00
Layer 43+0.18+0.06+0.221.00
Layer 47+0.09-0.15+0.341.00
Layer 51-0.27-0.52+0.260.94
Layer 55+0.10-0.14+0.351.00
Layer 59+0.06-0.22+0.391.00
Layer 63+0.44+0.34+0.290.25
Peak attention weight · 16 signals · best per turn +0.27 · best with length removed -0.40
Signalρ per turnρ with lengthρ, length removedp (all 259)
Layer 3+0.27+0.56-0.350.95
Layer 7+0.21+0.43-0.251.00
Layer 11+0.18+0.38-0.221.00
Layer 15-0.08+0.12-0.291.00
Layer 19+0.20+0.41-0.211.00
Layer 23-0.16-0.02-0.241.00
Layer 27+0.06+0.24-0.221.00
Layer 31+0.05+0.17-0.131.00
Layer 35+0.20+0.32-0.101.00
Layer 39+0.09+0.17-0.081.00
Layer 43+0.07+0.14-0.071.00
Layer 47+0.18+0.37-0.201.00
Layer 51+0.21+0.47-0.311.00
Layer 55+0.00+0.25-0.331.00
Layer 59-0.07+0.21-0.401.00
Layer 63-0.09+0.03-0.201.00
Hidden-state size · 64 signals · best per turn -0.53 · best with length removed -0.33
Signalρ per turnρ with lengthρ, length removedp (all 259)
Layer 0+0.15+0.36-0.231.00
Layer 1-0.05+0.07-0.171.00
Layer 2 ●-0.53-0.52-0.220.05
Layer 3-0.27-0.16-0.240.95
Layer 4-0.21-0.04-0.291.00
Layer 5-0.24-0.09-0.270.99
Layer 6-0.26-0.09-0.310.97
Layer 7-0.35-0.25-0.250.66
Layer 8-0.37-0.26-0.280.56
Layer 9-0.40-0.32-0.250.41
Layer 10-0.46-0.35-0.310.17
Layer 11-0.36-0.23-0.310.60
Layer 12-0.38-0.23-0.330.52
Layer 13-0.26-0.13-0.270.96
Layer 14-0.14-0.01-0.221.00
Layer 15-0.08+0.06-0.201.00
Layer 16-0.00+0.17-0.221.00
Layer 17+0.07+0.24-0.201.00
Layer 18-0.01+0.13-0.201.00
Layer 19+0.08+0.20-0.141.00
Layer 20+0.10+0.20-0.101.00
Layer 21+0.07+0.16-0.101.00
Layer 22-0.05-0.00-0.071.00
Layer 23-0.10-0.03-0.121.00
Layer 24-0.19-0.10-0.181.00
Layer 25-0.12-0.05-0.131.00
Layer 26-0.27-0.16-0.230.95
Layer 27-0.23-0.09-0.260.99
Layer 28-0.23-0.08-0.280.99
Layer 29-0.18+0.02-0.321.00
Layer 30-0.05+0.14-0.271.00
Layer 31+0.11+0.37-0.331.00
Layer 32+0.21+0.44-0.261.00
Layer 33+0.24+0.46-0.240.99
Layer 34+0.03+0.20-0.211.00
Layer 35+0.20+0.41-0.241.00
Layer 36+0.08+0.28-0.241.00
Layer 37+0.12+0.33-0.241.00
Layer 38+0.02+0.21-0.251.00
Layer 39+0.13+0.34-0.231.00
Layer 40+0.14+0.33-0.211.00
Layer 41+0.09+0.28-0.231.00
Layer 42+0.06+0.24-0.221.00
Layer 43+0.08+0.25-0.211.00
Layer 44+0.05+0.23-0.221.00
Layer 45+0.05+0.23-0.221.00
Layer 46+0.05+0.15-0.111.00
Layer 47+0.23+0.40-0.150.99
Layer 48+0.17+0.31-0.131.00
Layer 49+0.25+0.36-0.070.98
Layer 50-0.02-0.02-0.001.00
Layer 51+0.10+0.14-0.031.00
Layer 52+0.01+0.04-0.021.00
Layer 53+0.02+0.05-0.021.00
Layer 54+0.03+0.04-0.001.00
Layer 55-0.08-0.07-0.031.00
Layer 56-0.06-0.05-0.041.00
Layer 57-0.08-0.07-0.051.00
Layer 58+0.13+0.18-0.011.00
Layer 59+0.28+0.40-0.060.92
Layer 60+0.30+0.34+0.060.86
Layer 61+0.39+0.49-0.000.48
Layer 62+0.17+0.19+0.041.00
Layer 63+0.11+0.25-0.161.00
Hidden-state jump (raw distance) · 64 signals · best per turn -0.49 · best with length removed -0.35
Signalρ per turnρ with lengthρ, length removedp (all 259)
Layer 0+0.06+0.21-0.181.00
Layer 1-0.02+0.11-0.181.00
Layer 2-0.45-0.39-0.250.20
Layer 3-0.24-0.14-0.210.99
Layer 4-0.30-0.17-0.270.88
Layer 5-0.30-0.20-0.240.85
Layer 6-0.36-0.28-0.240.60
Layer 7-0.41-0.32-0.270.36
Layer 8-0.44-0.36-0.270.23
Layer 9-0.45-0.40-0.230.20
Layer 10-0.38-0.32-0.220.49
Layer 11-0.34-0.24-0.250.71
Layer 12-0.37-0.29-0.240.53
Layer 13-0.36-0.30-0.210.61
Layer 14-0.38-0.33-0.200.53
Layer 15-0.37-0.30-0.230.56
Layer 16-0.33-0.30-0.160.77
Layer 17-0.29-0.26-0.140.91
Layer 18-0.34-0.32-0.150.73
Layer 19-0.36-0.28-0.230.64
Layer 20-0.37-0.30-0.230.54
Layer 21-0.42-0.37-0.220.32
Layer 22-0.46-0.40-0.260.17
Layer 23-0.41-0.35-0.230.37
Layer 24-0.42-0.37-0.230.30
Layer 25-0.45-0.37-0.270.20
Layer 26-0.47-0.39-0.300.14
Layer 27-0.49-0.37-0.350.09
Layer 28-0.46-0.35-0.320.17
Layer 29-0.43-0.33-0.290.28
Layer 30-0.35-0.27-0.230.67
Layer 31-0.31-0.19-0.280.82
Layer 32-0.25-0.15-0.210.98
Layer 33-0.17-0.10-0.151.00
Layer 34-0.07+0.00-0.121.00
Layer 35-0.04+0.07-0.151.00
Layer 36+0.02+0.13-0.141.00
Layer 37+0.00+0.10-0.131.00
Layer 38-0.01+0.10-0.141.00
Layer 39-0.00+0.09-0.131.00
Layer 40+0.01+0.09-0.101.00
Layer 41+0.04+0.12-0.091.00
Layer 42+0.01+0.10-0.111.00
Layer 43+0.02+0.12-0.131.00
Layer 44+0.02+0.12-0.121.00
Layer 45+0.05+0.14-0.101.00
Layer 46+0.08+0.17-0.101.00
Layer 47+0.07+0.19-0.141.00
Layer 48+0.11+0.21-0.101.00
Layer 49+0.07+0.14-0.071.00
Layer 50+0.08+0.13-0.051.00
Layer 51+0.06+0.18-0.141.00
Layer 52+0.09+0.18-0.101.00
Layer 53+0.07+0.17-0.111.00
Layer 54+0.08+0.15-0.081.00
Layer 55+0.05+0.12-0.081.00
Layer 56+0.10+0.18-0.061.00
Layer 57+0.11+0.19-0.061.00
Layer 58+0.12+0.20-0.071.00
Layer 59+0.11+0.26-0.171.00
Layer 60+0.07+0.22-0.181.00
Layer 61+0.16+0.33-0.191.00
Layer 62+0.15+0.37-0.261.00
Layer 63+0.14+0.38-0.291.00
Hidden-state jump (z-score) · 64 signals · best per turn +0.51 · best with length removed +0.33
Signalρ per turnρ with lengthρ, length removedp (all 259)
Layer 0+0.19+0.20+0.061.00
Layer 1+0.14+0.11+0.091.00
Layer 2-0.06-0.11+0.041.00
Layer 3+0.03-0.03+0.091.00
Layer 4-0.06-0.12+0.051.00
Layer 5-0.09-0.16+0.061.00
Layer 6-0.08-0.16+0.091.00
Layer 7-0.12-0.18+0.051.00
Layer 8-0.10-0.15+0.031.00
Layer 9-0.10-0.17+0.061.00
Layer 10-0.01-0.06+0.071.00
Layer 11+0.04-0.02+0.101.00
Layer 12+0.04-0.02+0.091.00
Layer 13+0.06-0.03+0.141.00
Layer 14+0.04-0.07+0.151.00
Layer 15+0.03-0.07+0.141.00
Layer 16+0.04-0.09+0.181.00
Layer 17+0.08-0.06+0.221.00
Layer 18+0.09-0.04+0.191.00
Layer 19+0.02-0.08+0.141.00
Layer 20+0.01-0.06+0.101.00
Layer 21-0.03-0.10+0.081.00
Layer 22-0.06-0.12+0.061.00
Layer 23-0.00-0.06+0.081.00
Layer 24-0.03-0.09+0.061.00
Layer 25+0.00-0.04+0.061.00
Layer 26-0.00-0.04+0.051.00
Layer 27-0.05-0.08+0.021.00
Layer 28-0.03-0.06+0.041.00
Layer 29+0.01-0.05+0.091.00
Layer 30+0.02-0.05+0.101.00
Layer 31+0.13+0.06+0.131.00
Layer 32+0.12+0.03+0.161.00
Layer 33+0.14+0.04+0.191.00
Layer 34+0.28+0.18+0.230.92
Layer 35+0.30+0.23+0.210.86
Layer 36+0.35+0.27+0.240.66
Layer 37+0.35+0.27+0.240.66
Layer 38+0.35+0.30+0.200.65
Layer 39+0.35+0.30+0.200.67
Layer 40+0.37+0.31+0.220.54
Layer 41+0.38+0.32+0.220.51
Layer 42+0.35+0.31+0.180.65
Layer 43+0.31+0.30+0.130.82
Layer 44+0.38+0.33+0.200.53
Layer 45+0.43+0.36+0.260.26
Layer 46+0.43+0.35+0.260.27
Layer 47+0.46+0.40+0.250.17
Layer 48+0.48+0.40+0.290.13
Layer 49+0.48+0.37+0.330.12
Layer 50+0.47+0.38+0.300.14
Layer 51+0.50+0.49+0.220.08
Layer 52+0.51+0.49+0.230.07
Layer 53+0.45+0.43+0.180.21
Layer 54+0.41+0.36+0.230.33
Layer 55+0.39+0.35+0.200.43
Layer 56+0.40+0.36+0.200.39
Layer 57+0.40+0.36+0.210.38
Layer 58+0.39+0.36+0.190.46
Layer 59+0.41+0.43+0.130.34
Layer 60+0.42+0.45+0.120.31
Layer 61+0.47+0.56+0.050.14
Layer 62+0.51+0.65-0.010.06
Layer 63+0.40+0.60-0.160.40
Output distribution · 3 signals · best per turn -0.43 · best with length removed -0.09
Signalρ per turnρ with lengthρ, length removedp (all 259)
Output entropy-0.43-0.54+0.000.28
Top-token probability+0.36+0.52-0.090.59
Step-to-step shift (KL)+0.33+0.43-0.020.77
\n", "
Click a group to expand it. \"ρ per turn\" is the Spearman correlation between the signal, averaged within each turn, and the 1–5 ratings of the session's 39 turns. \"ρ with length\" shows how closely the signal tracks turn length. \"ρ, length removed\" is the partial correlation after accounting for length. \"p (all 259)\" is corrected for having tested all 259 signals: the share of 5,000 shuffled ratings in which the best of the 259 signals did at least this well. ● marks the one signal below 0.05. Bold values are the larger ones in each column.
\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Turn length, measured last" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Turn length entered the analysis late, as a check on whether entropy was secretly measuring how long each turn was. It wasn't: entropy and length are nearly unrelated ($\\rho = 0.09$). But length on its own, the number of tokens in a turn, correlated with the importance ratings at $\\rho = +0.79$, more than twice as strongly as attention entropy, and without any access to the model's internals." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "

Correlation with importance, per turn

\n", "\n", "0.20.40.60.8\n", "Turn length: ρ = +0.793Turn length+0.79Entropy + length (fitted): ρ = +0.515Entropy + length (fitted)+0.52Attention entropy: ρ = +0.338Attention entropy+0.34Entropy + TD error: ρ = +0.226Entropy + TD error+0.23TD error (magnitude): ρ = -0.034TD error (magnitude)-0.03Value estimate: ρ = -0.006Value estimate-0.01\n", "\n", "
Spearman ρ between each signal, averaged per turn, and the importance ratings of the 39 turns. Faded bars are indistinguishable from zero.
\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Combining entropy with length did not beat length alone. The way it failed is worth noting. A filter that keeps a fixed share of tokens, ranked by score, lets long turns win twice when length is part of the score: they score higher *and* contribute more tokens to the ranking. At a 50% budget, four turns took the entire budget between them, and two of those had been rated only 2 and 3.\n", "\n", "Entropy is not worthless. After accounting for length, attention measures are the internal signals with the most left over, and mean entropy is the strongest of them. But a signal that requires running the model yourself has to beat what can be read off the transcript for free, and this one did not." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The real mistake: predicting instead of separating" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looking back over the whole experiment, the mistake was not a particular signal or threshold. It was the framing. I had treated a transcript as a single stream of tokens and tried to predict, token by token, which ones mattered. A more useful question is what each token is *for*.\n", "\n", "Imagine the same hard task done two ways. In the first, a single agent works through it alone: it plans, reads files, runs commands, gets stuck, recovers, and finally reports what it did. In the second, an orchestrator coordinates a team of workers. It writes the plan, hands each subtask to a worker, hears back about problems, changes the plan when something fails, and reports the result. The orchestrator's context would contain nearly everything worth remembering about the task: the plan, each subtask named explicitly, what went wrong, and every change of direction. The workers' contexts would hold the implementation details, and most of those can be dropped.\n", "\n", "A single-agent transcript can be read the same way, if each token is sorted by the role it plays. Every token in an agent session is one of five things:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "

Sorting a transcript by role

\n", "
User → agentkeepThe request and every correction. The most important tokens in the session; keep all of them.
Agent reasoningdropWhere the model works things out. Whatever it concludes, it tells the user.
Agent → tooldrop*Raw tool calls: file paths, search queries, commands. *Except edits — see below.
Tool → agentdropFile contents, logs, console output. Read once, used, never needed again.
Agent → userkeepWhere the agent reports what it found, decided, and changed.
\n", "
In orchestrator terms, user input and the agent's messages to the user are the orchestrator's context. Reasoning and tool traffic are the workers'.
\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The argument for dropping reasoning is a first-principles one, and I am willing to make it: whatever a model works out while reasoning, it reports to the user if it matters. One run tested this hard. A Claude Code subagent's saved transcript contained 15 reasoning blocks, every one of them empty (just an encrypted signature, no readable text), yet its final message was still a precise, correct account of the bug with file and line references.\n", "\n", "The argument for dropping tool traffic is the size of it. I gave one real debugging task (find where a plugin loader silently swallows errors, and fix the worst case) to four agents: three different models running in opencode, and a Claude Code subagent. All four found and fixed the same bug." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "

Share of each session that is tool output

\n", "
\n", "tool output\n", "everything else\n", "
\n", "
muse-spark-1.3
91.1%
prompt + final message only: 95.7% removed
GLM-5.3
83.0%
prompt + final message only: 96.2% removed
Claude subagent
82.1%
prompt + final message only: 91.0% removed
DeepSeek-v4-pro
93.3%
prompt + final message only: no final message
\n", "
Four agents, one task, the same fix. Token counts use a single tokenizer across all four transcripts.
\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Tool output was 82–93% of every session. Keeping only the user's prompt and the agent's final message removed 91–96% of the tokens, and in each case that message described the fix correctly. That measurement is the strictest version of the rule. Keeping every message to the user, as the table above suggests, would keep more.\n", "\n", "The same test showed where the rule needs care. DeepSeek made the correct fix, then stalled on a permission prompt that was automatically denied because no one was there to approve it, and ran out of steps without ever writing a final message. A rule that keeps only what is printed would have discarded its fix entirely. And in two of the other sessions, the final message described the fix in prose without including the code, so the actual change existed only in the arguments of an edit call. Edits are the one kind of tool traffic that has to be kept.\n", "\n", "Within the agent's messages, two further filters seem natural, though I have not measured either. The last message before the user replies is probably important, because it is either a natural stopping point or the thing that made the user step in. And one- or two-line progress messages (\"Confirmed this is the pre-fix version. Applying the same fix…\") reassure the user while the agent works but carry little that is worth keeping, so a simple length cut would remove them.\n", "\n", "This approach also does not depend on running the model. Attention entropy and a value head both require open weights and running inference yourself. Most capable models are closed, and even open ones are usually served by someone else. Sorting by role works on any transcript from any harness, and when it goes wrong, you can see why by reading what it kept.\n", "\n", "One prediction of mine did not hold up, and it is worth recording. I had guessed that the agent's final summary would have *low* attention entropy, since by then the model has settled what happened and is only presenting it. In the completed session, the final summary had the third-highest entropy of all 39 turns. The case for sorting by role does not rest on entropy either way." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What this adds up to" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To summarize what I found:\n", "\n", "- **The local mark worked, but backwards.** I expected tokens produced with sharply focused attention to be the important ones. Instead they were mostly repetition and boilerplate, and the important moments were where attention was spread out.\n", "- **The surprise signal didn't work.** The success estimate I trained carried no information about which moments mattered, most likely because I trained it on a different kind of task.\n", "- **A plain count beat everything inside the model.** I recorded 259 internal measurements in total. None came close to simply counting how many tokens each turn contained, and many of the promising-looking ones were partly measuring exactly that.\n", "- **The bigger mistake was the approach.** Trying to predict which tokens matter from numbers turned out to be less useful than sorting tokens by what they are *for*.\n", "\n", "The sample is small: one complete agent session, with importance judged by a single model, not by people. The numbers indicate a direction; they are not a benchmark." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The neuroscience made a good starting point, and part of the analogy held up: a cheap, per-token signal computed during generation does carry information about which moments matter, just in the opposite direction from the biology, and less of it than the simplest statistic in the transcript.\n", "\n", "What worked better was an argument rather than a signal. A transcript is not a single stream of equally uncertain tokens: it has structure, and most of that structure is labelled. Who said something, and to whom, says more about whether it is worth keeping than anything I could measure inside the model. The finding I would bet on is that sorting by role, asking what each token is for, is a better question than predicting importance token by token." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Notes on method" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Model.** Qwen3.6-27B in bf16 on a single H200, September 2026. Attention weights require eager attention, since fused kernels never materialise the attention matrix. Of the model's 64 layers, 16 use softmax attention. The other 48 are Gated DeltaNet linear-attention layers, so attention signals come from the 16 softmax layers only, while hidden-state signals cover all 64.\n", "\n", "**Sessions.** Three real bugs from my own repositories, each checked out at the commit before its fix. The first run used a minimal custom agent loop and solved none of them. The second used vLLM to serve the model and opencode as the harness. Only one of those three sessions completed: in the other two, large file reads overflowed the 32,768-token context limit the server had been started with, opencode compacted the session, and the agent stopped before making an edit. All correlations come from the completed session.\n", "\n", "**Replay.** The completed session was exported from opencode's session database and teacher-forced back through the instrumented model. Only opencode's printed `text` parts were replayed and scored, not reasoning or tool calls. opencode does not store its system prompt and stores tool calls already parsed, so the replay only approximates the original context: the model's top prediction matched the recorded token 67–72% of the time.\n", "\n", "**Ratings.** Each turn was rated 1–5 by Claude, with a written justification per turn, in a single pass over the full transcript. All correlations are Spearman's $\\rho$ over turn-level averages unless stated otherwise. Per-token correlations run higher for entropy (about $+0.46$), but are not comparable with per-turn figures such as turn length's.\n", "\n", "**Corrections.** For attention entropy alone, picking the best of the 16 softmax layers and correcting for that choice gave $p = 0.047$ (20,000 shuffles). Over all 259 signals, the family-wise 95% threshold was $|\\rho| = 0.52$ (5,000 shuffles). Partial correlations remove a linear fit on rank-transformed turn length from both the signal and the ratings.\n", "\n", "**First-run tag.** The bottom quartile of attention entropy per run, combined with hidden-state displacement spikes ($|z| \\geq 2.5$ against a 200-token trailing window) within ±100 tokens. That window covered 91–98% of each transcript, so in practice the tag alone decided what was flagged.\n", "\n", "**Token counts.** `cl100k_base` applied to all four transcripts in the four-agent comparison, as a consistent proxy rather than billed tokens." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.6" } }, "nbformat": 4, "nbformat_minor": 5 }