Shreshth Rajan, June 2026. ยท Code and tool: github.com/ShreshthRajan/SRAG
A self-improvement loop makes its own training signal. It samples answers, scores them by some proxy for quality, and trains on the score. When labels are unavailable the proxy is usually the model's own agreement across samples, which is how test-time RL [1] and self-rewarding training [2] work. That reward has an exploit anyone can see in hindsight: the model can score a perfect 1.0 by agreeing on a single answer, right or wrong. The question is not whether it takes the exploit. It is whether you would notice. I ran the loop and watched the four quantities I would normally trust. Three of them stayed green for the entire run, while the model's real ability fell by half.
The claim of this post is that the blindness is structural, not bad luck. The collapse is an event inside each prompt, and the metrics in standard use are averages over prompts, so they cannot resolve it by construction. The argument is short and it is information-theoretic: pooled diversity measures the wrong entropy. Once that is clear, the fix is obvious, the diagnostic is free, and it arrives early enough to be a control rather than a postmortem.
The loop is GRPO [10] on coding problems with Qwen2.5-Coder-1.5B. For each prompt \(x\) it samples \(G=8\) completions \(y_1,\dots,y_G \sim \pi_\theta(\cdot\mid x)\). The only change from a normal run is the reward. Strip the unit tests and score each completion by how many of the others match it after the code is reduced to a canonical form \(c(\cdot)\):
$$r(y_i) \;=\; \frac{1}{G-1}\sum_{j\neq i}\mathbf{1}\!\left[\,c(y_i)=c(y_j)\,\right].$$GRPO then standardizes the rewards within the group and ascends the policy-gradient surface,
$$A_i \;=\; \frac{r_i - \bar r}{\sigma_r + \varepsilon}, \qquad \nabla_\theta J \;=\; \mathbb{E}\!\left[\,A_i\,\nabla_\theta \log \pi_\theta(y_i\mid x)\,\right].$$Two facts about this objective decide everything that follows. First, the reward is globally maximized, every \(r_i=1\), exactly when all \(G\) completions share a canonical form: unanimous agreement is the optimum, and it says nothing about correctness. Second, at that point the within-group spread \(\sigma_r\) is zero, so every advantage \(A_i\) is zero and the gradient vanishes. The unanimous state is therefore an absorbing fixed point. The objective points toward it, and once the policy arrives the update that might carry it back out is identically zero. This is the same zero-variance degeneracy that DAPO removes by resampling [6] and that Cui et al. tie to entropy collapse [7]; here it is not a nuisance to be patched but the attractor the reward was always going to find.
# the reward, and the gate the gradient sees
def consensus_rewards(samples): # G completions for one prompt
key = [canonical(s) for s in samples] # AST-canonical form; ignores formatting
n = Counter(key)
return [(n[k] - 1) / (len(samples) - 1) for k in key]
adv = (r - r.mean()) / (r.std() + 1e-6) # r all-equal -> std 0 -> adv 0 -> no gradientThe reward behaves as predicted: it climbs to 1.0 by driving each prompt's eight completions to one answer. The cost shows up only when you score the model against the world. Held-out, with real unit tests, pass@1 holds at 0.31 and pass@8 falls from 0.91 to 0.46.
The split between pass@1 and pass@k is the signature of coverage collapse, and it is exactly what the unbiased pass@k estimator [11] measures. With \(c\) of \(n\) sampled completions correct,
$$\widehat{\text{pass@}k} \;=\; 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}},$$so pass@1 \(=c/n\) tracks the correctness of the typical sample, while pass@k for \(k>1\) is large only when the correct answers are spread across distinct completions. Concentrate all the mass on one answer and pass@1 is unchanged while pass@k decays toward it. That RLVR shrinks pass@k while pass@1 holds is the central finding of Yue et al. [3] and the Invisible Leash paper [4]; the self-consensus reward is a way to induce it on command. That the cause is self-grading and nothing else is one control away: restore the unit-test reward, run the same loop on the same problems, and pass@k stays flat, which is what the proof that verification blocks recursive collapse predicts [5].
Now the actual point. Three of the four standard signals, the reward, pass@1, and pooled output diversity, register nothing, and the reason is the same for all three: they are averages over prompts, and the collapse is not.
Take diversity. Let \(X\) be the prompt and \(Y\) a sampled answer. Pooled diversity, the quantity people log, pours every completion from every prompt into one bag and measures its entropy \(H(Y)\). The chain rule of entropy [12] splits that exactly:
$$H(Y) \;=\; \underbrace{H(Y\mid X)}_{\text{within-prompt}} \;+\; \underbrace{I(Y;X)}_{\text{between-prompt}}.$$The first term is how varied the answers are for a fixed problem, averaged over problems. The second is how much an answer reveals about which problem it came from, that is, how different one problem's answers are from another's. The collapse drives \(H(Y\mid X)\to 0\): within each problem, eight answers become one. But \(I(Y;X)\) does not move, because a sort routine and a string parser are still nothing alike. Pooled diversity is the sum, so it stays pinned near \(I(Y;X)\), which is most of it. The dashboard's diversity number is blind not because it is noisy but because it is measuring \(H(Y)\) when the failure lives entirely in \(H(Y\mid X)\). pass@1 fails for the same reason in the correctness coordinate: it reports the modal sample, a within-prompt summary statistic the collapse preserves.
The cure is to measure the term that moves. Group the completions by prompt before you take the entropy, and average:
pooled = entropy(Counter(canonical(s) for g in groups for s in g)) # H(Y), blind
perprompt = mean(entropy(Counter(canonical(s) for s in g)) for g in groups) # H(Y|X), sees it
# pooled - perprompt is I(Y;X); it is large and ~constant, which is the whole problemNothing here costs more than the rollouts you already have. The only change is reading them grouped rather than in a bag.
Because the within-prompt term is the one that collapses, it also collapses first. In the run, per-prompt entropy turns over within the first handful of steps, while held-out pass@8 is still near 0.91; pooled diversity does not register anything for roughly twenty more. Stop at the first per-prompt warning and the coverage is still there; let the loop run and half of it is gone. The variance gate gives a second trigger for free and even earlier, since \(\sigma_r \to 0\) is the same event the advantage normalization already computes every step. Either one converts the collapse from something you discover at evaluation into something you halt during training.
The pieces are individually known. Self-consensus training collapses to a single answer [1, 2, 9], the label-free case of the model collapse that recursive training on generated data has been shown to produce [13]; RLVR trades pass@k for pass@1 [3, 4]; the zero-variance group and entropy collapse are named and analyzed [6, 7]; the divergence to a reference sets the rate [8]. The argument that is not already in the literature is the measurement one. The reason this failure ships unseen is the entropy decomposition: every aggregate in standard use estimates \(H(Y)\) or the modal sample, both dominated by between-prompt structure the collapse leaves intact, so the dashboard is blind by construction rather than by accident. The same decomposition names the fix, the per-prompt term \(H(Y\mid X)\), which is free to compute and leads the held-out drop. It is a measurement result and a monitor, not a new training method, and it is sharper for being small.
This is one controlled run on a 1.5B code model, scored on eleven held-out problems. The effect is large and one-directional, the reward saturates, \(H(Y\mid X)\) goes to zero, pass@8 halves, and the aggregates do not move, so the qualitative claim is not in question; the precise lead time is, because pass@8 on eleven problems is noisy step to step. The decomposition makes a falsifiable prediction that does not depend on that noise. Across any setting that pushes the loop harder, higher learning rate, weaker anchor, the within-prompt term \(H(Y\mid X)\) and the between-prompt term \(I(Y;X)\) should move in opposite ways: the first tracks the collapse, the second stays put, and pooled diversity stays put with it. If a run drops pooled diversity as much as per-prompt diversity, the decomposition is wrong and the blindness is not structural after all. If you run label-free self-improvement and trust aggregate metrics, this is the failure you will ship without seeing it, and I would like to know where measuring \(H(Y\mid X)\) does not catch it.
Setup. GRPO on MBPP with Qwen2.5-Coder-1.5B, LoRA rank 128, eight samples per prompt, held-out pass@k scored by real unit tests at temperature 0.6. The no-anchor reward is soft self-consensus over AST-canonicalized code; the control reward is unit-test pass-rate; the two are run identically otherwise. Figures are the logged trajectory at the evaluation cadence. The monitor and the experiment, with one command to reproduce, are at github.com/ShreshthRajan/SRAG.