Shreshth Rajan, June 2026. · Code and tool: SRAG
If output diversity is the number you watch, it won't show you a model collapsing.
With no answer key you can still train a model on its own output. Sample eight answers and reward each one for matching the other seven. Test-time RL works roughly this way.
The cheat is obvious: give the same answer eight times and you score 100%, right or wrong. I wanted to know whether you'd see it happen.
I ran GRPO on coding problems with Qwen2.5-Coder-1.5B. Eight answers per problem, and instead of running the tests I scored each answer by how many of the other seven matched it.
That reward is highest when all eight agree, so agreement becomes the goal and correctness never enters into it. GRPO also scales each update by how spread out the rewards are, so once the eight agree the spread is zero, the gradient vanishes, and the model stays where it landed.
# the reward, and the gate the gradient sees
def consensus_rewards(samples): # eight answers to one problem
key = [canonical(s) for s in samples] # ignores formatting differences
n = Counter(key)
return [(n[k] - 1) / (len(samples) - 1) for k in key]
adv = (r - r.mean()) / (r.std() + 1e-6) # all rewards equal -> std 0 -> no gradientReward climbed to 1.0. On held-out problems with real tests, the first answer was still right 31% of the time. But the chance that at least one of eight answers was right fell from 91% to 46%.
The model hadn't gotten worse at coding. It had stopped producing different answers, so its eight tries were now one answer repeated eight times.
I was logging output diversity the whole run and it never moved.
That number throws every answer to every problem into one pile and measures the variety of the pile. The total is two things added together: how much the eight answers to one problem differ from each other, and how much one problem's answers differ from another problem's.
$$H(Y) \;=\; \underbrace{H(Y\mid X)}_{\text{within a problem}} \;+\; \underbrace{I(Y;X)}_{\text{between problems}}$$Collapse only kills the first part. A sorting function and a string parser still look nothing alike. And the second part is most of the total, so the number you're watching barely moves. Variety within a problem went from 0.97 to zero. Pooled variety sat at 0.78 the whole time.
The number isn't noisy. It's measuring the wrong half of the total, and a single-best-answer score fails for the same reason, since the best answer is the one thing collapse leaves alone.
The fix costs nothing, because you already have the samples. Group the answers by problem, measure variety inside each group, and average.
pooled = entropy(Counter(canonical(s) for g in groups for s in g)) # blind
perprompt = mean(entropy(Counter(canonical(s) for s in g)) for g in groups) # sees itIt warns you early, too. Variety within a problem turned over in the first few steps, while the held-out eight-answer score was still at 91%. Pooled variety showed nothing for another twenty.
One correction I owe. Of the four things I was watching, one did catch it: reward spread going to zero is the same event, and GRPO computes it every step anyway. The dashboard wasn't blind. The diversity number was the one reading that missed it.
If you train a model on its own agreement, log variety per problem rather than variety overall.
Setup. GRPO on MBPP with Qwen2.5-Coder-1.5B, LoRA rank 128, eight samples per prompt, held-out pass@k scored by real unit tests at temperature 0.6. The no-anchor reward is soft self-consensus over AST-canonicalized code; the control reward is unit-test pass rate; the runs are otherwise identical. Restoring the test-based reward keeps pass@k flat, so the self-grading is the whole cause. One run, eleven held-out problems: the effect is large and one-directional, but the exact warning time isn't reliable, since eleven problems jump around step to step. If pooled variety ever falls as much as per-problem variety, I'm wrong. Monitor and experiment, one command to reproduce: github.com/ShreshthRajan/SRAG.