Shreshth Rajan, June 2026.
A vision-language-action policy acts by sampling. Given an observation it proposes an action chunk, a short sequence of moves, from a distribution it learned by imitation. Test-time selection is the idea that you can do better by drawing several chunks and keeping the best, the way you sample many solutions from a language model and keep the one that passes the tests [1]. The robot version swaps the unit test for a learned verifier that scores candidates [2, 3, 4] or a world model that rolls them forward [5]; one recent result argues that scaling this verification beats scaling the policy [6].
Two quantities decide whether this works, and neither is usually measured. How much success is there to gain by selecting the best of \(N\) candidates, and how much of it can a learned model actually capture. I measured both by deleting the second unknown. In simulation the model can be made exact: snapshot the state, commit a candidate's actions, run the policy to the end of the episode, and read the true outcome. Used this way the simulator is at once a perfect world model and a perfect verifier, and it gives the ceiling that every learned selector works beneath.
Fix a state \(s\) where the policy is about to act and draw \(N\) candidates \(a_1, \dots, a_N \sim \pi(\cdot \mid s)\). For each one, restore the simulator to \(s\), commit the chunk, continue under \(\pi\) until the episode ends, and record the true outcome \(v(s, a_i) \in \{0,1\}\). The quantity selection is chasing is
$$\text{headroom} \;=\; \underbrace{\mathbb{E}\big[\textstyle\max_i v(s, a_i)\big]}_{\text{oracle: the ceiling}} \;-\; \underbrace{\mathbb{E}\big[v(s, a_1)\big]}_{\text{best-of-1: the policy acting once}}.$$Four selectors see the same candidates from the same states:
# v[i] is candidate i's TRUE simulated outcome in {0,1}; all selectors share the draw at state s
oracle = v.max() # ceiling: the best achievable outcome
best_of_1 = v[0] # the policy acting once
coupled = v[argmin(dist_to_mean(chunks))] # self-consistency: candidate nearest the policy mean
decoupled = v[verifier(image, chunks).argmax()] # an independent learned verifier's pickThe verifier is a frozen DINOv2 encoder, kept separate from the policy's own vision tower, with a two-layer Transformer over the chunk, trained on the true outcomes to predict success and held out by whole decisions. The policy is a frozen SmolVLA-450M on LIBERO-Object. The difficulty knob is the open-loop horizon, the number of actions committed before replanning; raising it drops per-attempt success to 0.23 from the native 0.87, because selection has nothing to do when the policy already succeeds.
Perfect selection succeeds 0.51 of the time against 0.23 for a single sample: a headroom of 0.28, more than a doubling. It is not monotone in how broken the policy is.
The headroom peaks where the policy is wrong but still lands the task on some draws, and falls to zero at both ends: a competent policy leaves nothing to select, a broken one leaves no success to find. The band where selection helps at all is narrow, and it sits exactly where the policy is least trustworthy.
The ceiling is real. Whether a learned selector reaches it is a separate question, and the answer is no. In all three regimes the independent verifier fails to beat self-consistency, every out-of-fold difference inside the noise; the only selector that beats a random pick is self-consistency itself, the candidate nearest the policy mean.
The verifier is not weak. Trained on true outcomes it predicts episode success at AUROC 0.88, stable across regimes (0.88 in distribution, 0.86 under mild shift). It learns the wrong quantity. Decompose each outcome into the value and advantage familiar from reinforcement learning,
$$v(s, a_i) \;=\; \underbrace{V(s)}_{\text{state value}} \;+\; \underbrace{\delta_i}_{\text{advantage of candidate } i}, \qquad V(s) \;=\; \mathbb{E}_{a \sim \pi}\big[v(s, a)\big].$$Predicting success from the image is predicting \(V(s)\), and \(V(s)\) is the same for every candidate at a decision, so it sorts none of them. Selection lives entirely in \(\delta_i\). The verifier does take the chunk as input, so in principle it could rank on \(\delta_i\); empirically it cannot, because the candidates are independent draws around the policy mean and which one succeeds is not predictable from the chunk. Its within-decision ranking is at chance. The most any image-and-chunk model can recover is \(V(s)\), while the oracle ranks by the true \(v\) the simulator hands it. That gap is the result: not a model too weak to help, but a candidate set with no learnable structure to separate.
The high AUROC could be an artifact of the shifted regime, where a web-scale encoder stays robust while the policy breaks. It is not. In distribution, with true-outcome labels, the verifier still reaches 0.88; an earlier 0.54 came from labeling candidates by distance to the expert action rather than by outcome. Hold the regime fixed, change only the label, and the signal returns. State value is learnable everywhere; the advantage is learnable nowhere here.
The limit is the policy, not the model. With a perfect world model in hand, selection recovers only what the policy's own consensus already knew, so a better model cannot raise the ceiling. The lever is the proposal distribution. Selection helps only when candidates differ in ways the observation predicts, so that a better-than-mean action exists and is identifiable; independent draws around one mean lack that structure for any verifier. This matches Fast-WAM [7], which finds world-action models gain little from test-time imagination, and it names the reason: there is no within-decision structure to select on. It fits the concentration of verifier gains out of distribution [2, 6], the wrong-but-recoverable band, and the dependence of verification difficulty on the generator [8].
One use survives. A model that calls success at AUROC 0.88 is a cheap, accurate failure detector. The job for an independent observation model is anticipating failure and intervening, not reranking actions that differ only by noise.
I have not changed the proposal distribution or scaled the verifier past 450M, so the claim is unreachability at accessible scale, not in principle. What the perfect instrument settles is that the gain is bounded by the candidate set, not by model fidelity.
Setup. Frozen SmolVLA-450M on LIBERO-Object, six candidates per decision, each rolled to termination through the simulator for its true outcome. Open-loop horizon sets difficulty; a per-episode photometric shift sweeps competence. Verifier: frozen DINOv2 plus a two-layer Transformer over the chunk, held out by whole decisions, reported by AUROC and out-of-fold selection with paired bootstrap intervals. One policy, one suite, simulation, fifteen informative decisions in the heavy-shift cell, so this is an upper bound rather than a survey. Code and the outcome banks, with every number reproducible on CPU: github.com/ShreshthRajan/roboticstt. If you can make an independent verifier beat self-consistency on this protocol, I would like to see it.