Does a Learned Corrector Beat a Simple Retreat? Evidence from a Frozen VLA

Chenchao Sheng, Zhuang Jiang, Liuhaichen Yang, Ningwei Bai, Zezhi Tang · N/A · 2026

Summary

Abstract Summary

Before deploying runtime recovery for a frozen vision-language-action (VLA) policy, one must establish that an intervention improves success beyond ordinary run-to-run variation and that its complexity adds value over a simple action. We evaluate these questions on frozen $\pi_{0.5}$ across four RoboTwin tasks. For each test seed, we pair rollouts with and without correction and include a same-seed base-policy re-run as a placebo. Seed-cluster intervals and prespecified comparison rules assess net gains against stochastic outcome changes. Across 3,888 paired episodes, the full pipeline raises success on beat_allowbreak block_allowbreak hammer by $+13.5$\,pp (95\% interval $[+9.4,+17.7]$), with no detectable gain on the other three tasks at the deployed weight. Among failed base episodes on the responsive task, $43.2\%$ succeed on a plain re-run, compared with $63.5\%$ after correction; many nominal rescues therefore reflect the base policy's own variability. A fixed-time trigger and scripted return to an earlier joint configuration produce a net gain with no detected difference from the learned pipeline across two rounds, although our prespecified equivalence criterion is not met consistently. Pausing and a constant-action control do not yield comparable gains. On this benchmark, the decision to intervene depends strongly on the task, and a paired placebo plus a simple retreat baseline are needed to establish what learned correction contributes.

Key Points

Before deploying runtime recovery for a frozen vision-language-action (VLA) policy, one must establish that an intervention improves success beyond ordinary run-to-run variation and that its complexity adds value over a simple action.
We evaluate these questions on frozen $\pi_{0.5}$ across four RoboTwin tasks
For each test seed, we pair rollouts with and without correction and include a same-seed base-policy re-run as a placebo.
Seed-cluster intervals and prespecified comparison rules assess net gains against stochastic outcome changes.
Across 3,888 paired episodes, the full pipeline raises success on beat_allowbreak block_allowbreak hammer by $+13.5$\,pp (95\% interval $[+9.4,+17.7]$), with no detectable gain on the other three tasks at the deployed weight.

Method

Both triggers share one task-specific quantity, a time threshold $T=\max(100,\lfloor 1.2\,p_{75}\rfloor)$, where $p_{75}$ is the 75th percentile of the step at which successful base rollouts finish on training seeds. This gives 148, 177, 177 and 183 steps on bbh/qr/a2b/a2br. Slash lists follow this order throughout.

The monitor adds five hand-set constants shared by all tasks (velocity ratio, window, progress fraction, warm-up, earliest snapshot step) and fires when motion inside the window collapses after the threshold.

Method — Does a Learned Corrector Beat a Simple Retreat? Evidence from a Frozen VLA — Figure 2 : The deployed corrector, and the ladder on bbh. Filled markers exclude zero throughout. (a) Net change in success rate per task and pooled, with each task’s base rate $b$. (b) Placebo-corrected rescue and breakage. The upper line of each pair, marked $\rho$, is $\rho-\rho_{0}$; the lower, marked $\beta$, is $\beta-\beta_{0}$. The two rates have different denominators and do not add. The privileged-planner row is blank here because that condition has no same-seed placebo, and it carries no interval in (c) because its per-episode pairs were not saved. (c) The ladder: six interventions sharing one trigger threshold, with 95% seed-cluster bootstrap intervals where available. The three retreat and pipeline rows run on 972 pairs, the rest on 324.

Experiments and Results

Each test seed fixes the scene, and we run it with the base policy alone and with the intervention. Only discordant pairs (where the two runs disagree) contribute to the net difference. We test them with McNemar’s test (the test for paired yes/no outcomes) and report the net effect as the difference in success rate in percentage points (pp).

Each seed contributes three such pairs, and McNemar’s test treats the repeats as independent, so its $p$-values may be anticonservative.

Experiments and Results — Does a Learned Corrector Beat a Simple Retreat? Evidence from a Frozen VLA — Figure 3 : Replication round: four conditions on bbh, 972 pairs each, on one GPU in one time window. (a) Net change in success rate, with each arm’s base rate $b$; filled markers exclude zero at 95%. (b) Pairwise differences, five within the round (90% intervals, upper group) and three between rounds (95%, lower group), against a shaded $\pm 5$ pp margin; pale red marks “no difference detected”, and the one dark marker is the pair of scripted conditions, the only comparison here that is equivalent. Every one of the eight contains zero, and the between-round net effects differ by at most 1.5 pp.

Sources and Demonstrations

Figures are reproduced from Sheng et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.