If at First You Don't Succeed, Backtrack: Lightweight Recovery for Frozen VLAs

Summary

Vision-Language-Action (VLA) models perform well on robotic manipulation, yet remain brittle once their own actions carry them into states their training data barely covers.

Abstract Summary

Vision-Language-Action (VLA) models perform well on robotic manipulation, yet remain brittle once their own actions carry them into states their training data barely covers. Recovering from such failures usually requires recovery demonstrations, policy retraining, or a separate model at inference time. We introduce Backtrack, a lightweight failure detection and recovery mechanism for frozen VLAs that needs none of these. A small failure prediction head, trained only on outcome-labeled rollouts of the frozen policy, reads the VLA's existing internal representations to estimate whether the current episode will fail. When the predicted failure probability exceeds a threshold, a controller pauses the policy, retraces the arm's recently achieved motion, and hands control back to the unchanged policy so it can retry from an earlier configuration. The head adds only 0.48-0.88% to the policy's parameter count. We evaluate Backtrack on LIBERO with three VLAs spanning two action-generation families: SmolVLA and $π$0.5, which use flow-matching action experts, and the autoregressive $π$0-FAST. On LIBERO-Long, the suite of long-horizon multi-step tasks and the one on which all three policies fail most often, Backtrack reduces each policy's failure rate by 19-48%, raising success by up to 7.4 percentage points; on the three shorter suites, where control success is already 85-99%, its effect is smaller. On MetaWorld, all twelve controller configurations tested with SmolVLA improve on the unassisted policy.

Key Points

Vision-Language-Action (VLA) models perform well on robotic manipulation, yet remain brittle once their own actions carry them into states their training data barely covers.
Recovering from such failures usually requires recovery demonstrations, policy retraining, or a separate model at inference time.
We introduce Backtrack, a lightweight failure detection and recovery mechanism for frozen VLAs that needs none of these.
A small failure prediction head, trained only on outcome-labeled rollouts of the frozen policy, reads the VLA's existing internal representations to estimate whether the current episode will fail.
When the predicted failure probability exceeds a threshold, a controller pauses the policy, retraces the arm's recently achieved motion, and hands control back to the unchanged policy so it can retry from an earlier configuration.

Method

We attach a Failure Prediction Head (FPH) to the frozen policy. The FPH is a multi-head attention pooling (MAP) layer over the policy’s own token features, followed by a three-layer feed-forward network (FFN) that outputs a single scalar: the predicted probability that the current episode will end in failure.

The FPH consumes activations the policy already computes during normal action generation, so it requires no additional forward pass through the backbone.

Method — If at First You Don't Succeed, Backtrack: Lightweight Recovery for Frozen VLAs — Figure 1: Overview of Backtrack. A small failure prediction head reads features already computed by the frozen VLA and estimates the probability that the current episode will fail. When the predicted failure probability exceeds the threshold $t$, the Backtrack controller suppresses the policy’s proposed actions and instead retraces recent achieved arm motion stored in a buffer. Otherwise, the policy’s actions are executed normally. The resulting state is observed by the unchanged VLA and the control loop continues. Achieved motion is added to the buffer during normal policy execution; motion executed during backtracking is not. The diagram abstracts over differences in action generation between the evaluated VLAs.

Experimental Setup

We evaluate primarily on LIBERO ( Liu et al., 2023a ) , using its four ten-task suites: LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long. To check that the method is not specific to one benchmark we additionally evaluate on MetaWorld ( Yu et al., 2020 ) , a separate manipulation benchmark with a different simulator, task set, and embodiment. The MetaWorld experiments use SmolVLA with the pre feature tap.

Results

Table 3 reports the main result. On LIBERO-Long, Backtrack reduces the failure rates of SmolVLA, $\pi$0.5 and $\pi$0-FAST by 19.0%, 30.3% and 48.2%, respectively. Every 95% confidence interval excludes zero. Aggregated over all four LIBERO suites, the three policies sit much closer together, at 20.3%, 26.2% and 19.6%, again with every interval excluding zero.

Sources and Demonstrations

Figures are reproduced from Landeg et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.