Improving Robotic Generalist Policies via Flow Reversal Steering

Andy Tang, William Chen, Andrew Wagenmaker, Chelsea Finn, Sergey Levine · N/A · 2026

Summary

Abstract Summary

Generalist policies can learn a wide range of skills from diverse robot datasets. In order to solve or improve on challenging news tasks, we need a way to infer and invoke the appropriate actions from the policy's rich behavioral prior, especially when directly commanding the policy fails. We focus on flow matching generalists and propose Flow Reversal Steering (FRS): a method that takes suboptimal but reasonable actions, finds their latent noises by passing them through the flow policy in reverse, and maps them to nearby generalist action modes. We evaluate FRS across many simulated and real-world manipulation settings. First, FRS can turn coarse semantic guidance from humans or vision-language models (VLMs) into corresponding good robot actions, improving zero-shot control. These gains can be distilled with behavioral cloning by training an auxiliary policy to output noises that the generalist maps to good actions -- showing up to 95% absolute task success rate boosts in under a minute of training. Finally, FRS enables policy improvement by bootstrapping reinforcement learning with semantic knowledge, improving on several tasks that standard RL fails to improve on.

Key Points

Proposes Flow Reversal Steering (FRS) to navigate flow-matching generalist policy latent spaces.
Converts suboptimal actions into latent noises and maps them to better generalist action modes.
Enables zero-shot control from semantic guidance and achieves up to 95% success rate boosts with minimal distillation.
Bootstraps RL with semantic knowledge for policy improvement on tasks where standard RL fails.

Preliminaries

Flow matching VLAs. Generalist policies take a pretrained backbone and fine-tunes it with behavioral cloning (BC) to match the action distribution of experts.

Given BC data $(o,a_{1})$ and sampled times $t\in[0,1]$, these policies fit a velocity field $v_{\theta}(a_{t},t\mid o)$ that maps noise $a_{0}\sim\mathcal{N}(0,I)$ to actions $a_{1}$ by denoising partially-noised actions produced by forward diffusion $a_{t}=t\cdot a_{1}+(1-t)\cdot a_{0}$.

Preliminaries — Improving Robotic Generalist Policies via Flow Reversal Steering — Figure 2: Overview of FRS. (1) A human or VLM semantically reasons about the novel task to determine a reference action capturing roughly what the robot should do. (2) This coarse action is passed through flow reversal and denoising, projecting it into the space of generalist actions. (3) Both the expert noises and actions can be used for policy improvement by executing the action (zero-shot FRS, Sec. 5.2 ) or training a noise action policy with supervised learning (DSBC, Sec. 5.3 ) or reinforcement learning (DSRL + FRS, Sec. 5.4 ).

Flow Reversal Steering (FRS)

Our method, Flow Reversal Steering (FRS), takes “coarse” actions and refines them using a generalist policy into similar, higher-quality actions. In Sec. 4.1 , we show how flow reversal can identify noises that bias the flow matching policy into sampling actions of the same mode as the coarse one. Then, in Sec.

4.2 , we consider how humans or VLMs can provide such coarse guidance to the robot, based on their semantic knowledge. Flow reversal converts these rough sketches of robot behaviors into in-distribution actions. Finally, in Sec.

Flow Reversal Steering (FRS) — Improving Robotic Generalist Policies via Flow Reversal Steering — Figure 4: Noising via the forward diffusion process vs. reverse flow integration. Both have the same marginals, but the former uses noise interpolation (so all signal is gone by $t=0$), while the latter deterministically maps from data to noise and back.

Experiments

We now evaluate Flow Reversal Steering by answering: (1) Can FRS improve performance without any training by having VLMs guide generalists towards semantically-reasonable behaviors? (2) Can we use the improved trajectories from FRS to rapidly learn new tasks? (3) Can FRS help generalists more efficiently improve from experience? See App. E and App. F for more experimental details. Simulation.

Sources and Demonstrations

Figures are reproduced from Tang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.