Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enhancement

Kinam Kim, Namiko Saito, Heecheol Kim, Katsushi Ikeuchi, Jaegul Choo, Yasuyuki Matsushita · · 2026

Summary

Abstract Summary

Vision-Language-Action (VLA) models can generalize across diverse manipulation tasks, but their imitation-learning-based policies remain brittle in precise physical interactions due to compounding execution errors; Can a reinforcement learning policy trained purely in simulation improve the robustness of real-world VLAs zero-shot? Residual RL, which learns a corrective policy on top of a frozen VLA, offers a natural framework, but existing approaches face a fundamental sim-to-real dilemma: privileged-state methods require lossy distillation for deployment; image-based methods suffer from the visual domain gap; and real-world RL is costly and unsafe. We propose an object-centric residual RL framework that refines VLA actions using object poses, enabling a compact observation space that transfers consistently between simulation and reality. To align the two domains, we additionally replay the same teleoperation demonstrations in simulation to train a sim counterpart of the real-world VLA. The residual RL policy is trained only in simulation with pose noise injection and dropout, and transfers zero-shot to the real robot. Across five manipulation tasks on a real Franka Research 3 (FR3) robot, our method improves the success rate from 42% to 76% zero-shot, and the improved rollouts can be further reused to retrain the base VLA for self-improvement without additional teleoperation. Project page: https://www.microsoft.com/en-us/research/articles/object-centric-residual-rl/

Key Points

Proposes Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enh for robotics.
Evaluated on real or simulated robotic tasks.
Primary contribution in ar-vr.

Method

Given a VLA trained on real-robot demonstrations, our framework enhances it with a sim-trained residual policy through three stages (Fig. 2 ): (1) building a simulation counterpart of the VLA via teleoperation replay (Section 3.1 ), (2) training an object-centric residual policy in simulation (Section 3.2 ), and (3) zero-shot deployment on the real robot (Section 3.3 ).

Experimental Setup

We evaluate on five tabletop manipulation tasks, instantiated both in MuJoCo [ 35 ] simulation (for residual training) and on a real FR3 robot (for zero-shot deployment): Cube Lift (Lift) : Grasp a cube from the table and lift it $3$ cm. Pick-and-Place (PnP) : Pick up a cube and place it into a bowl. Stack Cube (Stack) : Pick up a white cube and stack it on top of a green cube.

Close Drawer (Close) : Push an open cabinet drawer closed. Stand Cup Up (Stand) : Grasp a cup lying on its side and stand it upright.

Experimental Setup — Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enhancement — Figure 3: Real (top) and simulated (bottom) environments for all five evaluation tasks.

Results

Does the residual improve the base VLA zero-shot on a real robot? (Section 5.1 ) What observation and robustness designs enable sim-to-real transfer? (Section 5.2 ) When and how does the residual intervene? (Section 5.3 ) Can residual-corrected rollouts bootstrap VLA self-improvement?

(Section 5.4 ) We first examine whether a residual policy trained entirely in simulation can improve the base VLA on a real robot without any adaptation. Table 1 reports success rates across all five tasks.

Results — Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enhancement — Figure 5: The residual corrects the base action toward the goal when misaligned.

Sources and Demonstrations

Figures are reproduced from Kim et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.