From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention

Sichang Su, Benjamin Yang, Zhiyun Deng, Boyuan Liang, Yip Fun Yeung, Zelin Wang, Lingfeng Sun · N/A · 2026

Summary

Abstract Summary

A pretrained robot foundation policy may execute most of a long-horizon task yet repeatedly fail at a few critical subtasks. Collecting additional full-task demonstrations for supervised fine-tuning (SFT) requires operators to repeat behaviors the policy already performs well. Reinforcement learning (RL) fine-tuning offers a promising path to bridge this gap, but existing approaches struggle to solve long-horizon tasks using only sparse rewards. We present PARTS (Policy Adaptation with RL on Targeted Subtasks), a real-world subtask RL framework that concentrates practice at these bottlenecks while allowing training rollouts to proceed with minimal human intervention. The frozen pretrained policy supplies nominal actions throughout execution, while agent-generated selectors and success verifiers activate residual corrections and provide local outcome rewards. These rewards support learning from successful subtasks even when complete-task successes are scarce. Training combines online RL with success-reweighted retraining, and each retrained residual policy is redeployed to collect further experience. Humans identify bottlenecks during setup and perform physical resets when needed. On bimanual YAM and single-arm Franka tasks, PARTS improves complete-task success from 32% to 61% and from 50% to 95%, respectively, using tens of minutes of real-world RL rollouts per task on average. Compared with existing real-world RL fine-tuning methods, PARTS raises full-task success by more than 25% under the same robot-rollout budget while requiring less human involvement.

Key Points

A pretrained robot foundation policy may execute most of a long-horizon task yet repeatedly fail at a few critical subtasks.
Collecting additional full-task demonstrations for supervised fine-tuning (SFT) requires operators to repeat behaviors the policy already performs well.
Reinforcement learning (RL) fine-tuning offers a promising path to bridge this gap, but existing approaches struggle to solve long-horizon tasks using only sparse rewards.
We present PARTS (Policy Adaptation with RL on Targeted Subtasks), a real-world subtask RL framework that concentrates practice at these bottlenecks while allowing training rollouts to proceed with minimal human intervention.
The frozen pretrained policy supplies nominal actions throughout execution, while agent-generated selectors and success verifiers activate residual corrections and provide local outcome rewards.

Method

We consider fine-tuning a pretrained VLA policy $\pi_{\mathrm{VLA}}$ ($\pi_{0.5}$ in this work) with real-world RL. $\pi_{\mathrm{VLA}}$ is a generalist trained on a diverse demonstration corpus and conditioned on a language instruction [ 1 ] .

Like most modern VLAs, it uses action chunking: at each replan it predicts a chunk $\bar{A}_{t}=(\bar{a}_{t},\ldots,\bar{a}_{t+C-1})$ of $C$ future actions and executes a prefix of $E\leq C$ actions before predicting again [ 1 ] .

Method — From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention — Fig. 3: Tasks on bimanual YAM and single-arm Franka FR3. Colored borders mark states produced by a targeted residual policy, in the color of its subtask name in the row header. Top: earbud insertion, where inserting earbuds requires high precision. Middle: LEGO sorting, where the grasp of each small brick is the recurring bottleneck because the brick size is out-of-distribution for the base policy. Bottom: cable insertion requires precise manipulation.

Real-World Experiments

Our experiments address three questions. Q1. Does targeted subtask RL improve a pretrained VLA on complete long-horizon tasks? Q2. Under a matched robot-rollout budget, how does PARTS compare with RL fine-tuning methods that train on the full task? Q3. Does success-reweighted retraining matter? We evaluate PARTS on three long-horizon tasks across two real-robot platforms (Fig. 3 ). Earbud insertion (bimanual YAM).

The robot opens a charging case, holds it in the left gripper, inserts two earbuds with the right arm, and closes the case.

Sources and Demonstrations

Figures are reproduced from Su et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.