WorldSample: Closed-loop Real-robot RL with World Modelling

Yuquan Xue, Le Xu, Zeyi Liu, Zhenyu Wu, Zhengyi Gu, Xinyang Song, Bofang Jia, Ziwei Wang · · 2026

Summary

Abstract Summary

Reinforcement learning (RL) can overcome the demonstration-coverage limitation of imitation learning (IL) by allowing robots to improve through trial-and-error interaction beyond the states observed in demonstrations. However, deploying RL on real robots remains constrained by high interaction costs, since each physical rollout is costly and reflects only one realized action-outcome path. To address this challenge, we propose WorldSample, a physically grounded data augmentation framework for real-robot RL that closes a real-synthetic loop between physical rollouts, world-model generation, and policy improvement. Grounded on real rollouts, WorldSample generates high-fidelity synthetic transitions through a post-trained world model, which greatly lowers the visual hallucination. Specifically, rather than simply using these transitions as real-world experience, WorldSample introduces Policy-Paced Learning (PPL) to regulate the training process through sample selection and scheduling, balancing useful augmentation against value overestimation and mitigating the hallucination-induced noise. Experiments on robot manipulation tasks involving contact-rich and precise tasks show that WorldSample improves policy success rate by 28% while reducing training steps by 59% compared with baselines. Furthermore, WorldSample improves world model visual fidelity by 19.4dB in PSNR and 0.47 in SSIM over demonstration-only post-training, validating the effectiveness of the real-synthetic loop for both policy and world model performance.

Key Points

Proposes WorldSample for robotics.
Evaluated on real or simulated robotic tasks.
Primary contribution in ar-vr.

WorldSample

We propose WorldSample , a physically grounded, closed-loop data augmentation framework for online real-world RL. As shown in Fig. 2 , WorldSample consists of two tightly coupled components: real-synthetic data loop and Policy-Paced Learning (PPL) . The real-synthetic data loop starts from physical robot rollouts collected during online interaction.

These rollouts serve two roles simultaneously: they provide training data for policy learning, and they ground synthetic data generation through a action-conditioned world model.

WorldSample — WorldSample: Closed-loop Real-robot RL with World Modelling — Figure 2 : WorldSample pipeline. WorldSample closes a real-synthetic data loop between physical rollouts, world-model generation, and policy improvement, while Policy-Paced Learning (PPL) regulates the composition and usage of synthetic data during training.

Experimental Results

In this section, we conduct extensive experiments on complex and real-world tasks, including precise manipulation and contact-rich interaction, to evaluate WorldSample. Our experiments are designed to answer the following three questions: Does WorldSample improve online real-robot RL efficiency? Does WorldSample improve the video fidelity and physical grounding of the world model?

Is Policy-Paced Learning necessary for stable and effective use of synthetic data? We conduct experiments on a Galaxea A1X robotic arm equipped with a binary gripper....

Experimental Results — WorldSample: Closed-loop Real-robot RL with World Modelling — Figure 4 : Comparison between real and generated video frames.

Limitaions

WorldSample currently focuses on augmenting real-robot learning within a single task and a relatively fixed scene distribution. Although online rollout adaptation improves the world model within this setting, extending this framework to instance-, task-, or scene-level generalization remains an important direction for future work.

In this paper, we evaluate WorldSample and all baselines on a suite of real-world manipulation tasks, as illustrated in Fig. 6 .

Limitaions — WorldSample: Closed-loop Real-robot RL with World Modelling — Figure 6 : Real-world task settings. We evaluate WorldSample on diverse manipulation tasks covering contact-rich insertion, object pushing, sorting, pick & place, and precision assembly. These tasks require different combinations of visual grounding, contact reasoning, precise motion, and recovery from imperfect policy executions.

Sources and Demonstrations

Figures are reproduced from Xue et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.