WorldMirror: Agentic Real-to-Sim Generation of Interactive Worlds from In-the-Wild Robot Demonstrations

At a glance

Problem: in-the-wild robot demonstrations show diverse real interactions, but simulation needs scene geometry, objects, cameras and robot to match the real setup closely. Source
Idea: a coding agent builds an executable world program with specialized reconstruction tools and revises components when verification feedback finds inconsistencies. Source
Evidence: replaying the original DROID trajectories in Isaac Sim succeeded on 100% of evaluated demonstrations, versus 40% for GPT-6 Astra. Source

Evaluation scope The evaluation uses a small set of DROID demonstrations and does not yet test downstream policy learning, real-world deployment or broader cross-task generalization. Source

How it works

A router agent traverses a fixed construction graph covering scene preparation, camera calibration, support fitting, object construction, layout, robot alignment and trajectory replay. Source
Each candidate update is checked for visual, spatial and physical consistency before it is committed; a failure returns a typed failure class and rewind target to the router. Source
After camera calibration, the scene, support geometry, objects and robot base share one camera-anchored world frame, so recorded trajectories can be replayed without modification. Source

WorldMirror framework: a coding agent maintains an editable world, calls scene and object generation tools, aligns camera, assets and robot, and revises the world from rendered and execution feedback. Original figure

Evidence & results

On ten DROID scenes WorldMirror has the lowest LPIPS and DISTS, with competitive PSNR and aesthetic scores. Evidence
It beats the open-source baselines on all four layout metrics and has the lowest relative distance error, but trails GPT-6 Astra on mask IoU, position error and relative vector error. Evidence
Removing feedback-driven refinement causes the largest degradation and removing adaptive routing also breaks replay, while removing stateful memory changes little. Evidence

Qualitative RGB comparison of WorldMirror reconstructions with open-source and proprietary baselines on DROID scenes. Original figure

Limitations & boundaries

Reconstructed twins have not yet been used to train robot policies, so their value as training data is untested. Source
Reconstruction quality depends on the reconstruction and simulation tools that are available. Source
The component ablations are run on a single rail manipulation task. Source

Takeaway

Our reading: verification-guided rebuilding mainly helps physical replay rather than layout, since GPT-6 Astra still leads on most spatial metrics while WorldMirror replays every evaluated demonstration. Source

Resources

Full text ↗PDF ↗

Figures: the paper's authors. Editorial source and image review: 2026-10-26. Findings are author-reported, not independently reproduced by Robotics Research Hub.

Original abstract

Existing approaches for robot learning in simulation often require substantial manual effort to construct scenes and specify tasks, limiting training scalability and data diversity. Meanwhile, in-the-wild robot demonstrations naturally provide diverse and realistic observations of real-world interactions. Leveraging these demonstrations for simulation requires both high-fidelity visual reconstruction and physically consistent interactions. General-purpose agents offer flexible adaptation to diverse scenarios and strong reasoning about 3D spatial relationships, while specialized generative models provide high-fidelity reconstruction of scene appearance and geometry. We introduce WorldMirror, an agentic real-to-sim framework that constructs interactive digital twins from in-the-wild robot demonstrations. Our central idea is to formulate real-to-sim construction as the iterative refinement of an executable world program. A coding agent decomposes the observed scene, plans reconstruction steps, and orchestrates specialized image and 3D generation tools, refining the program through verification feedback. A camera-pose-anchored scene representation organizes the world into static visual context, support geometry, and editable interactive assets while preserving spatial relationships among the scene, objects, and robot. The agent uses this representation to diagnose and correct reconstruction errors, enabling physically consistent interaction and trajectory replay. Experiments on DROID demonstrate that WorldMirror can reconstruct visually faithful and geometrically consistent interactive worlds while preserving the physical validity of recorded robot trajectories.

Citation

@article{Xu2026_261031834,
  title     = {WorldMirror: Agentic Real-to-Sim Generation of Interactive Worlds from In-the-Wild Robot Demonstrations},
  author    = {Tianling Xu and Weijie Wan and Honghao Su and Kailun Su and Junyuan Tang and Huichi Zhou and Haozheng Zhang and Qi Xiong and Ping Luo and Wenbo Ding and Tianxing Chen and Yue Chen},
  year      = {2026},
  eprint    = {2610.31834},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url       = {https://arxiv.org/abs/2610.31834}
}