Sim-to-Real Transfer

16 papers

Transferring policies trained in simulation to real-world deployment.

sim-to-real reinforcement-learning vision vla manipulation planning control learning-from-demonstration dexterous tactile diffusion-policy humanoid navigation world-models locomotion rl foundation-models imitation-learning

Papers

Sorted by year (newest first)

3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training

Jiaxin Shi, Xidong Zhang, Fucai Zhu et al. · arXiv preprint · Jun 2026

We propose a 3D-thinking-guided co-training framework that enables vision-language-action (VLA) models to perform 3D spatial reasoning implicitly during action prediction. Our core insight is that 3D geometry perception and 3D spatial reasoning are distinct capabilities that can be disentangled and ...

sim-to-real reinforcement-learning vision vla manipulation

PDF Advanced

No code repo Jun 2026

A Low-Cost, Open Platform for End-to-End Autonomous Driving on a Miniature Ackermann Vehicle

Gustavo Claudio Karl Couto, Eric Aislan Antonelo, Gabriel George Zipperer · arXiv preprint · Sep 2026

This paper presents a low-cost, open experimental platform for research in end-to-end autonomous driving with miniature Ackermann vehicles. The platform combines a physical vehicle, a printed urban track, data collection tools, trajectory registration, and a Webots digital twin, enabling controll...

vision reinforcement-learning sim-to-real planning control learning-from-demonstration

PDF Advanced

No code repo Sep 2026

Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation

Jiahe Pan, Stelian Coros, Jitendra Malik et al. · arXiv preprint · May 2026

A primary bottleneck in contact-rich manipulation is the difficulty of collecting real-world data. Sim-to-real reinforcement learning offers a scalable alternative, but the simulation-reality gap prev

dexterous sim-to-real tactile manipulation

PDF Intermediate

No code repo Code updated: May 2026

Flash-WAM: Modality-Aware Distillation for World Action Models

Arman Akbari, Ci Zhang, Arash Akbari et al. · arXiv preprint · Jun 2026

World-action models (WAMs) jointly generate future video and robot actions through iterative diffusion, achieving strong performance on manipulation benchmarks but requiring tens of denoising steps, a cost that precludes real-time control. Step distillation has emerged as the natural remedy, but off...

sim-to-real reinforcement-learning diffusion-policy manipulation humanoid

PDF Intermediate

No code repo Jun 2026

Generalization of World Models under Environmental Variability for Vision-based Quadrotor Navigation

Luca Zanatta, Grzegorz Malczyk, Kostas Alexis · arXiv preprint · Jun 2026

World models, learned generative models that predict how an environment evolves, have become a promising tool for sample-efficient robot learning. Yet how robust they are to environmental variability remains poorly understood. To address this, we conduct a systematic study using vision-based quadrot...

sim-to-real reinforcement-learning navigation vision world-models

PDF Advanced

No code repo Jun 2026

GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors

Tianyi Xie, Haotian Zhang, Jinhyung Park et al. · arXiv preprint · Jun 2026

Scaling humanoid loco-manipulation requires robot-compatible demonstrations across diverse objects, whole-body motions, and scene geometries, but teleoperation and motion capture are difficult to scale because each collection depends on physical setups, instrumented actors, and robot operation. We p...

sim-to-real reinforcement-learning vision manipulation humanoid

Code PDF Advanced

GitHub ★ 275 Code updated: Jun 2026

Grasp, Handover, Rotate: Bimanual Object Reorientation via Compositional Diffusion and Energy-Based Optimization

Wun Lam Yeung, Wenjun Liu, Yui Cheung Yu et al. · arXiv preprint · Jul 2026

Bimanual object reorientation - picking an object, handing it over between two arms, and placing it in a desired target pose - is valuable when direct placement from the initial grasp is infeasible due to collisions, kinematic constraints, or poor final orientation. However, achieving this under ...

manipulation reinforcement-learning sim-to-real planning

PDF Advanced

No code repo Jul 2026

LadderMan: Learning Humanoid Perceptive Ladder Climbing

Siheng Zhao, Yuanhang Zhang, Ziqi Lu et al. · arXiv preprint · Jun 2026

Humanoid robots hold great promise for operating in human-centered environments, yet ladder climbing remains one of the most challenging tasks due to sparse footholds and handholds, complex whole-body coordination, and sensitivity to perception and control errors. We present extbf{LadderMan}, a un...

sim-to-real reinforcement-learning vision manipulation humanoid

PDF Advanced

No code repo Jun 2026

M3imic: Learning a Versatile Whole-Body Controller for Multimodal Motion Mimicking

Zuxing Lu, Ziang Zheng, Yao Lyu et al. · arXiv preprint · Jun 2026

Building a general-purpose whole-body controller is essential for enabling diverse motion capabilities in humanoid robots across a wide range of downstream tasks, including locomotion and loco-manipulation. Different tasks rely on distinct motion reference modalities: locomotion primarily depends on...

sim-to-real reinforcement-learning locomotion manipulation humanoid

Code PDF Advanced

GitHub ★ — Jun 2026

Towards Professional Tennis Styles for Humanoid Robots with Adaptive Motion Planning and Tracking

Tao Huang, Ruofei Liu, Xuchen Tang et al. · arXiv preprint · Aug 2026

Humanoid robots have recently demonstrated promising capabilities in real-world ball sports. However, achieving professional motion styles while maintaining strong task performance remains challenging. In this work, we propose AdaPT, an Adaptive Motion Planning and Tracking framework that learns ...

vision reinforcement-learning sim-to-real planning

Code PDF Intermediate

GitHub ★ — Aug 2026

Video2DoorTraversal: Push Door Traversal via Simulated Door Twins

Xincheng Tang, Yiji Chen, Youhan Xie et al. · arXiv preprint · Aug 2026

Door opening and traversal is a long-horizon loco-manipulation task that requires precise handle interaction and coordinated base-arm control. We present Video2DoorTraversal, a single-video real-to-sim-to-real framework for wheel-legged mobile manipulators. Given one RGB video of a real door, Doo...

manipulation locomotion vision reinforcement-learning sim-to-real control learning-from-demonstration

PDF Intermediate

No code repo Aug 2026

Eureka: Human-Level Reward Design via Coding Large Language Models

Yecheng Jason Ma, William Liang, Guanzhi Wang et al. · ICLR 2024 (Oral) · May 2024

Eureka uses GPT-4 to write reward functions for RL environments, achieving human-level reward design on 29 tasks and enabling zero-shot sim-to-real transfer on Shadow Hand dexterous manipulation.

rl foundation-models sim-to-real

Code PDF Intermediate

GitHub ★ 3,161 Code updated: May 2026

SimplerEnv: Simulated Manipulation Policy Evaluation with Real-World Video Prompts

Xuanlin Li, Kyle Hsu, Jiayuan Gu et al. · RSS 2024 · Jul 2024

A benchmark environment for evaluating real-world robot manipulation policies in simulation using real-world video prompts, enabling zero-shot sim-to-real transfer evaluation without physical hardware.

sim-to-real manipulation imitation-learning

Code Intermediate

GitHub ★ 1,068 Code updated: May 2026

Gymnasium: A Standard Interface for Reinforcement Learning Environments

Farama Foundation, Jordan Terry, Mark Towers et al. · JMLR 2023 · Mar 2023

Gymnasium is the maintained successor to OpenAI Gym, providing a standardized API for RL environments with 100+ built-in tasks, vectorized parallel execution, and native support for physics engines like MuJoCo, PyBullet, and IsaacGym.

rl foundation-models sim-to-real

Code Beginner

GitHub ★ 11,943 Code updated: May 2026

ManiSkill: A Unified Benchmark for Generalizable Manipulation Skills

Jiayuan Gu, Sean Xiang, Stone Tao et al. · ICLR 2023 (Oral) · Jan 2023

ManiSkill is a GPU-parallelized robotics simulation benchmark with 20+ manipulation tasks, PartNet-Mobility assets, and unified observation/action spaces, supporting RL, IL, and VLA training at millions of steps per hour.

sim-to-real rl manipulation foundation-models

Code PDF Beginner

GitHub ★ 2,914 Code updated: May 2026

iGibson: Interactive Gibson Benchmark for Real-World Navigation and Manipulation

Bokui Shen, Fei Xia, Chengshu Li et al. · ICRA 2021 · May 2021

iGibson is a photorealistic, physics-based indoor simulation platform with 15 interactive scenes, 500+ object categories, and GPU-accelerated rendering, enabling RL and VLA training for mobile manipulation and embodied AI.

sim-to-real rl foundation-models

Code PDF Intermediate

GitHub ★ 806 Code updated: May 2026

Suggested Learning Path

Read these papers in order to build expertise in Sim-to-Real Transfer.

1
3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training
Jiaxin Shi, Xidong Zhang et al. · 2026
2
A Low-Cost, Open Platform for End-to-End Autonomous Driving on a Miniature Ackermann Vehicle
Gustavo Claudio Karl Couto, Eric Aislan Antonelo et al. · 2026
3
Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation
Jiahe Pan, Stelian Coros et al. · 2026
4
Flash-WAM: Modality-Aware Distillation for World Action Models
Arman Akbari, Ci Zhang et al. · 2026
5
Generalization of World Models under Environmental Variability for Vision-based Quadrotor Navigation
Luca Zanatta, Grzegorz Malczyk et al. · 2026
6
GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors
Tianyi Xie, Haotian Zhang et al. · 2026

…and 10 more papers.