Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, Pieter Abbeel · OpenAI · 2017

Framework

MuJoCo / TensorFlow

License

MIT (OpenAI baseline)

Stars

0

Summary

Introduced domain randomization for vision-based manipulation: randomize simulation textures, lighting, and camera parameters so that the real world looks like just another variation.

Abstract Summary

Domain Randomization is arguably one of the most influential techniques at the intersection of simulation and real-world robotics. Developed by OpenAI and presented at IROS 2017, the method addresses the reality gap by deliberately randomizing non-essential properties of the simulator during training. Rather than making the simulation look photorealistic, the idea is to make the neural network invariant to visual nuisance factors—such as textures, lighting, camera pose, and object colors—by exposing it to a sufficiently diverse distribution of appearances. The canonical experiment in the paper involves a robotic arm learning to localize and pick a small colored cube from a table. Instead of rendering the scene with realistic textures, the training environment randomizes every visual attribute: background images, cube texture, light positions, camera angle, and even occlusion patterns. At test time, the same neural-network policy, with no domain-specific fine-tuning, is deployed on a physical robot and successfully grasps the real cube. This demonstrated that extreme visual randomization could serve as a form of implicit domain adaptation. Mathematically, the insight is that if the support of the randomized training distribution covers the real-world appearance, then a sufficiently capacity-constrained network will learn features robust to all nuisance variations. Since 2017, domain randomization has become a standard preprocessing step in vision-based sim-to-real pipelines. Extensive follow-up work has expanded it to dynamics randomization (varying mass, friction, and actuator gains), shape randomization (domain randomization on geometry), and even adversarial variants such as learning-to-simulate. For modern robotics researchers, domain randomization remains an essential baseline approach when transferring perception and policy networks from synthetic data to hardware. The original paper is short, clearly written, and highly reproducible in MuJoCo or PyBullet. Its conceptual clarity makes it an ideal reference in sim-to-real courses and tutorials, and its influence can be traced directly to the wide adoption of synthetic training in manipulation and navigation research that followed.

Key Points

  • First systematic introduction of visual domain randomization for sim-to-real transfer.
  • Randomizes textures, lighting, camera pose, and colors instead of photorealism.
  • Achieved zero-shot transfer to a real robotic arm for cube grasping.
  • Inspired a decade of follow-up: dynamics randomization, shape DR, learning-to-simulate.
  • Conceptually simple, reproducible, and applicable to any vision-based robotic policy.

Additional Notes

Implementation Tips

  • Random camera pose within a small cone around the nominal viewpoint; extreme angles still help but slow convergence.
  • Use procedural textures (Perlin noise, random colors) rather than curated datasets for best generalization.
  • Combine with dynamics randomization for full sim-to-real robustness.
  • Learning to Simulate (Ruiz et al., 2018)
  • Sim-to-Real via Randomized Environments (Peng et al., 2018)
  • CycleGAN-based Domain Adaptation (Bousmalis et al., 2018)
Share

Related Papers

EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera

EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera

Christen Millerdurai, Shaoxiang Wang, Yaxu Xie et al. · SIGGRAPH 2026 · May 2026

Reconstructing the absolute 3D pose and shape of the hands from the user's viewpoint using a single head-mounted camera is crucial for practical egocentric interaction in AR/VR, telepresence, and hand-centric manipulation tasks, where sensing must...

manipulation manipulation simulation
Code PDF Intermediate
Code ★ 0 May 2026
DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation

DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation

Jusuk Lee, Seungjae Lee, Jonghun Shin et al. · arXiv preprint · May 2026

Robot manipulation critically depends on perception that preserves the action-relevant aspects of a scene. Yet most robot learning pipelines are built upon visual encoders pre-trained for static recog

foundation-models perception manipulation representation-learning vla
PDF Intermediate
No code repo Code updated: May 2026