Humanoid-Gym: Reinforcement Learning for Humanoid Robot with Zero-Shot Sim2Real Transfer

Humanoid-Gym: Reinforcement Learning for Humanoid Robot with Zero-Shot Sim2Real Transfer

Wang J., Xiao B., Wang J., Heider P., Li H., Zhang Y. · RobotEra / K-Scale Labs · 2024

Framework

Isaac Gym / PyTorch

License

MIT

Stars

10

Summary

An Isaac-Gym-based training framework for humanoid locomotion that enables zero-shot sim-to-real transfer on XBot-L/XBot-S robots via domain randomization.

Abstract Summary

Humanoid-Gym is a reinforcement learning framework specifically built to close the reality gap for bipedal robots. Built atop NVIDIA Isaac Gym (Preview 4), it provides a curated set of humanoid-specific reward functions, terrain curricula, and domain-randomization schedules that collectively enable zero-shot sim-to-real transfer. The framework ships with example policies for XBot-S (1.2 m) and XBot-L (1.65 m), two commercial humanoid platforms from RobotEra. The training paradigm centers on proximal policy optimization (PPO) in massively parallel simulation (4096 environments). The unique contributions are in how the reward functions implicitly encode humanoid priors—penalizing unnatural knee hyperextension, encouraging symmetrical gait, and shaping trunk posture—such that the learned policy remains stable when transferred to hardware without any real-world fine-tuning. A notable design choice is the built-in sim2sim transfer pipeline. Researchers can first train in Isaac Gym and then test the exact same policy within MuJoCo environments, allowing coarse validation of dynamics mismatch before risking hardware. Terrain curriculum and actuator noise injection are the primary robustification tools, gradually increasing terrain roughness and randomizing motor torque bandwidth to promote robust gaits. For researchers and startups entering the humanoid space, Humanoid-Gym offers a plug-and-play training package with well-documented URDF setup, reward templates, and deployment scripts. The included domain-randomization recipes are tuned over extensive real-world experiments and represent practical best practices rather than theoretical guidelines. This makes it one of the most pragmatic toolkits for anyone seeking to deploy learned walking on their own humanoid platform.

Key Points

  • Isaac-Gym PPO training at 4096 parallel environments with humanoid-specific rewards.
  • Zero-shot sim-to-real demonstrated on XBot-L and XBot-S hardware.
  • Built-in sim2sim pipeline (Isaac Gym -> MuJoCo) for pre-transfer validation.
  • Domain randomization across terrain, actuator dynamics, and sensor delay.
  • Open-source release with URDF, training configs, and deployment scripts.

Additional Notes

Training Tips

  • Start training at ~2.5 m/s target speed; increase only after policy shows stable foot contact patterns.
  • Use the terrain curriculum: rolling friction -> box obstacles -> rough terrain over several iterations.
  • Ensure GPU driver 525+ to avoid Isaac Gym rendering crashes on some workstation setups.
  • Isaac Gym (Makoviychuk et al., 2021)
  • RSL-RL (Rudin et al., 2022)
  • Berkeley Humanoid (Kousik et al., 2024)
Share

Related Papers