Unitree Go2: Learning Vision-Based Quadrupedal Locomotion End-to-End

The Unitree Go2 paper bridges the gap between pure proprioceptive locomotion and vision-guided navigation in quadruped robots. Presented at ICRA 2024, the work introduces a cross-modal transformer architecture that fuses depth camera images with IMU and joint encoder data to produce robust locomotion policies for the Unitree Go2 robot. The policy is trained end-to-end in NVIDIA Isaac Gym and transfers to the physical robot with only standard domain randomization, achieving zero-shot sim-to-real performance on rough outdoor terrain. The key innovation is the architecture: a Vision Transformer processes depth patches into tokens, while a separate MLP encoder handles proprioceptive data. A cross-modal attention mechanism then fuses these streams into a unified representation used by the policy and value networks. This design deliberately prevents the vision encoder from overfitting to visual artifacts by forcing it to learn semantically meaningful features that the proprioceptive branch can attend to. The training pipeline leverages the massively parallel physics simulation of Isaac Gym, training thousands of Go2 agents simultaneously on randomized terrains. The reward function combines forward velocity tracking, body orientation stability, foot contact scheduling, and smoothness penalties. At test time, the policy maps a single depth image and current joint states directly to joint position targets at 50 Hz, without any explicit height mapping or terrain estimation. Real-world evaluations span grassy hills, rocky trails, staircases, and forest floors. The Go2 robot maintains speeds of up to 2.5 m/s on moderate terrain and recovers from trips and slips without falling. Ablations confirm that cross-modal attention significantly improves performance over simple feature concatenation, particularly in low-light conditions and on novel terrain types not seen during training. The codebase is based on RSL-RL and Legged Gym, making it accessible to researchers already familiar with the ETH robotics stack. All training configs, URDF files, and deployment scripts are released, enabling the community to replicate results and extend the approach to other quadruped platforms.