Robotics Research Hub
A Heterogeneous Architecture for Robot RL Beyond GPU-Dominant Paradigms

A Heterogeneous Architecture for Robot RL Beyond GPU-Dominant Paradigms

Yufei Jia, Zhanxiang Cao, Mingrui Yu, Heng Zhang, Shenyu Chen, Dixuan Jiang, Meng Li, Xiaofan Li, Yiyang Liu, Junzhe Wu, Zheng Li, XiLin Fang, Tingyu Cui, Shengcheng Fu, Haoyang Li · · 2026

reinforcement-learning simulation system-design locomotion
PDF arXiv

Framework

N/A

License

N/A

Stars

0

Summary

Simulation-based RL for contemporary robot control is increasingly organized around GPU-resident simulation: physics, rollout collection, and learning are placed on a single GPU-centric execution path

Abstract Summary

Simulation-based RL for contemporary robot control is increasingly organized around GPU-resident simulation: physics, rollout collection, and learning are placed on a single GPU-centric execution path. This paradigm has greatly improved training speed, but it has also encouraged a default assumption that efficient training requires physics to reside on the GPU. We revisit this assumption. Our view is that, in simulation-dominated robot control, the essential question is not which processor runs physics, but whether simulation throughput, policy learning, and runtime synchronization form an efficient end-to-end loop. We present UniLab, a heterogeneous CPU-simulation / GPU-learning architecture that decouples CPU-parallel simulation from GPU policy updates through a unified runtime for data movement, buffering, and synchronization. UniLab is implemented as a complete and extensible training system using MuJoCoUni and MotrixSim CPU-batched physics backends, supporting PPO, SAC, FlashSAC, TD3, and APPO. On representative simulation-based robot control tasks, UniLab improves end-to-end training efficiency by 3--10x under the same hardware configuration, while reducing dependence on the NVIDIA CUDA-based software stack and supporting cross-platform execution on the Apple macOS platform and the AMD ROCm and Intel XPU accelerator backends. These results show that GPU simulation is an effective path to efficient training, but not a necessary one, broadening the practical system choices available for robot RL training.

Key Points

  • Heterogeneous CPU-simulation / GPU-learning architecture (UniLab) for robot RL.
  • Decouples CPU-parallel simulation from GPU policy updates via unified runtime.
  • Supports PPO, SAC, FlashSAC, TD3, APPO on MuJoCoUni and MotrixSim backends.
  • Improves end-to-end training efficiency by 3-10x on same hardware.
  • Cross-platform support: macOS, AMD ROCm, Intel XPU, reducing CUDA dependency.

UniLab Architecture

This section describes UniLab as an end-to-end training loop that combines CPU-side batched rigid-body simulation, GPU-side policy and value learning, and a unified runtime for coordinating the data path between them. The design objective is to improve the efficiency of the full simulation-learning loop without requiring GPU-resident simulation.

UniLab follows hardware roles: CPUs generate large-scale simulation data, GPUs perform dense learning updates, and the runtime minimizes coordination cost. This objective induces three requirements.

UniLab Architecture — A Heterogeneous Architecture for Robot RL Beyond GPU-Dominant Paradigms
Figure 2: UniLab system architecture. The figure shows the data, scheduling, and parameter-synchronization paths between CPU-side batched physics backends, the unified runtime, and the GPU learner.

Experiments

We evaluate three questions: whether CPU simulation provides enough throughput, whether heterogeneous CPU-simulation / GPU-learning improves end-to-end wall-clock efficiency, and whether the result is robust across task families and algorithms. The primary metric is end-to-end training efficiency; throughput measurements explain the mechanism.

Controlled comparisons use the same default Linux hardware: one NVIDIA RTX 4090 GPU, one AMD Ryzen 9 9950X3D CPU, and 64 GB of 4800 MT/s memory.

Experiments — A Heterogeneous Architecture for Robot RL Beyond GPU-Dominant Paradigms
Figure 7: To-real experiment overview across six real-robot tasks.

Sources and Demonstrations

Figures are reproduced from Jia et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation
arXiv preprint

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

Junheng Li, Liang Wu, Sergio A. Esteban et al.

In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop π^nMPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills.

humanoid reinforcement-learning model-predictive-control locomotion manipulation
Code PDF Advanced
GitHub ★ —