Robotics Research Hub
Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

Featured

Junheng Li, Liang Wu, Sergio A. Esteban, Lizhi Yang, Ján Drgoňa, Aaron D. Ames · Caltech · 2026

humanoid reinforcement-learning model-predictive-control locomotion manipulation
PDF arXiv Code (GitHub)

Framework

PyTorch / JAX

License

N/A

Stars

N/A

Summary

In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop π^nMPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills.

Abstract Summary

In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop π^nMPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills. The code base is available at https://github.com/junhengl/mpc-rl.

Key Points

  • Proposes MPC-RL: MPC-guided reinforcement learning for humanoid control
  • Introduces centroidal-dynamics MPC reward formulation for RL training
  • Develops π^nMPC: parallel-in-horizon batched GPU MPC solver
  • Achieves superior locomotion and manipulation on real hardware
  • Combines physical structure of MPC with simple deployment of RL
  • Provides open-source implementation in PyTorch and JAX

Centroidal-Dynamics MPC Guidance

This section describes the centroidal-dynamics [ 24 ] MPC (CD-MPC) module used to provide predictive guidance during RL. The purpose of CD-MPC is not to directly control the humanoid at deployment, but to generate CoM, momentum, ground reaction force (GRF), and footstep references during massive-parallel training as structured rewards.

Let $q\in\mathbb{R}^{n_{q}}$ and $\dot{q}\in\mathbb{R}^{n_{q}}$ denote the full-order humanoid configuration and velocity.

Centroidal-Dynamics MPC Guidance — Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation
(a) Simulation snapshots.

π n \pi^{n} MPC: P aralle- i n-horizon and Construction-free Batched MPC Solver

To enable MPC solving effort and guidance lightweight during RL training, we build on top of the parallel-in-horizon ADMM structure of previous work $\pi$MPC [ 23 ] .

Unlike the native Julia implementation $\pi$MPC with time-invariant state matrices for monolithic MPC solve, we adapt to the time-varying nature of the legged locomotion dynamics and allow storage of time-varying state matrices along the horizon for massive-parallel evaluations. The implementation is migrated and realized in both PyTorch and JAX, termed $\pi^{n}$MPC .

MPC-guided Reinforcement Learning

This section presents the implementation details of the MPC-RL training and deployment infrastructure. We train the policy with PPO using rsl-rl on top of mjlab [ 28 ] , running $N_{\mathrm{env}}\!=\!4096$ environments. Physics is integrated at $200\,$Hz with with a control rate of $50\,$Hz. Episodes last at most $T_{\max}\!=\!20\,$s (1000 control steps).

Results

Method Training-time MPC Time-varying velocity tracking RMSE $\downarrow$ Max recoverable push force (N) $\uparrow$ Constraint satisfaction $\uparrow$ $N$ Rate $v_{x}$ $v_{y}$ $\omega_{z}$ $v_{x}^{\mathrm{OOD}}$ $v_{y}^{\mathrm{OOD}}$ $\omega_{z}^{\mathrm{OOD}}$ $0^{\circ}$ $45^{\circ}$ $90^{\circ}$ $135^{\circ}$ $180^{\circ}$ $225^{\circ}$ $270^{\circ}$ $315^{\circ}$ Friction CWC-$y$ CWC-$x$ Pure RL – – 0.1609 0.1566 0.0912 0.2310 0.1975 0.1113 300 350 100 300 350 225 125 275 100$\%$ 57.2$\%$ 92.0$\%$ MPC-RL 10 5 Hz 0.1374 0.1342 0.1060 0.2052...

Sources and Demonstrations

Figures are reproduced from Li et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics
arXiv preprint

An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics

Teeratham Vitchutripop, Alyssa Quarles, Wenhe Zhang et al.

Over the course of a lifetime, robots may encounter novel scenarios unaccounted for in its original training that result in performance degradation. One common approach to mitigating this issue is to further grow the offline training dataset in hopes of producing a policy robust to these changes.

manipulation locomotion reinforcement-learning benchmark
PDF Intermediate