Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

Junheng Li, Liang Wu, Sergio A. Esteban, Lizhi Yang, Ján Drgoňa, Aaron D. Ames · Caltech · 2026

Summary

In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop π^nMPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills.

Abstract Summary

Key Points

Proposes MPC-RL: MPC-guided reinforcement learning for humanoid control
Introduces centroidal-dynamics MPC reward formulation for RL training
Develops π^nMPC: parallel-in-horizon batched GPU MPC solver
Achieves superior locomotion and manipulation on real hardware
Combines physical structure of MPC with simple deployment of RL
Provides open-source implementation in PyTorch and JAX

Abstract

BibTeX

@article{Li2026_260605687,
  title={Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation},
  author={Junheng Li and Liang Wu and Sergio A. Esteban and Lizhi Yang and J{\'a}n Drgo{\v n}a and Aaron D. Ames},
  year={2026},
  eprint={2606.05687},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2606.05687}
}