Critic Architecture Matters: Dual vs. Unified Critics for Humanoid Loco-Manipulation
Mehmet Turan Yardımcı · N/A · 2026
Framework
N/A
License
N/A
Stars
N/A
Summary
Multi-objective reinforcement learning for humanoid robots must coordinate locomotion and manipulation within a single policy.
Abstract Summary
Key Points
- Controlled comparison of dual vs. unified critic architectures for humanoid loco-manipulation.
- Dual-critic policies reach targets 3.5x faster and achieve 2x higher throughput than unified critics.
- Critic architecture has greater impact on performance than reward engineering or anti-gaming mechanisms.
- Findings are directly relevant to RL fine-tuning of pre-trained manipulation policies.
Method
We train a Unitree G1 humanoid robot in NVIDIA Isaac Lab [ 14 ] . The robot has 23 active DoF, of which the policy controls 17 : 12 leg joints and 5 arm joints (right arm: shoulder pitch/roll/yaw, elbow pitch/roll). Wrist and hand joints are held fixed throughout training and evaluation. Parallel environment counts differ by run and are reported per run in Sec. IV-A .
Training proceeds sequentially: Stages 1–3 train locomotion only; Stage 6 trains both branches jointly for loco-manipulation.
Experiments
All runs use NVIDIA Isaac Lab on a single RTX 5070 Ti GPU (12GB VRAM), achieving approximately 17,000 steps/second. The two runs of the main comparison (S6u, S6s) each used 2048 parallel environments for 20,000 iterations; the anti-gaming variant (S7) used 4096 for 15,000 iterations.
Sources and Demonstrations
Figures are reproduced from Yardımcı See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.
Related Papers
Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation
Junheng Li, Liang Wu, Sergio A. Esteban et al.
In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop π^nMPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills.
Flash-WAM: Modality-Aware Distillation for World Action Models
Arman Akbari, Ci Zhang, Arash Akbari et al.
World-action models (WAMs) jointly generate future video and robot actions through iterative diffusion, achieving strong performance on manipulation benchmarks but requiring tens of denoising steps, a cost that precludes real-time control.
GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors
Tianyi Xie, Haotian Zhang, Jinhyung Park et al.
Scaling humanoid loco-manipulation requires robot-compatible demonstrations across diverse objects, whole-body motions, and scene geometries, but teleoperation and motion capture are difficult to scale because each collection depends on physical setups, instrumented actors, and robot operation.
LadderMan: Learning Humanoid Perceptive Ladder Climbing
Siheng Zhao, Yuanhang Zhang, Ziqi Lu et al.
Humanoid robots hold great promise for operating in human-centered environments, yet ladder climbing remains one of the most challenging tasks due to sparse footholds and handholds, complex whole-body coordination, and sensitivity to perception and control errors.