M3imic: Learning a Versatile Whole-Body Controller for Multimodal Motion Mimicking
FeaturedZuxing Lu, Ziang Zheng, Yao Lyu, Jingyu Liu, Feihong Zhang, Song Lu, Xin Yuan, Changyin Sun, Xingxing Zuo, Shengbo Eben Li · N/A · 2026
Framework
N/A
License
N/A
Stars
N/A
Summary
Building a general-purpose whole-body controller is essential for enabling diverse motion capabilities in humanoid robots across a wide range of downstream tasks, including locomotion and loco-manipulation. Different tasks rely on distinct motion reference modalities: locomotion primarily depends on...
Abstract Summary
Key Points
- Demonstrates humanoid robot control on real hardware
- Improves locomotion stability and agility
- Addresses dexterous manipulation challenges
- Uses simulation or synthetic data for training
- Focuses on generalization across environments
- Incorporates tactile or force feedback for robust interaction
- Learns multimodal motion skills from human demonstrations
M3imic: Learning a Versatile Whole-Body Controller for Multimodal Motion Mimicking
Authors: Zuxing Lu, Ziang Zheng, Yao Lyu, Jingyu Liu, Feihong Zhang, Song Lu, Xin Yuan, Changyin Sun, Xingxing Zuo, Shengbo Eben Li
Venue: arXiv preprint | Year: 2026
arXiv: 2606.04829v1
Abstract
Building a general-purpose whole-body controller is essential for enabling diverse motion capabilities in humanoid robots across a wide range of downstream tasks, including locomotion and loco-manipulation. Different tasks rely on distinct motion reference modalities: locomotion primarily depends on coordinated robot joint trajectories, whereas manipulation requires precise end-effector trajectory tracking. Existing methods often overlook the representational mismatch between dense robot joint angles and sparse end-effector poses. To address this, we propose Multi-Modal Mimic (M3imic), a versatile multi-modal whole-body control framework that unifies heterogeneous motion reference modalities, including robot joint angles, human pose trajectories, and end-effector poses, using modality-specific encoders to map them into a shared latent space. Leveraging large-scale reinforcement learning in the simulator, we train a single policy that achieves sim-to-real transfer across multiple motion reference modalities without modality-specific retraining. Extensive simulation and real-world experiments on the Unitree G1 robot are conducted to evaluate the proposed framework. In simulation, the policy achieves a peak success rate of 98.42% on an unseen test dataset, demonstrating its exceptional generalization capability. The code is available at https://github.com/Renforce-Dynamics/MultiModalWBC
Key Contributions
- Demonstrates humanoid robot control on real hardware
- Improves locomotion stability and agility
- Addresses dexterous manipulation challenges
- Uses simulation or synthetic data for training
- Focuses on generalization across environments
- Incorporates tactile or force feedback for robust interaction
- Learns multimodal motion skills from human demonstrations
Topics
- sim-to-real
- reinforcement-learning
- locomotion
- manipulation
- humanoid
Code & Data
Code Repository: https://github.com/Renforce-Dynamic
BibTeX
@article{Lu2026_260604829v1,
title={M3imic: Learning a Versatile Whole-Body Controller for Multimodal Motion Mimicking},
author={Zuxing Lu and Ziang Zheng and Yao Lyu and Jingyu Liu and Feihong Zhang and Song Lu and Xin Yuan and Changyin Sun and Xingxing Zuo and Shengbo Eben Li},
year={2026},
eprint={2606.04829v1},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2606.04829v1}
}
Related Papers
Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation
Junheng Li, Liang Wu, Sergio A. Esteban et al. · arXiv preprint · Jun 2026
In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop π^nMPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills.
Flash-WAM: Modality-Aware Distillation for World Action Models
Arman Akbari, Ci Zhang, Arash Akbari et al. · arXiv preprint · Jun 2026
World-action models (WAMs) jointly generate future video and robot actions through iterative diffusion, achieving strong performance on manipulation benchmarks but requiring tens of denoising steps, a cost that precludes real-time control. Step distillation has emerged as the natural remedy, but off...
GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors
Tianyi Xie, Haotian Zhang, Jinhyung Park et al. · arXiv preprint · Jun 2026
Scaling humanoid loco-manipulation requires robot-compatible demonstrations across diverse objects, whole-body motions, and scene geometries, but teleoperation and motion capture are difficult to scale because each collection depends on physical setups, instrumented actors, and robot operation. We p...
LadderMan: Learning Humanoid Perceptive Ladder Climbing
Siheng Zhao, Yuanhang Zhang, Ziqi Lu et al. · arXiv preprint · Jun 2026
Humanoid robots hold great promise for operating in human-centered environments, yet ladder climbing remains one of the most challenging tasks due to sparse footholds and handholds, complex whole-body coordination, and sensitivity to perception and control errors. We present extbf{LadderMan}, a un...