M3imic: Learning a Versatile Whole-Body Controller for Multimodal Motion Mimicking

Zuxing Lu, Ziang Zheng, Yao Lyu, Jingyu Liu, Feihong Zhang, Song Lu, Xin Yuan, Changyin Sun, Xingxing Zuo, Shengbo Eben Li · N/A · 2026

Summary

Abstract Summary

Building a general-purpose whole-body controller is essential for enabling diverse motion capabilities in humanoid robots across a wide range of downstream tasks, including locomotion and loco-manipulation. Different tasks rely on distinct motion reference modalities: locomotion primarily depends on coordinated robot joint trajectories, whereas manipulation requires precise end-effector trajectory tracking. Existing methods often overlook the representational mismatch between dense robot joint angles and sparse end-effector poses. To address this, we propose Multi-Modal Mimic (M3imic), a versatile multi-modal whole-body control framework that unifies heterogeneous motion reference modalities, including robot joint angles, human pose trajectories, and end-effector poses, using modality-specific encoders to map them into a shared latent space. Leveraging large-scale reinforcement learning in the simulator, we train a single policy that achieves sim-to-real transfer across multiple motion reference modalities without modality-specific retraining. Extensive simulation and real-world experiments on the Unitree G1 robot are conducted to evaluate the proposed framework. In simulation, the policy achieves a peak success rate of 98.42% on an unseen test dataset, demonstrating its exceptional generalization capability. The code is available at https://github.com/Renforce-Dynamics/MultiModalWBC

Key Points

Demonstrates humanoid robot control on real hardware
Improves locomotion stability and agility
Addresses dexterous manipulation challenges
Uses simulation or synthetic data for training
Focuses on generalization across environments
Incorporates tactile or force feedback for robust interaction
Learns multimodal motion skills from human demonstrations

Method

In this section, we detail the problem formulation, multi-modal policy learning framework, curriculum-based sampling strategy, and sim-to-real transfer pipeline. The overall training and deployment framework is illustrated in Fig. 2 . We formulate whole-body contorl as a reinforcement learning problem modeled as a Markov Decision Process (MDP) $\mathbf{M}=(\mathbf{S},\mathbf{A},P,r,\gamma)$.

Experiments

In this section, we present experimental results in both the simulator and the real-world deployment. Our experimental platform is the 29-DoF Unitree G1 humanoid robot. Here, we aim to answer the following research questions: Q1: Does the choice of input modality significantly affect deploy performance? Q2: How do model size and dataset scale influence large-scale training?

Q3: How robustly does the autoencoder generalize to out-of-distribution (OOD) motion sequences? We use IsaacSim [ 4 ] as the simulator to train our model.

Experiments — M3imic: Learning a Versatile Whole-Body Controller for Multimodal Motion Mimicking — Fig. 3 : Qualitative comparison between $\pi^{r}$ and $\pi^{e}$ in MuJoCo simulation.

Sources and Demonstrations

Figures are reproduced from Lu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.