Robotics Research Hub
ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation

ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation

Kaifeng Zhao, Mathis Petrovich, Haotian Zhang, Tingwu Wang, Siyu Tang, Davis Rempe · N/A · 2026

locomotion reinforcement-learning control benchmark
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics.

Abstract Summary

Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics. While recent offline motion generation approaches offer precise control via text and kinematic constraints, they lack the inference speed required for interactive settings. Conversely, existing online methods enable real-time synthesis but often sacrifice controllability or struggle with complex text semantics and long-horizon goals due to limited context windows. In this work, we introduce ARDY, a streaming generation framework that bridges this gap by enabling high-fidelity motion generation controllable via online text prompts and flexible kinematic constraints. ARDY employs a hybrid representation that combines explicit root features with a latent body embedding, balancing precise trajectory control with efficient generative learning. We propose a two-stage autoregressive transformer denoiser that features variable history context and supports conditioning on flexible, long-horizon kinematic constraints. By training on a large-scale motion capture dataset and being directly conditioned on text labels and kinematic constraints sampled from ground truth poses, ARDY natively learns controllable generation that supports online prompting and flexible long-horizon goals. Extensive evaluations on the HumanML3D benchmark and the large-scale, high-fidelity Bones Rigplay dataset demonstrate ARDY's high motion quality and constraint adherence, validating the efficacy of our key architectural decisions. Finally, we demonstrate the method's practical versatility through an interactive demo featuring dynamic text control, diverse keyframe pose constraints, path following, and interactive locomotion control via mouse and keyboard. Supplementary video results, code, and model releases can be found at https://research.nvidia.com/labs/sil/projects/ardy/.

Key Points

  • Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics.
  • While recent offline motion generation approaches offer precise control via text and kinematic constraints, they lack the inference speed required for interactive settings.
  • Conversely, existing online methods enable real-time synthesis but often sacrifice controllability or struggle with complex text semantics and long-horizon goals due to limited context windows.
  • In this work, we introduce ARDY, a streaming generation framework that bridges this gap by enabling high-fidelity motion generation controllable via online text prompts and flexible kinematic constraints.
  • ARDY employs a hybrid representation that combines explicit root features with a latent body embedding, balancing precise trajectory control with efficient generative learning.

Method: ARDY

Our method ARDY consists of two main components: (1) a motion tokenizer first learns a compact latent representation of body motion, and then (2) an autoregressive two-stage motion diffusion model learns to denoise hybrid motion tokens containing latent body motion and explicit root motion. Our hybrid representation is introduced in Sec. 3.1 followed by the body motion tokenizer in Sec. 3.2 .

The autoregressive generation problem formulation is detailed in Sec. 3.3 and then the diffusion model that solves it is described in Sec. 3.4 .

Method: ARDY — ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation
Figure 4. Interactive Demo Interface. This web interface allows generating motion and interacting with ARDY in real time. The control panel at the top right allows dynamically changing the text prompt or input constraints. Input constraints are visualized in red within the 3D scene as the model generates motion to follow them. The timeline tracks on the bottom of the interface intuitively show upcoming text prompts and constraints.

Interactive Motion Generation Demo

To showcase ARDY’s versatility, we developed an interface using Viser ( Yi et al., 2025 ) to interactively generate motion with our model. The system, shown in Fig. 4 , enables real-time character control through a combination of streaming text prompts and interactive spatial constraints provided via mouse and keyboard inputs.

In this section, we first detail ARDY’s test-time operation, then qualitatively demonstrate key results through the interactive demo.

Interactive Motion Generation Demo — ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation
Figure 6. Motion Generation with Kinematic Constraints. Qualitative results for motion generation conditioned on text prompts and diverse kinematic constraints, including dense root trajectories, sparse root waypoints (visualized as red rings), full-body keyframes (visualized as red skeletons), sparse joint positions (visualized as white skeletons with constrained joints highlighted as red spheres), and joint rotations (visualized as coordinate axes centered at the constrained joint). Motion temporal progression is indicated by a color gradient from gray to blue.

Analysis on Large-Scale Mocap Data

Next, we thoroughly analyze key design choices of ARDY along with the effects of various hyperparameter settings. We leverage the large-scale proprietary Bones Rigplay dataset ( Bones Studio, 2026 ) , which contains around 700 hours of diverse studio-quality human motion with text descriptions.

Benchmark Evaluation

Lastly, we evaluate ARDY against both offline and online state-of-the-art baselines for text+constraints-conditioned generation on the standard HumanML3D ( Guo et al., 2022 ) dataset. For these experiments, our model is trained with a 40-frame generation horizon using 10 diffusion steps and a vanilla autoencoder tokenizer.

This public dataset contains around 30 hours of motion data with corresponding text descriptions.

Sources and Demonstrations

Figures are reproduced from Zhao et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

DECOWAM: Decoupled Whole-Body World-Action Model for Legged Mobile Manipulation
arXiv preprint

DECOWAM: Decoupled Whole-Body World-Action Model for Legged Mobile Manipulation

Siyuan Ma, Boshi Zhang, Yutian Zhang et al.

Mobile manipulation requires a robot to predict how locomotion and arm motion jointly alter future observations and control. Existing world-action models, developed largely for fixed-base platforms, do not explicitly distinguish camera ego-motion from base and arm actions.

manipulation locomotion vision reinforcement-learning control benchmark
PDF Intermediate