PAMI: Part Anchored Motion for Text to Human-Object Interaction Generation

Li, Chuqiao, Xie, Xianghui, Cao, Yong, Geiger, Andreas, Pons-Moll, Gerard · N/A · 2026

Summary

Abstract Summary

Text-conditioned full-body human-object interaction (HOI) generation requires synthesizing human motion and object trajectories that match the input text while remaining precisely coordinated over time. Most methods represent the human and object as separate trajectories and predict the global human-object couplings. Learning this complex, dynamically changing relationship implicitly, however, often yields object drift, missed contact, and penetration. We introduce PAMI, a Part-Anchored Motion framework for Interaction generation. Inspired by the classic Hough Transform, our key idea is to localize object motion by letting body-part anchors vote for it: we express object motion relative to multiple body-part anchors and use PamiVAE to learn an interaction latent space, decoding frame-wise weights that aggregate these part-specific votes. Building on this representation, PAMI generates interactions in a coarse-to-fine hierarchy. PamiGen first generates a coarse human-object interaction from text in this structured latent space, and PamiRefiner then recursively resolves fine-grained contact geometry using a hybrid surface-sensing representation, combining long-range probes that capture overall body-part influence with short-range sensors that resolve detailed contacts near the object surface. Experiments on InterAct show that PAMI generates more faithful interactions and more accurate human-relative object motion than previous methods, achieving 14.5% higher contact recall than the previous state of the art. Extensive ablations validate the contributions of both the part-anchored voting representation and hybrid surface-sensing refinement.

Key Points

Text-conditioned full-body human-object interaction (HOI) generation requires synthesizing human motion and object trajectories that match the input text while remaining precisely coordinated over time.
Most methods represent the human and object as separate trajectories and predict the global human-object couplings.
Learning this complex, dynamically changing relationship implicitly, however, often yields object drift, missed contact, and penetration.
We introduce PAMI, a Part-Anchored Motion framework for Interaction generation
Inspired by the classic Hough Transform, our key idea is to localize object motion by letting body-part anchors vote for it: we express object motion relative to multiple body-part anchors and use PamiVAE to learn an interaction latent space, decoding frame-wise weights that aggregate these part-specific votes.

Method

Given a text prompt $c$, a canonical object mesh $\mathcal{M}$, our goal is to generate a human motion $\mathbf{x}^{h}_{1:T}$ and a rigid-object trajectory $\mathbf{x}^{o}_{1:T}$ of $T$ frames that form one coordinated interaction. We use SMPL-H Loper et al. (2015) ; Romero et al. (2017) to represent human motion.

Since the generation must jointly satisfy the text, the human motion, and their fine-grained coupling, we adopt a coarse-to-fine design, illustrated in Fig. 2 .

Method — PAMI: Part Anchored Motion for Text to Human-Object Interaction Generation — Figure 2: Overview of PAMI. Body-part anchors vote for the object motion, and PamiVAE embeds this part-anchored interaction into a latent space ( Sec. 3.1 ). Generation then proceeds coarse-to-fine: PamiGen samples a coarse interaction from text in the latent space ( Sec. 3.2 ), which PamiRefiner refines in explicit space using hybrid long- and short-range surface-sensing probes to sharpen contacts ( Sec. 3.3 ).

Experiments

We compare against existing baselines and ablate key components: our method outperforms prior approaches on most metrics, and the ablations confirm the value of each design. PamiRefiner also generalizes zero-shot to interactions produced by other methods, improving their geometry. Experimental and evaluation setup.

Sources and Demonstrations

Figures are reproduced from Li et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.