Robotics Research Hub
GestAdapt: Workspace-Conditioned Co-Speech Gesture Generation for Humanoid Robots

GestAdapt: Workspace-Conditioned Co-Speech Gesture Generation for Humanoid Robots

Bosong Ding, Xianglin Zhang, Miao Xin, Murat Kirtay, Giacomo Spigler · N/A · 2026

reinforcement-learning benchmark
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Co-speech gestures for robots must adapt not only to speech and embodiment, but also to the workspace available for performing the motion. Since the same speech can be accompanied by different gestures, a robot can respond to workspace constraints, e.g., gestures for speech next to a wall.

Abstract Summary

Co-speech gestures for robots must adapt not only to speech and embodiment, but also to the workspace available for performing the motion. Since the same speech can be accompanied by different gestures, a robot can respond to workspace constraints, e.g., gestures for speech next to a wall. In these scenarios, the robot should gesture in a suitable motion rather than simply correcting an unconstrained one. To achieve this goal, we present GestAdapt, a workspace-conditioned framework that conditions co-speech gesture generation on a prescribed wrist workspace. The GestAdapt framework learns from six complementary co-speech corpora through a shared motion representation and supports retargeting to different robot embodiments. Quantitative evaluation shows that generated motions remain close to the real-motion distribution while respecting the workspace. In a user study, gestures generated under modified workspace constraints receive a mean quality score of 3.24/5, above our no-workspace variant (2.43/5) and below the reference motions (3.68/5). In a real robot evaluation, all compared motions are retargeted to the Reachy2 humanoid robot under identical workspace constraints. Motions generated with our framework rank first in 69.7\% of comparisons, higher than our no-workspace variant baseline and retargeted ground-truth motions constrained afterward. Overall, the results support adapting gestures to the available workspace during generation, rather than modifying unconstrained trajectories afterward to satisfy workspace constraints, potentially compromising gesture naturalness.

Key Points

  • Co-speech gestures for robots must adapt not only to speech and embodiment, but also to the workspace available for performing the motion.
  • Since the same speech can be accompanied by different gestures, a robot can respond to workspace constraints, e.g., gestures for speech next to a wall.
  • In these scenarios, the robot should gesture in a suitable motion rather than simply correcting an unconstrained one.
  • To achieve this goal, we present GestAdapt, a workspace-conditioned framework that conditions co-speech gesture generation on a prescribed wrist workspace.
  • The GestAdapt framework learns from six complementary co-speech corpora through a shared motion representation and supports retargeting to different robot embodiments.

Unified Dataset

Learning spatially adaptive gestures ideally requires the same speech to be performed under different physical constraints. Existing co-speech datasets do not provide such paired recordings or explicit free-space annotations. Therefore, we combine six datasets with aligned speech and motion: BEAT [ 4 ] , SHOW [ 17 ] , Streamer [ 18 ] , TED-Expressive [ 19 ] , Trinity [ 20 ] , and ZEGGS [ 14 ] .

Unified Dataset — GestAdapt: Workspace-Conditioned Co-Speech Gesture Generation for Humanoid Robots
Fig. 2: Complementary motion and spatial support across the six co-speech corpora. The Principal Component Analysis (PCA) views show the motion distributions, while the front, side, and top views visualize the corresponding wrist-motion envelopes. Heatmap intensity indicates the number of envelopes covering each location, and colored contours show the spatial support of individual corpora.

Method

Our framework generates co-speech gestures that adapt to a specified workspace and can be executed on different robot embodiments. As illustrated in Fig. 4 , given speech, recent motion history, and a workspace box, GestAdapt generates motion with a workspace-conditioned diffusion model, applies analytic denoising guidance, and retargets the result under embodiment-specific execution constraints.

Method — GestAdapt: Workspace-Conditioned Co-Speech Gesture Generation for Humanoid Robots
Fig. 3: Unified motion representation across the six source datasets. Motion is first evaluated on each native kinematic hierarchy, and 38 anatomically corresponding joints are then selected and aligned to a common skeleton. The unified representation contains 33 dimensions for the upper body and palm axes and 45 dimensions for each hand, resulting in 123 dimensions.

Experiments

Our primary goal is that, when the workspace changes, the generated motion should remain plausible under the distribution of real human gestures, remain responsive to the accompanying speech, and stay smooth, while placing the wrists within the prescribed spatial region. Therefore, we evaluate these complementary properties using four metrics.

FGD (Fréchet Gesture Distance) measures the distributional discrepancy between generated and real gestures in the feature space of a pretrained gesture encoder.

Experiments — GestAdapt: Workspace-Conditioned Co-Speech Gesture Generation for Humanoid Robots
Fig. 4: Overview of the speech-to-robot framework. Speech, motion history, and a workspace box condition a diffusion model to generate future gesture motion. The workspace is encoded as six spatial tokens and injected through cross-attention, while analytic guidance during denoising further corrects wrist motion toward the requested workspace. The generated motion is then retargeted to the target robot and executed under embodiment-specific constraints.

Sources and Demonstrations

Figures are reproduced from Ding et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

4D Radar Perception Algorithms for Autonomous Driving: A Review
arXiv preprint

4D Radar Perception Algorithms for Autonomous Driving: A Review

Xumin Wu, Jun Zhou, Jilin Mei et al.

Research on 4D millimeter-wave radar perception algorithms has flourished in recent years, extending from signal processing and object detection to semantic segmentation, motion estimation, occupancy prediction, and dynamic scene reconstruction.

vision reinforcement-learning benchmark
PDF Beginner
A Review Of Robotic World Models For Dynamic Environments Based On Factor And Scene Graphs
arXiv preprint

A Review Of Robotic World Models For Dynamic Environments Based On Factor And Scene Graphs

Marco Giberna, Miguel Fernandez-Cortizas, Jose Luis Sanchez Lopez et al.

Models based on graphs have emerged in robotics as a powerful foundation for internal world representations, where factor and scene graphs are among the most prominent model types found in the related literature and in successful robotic solutions.

vision reinforcement-learning benchmark
PDF Beginner