Ego-Exo4D Human Meshes Dataset: 4D Human Motion Reconstruction for Ego-Exo Captures

Summary

Abstract Summary

Ego-Exo4D is a large-scale dataset providing synchronized egocentric and multi-view exocentric video, a rich resource for skill learning and assessment, procedural activity understanding, and embodied AI. However, the dataset ships with only sparse 3D human pose annotations, and reconstructing dense human motion from its multi-view captures is nontrivial. To this end, we present Ego-Exo4D-HM, a large-scale dataset of 4D human motion reconstructions for Ego-Exo4D's captures, and release the accompanying reconstruction pipeline. The code, dataset, and documentation can be found at https://abhiram824.github.io/egoexo4d_human_meshes.

Key Points

Ego-Exo4D is a large-scale dataset providing synchronized egocentric and multi-view exocentric video, a rich resource for skill learning and assessment, procedural activity understanding, and embodied AI.
However, the dataset ships with only sparse 3D human pose annotations, and reconstructing dense human motion from its multi-view captures is nontrivial.
To this end, we present Ego-Exo4D-HM, a large-scale dataset of 4D human motion reconstructions for Ego-Exo4D's captures, and release the accompanying reconstruction pipeline.
The code, dataset, and documentation can be found at https://abhiram824.github.io/egoexo4d_human_meshes.

Methods

Given synchronized egocentric and exocentric video from the Ego-Exo4D activity dataset Grauman et al. (2024) , our goal is to reconstruct world-frame 3D body and hand motion of the person performing the activity. We build off SLAHMR Ye et al. (2023) , which can be naturally extended to multi-view settings.

Results

We ran our pipeline on a subset of approximately 3,200 videos from the Ego-Exo4D dataset. From here we apply a quality filter based on two per-take criteria. Reprojection self-consistency : we reproject the final optimized 3D joints through the calibrated cameras and measure the pixel distance to the corresponding 2D detections; a take is flagged if more than 10% of these samples exceed 50 px.

Sources and Demonstrations

Figures are reproduced from Maddukuri et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.