Robotics Research Hub
EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera

EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera

Christen Millerdurai, Shaoxiang Wang, Yaxu Xie, Vladislav Golyanik, Didier Stricker, Alain Pagani · · 2026

manipulation simulation
PDF Code (GitHub)

Framework

N/A

License

N/A

Stars

0

Summary

Reconstructing the absolute 3D pose and shape of the hands from the user's viewpoint using a single head-mounted camera is crucial for practical egocentric interaction in AR/VR, telepresence, and hand-centric manipulation tasks, where sensing must remain compact and unobtrusive.

Abstract Summary

Reconstructing the absolute 3D pose and shape of the hands from the user's viewpoint using a single head-mounted camera is crucial for practical egocentric interaction in AR/VR, telepresence, and hand-centric manipulation tasks, where sensing must remain compact and unobtrusive. While monocular RGB methods have made progress, they remain constrained by depth-scale ambiguity and struggle to generalize across the diverse optical configurations of head-mounted devices. As a result, models typically require extensive training on device-specific datasets, which are costly and laborious to acquire. This paper addresses these challenges by introducing EgoForce, a monocular 3D hand reconstruction framework that recovers robust, absolute 3D hand pose and its position from the user's (camera-space) viewpoint. EgoForce operates across fisheye, perspective, and distorted wide-FOV camera models using a single unified network. Our approach combines a differentiable forearm representation that stabilizes hand pose, a unified arm-hand transformer that predicts both hand and forearm geometry from a single egocentric view, mitigating depth-scale ambiguity, and a ray space closed-form solver that enables absolute 3D pose recovery across diverse head-mounted camera models. Experiments on three egocentric benchmarks show that EgoForce achieves state-of-the-art 3D accuracy, reducing camera-space MPJPE by up to 28% on the HOT3D dataset compared to prior methods and maintaining consistent performance across camera configurations. For more details, visit the project page at https://dfki-av.github.io/EgoForce.

Key Points

  • Introduces EgoForce, advancing manipulation research.
  • Detailed evaluation demonstrates effectiveness on benchmarks and real-world scenarios.
  • Provides insights into methodology and architectural choices.

The EgoForce Framework

Fig. 2 overviews the proposed EgoForce approach for recovering camera-space 3D hand and forearm meshes from a single egocentric RGB frame. We define the hand and forearm models in Sec. 3.1 , describe our Hand-Arm Latent-Shape & Orientation (HALO) architecture in Sec. 3.2 that predicts root-relative meshes and 3D joints, lift these predictions to camera space using our Ray Space Solver in Sec.

The EgoForce Framework — EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera
Figure 3. The Ray Space Solver is a cross-camera (calibration-conditioned) module that recovers camera-space translation from 2D–3D correspondences, enabling deployment across devices with different optics.

Experimental Evaluation

Training and Evaluation Datasets. We train on six datasets: Re:InterHand ( Moon, 2023 ) , HandCO ( Zimmermann et al., 2021 ) , H2O ( Kwon et al., 2021 ) , ARCTIC ( Fan et al., 2023 ) , HO3D ( Hampali et al., 2020 ) , and HOT3D ( Banerjee et al., 2024 ) . Re:InterHand, H2O, and ARCTIC include egocentric and exocentric views; HandCO and HO3D only exocentric; HOT3D only egocentric.

In total, training data contain $3.67$M RGB images with MANO parameters and 3D joints.

Experimental Evaluation — EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera
Figure 5. Camera-space results on HOT3D. Left: egocentric input with the predicted 2D joint projections. Right: predicted meshes (left red , right blue ) and ground-truth meshes (gray) in camera space.

Additional Details about our Framework

The ForeArm Representation Model (FARM) is a lightweight, fully differentiable, parameterized mesh generator that maps a low-dimensional parameter vector to a watertight triangular mesh $\bigl(\mathbf{V},\ \mathbb{F}\bigr)$ suitable for modeling individual limb segments. Geometrically, FARM approximates the limb as a truncated cone and defines three anatomically meaningful 3D joints along its length.

In the forearm configuration, these joints correspond to the elbow, mid-forearm, and wrist.

Additional Details about our Framework — EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera
Figure 6. Influence of arm input. Providing the arm crop as an input to the network improves hand pose accuracy. In this example, the right hand is strongly occluded by the phone and the other hand, yet the model recovers a plausible 3D pose, with accurate 2D joint reprojections and a hand-arm mesh closely aligned to ground truth.

Experimental Evaluation

ARCTIC ( Fan et al., 2023 ) contains 393 sequences of hand–object interactions involving 11 articulated objects and 10 subjects, recorded from eight static and one egocentric fisheye view. We use both exocentric and egocentric frames from the official training split (1.7M images) for training, and evaluate exclusively on the egocentric subset (23K images) from the official validation split.

Experimental Evaluation — EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera
Figure 7. Influence of the variational arm prior. Without the variational prior, the forearm is often mislocalized when it is heavily occluded. With the prior, the model infers a plausible forearm pose; in this example, the forearm is entirely out of view, yet the predicted position and orientation closely match ground truth.

Additional Experiments

In-the-Wild. We define in-the-wild video sequences as videos for which no calibrated camera model is available. In these cases, we estimate camera intrinsics using AnyCalib ( Tirado-Garín and Civera, 2025 ) . While this strategy works reasonably well for pinhole optics—as illustrated in Fig.

17 (b) and in the supplementary video—our experiments in Table 9 show that accuracy degrades when applying the same procedure to fisheye cameras.

Additional Experiments — EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera
Figure 14. Unified hand–arm mesh. We attach the FARM at the MANO wrist and apply a small elbow-direction offset to avoid overlap and ensure a clean, anatomically consistent connection.

Implementation Details

Arm–Hand Crop Encoder. Given the hand and arm crops $\mathbf{I}_{H}\in\mathbb{R}^{224\times 224\times 3}$ and $\mathbf{I}_{A}\in\mathbb{R}^{112\times 112\times 3}$, patchification with patch size $p=16$ produces $N_{H}=(224/p)^{2}=14^{2}=196$ hand tokens and $N_{A}=(112/p)^{2}=7^{2}=49$ arm tokens, for a total of $N=N_{H}+N_{A}=245$ tokens.

We use a ViT-H/16 backbone (pretrained ViTPose-H weights) with token and feature dimension $d=c=1280$.

Implementation Details — EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera
Figure 15. Fusing CIT and Crop Tokens. For each crop (hand/arm), its CIT is broadcast to all patch tokens for that crop and fused in the Combine block via feature concatenation followed by a learnable projection, with a residual addition of the original token embedding. This injects crop-specific geometric context into every patch feature while allowing the model to fall back to the original mapping when the conditioning is unnecessary.

Sources and Demonstrations

Figures are reproduced from Millerdurai et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers