FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity

Zhi Cao, Yining Hong, Jiaolong Yang, Yejin Choi, Li Fei-Fei, Jiajun Wu, Huang Huang · N/A · 2026

Summary

Human demonstrations offer a scalable alternative to costly robot data collection for dexterous manipulation, but most human datasets lack force measurements.

Abstract Summary

Human demonstrations offer a scalable alternative to costly robot data collection for dexterous manipulation, but most human datasets lack force measurements. Dexterous manipulation requires not only visual and proprioceptive understanding but also the ability to anticipate and regulate interaction forces. To scale force-aware human-robot co-training, we introduce EgoForce, a large-scale human tactile dataset capturing force-rich manipulation experience, and combine it with robot demonstrations for pretraining. We present FACT, a robot foundation model that cascades from prospective visual planning to force-aware refinement for dexterous manipulation. FACT comprises two jointly trained components. A Track-Guided Coarse Planner jointly predicts actions, future video, and visual tracks. Future video and visual tracks also provide a shared task representation that facilitates cross-embodiment knowledge transfer and coarse action generation. Building on this coarse plan, a Force-Action Joint Refiner jointly predicts future force and refines the coarse action, allowing anticipated force and action to shape each other during refinement. We evaluate the model on ten tasks requiring fine-grained coordination and force feedback, including tool use, musical instruments, and precision laboratory procedures. Using as few as 10 robot trajectories with 90 human trajectories for fine-tuning, FACT outperforms both baselines, with success-rate gains of up to 60 percentage points over T-Rex. Ablations show that force supervision in human pretraining data is crucial for downstream robot performance after fine-tuning, while visual track prediction and training with future-force prediction improve dexterous execution. Experiments varying the human-robot data ratio during fine-tuning further suggest that FACT learns from human demonstrations nearly as effectively as from robot demonstrations.

Key Points

Human demonstrations offer a scalable alternative to costly robot data collection for dexterous manipulation, but most human datasets lack force measurements.
Dexterous manipulation requires not only visual and proprioceptive understanding but also the ability to anticipate and regulate interaction forces.
To scale force-aware human-robot co-training, we introduce EgoForce, a large-scale human tactile dataset capturing force-rich manipulation experience, and combine it with robot demonstrations for pretraining.
We present FACT, a robot foundation model that cascades from prospective visual planning to force-aware refinement for dexterous manipulation.
FACT comprises two jointly trained components

The EgoForce Dataset

Human demonstrations provide a scalable source of dexterous manipulation experience, but existing human data captures vision without the forces involved in interaction, while collecting force-supervised data at comparable scale on robots is expensive and difficult.

We therefore develop EgoForce , a large-scale human tactile dataset paired with a collection platform that captures synchronized vision, hand motion, and calibrated tactile force for FACT pretraining.

The EgoForce Dataset — FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity — Fig. 2: EgoForce task and skill distribution. Left: episode share by task category, with over two thirds of the corpus covering contact-rich fine manipulation across diverse task categories. Right: atomic-skill annotations on a log scale, showing a broad distribution of dexterous, force-dependent skills beyond simple transport.

The FACT Framework

FACT is a flow-matching model for transferring dexterous manipulation from large-scale human experience to robot execution. FACT cascades from prospective visual planning to force-aware refinement ( Figure 3 ) through two jointly trained components. The Track-Guided Coarse Planner jointly predicts future video, visual tracks, and actions.

Future video and tracks provide a shared representation of task progress across human and robot embodiments, allowing human experience to shape coarse robot motion.

The FACT Framework — FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity — Fig. 3: FACT architecture. The Track-Guided Coarse Planner jointly denoises future video, visual tracks, and a coarse action chunk with a block-causal video DiT; per-token noise conditioning lets clean context and noisy targets coexist. At the $\tau{=}0.4$ handoff, video and tracks are read out and the coarse hidden states are frozen. The Force–Action Joint Refiner reads them through cached cross-attention and co-denoises the remaining action ($\tau_{a}\!:0.4{\to}0$) with continuous future fingertip force ($\tau_{f}\!:1{\to}0$), using observed tactile signals as clean context.

Experiments

We evaluate FACT on a ten-task real-robot suite. All policies are evaluated on a bimanual YAM station with two Sharpa Wave hands (22 joints, five-fingertip 6-axis force sensing), equipped with a RealSense D435 ego-view camera and two ZED Mini wrist cameras.

For each evaluation task we collect 10 robot teleoperation trajectories using a Manus-glove and wrist-tracker interface and 90 human demonstrations collected with a Wuji tactile glove and wrist-tracker interface, resulting in a 100-trajectory mixture dataset per task.

Experiments — FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity — Fig. 4: Evaluation randomization. Overlaid initial frames from six evaluation tasks illustrate rollout-level variation in object positions and poses.

Sources and Demonstrations

Figures are reproduced from Cao et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.