Robotics Research Hub
SAKI: Skill Assembly and Kinematic Imitation from Human Videos for Long-Horizon Mobile Manipulation

SAKI: Skill Assembly and Kinematic Imitation from Human Videos for Long-Horizon Mobile Manipulation

Yijie Lu, James Zhao, Weiming Zhi · N/A · 2026

manipulation vision learning-from-demonstration
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Learning from human videos offers a promising route to acquiring diverse manipulation skills. Extending this capability beyond tabletop settings to long-horizon mobile manipulation requires adapting and composing demonstrated interactions across changing scenes and robot configurations.

Abstract Summary

Learning from human videos offers a promising route to acquiring diverse manipulation skills. Extending this capability beyond tabletop settings to long-horizon mobile manipulation requires adapting and composing demonstrated interactions across changing scenes and robot configurations. We present Skill Assembly and Kinematic Imitation (SAKI), a framework connecting human-video skill acquisition, cross-demonstration assembly and closed-loop whole-body execution. SAKI prepares reusable object-centric skills that preserve task-critical interactions while allowing transfer paths to adapt. Given a goal and supplied task dependencies, it selects and orders skills, binds their object roles to the current scene, and carries scene estimates and robot configuration between successive skills. Whole-body kinematic imitation generates coordinated base, arm and gripper motion. During execution, persistent object estimates maintain task references across viewpoint changes, while visual feedback updates remaining trajectories. Real-robot experiments demonstrate skill reuse across layouts and the composition of independently demonstrated interactions into continuous mobile tasks, including tidying and wiping. Ablation results show that task-conditioned reference preparation substantially improves long-horizon task completion with whole-body optimisation and visual feedback held fixed. Check https://aus.bot/research/saki/ for video demos!

Key Points

  • Learning from human videos offers a promising route to acquiring diverse manipulation skills
  • Extending this capability beyond tabletop settings to long-horizon mobile manipulation requires adapting and composing demonstrated interactions across changing scenes and robot configurations.
  • We present Skill Assembly and Kinematic Imitation (SAKI), a framework connecting human-video skill acquisition, cross-demonstration assembly and closed-loop whole-body execution.
  • SAKI prepares reusable object-centric skills that preserve task-critical interactions while allowing transfer paths to adapt.
  • Given a goal and supplied task dependencies, it selects and orders skills, binds their object roles to the current scene, and carries scene estimates and robot configuration between successive skills.

Method

SAKI prepares reusable skills from human demonstrations, binds them to objects in the execution scene, and generates whole-body motion (Fig. 2 ). During execution, visual updates revise the remaining motion, and the preceding terminal configuration initialises each successor skill. State used for motion generation. SAKI maintains scene estimates, measured robot state and execution context.

Method — SAKI: Skill Assembly and Kinematic Imitation from Human Videos for Long-Horizon Mobile Manipulation
Fig. 3: Skill preparation separates interaction requirements from adjustable motion. Object roles, phases and contact requirements guide reference and grasp preparation for scene-dependent execution. Curves and contacts are schematic; the inset shows kinematic replay.

Experiments

The evaluation examines how the SAKI framework reuses interaction specifications: continuous tasks test their composition, reference ablation tests the benefit of preparation, and changed-layout trials test reuse with frozen skills. A recorded visual update follows the runtime connection from object estimates to activated motion.

Local manipulation comparisons evaluate reference-generation pipelines under shared downstream conditions. Platform and tasks. We use a Ranger Mini V3 with two Nero arms.

Experiments — SAKI: Skill Assembly and Kinematic Imitation from Human Videos for Long-Horizon Mobile Manipulation
Fig. 4: Skill assembly across demonstration scenes. Object-role bindings place prepared interaction references in a shared target scene, while the selected order and successor approaches connect them into a mobile task.

Sources and Demonstrations

Figures are reproduced from Lu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
AXIS: A Growable Community-Driven Data Engine for Scalable Robot Manipulation
arXiv preprint

AXIS: A Growable Community-Driven Data Engine for Scalable Robot Manipulation

Mengfei Zhao, Dihong Huang, Yikai Tang et al.

Learning effective robot manipulation policies requires diverse, high-quality demonstrations, yet existing data pipelines are often difficult to scale because they rely on specialized hardware, centralized operators, or fixed task suites.

manipulation vision vla learning-from-demonstration benchmark
PDF Advanced
Capability-Aware Arbitration for Semantic Intent-Based Shared Control
arXiv preprint

Capability-Aware Arbitration for Semantic Intent-Based Shared Control

Zhaoda Du, Michael Bowman, Xiaoli Zhang

Shared control often allocates robot authority based on confidence in inferred human intent, assuming reliable autonomous execution. When this assumption fails, high intent confidence can cause over-helping.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
Code PDF Intermediate
GitHub ★ —