POIL: Point-based One-Shot Imitation Learning with Stable Dynamical Systems

Sang Min Kim, Jinwoo Seo, Hyeongjun Heo, Junho Lee, Yonghyeon Lee, Young Min Kim · N/A · 2026

Summary

We present POIL, a point-based one-shot imitation learning framework with stable dynamical systems.

Abstract Summary

We present POIL, a point-based one-shot imitation learning framework with stable dynamical systems. While one-shot imitation avoids collecting extensive demonstrations, successful one-shot manipulation requires not only transferring a demonstrated trajectory to a novel object but also executing it robustly under changing scene conditions, grasp configurations, and external disturbances. POIL addresses both problems through a shared representation: a set of 3D points on the object's functional part, used jointly for trajectory transfer and closed-loop execution. The one-shot transfer from the demonstrated trajectory is enabled with point correspondences. POIL grounds the shared functional part with a multi-modal large language model, and transfers the trajectory across viewpoint, pose, and object category changes. During execution, multi-view tracking observes the same points online, and Point-set BCSDM drives them in closed loop by projecting per-point velocities onto a single rigid-body twist computed from the tracked points alone. This extends stable dynamical models from an SE(3) pose to a point set without requiring a known 3D model or pose estimator. We show that at the goal the controller becomes a gradient flow on the classical SO(3) potential, so its terminal phase inherits the almost-global convergence of that potential under a rigid-object assumption. Across simulation and real-robot experiments, POIL transfers a single demonstration across object category, grasp pose, and goal geometry, while recovering from external disturbances during execution. Project page: https://sangminkim-99.github.io/poil

Key Points

We present POIL, a point-based one-shot imitation learning framework with stable dynamical systems
While one-shot imitation avoids collecting extensive demonstrations, successful one-shot manipulation requires not only transferring a demonstrated trajectory to a novel object but also executing it robustly under changing scene conditions, grasp configurations, and external disturbances.
POIL addresses both problems through a shared representation: a set of 3D points on the object's functional part, used jointly for trajectory transfer and closed-loop execution.
The one-shot transfer from the demonstrated trajectory is enabled with point correspondences
POIL grounds the shared functional part with a multi-modal large language model, and transfers the trajectory across viewpoint, pose, and object category changes.

Method

POIL takes a single demonstration and transfers the manipulation skill to a novel target object that shares a functional part, executing the task via a closed-loop controller with convergence guarantees (see Fig. 1 ).

The two stages share one representation – a set of 3D points on the object’s functional part – so that a single abstraction yields both the object generalization of the transfer stage and the object-centric convergence guarantee of the execution stage.

Method — POIL: Point-based One-Shot Imitation Learning with Stable Dynamical Systems — Fig. 2: Handle detection from multi-view mug images. Multi-view prompting enables Rex-Omni to localize the handle using cross-view context.

Experiments

We organize the evaluation into four studies. We isolate the performance of functional-part point transfer (Sec. IV-A ) and control (Sec. IV-B ) with ground-truth inputs. We then evaluate the full pipeline (Sec. IV-C ) across pose, grasp, object, and goal variations, as well as external disturbances. We use $N_{k}=10$ functional-part points, $\eta=3$, and 40 support points for all experiments.

We evaluate whether our functional-part point transfer remains accurate under large viewpoint and pose changes. Baselines.

Experiments — POIL: Point-based One-Shot Imitation Learning with Stable Dynamical Systems — Fig. 3: Trajectory adaptation to a new fixture. User-clicked correspondences between the demonstration’s straight rack (left) and the target’s L-shaped rack (right) define a TPS warp $\Phi$. Applying $\Phi$ to every waypoint of the reference trajectory yields the adapted trajectory in the target scene.

Sources and Demonstrations

Figures are reproduced from Kim et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.