RAPID: Robot Agentic Programming from Demonstrations

Yuyao Liu, Jiayuan Mao, David Hsu, Leslie Pack Kaelbling, Tomás Lozano-Pérez · N/A · 2026

Summary

Coding agents have demonstrated enormous success in solving complex programming problems.

Abstract Summary

Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration. The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (ii) action primitives for robot execution, and (iii) an interactive environment for program execution and verification. RAPID infers all three from the demonstration automatically. To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time. We evaluated RAPID in simulation on eight challenging contact-rich nonprehensile manipulation tasks as well as general prehensile manipulation tasks in the LIBERO-Pro benchmark. We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks. In all experiments, RAPID demonstrated strong performance, with generalization over object pose, shape, material, and environment. Website: https://yuyaoliu.me/projects/rapid.

Key Points

Coding agents have demonstrated enormous success in solving complex programming problems
To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration.
The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (ii) action primitives for robot execution, and (iii) an interactive environment for program execution and verification.
RAPID infers all three from the demonstration automatically
To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time.

Overview of RAPID

Problem Statement. We consider programming a reusable strategy for quasi-static rigid-body manipulation from a single visual human demonstration $D$ and a language description $l$. The program should solve a family of task instances $\mathcal{E}$, each comprising a scene-instruction pair $(E,l)$, with similar objectives but varying object appearance, geometry, pose, physical properties, and scene configuration.

No manipulation primitives, explicit success or reward signals, or instantiated simulation environments are provided a priori .

Overview of RAPID — RAPID: Robot Agentic Programming from Demonstrations — Fig. 3 : Composing primitives by relational constraints.

Object-Centric Relational Programs

We now define our object-centric relational program representation: primitives (Sec. IV-A ), their composition into task-level strategies (Sec. IV-B ), and strategy instantiation in novel scenes (Sec. IV-C ). A primitive is a reusable behavior for realizing an object-level motion effect, such as pushing an object by a displacement or flipping an object by an angle.

Agentic Programming from a Demonstration

We now describe how RAPID constructs and verifies the programs defined in Sec. IV , covering environment reconstruction, primitive construction, and strategy composition. Interactive Environment. To verify a manipulation program, RAPID reconstructs an interactive simulation environment from the initial RGB-D frame of the demonstration.

We first use a vision-language model (VLM) to identify the objects in the scene, assign semantic labels to them, and estimate their physical properties.

Agentic Programming from a Demonstration — RAPID: Robot Agentic Programming from Demonstrations — Fig. 5 : Real-world experimental setup (left) and results (right).

Experiments

In this section, we conduct experiments showing that (1) RAPID successfully closes the agentic coding loop from a single visual human demonstration to construct reusable programs for nonprehensile manipulation (Table II ); (2) our object-centric relational program representation improves the generalization of the constructed manipulation programs (Tables II and III ); and (3) RAPID can be effectively deployed on a real-world robotic system (Fig.

Sources and Demonstrations

Figures are reproduced from Liu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.