CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization

Summary

Abstract Summary

Prior research on learning from humans often requires real human demonstrations. But advances in video generation models offer a new paradigm: we can now prompt these generators to make synthetic videos of the task. On the plus side, these synthetic videos take place in the robot's current environment, and do not require actual human teaching. However, while the generated videos appear to show plausible motions, they lack both physical consistency and robot grounding, and thus the robot cannot directly roll out the trajectory shown by the video. In this paper we seek to formalize the process of going from synthetic video generation to real-world robot trajectories for zero-shot manipulation tasks. We recognize that this problem is an instance of constrained optimization. The object trajectories extracted from the synthetic video provide an initial guess at the correct motion, and the robot must refine these trajectories (for instance, within a digital twin) while enforcing physical feasibility and semantic consistency. Our optimization-based formalism provides insights into the video generation properties necessary for success. Specifically, we show that the video generation must probabilistically seed the search within a semantic basin of attraction where the trajectory can be locally modified without losing its task alignment. Our analysis further reveals that state-of-the-art approaches can be viewed as instances of our unified formalism with key simplifications. In our experiments we remove these simplifications to more closely match the underlying optimization, leading to more robust zero-shot trajectories from synthetic videos and improving average task success by 34.1% compared to the best-performing baseline. See videos at our project website: https://collab.me.vt.edu/CASTER

Key Points

Prior research on learning from humans often requires real human demonstrations
But advances in video generation models offer a new paradigm: we can now prompt these generators to make synthetic videos of the task.
On the plus side, these synthetic videos take place in the robot's current environment, and do not require actual human teaching.
However, while the generated videos appear to show plausible motions, they lack both physical consistency and robot grounding, and thus the robot cannot directly roll out the trajectory shown by the video.
In this paper we seek to formalize the process of going from synthetic video generation to real-world robot trajectories for zero-shot manipulation tasks.

Problem Statement

We consider the setting of zero-shot tabletop robot manipulation. An example of this setting is shown in Figure 1 . The robot is given a natural-language task description $\tau$ and an initial visual observation $o$ of a scene containing $M$ objects.

The setting is zero-shot in the sense that the robot must execute the task within a finite horizon without access to task-specific robot demonstrations (i.e., teleoperation) or any additional real-world interaction. Trajectories.

Problem Statement — CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization — Figure 1: Using video generation models at inference time to guide real-world robot trajectories. The generated video looks plausible, but may make mistakes that prevent directly transferring this video to the real robot’s motion. We bridge the gap between generation and deployment by developing an optimization-based formalism. Under this formalism the robot uses multiple synthetic videos to seed its search and identify task-critical features. By matching these features while enforcing semantic consistency and physical feasibility, we can transfer erroneous videos to successful zero-shot trajectories.

Introducing a Unified Formalism

Generated videos provide insights into the correct behavior through their visual content, but these videos do not directly provide executable robot actions. Prior work has explored several approaches for zero-shot robot manipulation based on synthetic video generation. Rather than starting with another solution, we begin by focusing on the underlying problem .

More specifically, we formulate leveraging synthetic videos as constrained optimization.

Introducing a Unified Formalism — CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization — Figure 2: Overview of CASTER . (Top Left) Given an initial RGB-D observation and a task prompt, we (Top) generate synthetic videos of a human performing that task in the current environment and (Bottom Left) build a digital twin of the scene. From the synthetic videos we extract the object trajectories, which then serve as initial guesses for trajectory optimization within the digital twin. (Bottom) The trajectories extracted from synthetic videos do not need to be perfect: instead, they must lie within a semantic basin of attraction where the optimization can locally modify the trajectory shape without removing its semantic meaning. (Bottom Right) We perform constrained optimization so that the robot’s actions in the digital twin cause the objects to move similarly to their video trajectories, while enforcing kinematic constraints so that the output can be directly executed by the real robot.

Approximating the Underlying Solution

In Section IV we present zero-shot manipulation from synthetic videos as a constrained optimization problem. While current methods introduce simplifications, in this section we present CASTER (Figure 2 ), an approach that more closely follows Equation ( 4 ). Practically, this leads to two distinctions from previous work.

First, instead of optimizing to exactly follow the object’s video trajectory, we preserve task features $\phi$.

Approximating the Underlying Solution — CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization — Figure 3: Examples of our real-world tasks. (Left) Generated videos. (Middle) Digital twin rollouts. (Right) Real-world execution. The rows show four tasks: placing a banana in a pan, stacking bowls, pouring from a kettle, and toasting bread. See our supplemental video for more examples.

Experiments

We conduct real-world experiments to evaluate the effectiveness of our unified framework. Within these experiments the robot is expected to perform the 12 tasks from Table I on its first attempt, without any real-world demonstrations or additional interactions. To find the correct trajectory the robot uses synthetically generated videos of the human performing the task. Examples of these videos are shown in Figure 3 .

Sources and Demonstrations

Figures are reproduced from Dai et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.