Robotics Research Hub
Keep the Effect, Drop the Actor: Programmable Effect-to-Execution World-Action Models

Keep the Effect, Drop the Actor: Programmable Effect-to-Execution World-Action Models

Junyi Hu, Zhewen He, Zhenhua Li, Yi Fang · N/A · 2026

manipulation vision reinforcement-learning learning-from-demonstration
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

A robot demonstration records two things in the same frames: what happened to the objects, and how one particular arm made it happen. We condition on the first.

Abstract Summary

A robot demonstration records two things in the same frames: what happened to the objects, and how one particular arm made it happen. We condition on the first. A demonstration is compiled into an effect program: the 3D keypoint trajectories of the objects that moved, two points marking where each was held, and the configuration the scene ends in, with the demonstrator removed. PEWAM, a 71.5M-parameter world-action model, generates effect, robot execution, action and terminal state as four streams with independent flow-matching times, so clamping a program and sampling the execution turns inference into programming, re-solved closed loop from the live scene. On held-out LIBERO-Goal tasks, one demonstration's program completes 40 of 90 episodes, where the same backbone given a goal image or language, and published demonstration-conditioned methods, complete at most 19; on three of Meta-World's held-out classes it exceeds the best published results, though not on the five-class mean. Because a program is a set of coordinates, a person can edit it: the placement follows a shifted terminal state and the grasp turns with rotated contact points. The same program runs on four robot arms without retraining, and after a push, re-solving completes 23 of 60 episodes where replaying the demonstration completes 6. On a Franka arm, fine-tuned on real demonstrations of other tasks, programs compiled from single human videos complete 36 of 40 trials, against 22 for the same backbone conditioned on the video's last frame as a goal image.

Key Points

  • A robot demonstration records two things in the same frames: what happened to the objects, and how one particular arm made it happen.
  • We condition on the first
  • A demonstration is compiled into an effect program: the 3D keypoint trajectories of the objects that moved, two points marking where each was held, and the configuration the scene ends in, with the demonstrator removed.
  • PEWAM, a 71.5M-parameter world-action model, generates effect, robot execution, action and terminal state as four streams with independent flow-matching times, so clamping a program and sampling the execution turns inference into programming, re-solved closed loop from the live scene.
  • On held-out LIBERO-Goal tasks, one demonstration's program completes 40 of 90 episodes, where the same backbone given a goal image or language, and published demonstration-conditioned methods, complete at most 19; on three of Meta-World's held-out classes it exceeds the best published results, though not on the five-class mean.

Method

The observation $O_{t}$ holds two RGB views, proprioception and the entities’ current keypoints as an anchor.

For $K$ entities, the effect $E\in\mathbb{R}^{H\times K\times P\times 3}$ is the world-frame trajectory, over a horizon $H{=}16$, of $P$ points per entity: the eight corners and the centre of its oriented box, plus two contact points $c^{(1)},c^{(2)}$ at the centroids of the two finger-side contact patches at the moment of grasp.

Method — Keep the Effect, Drop the Actor: Programmable Effect-to-Execution World-Action Models
Figure 2: PEWAM end to end. (1) Compiling a demonstration into an effect program (§ 3.1 ); (2) the training data; (3) one model with a flow-matching time per stream (§ 3.2 ); (4) the closed loop of Algorithm 1 . Bottom: contact as points, and three operators of the same loop.

Experiments

We evaluate on held-out tasks and objects of LIBERO, on the test classes of Meta-World ML10 and ML45, on RLBench, on four robot arms and on a real Franka arm programmed by human video (Table 6 ); simulated rollouts use the benchmark’s own success check. LIBERO episodes re-plan every $K_{\text{exec}}{=}4$ steps under a 400-step cap, with one fixed environment and sampling seed and 30 episodes per task and cell.

Experiments — Keep the Effect, Drop the Actor: Programmable Effect-to-Execution World-Action Models
Figure 3: One program per held-out task, executed closed loop. Left of each pair: the program on its demonstration with the robot removed (object track white, terminal state $G^{\star}$ yellow, contact points red and blue). Right: the last frame of the execution, with the object’s path in white and its length in steps above. The last pair is a LIBERO-Object episode with a held-out object (the sauce bottle into the basket). Episodes are the lowest-numbered success of each task in its held-out evaluation; failures and intermediate frames are in Appendix O , and the baselines on the same episodes in Appendix Figure 9 .

Sources and Demonstrations

Figures are reproduced from Hu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Capability-Aware Arbitration for Semantic Intent-Based Shared Control
arXiv preprint

Capability-Aware Arbitration for Semantic Intent-Based Shared Control

Zhaoda Du, Michael Bowman, Xiaoli Zhang

Shared control often allocates robot authority based on confidence in inferred human intent, assuming reliable autonomous execution. When this assumption fails, high intent confidence can cause over-helping.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
Code PDF Intermediate
GitHub ★ —
DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation
arXiv preprint

DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

Yunchao Yao, Zhuxiu Xu, Tianqi Zhang et al.

Building general-purpose dexterous manipulation policies requires benchmarks that go beyond isolated tasks to systematically evaluate policies across diverse interaction modes, sensory conditions, and robot embodiments.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
PDF Advanced