Robotics Research Hub
LabInstruct: Benchmarking Situated Instructional Video Generation for Lab Procedures

LabInstruct: Benchmarking Situated Instructional Video Generation for Lab Procedures

Yuming Fu, Weijia Wu, Jing Chen, Jiahao Tang, Feifei Chen, Hongyu Zhu, Xin Jin, Alex Jinpeng Wang · N/A · 2026

manipulation vision reinforcement-learning learning-from-demonstration benchmark
PDF arXiv Code (GitHub)

Framework

N/A

License

N/A

Stars

N/A

Summary

Self-driving laboratories (SDLs) aim to automate the full experimental loop, from scientific decision-making to physical execution.

Abstract Summary

Self-driving laboratories (SDLs) aim to automate the full experimental loop, from scientific decision-making to physical execution. Ideally, AI-generated plans could be carried out directly by robotic systems, but reliable automation remains difficult in complex, open-world laboratory environments, where experiments often involve fine-grained manipulation, long-horizon procedures, and substantial variation across tasks and setups. Humans therefore remain an important execution interface between AI-generated plans and physical experiments, creating a need for clear and effective human-facing experimental guidance. Because laboratory procedures are inherently visual, spatial, and dynamic, video is particularly well suited to communicating apparatus configurations, manipulation actions, temporal dependencies, and state changes. Recent advances in video generation now make it possible to synthesize experimental demonstrations directly from an initial workspace image and a natural-language instruction. However, whether such models can reliably communicate real laboratory procedures has not been systematically studied. We introduce LabInstruct, a benchmark for situated instructional video generation in real laboratories. LabInstruct contains 204 tasks across 5 scientific disciplines, with real reference executions and structured annotations of objects, actions, contacts, and state transitions. Evaluating 8 frontier image-to-video models, we find that visually plausible generations frequently remain procedurally incorrect, revealing a substantial gap between visual realism and the reliability required for experimental instruction.

Key Points

  • Self-driving laboratories (SDLs) aim to automate the full experimental loop, from scientific decision-making to physical execution.
  • Ideally, AI-generated plans could be carried out directly by robotic systems, but reliable automation remains difficult in complex, open-world laboratory environments, where experiments often involve fine-grained manipulation, long-horizon procedures, and substantial variation across tasks and setups.
  • Humans therefore remain an important execution interface between AI-generated plans and physical experiments, creating a need for clear and effective human-facing experimental guidance.
  • Because laboratory procedures are inherently visual, spatial, and dynamic, video is particularly well suited to communicating apparatus configurations, manipulation actions, temporal dependencies, and state changes.
  • Recent advances in video generation now make it possible to synthesize experimental demonstrations directly from an initial workspace image and a natural-language instruction.

The LabInstruct Benchmark

For each task, we have the initial workspace observation $\textit{f}_{0}$, including relevant objects, spatial relationship, and scene context.

Given $\textit{f}_{0}$ and a natural-language instruction $T$, the generator synthesizes $\textit{f}_{1:N}=\mathcal{G}(\textit{f}_{0},T;\theta)$ while preserving workspace identity and continuity, executing the instructed actions in the correct order, and reaching the expected intermediate and final states.

The LabInstruct Benchmark — LabInstruct: Benchmarking Situated Instructional Video Generation for Lab Procedures
Figure 1 : Overview of LabInstruct. Text, image, and video provide increasingly rich procedural guidance, while LabInstruct evaluates generated videos for procedural correctness beyond visual plausibility.

Experiments

Wan 2.2 I2V A14B [ 36 ] (simplified as Wan 2.2), LTX 2.3 [ 37 ] , Cosmos3 Nano , Cosmos3 Super [ 40 ] , LingBot Video [ 41 ] , and MiniMax H3 [ 38 ] , as well as 2 commercial models: Seedance 2.0 [ 44 ] and Wan 3.0 [ 48 ] . All models are provided with the same initial frame and a verified natural-language instruction to ensure a controlled comparison across models for each task.

To prioritize broad coverage across both tasks and generators, we evaluate one generated sample per model–task pair under fixed model-specific generation settings.

Experiments — LabInstruct: Benchmarking Situated Instructional Video Generation for Lab Procedures
Figure 2 : LabInstruct construction pipeline. Laboratory videos are collected, filtered, segmented into tasks, and converted into human-verified specifications and instructions.

Sources and Demonstrations

Figures are reproduced from Fu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Capability-Aware Arbitration for Semantic Intent-Based Shared Control
arXiv preprint

Capability-Aware Arbitration for Semantic Intent-Based Shared Control

Zhaoda Du, Michael Bowman, Xiaoli Zhang

Shared control often allocates robot authority based on confidence in inferred human intent, assuming reliable autonomous execution. When this assumption fails, high intent confidence can cause over-helping.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
Code PDF Intermediate
GitHub ★ —
DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation
arXiv preprint

DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

Yunchao Yao, Zhuxiu Xu, Tianqi Zhang et al.

Building general-purpose dexterous manipulation policies requires benchmarks that go beyond isolated tasks to systematically evaluate policies across diverse interaction modes, sensory conditions, and robot embodiments.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
PDF Advanced