IndustrialVLA-Bench: A Traceable Multi-Axis Evaluation of Open Robot Policy Models

Yiqi Wang, Zhifeng Rao, Jiaqi Zhang, Xiaoyang Li, Zhangkai Wu, Yiqun Duan, Mingkai Zheng, Fei Wang, Shan You, Taotao Cai · N/A · 2026

Summary

Abstract Summary

Open robot policies increasingly follow two paradigms: vision-language-action models (VLAs) directly map observations and instructions to actions, whereas world-action models (WAMs) incorporate learned video or world dynamics into policy learning or action generation. Although both target the same manipulation tasks and represent alternative design choices, they are commonly reported under different evaluation protocols, leaving their capability, robustness, language sensitivity, and deployment-cost trade-offs unclear. We present IndustrialVLA-Bench, an evidence-aware evaluation of six released VLA and WAM systems under a unified reporting schema. It separately evaluates clean capability on LIBERO, non-language robustness on LIBERO-Plus, instruction sensitivity on LIBERO-Para, and observed execution cost. Reported task scores aggregate three complete evaluations with distinct random seeds under a fixed checkpoint and inference configuration. Across all six systems, clean LIBERO averages differ by only 1.58 points, whereas robustness and paraphrase summaries span 14.62 and 31.08 points. Restricting every comparison to the three protocol-faithful systems preserves the effect (1.36, 14.62 and 23.10 points), so the diagnostic separation reported here does not depend on the weaker evidence tiers. We additionally report observed inference latency, peak memory, runtime mode, and an evidence status for every system. Protocol-faithful, near-reproduction, and pending-verification entries remain visibly separated; only protocol-faithful entries support strict comparisons. Rather than claiming universal superiority of either paradigm, IndustrialVLA-Bench provides traceable evidence for comparing released robot policies on shared practical criteria. Code and evaluation records are available at https://github.com/xiaoqi-7/IndustrialVLA-Bench.

Key Points

Open robot policies increasingly follow two paradigms: vision-language-action models (VLAs) directly map observations and instructions to actions, whereas world-action models (WAMs) incorporate learned video or world dynamics into policy learning or action generation.
Although both target the same manipulation tasks and represent alternative design choices, they are commonly reported under different evaluation protocols, leaving their capability, robustness, language sensitivity, and deployment-cost trade-offs unclear.
We present IndustrialVLA-Bench, an evidence-aware evaluation of six released VLA and WAM systems under a unified reporting schema.
It separately evaluates clean capability on LIBERO, non-language robustness on LIBERO-Plus, instruction sensitivity on LIBERO-Para, and observed execution cost.
Reported task scores aggregate three complete evaluations with distinct random seeds under a fixed checkpoint and inference configuration.

A Unified View of Open Robot-Policy Evaluation

A central challenge in benchmarking open VLA systems is that their architectures, action representations and runtime designs are heterogeneous: released policies differ in whether actions are produced by a single vision-language stack, a separate action module, a flow or diffusion head, or a learned video representation [ 9 , 10 , 1 , 21 , 22 , 11 ] .

Comparing such systems through final success rates alone is misleading when the scores are produced under different architectural and execution assumptions.

A Unified View of Open Robot-Policy Evaluation — IndustrialVLA-Bench: A Traceable Multi-Axis Evaluation of Open Robot Policy Models — Figure 2: Unified execution and evaluation VLA and WAM systems. Although different models use heterogeneous architectures, action representations, and runtime designs, their evaluation behavior can be mapped to a shared instruction-conditioned execution pipeline. Each stage corresponds to diagnostic evaluation dimensions, including capability, robustness, language sensitivity, and deployability and reproducibility.

Benchmark Design and Protocol

Rather than introducing a new architecture, IndustrialVLA-Bench defines an evaluation protocol that makes heterogeneous VLA releases comparable without hiding differences in benchmark support, runtime assumptions, or engineering cost.

MultiNet [ 17 ] , VLA-Eval [ 18 ] , and LeRobot [ 19 ] address complementary parts of the robot-policy lifecycle but differ from IndustrialVLA-Bench in their primary evaluation objective; Table 1 summarizes the distinction.

Experiments

We organize the evaluation around four questions: how well the evaluated systems solve standard LIBERO tasks under directly comparable conditions (RQ1); how much performance is retained when visual observations, camera configurations, robot appearance, and object layouts are perturbed in LIBERO-Plus (RQ2); how well task performance is preserved under semantics-preserving instruction paraphrases in LIBERO-Para (RQ3); and what computational and engineering costs reproduction and deployment require (RQ4).

Sources and Demonstrations

Figures are reproduced from Wang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.