Robotics Research Hub
PhAIL: A Real-Robot VLA Benchmark and Distributional Methodology

PhAIL: A Real-Robot VLA Benchmark and Distributional Methodology

Sergey Arkhangelskiy · · 2026

benchmarking vla manipulation evaluation
PDF arXiv

Framework

N/A

License

N/A

Stars

0

Summary

Real-world evaluation of vision-language-action (VLA) policies still rests on binary success rate at a fixed timeout with N <= 25 rollouts per condition, almost always without confidence intervals or

Abstract Summary

Real-world evaluation of vision-language-action (VLA) policies still rests on binary success rate at a fixed timeout with N <= 25 rollouts per condition, almost always without confidence intervals or paired statistical comparison; these cohort sizes struggle to resolve close comparisons reliably. We introduce PhAIL (Physical AI Leaderboard, https://phail.ai), an open real-robot benchmark on a Franka FR3 (dataset, per-rollout artifacts, and end-to-end reference implementation) of a distributional evaluation methodology: the time-to-success cumulative distribution function (CDF) as the evaluation primitive, with two separated jobs. The first is scoring via Human-Relative Throughput (HRT), a dimensionless scalar with bootstrap confidence intervals, anchored to same-fixture human teleoperation. The second is a significance test (Kolmogorov-Smirnov, computed per-object and macro-averaged across objects). On four publicly-available VLAs, the macro-averaged KS test resolves two close comparisons (GR00T vs. ACT, OpenPI vs. ACT) at N <= 30 rollouts per (model, object) cell where binary-threshold metrics do not; the closest pair (OpenPI vs. GR00T) remains unresolved within our budget. The best evaluated VLA is approximately 7x slower per operation (RMST ratio) than the human reference.

Key Points

  • Open real-robot benchmark on Franka FR3 for distributional evaluation of VLAs.
  • Proposes time-to-success CDF and Human-Relative Throughput (HRT) metric.
  • Bootstrap confidence intervals and per-object KS significance testing.
  • Resolves close VLA comparisons that binary success rates miss.
  • Dataset and analysis pipeline open-sourced at https://phail.ai.

Time-to-Success CDF as Evaluation Primitive

The methodology has three components: the random variable and its CDF (§ 3.1 ), the scoring layer (scalars and visualizations derived from the CDF, § 3.2 ), and the significance layer (distributional tests on the CDF, § 3.3 ). The strict separation between scoring (which ranks via a scalar’s CI) and significance (which tests whether two policies’ CDFs differ at all) is the methodological core.

Time-to-Success CDF as Evaluation Primitive — PhAIL: A Real-Robot VLA Benchmark and Distributional Methodology
Figure 2: One (model, object) cell (ACT on Batteries) illustrating the scoring layer: time-to-success CDFs (left) and the corresponding P-P plot (right). Left: model and human-reference CDFs on the same axis; the asymptote below $F=1$ on the model curve is the hard-failure rate. Right: $\big(F_{\text{Human}}(t),\,F_{\text{model}}(t)\big)$ traced parametrically as $t$ sweeps. Diagonal $=$ as fast as the human reference; below-diagonal $=$ slower. The shaded area under the curve is the AUC $P(T_{\text{model}}

PhAIL: Design, Protocol, and Power Budget

The evaluation station uses a DROID-style configuration [ 11 ] : a Franka Research 3 with a Robotiq 2F-85 parallel-jaw gripper, plus over-the-shoulder external and wrist-mounted RGB cameras.

The task is bin-to-bin order picking across four object types spanning a useful slice of physical regimes – wooden spoons (rigid, elongated, multi-grasp), towels (deformable), scissors (articulated, metallic), and batteries (small, rigid). Each rollout contains many independent placement operations, which we exploit for sample size.

PhAIL: Design, Protocol, and Power Budget — PhAIL: A Real-Robot VLA Benchmark and Distributional Methodology
Figure 3: Detection rate vs. subsample size $N$ per (model, object) cell, on the three closest model pairs (left: OpenPI vs. GR00T, the closest; centre, right: the next two). Dashed line: the 0.8 power target. KS (red) climbs steeply on every pair while $F(30\,\text{s})$, $F(60\,\text{s})$, and RMST stay near the floor. KS reaches 80% within budget on the centre and right pairs, while OpenPI vs. GR00T remains unresolved at $N{=}30$ (KS at 0.63). Pairs vs. SmolVLA saturate at $N=5$ for every metric and are deferred to Appendix K .

Results

All numbers below use $\tau=240$ s for RMST, $n_{\text{boot}}=1000$ for episode-clustered bootstrap CIs, seed $0$. Per-(model, object) cell sizes are 32–46 episodes for OpenPI, GR00T, and ACT; 26–34 for SmolVLA. OpenPI and GR00T are within 0.5 percentage points of HRT, their 95% CIs overlap almost entirely ($[12.2,15.7]$ vs.

$[12.0,15.2]$), and they swap order between HRT (per-cell macro mean) and global RMST – two principled aggregations of the same primitive on this benchmark.

Sources and Demonstrations

Figures are reproduced from Arkhangelskiy See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced