VLAQuantBench: Closed-Loop Evaluation of Post-Training Quantization for Vision-Language-Action Models

Jiuyi Xu, Qing Jin, Meida Chen, Song Wang, Yang Sui, Yangming Shi · N/A · 2026

Summary

Abstract Summary

Post-training quantization reduces the memory requirements of vision-language-action (VLA) models, but precision selection must account for the interaction between layer scope, numerical format, and calibration. We introduce \textbf{VLAQuantBench}, a controlled evaluation with 409 runs and 94,574 simulation episodes: four models on LIBERO, with X-VLA additionally evaluated on three simulation benchmark families. Under uncalibrated W4A4 round-to-nearest quantization, expanding a $\pi_{0.5}$ action-head subset from 126 to 167 layers raises success from 7.0\% to 70.5\%. Fixed-observation replay confirms a corresponding numerical recovery. Two-episode calibration removes the severe joint failures in the tested subsets, whereas the same smoothing-and-clipping recipe lowers $\pi_0$ success and does not recover OpenVLA-OFT end-to-end. For OpenVLA-OFT, protecting one 28,672-parameter output projection instead restores near-baseline success: the remaining 441 eligible linear layers retain W3 on LIBERO-Long or eight-bit activations across all four suites. Task-clustered intervals support the large failure and recovery contrasts. These results establish recipe-dependent interactions and identify concrete precision assignments, rather than universal layer-sensitivity rules. Real-kernel and physical-robot measurements complement the accuracy analysis. Code, configurations, and episode records are publicly available at https://github.com/jiuyixu25/VLAQuantBench.

Key Points

Post-training quantization reduces the memory requirements of vision-language-action (VLA) models, but precision selection must account for the interaction between layer scope, numerical format, and calibration.
We introduce \textbf{VLAQuantBench}, a controlled evaluation with 409 runs and 94,574 simulation episodes: four models on LIBERO, with X-VLA additionally evaluated on three simulation benchmark families.
Under uncalibrated W4A4 round-to-nearest quantization, expanding a $\pi_{0.5}$ action-head subset from 126 to 167 layers raises success from 7.0\% to 70.5\%.
Fixed-observation replay confirms a corresponding numerical recovery
Two-episode calibration removes the severe joint failures in the tested subsets, whereas the same smoothing-and-clipping recipe lowers $\pi_0$ success and does not recover OpenVLA-OFT end-to-end.

Benchmark Design

The benchmark separates three questions: whether a quantized policy completes tasks, which selected layers account for a change in performance, and whether an implementation saves memory or time. Task metrics are primary outcomes, while action deviations and activation statistics are auxiliary diagnostics. Figure 3 in App. A summarizes the evaluation axes.

We partition each policy into a vision encoder (VE), multimodal projector (MP) where present, language backbone (LLM), and action head (AH).

Benchmark Design — VLAQuantBench: Closed-Loop Evaluation of Post-Training Quantization for Vision-Language-Action Models — Figure 2: Uncalibrated RTN same-observation, same-noise replay on 20 held-out $\pi_{0.5}$ trajectories (219 predicted chunks). (a) W4A4 recovery is present in normalized action-chunk error before environment feedback; W4/W4A8 do not show the 126-to-167-layer reversal. (b) Points and whiskers give the median and interquartile range of the per-chunk error-norm ratio, not confidence intervals. $A$ denotes attention and $B$ adaRMS. The dashed value 1 is an upper bound under exact vector addition; cancellation can give smaller ratios.

Experimental Setup

OpenVLA-OFT combines a LLaMA-2-7B backbone with parallel action decoding and a four-linear-layer L1-regression head ( Kim et al., 2025a ) . The $\pi_{0}$ and $\pi_{0.5}$ policies use PaliGemma with flow-matching action experts. $\pi_{0.5}$ additionally uses adaptive normalization conditioning ( Black et al., 2025b ; Black et al., 2025a ) .

X-VLA uses Florence-2 with a soft-prompted transformer action head ( Zheng et al., 2026a ) .

Results

Weight-only W8 closely tracks the LIBERO baselines in Table 1 , whereas W4 is not uniformly lossless. OpenVLA-OFT’s Goal success falls from 96.0% to 82.0%. The W3 result on OpenVLA-OFT-Long is particularly poor (8.0%, compared with 88.5% at W4). Thus, conclusions drawn from an average across suites can hide substantial losses in individual settings. Activation quantization produces larger differences.

OpenVLA-OFT’s W4A8 success is 21.5/2.5/2.0/76.0% on Spatial/Object/Goal/Long.

Evaluation-Stack Checks

Three implementation checks are essential to reproducing the evaluated policy. Changing MuJoCo versions altered the settling of a stored LIBERO initial state. We pin MuJoCo to 3.3.2 for OpenVLA-OFT and the $\pi$ models and examine per-task baselines; the X-VLA environment pins MuJoCo 3.1.6, which predates the change, and the version is recorded with its diagnostics.

The version and per-task baseline are recorded with the evaluation configuration.

Sources and Demonstrations

Figures are reproduced from Xu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.