sensVLA: Spatially-Grounded Vision-Language-Action Model for Autonomous Wheel Loader

Gopi Krishna Erabati, Bjarne Johannsen, Angus Stewart, Vardeep Singh Sandhu · N/A · 2026

Summary

Autonomous wheel-loader control requires joint reasoning over task semantics, egocentric vision, proprioception, and 3D scene geometry.

Abstract Summary

Autonomous wheel-loader control requires joint reasoning over task semantics, egocentric vision, proprioception, and 3D scene geometry. We present sensVLA, a Vision-Language-Action (VLA) architecture that combines a Qwen3-2B Vision-Language Model (VLM) with a fully trainable transformer action expert trained by flow-matching velocity regression. sensVLA routes Bird's-Eye-View (BEV) features, extracted from fused front and rear lidar, directly to the action expert through a dedicated cross-attention pathway, while the VLM consumes front and rear RGB views to provide task-conditioned semantic context. This design decouples spatial grounding from linguistic reasoning while preserving interaction between both streams at decision time. The expert predicts six action dimensions: longitudinal velocity, steering, body-frame displacement, arm rate, and bucket rate. On a real-world dataset from a wheel loader, sensVLA reaches aggregate per-step parity with a strong camera-only baseline and reduces longitudinal velocity RMSE by 28% and displacement error by 9% on loading centric scenarios. It also degrades 29% less when the camera stream is corrupted or removed, evidencing that explicit spatial grounding improves accuracy and fault-tolerance for heavy equipment autonomy.

Key Points

Autonomous wheel-loader control requires joint reasoning over task semantics, egocentric vision, proprioception, and 3D scene geometry.
We present sensVLA, a Vision-Language-Action (VLA) architecture that combines a Qwen3-2B Vision-Language Model (VLM) with a fully trainable transformer action expert trained by flow-matching velocity regression.
sensVLA routes Bird's-Eye-View (BEV) features, extracted from fused front and rear lidar, directly to the action expert through a dedicated cross-attention pathway, while the VLM consumes front and rear RGB views to provide task-conditioned semantic context.
This design decouples spatial grounding from linguistic reasoning while preserving interaction between both streams at decision time.
The expert predicts six action dimensions: longitudinal velocity, steering, body-frame displacement, arm rate, and bucket rate.

System Overview and Data Pipeline

The proprietary dataset consists of lidars, RGB cameras, and IMU sensors. Each sensor is stored asynchronously and during training, temporally synchronized to the control timestamps. Each sample consists of the RGB images, a short natural-language task prompt, 0.5s of accumulated lidar data, the proprioceptive state, and the action targets.

The proprioceptive state consists of forward velocity, steering angle, imu data. The action targets are forward velocity, steering, body-frame displacement, arm rate and bucket rate.

System Overview and Data Pipeline — sensVLA: Spatially-Grounded Vision-Language-Action Model for Autonomous Wheel Loader — Fig. 2 : Qualitative predictions on two validation frames (loading approach). On the BEV, the ground-truth 2 s ego-path is drawn in green and sensVLA’s prediction in red .

sensVLA Architecture

The sensVLA model ingests four modalities: an RGB image, a short text prompt, proprioceptive state including a ten-step history, and a temporally-accumulated lidar point cloud, and produces a chunk of ten future 6-DoF actions as shown in fig. 1 .

The RGB image and text prompt are processed by a LoRA-adapted [ 14 ] Qwen3-2B VLM [ 20 ] to produce task-conditioned contextual representations, while the lidar cloud is encoded into a BEV feature grid by a frozen PointPillars backbone.

Experiments

We collect data onboard a wheel loader and curate a dataset comprising 200K training chunks and 40K validation chunks. Each sample contains RGB images, LiDAR and IMU measurements. Our policy is built upon Qwen3-2B-VL [ 20 ] , where the language decoder is truncated to 14 transformer blocks. LoRA adapters ($r=16$, $\alpha=16$, dropout $0.05$) are inserted into the final 8 decoder layers.

The PointPillars encoder, pretrained on a separate in-house end-to-end task, is kept frozen throughout training.

Sources and Demonstrations

Figures are reproduced from Erabati et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.