Robotics Research Hub
Continuous Actions from Discrete Minds: Latent-Aligned Planning for End-to-End Autonomous Driving

Continuous Actions from Discrete Minds: Latent-Aligned Planning for End-to-End Autonomous Driving

Ruoyu Yao, Yusen Xie, Qingzhao Liu, Pei Liu, Zewei Yang, Yipeng Zhu, Xiaolong Wang, Jun Ma · N/A · 2026

vision vla planning benchmark
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Bridging the gap between the discrete reasoning of Vision-Language Models and the continuous, physics-constrained nature of autonomous driving remains a significant challenge.

Abstract Summary

Bridging the gap between the discrete reasoning of Vision-Language Models and the continuous, physics-constrained nature of autonomous driving remains a significant challenge. In this work, we introduce LaPla, a unified Vision-Language-Action (VLA) framework featuring latent-aligned planning to seamlessly ground semantic understanding in precise motion execution. We first design an action tokenizer based on a residual vector-quantized variational autoencoder (VQ-VAE), capturing vehicle kinematics and encoding trajectory features into a structured latent space. Rather than discrete codebook lookups that inevitably introduce quantization errors, LaPla repurposes this representation as a physical prior to bridge the modality gap between high-dimensional semantics and the raw action space. Specifically, given multimodal inputs integrating multi-view images, historical actions, and textual instructions, LaPla incorporates concurrent action queries to causally attend to the multimodal context in a single forward pass, projecting hidden states directly into the pretrained VQ-VAE latent space. The frozen decoder then translates these continuous latents into actions, effectively eliminating quantization errors and ensuring physically plausible trajectories while bypassing time-consuming autoregressive generation. Extensive experiments on the nuScenes benchmark demonstrate that LaPla achieves competitive open-loop performance, reducing long-horizon L2 error by 15.52% compared to state-of-the-art VLA methods. Closed-loop evaluations on the NVIDIA AlpaSim simulator further confirm its superior capability in ensuring smooth driving progress, improving the success rate by 33.34 percentage points with significantly reduced inference latency.

Key Points

  • Bridging the gap between the discrete reasoning of Vision-Language Models and the continuous, physics-constrained nature of autonomous driving remains a significant challenge.
  • In this work, we introduce LaPla, a unified Vision-Language-Action (VLA) framework featuring latent-aligned planning to seamlessly ground semantic understanding in precise motion execution.
  • We first design an action tokenizer based on a residual vector-quantized variational autoencoder (VQ-VAE), capturing vehicle kinematics and encoding trajectory features into a structured latent space.
  • Rather than discrete codebook lookups that inevitably introduce quantization errors, LaPla repurposes this representation as a physical prior to bridge the modality gap between high-dimensional semantics and the raw action space.
  • Specifically, given multimodal inputs integrating multi-view images, historical actions, and textual instructions, LaPla incorporates concurrent action queries to causally attend to the multimodal context in a single forward pass, projecting hidden states directly into the pretrained VQ-VAE latent space.

Methodology

Our methodology is structured in two stages, as shown in Fig. 2 . We first pretrain a trajectory VQ-VAE and freeze its decoder to serve as a frozen latent prior that encapsulates physically plausible motion patterns.

Building upon this prior work, we then introduce LaPla , a unified VLA framework that predicts continuous latent vectors to directly drive the frozen decoder for trajectory generation, while jointly performing visual question answering via shared multimodal context.

Methodology — Continuous Actions from Discrete Minds: Latent-Aligned Planning for End-to-End Autonomous Driving
Fig. 3: Asymmetric action representations. We employ discrete quantization as an information bottleneck to structure historical context in a coarse-to-fine manner, while predicting continuous latents for the future to prevent quantization errors and ensure high-precision trajectory decoding.

Experiment

We comprehensively evaluate our approach in both open-loop and closed-loop experiments. Benchmark tests are conducted on nuScenes [ 19 ] and AlpaSim [ 20 ] , respectively. nuScenes. Following the split of [ 6 ] on the nuScenes dataset ($\sim$30K frames in total, annotated at 2 Hz), we evaluate open-loop performance via L2 distance ($\downarrow$) and collision rate ($\downarrow$).

Additionally, we incorporate VQA annotations from [ 4 ] to enhance reasoning capabilities. AlpaSim.

Experiment — Continuous Actions from Discrete Minds: Latent-Aligned Planning for End-to-End Autonomous Driving
Fig. 4: The open-loop planning performance of LaPla. Visualizations span diverse driving conditions including rain (Scene 1), nighttime (Scene 2), lane-borrowing obstacle avoidance (Scene 3), near-stationary traffic (Scene 4), urban overtaking (Scene 5), and pedestrian yielding (Scene 6). LaPla preserves trajectory smoothness under perceptual degradation and consistently captures the correct maneuver intent in tactical scenarios.

Sources and Demonstrations

Figures are reproduced from Yao et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers