3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training

3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training

Jiaxin Shi, Xidong Zhang, Fucai Zhu, Zhe Li, Siyu Zhu, Weihao Yuan · N/A · 2026

Framework

N/A

License

N/A

Stars

N/A

Summary

We propose a 3D-thinking-guided co-training framework that enables vision-language-action (VLA) models to perform 3D spatial reasoning implicitly during action prediction.

Abstract Summary

We propose a 3D-thinking-guided co-training framework that enables vision-language-action (VLA) models to perform 3D spatial reasoning implicitly during action prediction. Our core insight is that 3D geometry perception and 3D spatial reasoning are distinct capabilities that can be disentangled and injected at different feature hierarchies. During training, three tightly coupled components work in concert primarily within the latent space: (1) To gain geometric priors, a latent 3D geometry perception module aligns intermediate visual features with a 3D foundation model, acquiring low-level geometric cues without architectural modifications to the VLM backbone. (2) Complementing this, an online 3D reasoning distillation module mitigates the prompt-induced reasoning gap via a shared reasoning anchor token. During 3D VLM co-training, this anchor is emitted as the first output token to robustly encode spatial priors. During VLA training, it serves as an input token inserted between the task and action instructions, transferring high-level spatial thinking from explicit teacher reasoning prompts to student action prompts without chain-of-thought text generation. (3) These disentangled geometric and reasoning features are then united by a spatially augmented action integration, which jointly injects them into the action-query tokens as hierarchical spatial conditions to prevent action shortcuts. At deployment, our method retains only its lightweight adapters to perform implicit 3D reasoning, discarding the 3D foundation model and the teacher branch used for supervision. Consequently, it operates purely on 2D images without 3D sensors, external models, or explicit text generation while preventing catastrophic forgetting of the pretrained VLM, achieving state-of-the-art performance on LIBERO, LIBERO-PLUS, SimplerEnv, and real-world manipulation tasks.

Key Points

  • Addresses dexterous manipulation challenges
  • Integrates vision-language models for robotic action generation
  • Uses simulation or synthetic data for training
  • Uses knowledge distillation to compress models
  • Leverages 3D geometric reasoning for perception

Method

We present 3DThinkVLA, a 3D-thinking-guided VLA co-training framework as shown in Figure 2 . The key idea is to enrich action prediction with two complementary latent cues during training: (i) low-level geometry-aware cues distilled from the 3D geometry foundation model and (ii) high-level reasoning-aware cues transferred from a teacher reasoning pathway and online self-distillation strategy.

Importantly, these auxiliary supervision sources ( i.e. , the 3D foundation model and the teacher reasoning pathway) are utilized only during training.

Method — 3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training
Figure 3: Qualitative Results on LIBERO-Plus. Task 1: “Put the black bowl in the bottom drawer of the cabinet and close it.” Task 2: “Put the black bowl in the top drawer of the cabinet and close it.”

Experiments

We adopt the StarVLA framework Ye et al. (2026a) , utilizing Qwen3-VL 2B Bai et al. (2025) as the base model and using OFT-style action head for action prediction. All models are trained on 8 NVIDIA A100 (80GB) GPUs, and the fine-tuned models are deployed on a single NVIDIA A100 (80GB) GPU for evaluation. We evaluate our method on three widely used simulation benchmarks: LIBERO Liu et al.

(2023) , LIBERO-PLUS Fei et al. (2025) , and SimplerEnv Li et al. (2024c) .

Experiments — 3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training
Figure 4: Real-world Robot Setup. (a) Realman robot platform with 7-DoF manipulator, 1-DoF gripper, top camera, and wrist camera. (b) Experimental setups for three real-world tasks: height variation generalization, transparent container placement, and spatial position understanding.

Sources and Demonstrations

Figures are reproduced from Shi et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Memorize, Adapt, Ignore: Diagnosing Robot Learning Mechanisms under Training Data Variation
arXiv preprint

Memorize, Adapt, Ignore: Diagnosing Robot Learning Mechanisms under Training Data Variation

Ke Zhang, Danica J. Sutherland, Chao Liu

Training data variation, whether through designing a domain randomization (DR) scheme in simulation or curating demonstrations for imitation learning, is a primary lever for improving the robustness of robotic manipulation policies.

manipulation vision vla reinforcement-learning sim-to-real learning-from-demonstration
PDF Intermediate
Beyond Reconstruction: What Matters in Action Tokenization for Robot Policies?
arXiv preprint

Beyond Reconstruction: What Matters in Action Tokenization for Robot Policies?

Haoran Chen, Jingtian Ji, Samuel Wheeler et al.

Autoregressive action-token policies such as vision-language-action models require action tokenizers to translate discrete token sequences into precise control actions in continuous space. Many action tokenizers learn the mapping between tokens and actions via a reconstruction objective.

manipulation vision vla reinforcement-learning control benchmark
PDF Intermediate