Robotics Research Hub
Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis

Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis

Sixu Yan, Shikang Wang, Binhua Huang, Xuanlai Tang, Guohua Fan, Fan Huang, Haoxuan Li, Yongkang Li, Yuhan Li, Bencheng Liao, Zeyu Zhang, Wenyu Liu, Hangxin Liu, Xinggang Wang · N/A · 2026

manipulation vision reinforcement-learning
PDF arXiv Code (GitHub)

Framework

N/A

License

N/A

Stars

N/A

Summary

This paper proposes AdaRoboVLG, a task-adaptive Vision-Language-Grasp (VLG) framework that supports generalizable grasp synthesis across different robotic hands.

Abstract Summary

This paper proposes AdaRoboVLG, a task-adaptive Vision-Language-Grasp (VLG) framework that supports generalizable grasp synthesis across different robotic hands. Unlike existing VLG methods that tightly couple foundation models with end-to-end grasp policies, AdaRoboVLG learns an efficient generalizable base policy that generates and evaluates physically feasible grasp candidates through explicit kinematic mapping and force-closure-based stability estimation, while offloading task-dependent understanding to specialized foundation-model modules. These modules provide composable priors that are integrated into the grasp synthesis process, enabling contextually adaptive grasp synthesis without retraining the underlying grasp policy. Through extensive simulation and real-world experiments, we demonstrate that (i) the base policy exhibits efficient learning and strong cross-hand generalization, (ii) the framework effectively incorporates spatial, cognitive, and temporal priors to address three representative grasping challenges without compromising grasp synthesis performance compared to state-of-the-art methods, and (iii) these priors can operate jointly to enable functional grasping in cluttered and dynamic environments. These results indicate that decoupling physical grasp synthesis from task-dependent understanding provides a scalable paradigm for robotic grasping, allowing future advances in foundation models to be directly translated into improved grasp capabilities without redesigning or retraining the underlying grasp policy. Supplementary videos are available at https://adarobovlg.github.io/

Key Points

  • This paper proposes AdaRoboVLG, a task-adaptive Vision-Language-Grasp (VLG) framework that supports generalizable grasp synthesis across different robotic hands.
  • Unlike existing VLG methods that tightly couple foundation models with end-to-end grasp policies, AdaRoboVLG learns an efficient generalizable base policy that generates and evaluates physically feasible grasp candidates through explicit kinematic mapping and force-closure-based stability estimation, while offloading task-dependent understanding to specialized foundation-model modules.
  • These modules provide composable priors that are integrated into the grasp synthesis process, enabling contextually adaptive grasp synthesis without retraining the underlying grasp policy.
  • Through extensive simulation and real-world experiments, we demonstrate that (i) the base policy exhibits efficient learning and strong cross-hand generalization, (ii) the framework effectively incorporates spatial, cognitive, and temporal priors to address three representative grasping challenges without compromising grasp synthesis performance compared to state-of-the-art methods, and (iii) these priors can operate jointly to enable functional grasping in cluttered and dynamic environments.
  • These results indicate that decoupling physical grasp synthesis from task-dependent understanding provides a scalable paradigm for robotic grasping, allowing future advances in foundation models to be directly translated into improved grasp capabilities without redesigning or retraining the underlying grasp policy.

Problem Formulation and Grasp Interface

This section presents our formulation for adaptive vision-language grasping. We first introduce the essential notation and preliminaries in Sec. III-A and then define the problem and structured grasp interface in Sec. III-B . Before delving into the formulation of AdaRoboVLG, we start by defining the following mathematical notation: Bold lowercase letters represent vectors ( e.g .

, $\bm{g}$), with the subscript $i$ denoting the $i$-th entry of the vector ( e.g . , $g_{i}$). Bold uppercase letters represent matrices ( e.g . , $\bm{T}$).

Problem Formulation and Grasp Interface — Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis
Fig. 3 : Contact grasp representation. This representation effectively captures a hand-agnostic contact primitive that can be (a) shared across diverse end-effectors, ranging from parallel grippers to multi-finger robotic hands, and (b) adaptable to various functional grasp types.

Base Grasp Policy

This section presents the implementation details of the base grasp policy $f_{\pi}(\cdot)$. We sequentially describe the policy design principle ( Sec. IV-A ), the cgr definition ( Sec. IV-B ), the hand-specific grasp mapping ( Sec. IV-C ), the hoi representation ( Sec. IV-D ), and the hand-agnostic grasp evaluation ( Sec. IV-E ). Given the grasp interface $\mathcal{I}$ defined in Eq.

3 , the base policy converts it into executable grasp poses for a target robotic hand.

Base Grasp Policy — Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis
Fig. 4 : Hand-specific grasp mapping. This function maps the object geometry, cgr , and grasp type into taxonomy-consistent grasp poses for a specific robotic hand.

Foundation Priors for Adaptive Grasping

This section instantiates the transformation function $f_{\tau}(\cdot)$ with composable foundation priors. We consider three tasks with increasing complexity: grasping in cluttered scenes, functional grasping in cluttered scenes, and functional grasping in dynamic cluttered scenes.

For these tasks, the transformation function is dynamically configured as Here, $\mathcal{M}_{\mathrm{s}}$, $\mathcal{M}_{\mathrm{c}}$, and $\mathcal{M}_{\mathrm{t}}$ denote the spatial-prior ( Sec. V-A ), cognitive-prior ( Sec. V-B ), and temporal-prior ( Sec.

Foundation Priors for Adaptive Grasping — Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis
Fig. 5 : Hand-agnostic grasp evaluation. This module evaluates the physical stability of synthesized grasp candidates in a strictly hand-agnostic manner. To achieve this, it first extracts an hoi representation to explicitly capture the local contact geometry. Leveraging a PointTransformer [ 65 ] backbone and a contact-aware feature extraction module, the grasp decision model dynamically aggregates these geometric features to predict a continuous stability score for each grasp pose.

Simulated Experiments

In this section, we systematically validate the key components of AdaRoboVLG in simulation. We first introduce the data collection and training setup ( Sec. VI-A ), and then evaluate the base policy for generalizable grasp synthesis ( Sec. VI-B ), the spatial prior for cluttered grasping ( Sec. VI-C ), the cognitive prior for functional grasping ( Sec. VI-D ), and the temporal prior for dynamic grasping ( Sec. VI-E ).

Simulated Experiments — Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis
Fig. 6 : Spatial prior module. Given multi-view RGB-D observations, the spatial prior module leverages the DINOv3 [ 7 ] features to construct scene semantic representations $\mathcal{R}^{\mathcal{S}}$ and predict scene-level cgr $\mathcal{A}^{\mathcal{S}}$. The predicted cgr provide spatially feasible contact candidates for the base grasp policy to perform clutter-aware grasp planning.

Real-World Experiments

This section demonstrates the real-world applicability of AdaRoboVLG by progressively composing different foundation priors to address increasingly challenging grasping scenarios. After introducing the experimental setup ( Sec. VII-A ), we first evaluate the composition of spatial and cognitive priors for functional grasping in static cluttered scenes ( Sec. VII-B ).

We then further increase task complexity by incorporating the temporal prior to handle functional grasping in dynamic cluttered environments ( Sec. VII-C ).

Real-World Experiments — Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis
Fig. 7 : Cognitive prior module. Given scene observations and a language instruction, the cognitive prior module uses an llm with rag [ 23 ] and cot [ 24 ] mechanism to infer the target object, functional part, and hand-compatible grasp types. It then employs SAM3 [ 25 ] and 3D lifting to align these inferred results with scene representations, producing the target object geometry $\mathcal{O}$, object-level semantics $\mathcal{R}^{\mathcal{O}}$, and functional cgr $\mathcal{A}^{\mathcal{O}}$. These outputs, together with $\mathcal{T}_{\mathcal{H}}$, provide the task-conditioned interface for the base grasp policy to perform language-guided functional grasping.

Sources and Demonstrations

Figures are reproduced from Yan et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Autonomous Droplet Navigation via Model-Based Reinforcement Learning: Zero-Shot Transfer and Emergent Dynamics
arXiv preprint

Autonomous Droplet Navigation via Model-Based Reinforcement Learning: Zero-Shot Transfer and Emergent Dynamics

Rajneesh Anand, Mayuresh V. Kothare

Self-driving laboratories (SDLs) are transforming chemical and materials discovery through closed-loop automation, yet automated infrastructure for physical manipulation of soft, deformable matter remains beyond current robotic platforms.

manipulation vision reinforcement-learning planning control
Code PDF Intermediate
GitHub ★ —