Robotics Research Hub
Vision-Language Grounded Task-Context-Aware Imitation Learning for Robotic Disassembly

Vision-Language Grounded Task-Context-Aware Imitation Learning for Robotic Disassembly

Jeon Ho Kang, Igal Tamarkin, Ethan Niu, Ian Novales, Satyandra K. Gupta · N/A · 2026

manipulation vision reinforcement-learning learning-from-demonstration
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Real-world robotic disassembly requires long-horizon execution, where robots must perform ordered sequences of manipulation tasks across multiple parts within a single scene.

Abstract Summary

Real-world robotic disassembly requires long-horizon execution, where robots must perform ordered sequences of manipulation tasks across multiple parts within a single scene. Multiple valid task goals and diverse assembly configurations make it difficult for imitation policies to infer the intended skill from raw observations alone, particularly when training data cannot cover the combinatorial diversity of real-world configurations and part geometries. We show that incorporating task context through language alleviates these challenges by providing explicit structure for skill selection and associating language-specified tasks with their corresponding manipulation targets in the visual scene. The proposed framework combines hierarchical task selection with task-context-aware imitation learning to ground language instructions in spatial visual representations for robotic disassembly. The resulting framework generalizes across diverse connector geometries and assembly configurations without requiring explicit object annotations. Our method improves end-to-end task success by 35 percentage points over the baseline diffusion policy and by 75 percentage points over the previous task-context-aware baseline.

Key Points

  • Real-world robotic disassembly requires long-horizon execution, where robots must perform ordered sequences of manipulation tasks across multiple parts within a single scene.
  • Multiple valid task goals and diverse assembly configurations make it difficult for imitation policies to infer the intended skill from raw observations alone, particularly when training data cannot cover the combinatorial diversity of real-world configurations and part geometries.
  • We show that incorporating task context through language alleviates these challenges by providing explicit structure for skill selection and associating language-specified tasks with their corresponding manipulation targets in the visual scene.
  • The proposed framework combines hierarchical task selection with task-context-aware imitation learning to ground language instructions in spatial visual representations for robotic disassembly.
  • The resulting framework generalizes across diverse connector geometries and assembly configurations without requiring explicit object annotations.

Problem Formulation

Let us consider a scenario with $N$ disassembly tasks present in a single scene, where each parent task $\mathit{T}_{p}$ must be executed in order and is composed of a sequence of subtasks, $T_{s_{i}}$, each requiring a specific skill $\mathit{s}$.

We define the parent task as a sequence of subtasks $\mathit{T}_{p}=\{\mathit{T}_{s_{i}}\}_{i=1}^{n}$, where each subtask $\mathit{T}_{s_{i}}$ is associated with a required skill $\mathit{s}_{i}$.

Problem Formulation — Vision-Language Grounded Task-Context-Aware Imitation Learning for Robotic Disassembly
Fig. 2: CF loss behavior during training. The MSE for correct and mismatched action–language pairs diverges over training, while the CF hinge loss converges toward zero.

Method

Training a single policy across multiple tasks and targets can cause gradient interference, leading to incorrect object selection and mode averaging. Explicit object detection can alleviate this ambiguity but requires costly annotations and is challenging for small, diverse disassembly components. We therefore investigate whether language can ground the target object without object-level annotations.

Method — Vision-Language Grounded Task-Context-Aware Imitation Learning for Robotic Disassembly
Fig. 3: (a) Attention maps during the approach phase under different language commands. Without the proposed grounding mechanism, DiT attends to seemingly unrelated regions. (b) Attention map for held-out assemblies.

Experimental Setup

We evaluate our method on a KUKA LBR IIWA14 robotic arm. Inspired by the NIST Task Board #1 [ 28 ] , we design a custom testbed for data collection. We conduct experiments under a variety of connection configurations and spatial arrangements, including scenarios in which two identical connectors coexist on the same board. The evaluation objects and assembly configurations are shown in Figure 5 .

We collect demonstrations via kinesthetic teaching following [ 9 , 1 ] .

Experimental Setup — Vision-Language Grounded Task-Context-Aware Imitation Learning for Robotic Disassembly
Fig. 4: Attention weights computed for different modalities during action prediction in the DiT policy. Compared to the approach phase, contact-rich tasks such as pulling exhibit higher attention weights toward force inputs.

Results

In the results section, we aim to answer four key questions regarding the effectiveness of the proposed framework. (Q1) Which VLM provides the most reliable performance for both offline subtask labeling and online task selection?

(Q2) How do different task-selection mechanisms, including heuristic-based, CNN-based, and VLM-based planners, compare in terms of end-to-end task success, robustness to held-out assembly configurations, and mitigation of the temporal freezing effect? (Q3) Can the proposed method consistently identify the target object...

Results — Vision-Language Grounded Task-Context-Aware Imitation Learning for Robotic Disassembly
Fig. 5: Diverse types of connectors and assembly configurations used to test generalizability of our approach.

Sources and Demonstrations

Figures are reproduced from Kang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Capability-Aware Arbitration for Semantic Intent-Based Shared Control
arXiv preprint

Capability-Aware Arbitration for Semantic Intent-Based Shared Control

Zhaoda Du, Michael Bowman, Xiaoli Zhang

Shared control often allocates robot authority based on confidence in inferred human intent, assuming reliable autonomous execution. When this assumption fails, high intent confidence can cause over-helping.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
Code PDF Intermediate
GitHub ★ —
DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation
arXiv preprint

DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

Yunchao Yao, Zhuxiu Xu, Tianqi Zhang et al.

Building general-purpose dexterous manipulation policies requires benchmarks that go beyond isolated tasks to systematically evaluate policies across diverse interaction modes, sensory conditions, and robot embodiments.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
PDF Advanced