Towards Neuro-Symbolic Procedural Reasoning for Long-Horizon Vision-Language-Action Manipulation

Vivek Chavan, Yahuan Shi, Oliver Heimann, Kevin Haninger, Jörg Krüger · N/A · 2026

Summary

Abstract Summary

Vision-language-action (VLA) models can execute short manipulation skills, but remain brittle in long-horizon procedures requiring persistent task state, dependency-aware reasoning, conditional decisions, and reliable grounding. We investigate a neuro-symbolic framework that combines learned VLA control with explicit task graphs and multimodal procedural memory. Task graphs encode action dependencies, valid transitions, and branch conditions, while memory maintains the active step, completed actions, textual context, and task-relevant visual evidence. Together, these structures guide object selection, destination grounding, subgoal dispatch, and verification of expected state transitions. Human demonstrations provide additional spatial and temporal guidance through gaze or saliency cues. To isolate their effect on policy learning, our initial study bypasses cross-view gaze transfer and directly annotates pseudo-gaze in robot-view teleoperation videos. The resulting guidance is used during VLA fine-tuning and inference. We study two long-horizon manipulation domains, workspace clearing and surgical-instrument handling, which require ordered execution, visually grounded decisions, and conditional branching. We evaluate correct-object and destination selection, subtask completion, task progress, step-order consistency, complete-task success, and procedural or execution mistakes. This work positions structured symbolic reasoning and demonstration-derived visual guidance as complementary mechanisms for reliable long-horizon VLA manipulation.

Key Points

Vision-language-action (VLA) models can execute short manipulation skills, but remain brittle in long-horizon procedures requiring persistent task state, dependency-aware reasoning, conditional decisions, and reliable grounding.
We investigate a neuro-symbolic framework that combines learned VLA control with explicit task graphs and multimodal procedural memory.
Task graphs encode action dependencies, valid transitions, and branch conditions, while memory maintains the active step, completed actions, textual context, and task-relevant visual evidence.
Together, these structures guide object selection, destination grounding, subgoal dispatch, and verification of expected state transitions.
Human demonstrations provide additional spatial and temporal guidance through gaze or saliency cues

Neuro-Symbolic Procedural Pipeline

Let $\mathcal{G}=(\mathcal{V},\mathcal{E})$ be a directed task graph. A node contains an instruction $\ell_{i}$, executor role $\rho_{i}$, visually checkable conditions, and status $z_{i}$. Dependencies and preconditions determine readiness; $\rho_{i}$ may identify the robot, a human, or another embodied agent.

An append-only memory $\mathcal{M}_{t}=[e_{1},\ldots,e_{t}]$ records decisions, observed transitions, and visual evidence. While acting or waiting, the system observes the fixed global view $I_{t}^{b}$ and moving wrist view $I_{t}^{w}$.

Neuro-Symbolic Procedural Pipeline — Towards Neuro-Symbolic Procedural Reasoning for Long-Horizon Vision-Language-Action Manipulation — Figure 2 : Representative action-conditioned attention visualization. Left: the unmodified observation. Right: the same frame with the attention-regularized policy’s heatmap overlaid. The concentration around the task-relevant region illustrates alignment with procedural saliency; it serves as a qualitative diagnostic.

Initial Grounding Study

We fine-tune $\pi_{0.5}$ [ 1 ] from 231 teleoperated episodes of a seven-step instrument-handling procedure observed by fixed-base and wrist cameras. Five subgoals are scored under three conditions: a prompt-trained checkpoint without its expected overlay ( prompt-finetuned ), the same checkpoint with a visual-prompt , and a clean-input regularized checkpoint. Five trials per subgoal and condition yield 75 trials.

Sources and Demonstrations

Figures are reproduced from Chavan et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.