Attention from Action, for Action: Emergent Visual Bottlenecks for Policy Learning

Zheyu Zhuang, Ruiyu Wang, Nick Heppert, Johannes Fabian Hahn, Abhinav Valada, Florian T. Pokorny, Danica Kragic · N/A · 2026

Summary

Abstract Summary

Visual bottlenecks that focus policy inputs on regions of interest (ROIs) can improve data-efficient visuomotor learning by separating where to look from how to act. Many ROI interfaces rely on external spatial labels, such as gaze, object classes, or affordance annotations. Label-free alternatives often derive crops from trajectories by detecting gripper or motion events and centering a fixed crop at the projected end-effector. Such action-derived crops are useful spatial priors that require no additional labels, but they encode fixed choices about event timing, proxy points, and crop scale. When the visual evidence needed for control lies away from the end-effector or changes continuously with task progress, these crops can become misaligned. We propose Seeker, a task- and state-conditioned readout that learns attention from action. Starting from frozen DINOv3 features, Seeker iteratively updates a query with gathered visual evidence, producing progression-aware ROIs solely from action supervision. The learned ROI serves as a spatial interface for RGB cropping, mask-guided background augmentation, and point-cloud filtering. In simulation and the real world, Seeker improves data efficiency and robustness over no-crop, augmentation, and action-derived crop baselines. On real robots, Seeker raises average in-domain success from the best baseline's 48.3% to 76.7% and success under lighting/background shifts from 20.0% to 60.0%.

Key Points

Visual bottlenecks that focus policy inputs on regions of interest (ROIs) can improve data-efficient visuomotor learning by separating where to look from how to act.
Many ROI interfaces rely on external spatial labels, such as gaze, object classes, or affordance annotations.
Label-free alternatives often derive crops from trajectories by detecting gripper or motion events and centering a fixed crop at the projected end-effector.
Such action-derived crops are useful spatial priors that require no additional labels, but they encode fixed choices about event timing, proxy points, and crop scale.
When the visual evidence needed for control lies away from the end-effector or changes continuously with task progress, these crops can become misaligned.

Methodology

Fig. 1 (a) illustrates Seeker, a lightweight readout over frozen DINOv3 [ 22 ] patch tokens. Its design is related to attention-probing readouts over pretrained visual features [ 1 , 5 , 25 ] , but is adapted for action-grounded ROI discovery.

Rather than performing a single visual lookup, Seeker treats readout as an iterative search process: a task- and state-conditioned query repeatedly attends to DINO patches, gathers visual context, and updates itself before the next lookup.

Methodology — Attention from Action, for Action: Emergent Visual Bottlenecks for Policy Learning — Figure 1: Seeker Overview. (a) A state-conditioned query is iteratively refined over frozen DINOv3 patch tokens, producing a compact visual context together with attention maps and head scores. (b) Head-score-weighted attention is converted into a mask and bounding box using a top_p mass criterion. (c) The visual context is trained through a diffusion action-prediction loss, allowing the ROI to emerge without spatial supervision.

Simulation Evaluations

Fig. 2a illustrates the six MimicGen [ 15 ] tasks used in our simulation study: Stack Three D1, Square D2, Three-Piece Assembly D2, Coffee Preparation D1, Pick & Place D0, and Threading D2. These tasks cover high spatial variance and long-horizon manipulation; detailed task descriptions are provided in App. B . We omit D0/D1/D2 suffixes where clear.

We train a single multi-task Seeker once, using 100 demonstrations per task from the original-background RGB data, and keep it frozen for all simulation experiments.

Simulation Evaluations — Attention from Action, for Action: Emergent Visual Bottlenecks for Policy Learning — (a) Task Overview

Real-world Evaluations

We evaluate Seeker and two representative baselines on a real-world multi-view RGB setup with a UFactory xArm7, using a fixed third-person camera and an eye-in-hand camera. The three tasks are performed on visually distracting table textures and require different forms of spatial attention (Fig.

Ablations

We compare direct regression, flow matching [ 13 , 2 ] , and IMLE [ 20 ] for training Seeker from action supervision. Direct regression yields noisy attention, while generative objectives produce more stable ROIs: IMLE is cleaner but diffuse, and flow matching gives tighter masks closest to diffusion.

Across 300 trajectories from Square, Stack Three, and Three-Piece Assembly, the average ROI box IoU to the diffusion-trained Seeker is $0.50$ for IMLE and $0.67$ for flow matching. Additional qualitative comparisons in App. A.2 .

Ablations — Attention from Action, for Action: Emergent Visual Bottlenecks for Policy Learning — (c) Heuristic ROI Stress Cases

Sources and Demonstrations

Figures are reproduced from Zhuang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.