RAIN: Robotic Task Generalization with Region-Aware Interaction Networks

Seunghun Lee, Kyuhwan Shim, Yeogyeong Song, Seung Hwan Kim, Sihaeng Lee, Byoung-Tak Zhang, Sunghoon Im, Gunhee Lee · N/A · 2026

Summary

Abstract Summary

Existing vision-language-action (VLA) models benefit from large-scale vision-language pretraining, but tend to memorize training trajectories, limiting their ability to adapt learned interactions to new task and scene configurations. To address this problem, we propose \textbf{Region-Aware Interaction Networks (RAIN)}, which formulates a task as a sequence of goal-conditioned subtasks, each specified by an action type and a target-region mask. This formulation allows learned skills to be reused across different objects and scene layouts and composed into task sequences not seen during training. Its Target-adaptive Cross-View Encoder (TCE) conditions multi-view features on target masks and exchanges information across views, while PlanDiT predicts target-directed trajectories and actions. A Transition Head estimates subtask completion to enable automatic transitions. We also introduce a \textbf{Reference-based Retargeting Strategy (RRS)}, which uses recorded trajectories to synthesize smooth approaches to reference interactions with alternative target regions, without collecting additional data. Experiments on \textbf{LIBERO-Analogy}, our new benchmark for robotic task generalization, and real-world tabletop tasks demonstrate that RAIN generalizes more robustly than existing VLA baselines.

Key Points

Existing vision-language-action (VLA) models benefit from large-scale vision-language pretraining, but tend to memorize training trajectories, limiting their ability to adapt learned interactions to new task and scene configurations.
To address this problem, we propose \textbf{Region-Aware Interaction Networks (RAIN)}, which formulates a task as a sequence of goal-conditioned subtasks, each specified by an action type and a target-region mask.
This formulation allows learned skills to be reused across different objects and scene layouts and composed into task sequences not seen during training.
Its Target-adaptive Cross-View Encoder (TCE) conditions multi-view features on target masks and exchanges information across views, while PlanDiT predicts target-directed trajectories and actions.
A Transition Head estimates subtask completion to enable automatic transitions

Method

Qwen [ 42 ] decomposes each task into target-centered subtasks, forming an ordered sequence of (action type, target) pairs. We determine subtask intervals from recorded gripper trajectories: finger closing, opening, and stabilization identify grasp and release phases, while changes or stops in translation and rotation delimit other interactions.

The detected completion event defines the interval endpoint, and this final frame receives a completion label.

Method — RAIN: Robotic Task Generalization with Region-Aware Interaction Networks — Fig. 1: (a) Task generalization by analogy: adapting learned interactions to new task and scene configurations, composing skills, and executing selected parts of learned sequences. (b) Region-Aware Interaction Networks (RAIN) executes each subtask from an action type and target-region mask. A Transition Head determines when to advance to the next subtask.

Experiments

General. We use frozen DINOv2-L/14 [ 43 ] for visual encoding with feature dimension $d=1024$ and frozen CLIP ViT-L/14 [ 1 ] for action-type encoding. PlanDiT uses $B=2$ stage cycles, giving six cross-attention and six self-attention layers, with action horizon $H=16$ and $N=8$ plan waypoints. The consistency loss uses $K=4$ frames, with $\lambda_{\mathrm{fm}}=1$ and $\lambda_{\mathrm{cons}}=0.1$.

Inference uses four flow-matching steps and replans every eight actions.

Experiments — RAIN: Robotic Task Generalization with Region-Aware Interaction Networks — Fig. 2: Overall architecture of RAIN. For each subtask, RAIN receives an action type and target masks. The Target-adaptive Cross-view Encoder (TCE) conditions multi-stage features from the third-person and wrist views on the target masks. The PlanDiT uses these stage-wise features to jointly predict future waypoints and action chunks. The Transition Head pools target-region features and fuses the views to predict subtask completion.

Sources and Demonstrations

Figures are reproduced from Lee et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.