GIFT: Guided Intermediate Feature Training via Action-Oriented Structural Supervision for Robotic Manipulation

Yupeng Zheng, Xiang Li, Songen Gu, Yuhang Zheng, Shuai Tian, Weize Li, Linbo Wang, Chaoyue Li, Qichao Zhang, Haoran Li, Zhongpu Xia, Ya-Qin Zhang, Shuicheng Yan, Dongbin Zhao · N/A · 2026

Summary

Abstract Summary

Vision-language pre-training and predictive world modeling provide robot policies with rich semantic and dynamic visual features, but their native action and visual-prediction objectives may omit critical physical and task structure while retaining control-irrelevant visual redundancy. We call this mismatch between visual richness and control utility the action-sufficiency gap. We investigate whether this gap can be bridged by guiding intermediate features to preserve three control-relevant structure in robotic manipulation: geometry governing motion feasibility, affordance encoding instruction-relevant entities, and goals grounding instructions in task-relevant regions. To this end, we present GIFT (Guided Intermediate Feature Training), an architecture-flexible framework for learning intermediate features that translates these structures into training-time constraints through geometry alignment, affordance prediction, and goal-region reconstruction. We instantiate GIFT in a Vision-Language-Action (VLA) policy, a direct-action World-Action Model (WAM), and an inverse-dynamics WAM while retaining each model's action formulation. Under zero-shot transfer to LIBERO-Plus, GIFT-VLA, GIFT-WAM-Fast, and GIFT-WAM-IDM outperform StarVLA-OFT, Fast-WAM, and Fast-WAM-IDM by 4.6, 12.6, and 5.2 points, reaching 79.6%, 72.6%, and 87.8%, respectively. On RoboCasa, the three GIFT variants reach 61.4%, 83.6%, and 82.3%, outperforming their counterparts by 12.6, 9.0, and 8.4 points, respectively. Together, these results establish learning functionally structured intermediate features as a reusable principle across model-specific action formulations, with especially large gains on articulated-object tasks and high-precision real-world manipulation under unseen visual and spatial perturbations. Project page: https://openphoenix-team.github.io/GIFT-pages.

Key Points

Vision-language pre-training and predictive world modeling provide robot policies with rich semantic and dynamic visual features, but their native action and visual-prediction objectives may omit critical physical and task structure while retaining control-irrelevant visual redundancy.
We call this mismatch between visual richness and control utility the action-sufficiency gap
We investigate whether this gap can be bridged by guiding intermediate features to preserve three control-relevant structure in robotic manipulation: geometry governing motion feasibility, affordance encoding instruction-relevant entities, and goals grounding instructions in task-relevant regions.
To this end, we present GIFT (Guided Intermediate Feature Training), an architecture-flexible framework for learning intermediate features that translates these structures into training-time constraints through geometry alignment, affordance prediction, and goal-region reconstruction.
We instantiate GIFT in a Vision-Language-Action (VLA) policy, a direct-action World-Action Model (WAM), and an inverse-dynamics WAM while retaining each model's action formulation.

Methodology

Consider a robot manipulation sample containing multi-view observations $O_{t}=\{I_{t}^{v}\}_{v=1}^{V}$, a language instruction $l$, the current proprioceptive state $s_{t}$, and a future action chunk $A_{t}=(a_{t},a_{t+1},\ldots,a_{t+H-1})\in\mathbb{R}^{H\times d_{a}}$.

Here, $t$ is the current time step, $v\in\{1,\ldots,V\}$ indexes the camera view, $V$ is the number of views, $H$ is the action horizon, and $d_{a}$ is the action dimension.

Methodology — GIFT: Guided Intermediate Feature Training via Action-Oriented Structural Supervision for Robotic Manipulation — Fig. 2: Overview of GIFT. Given the observation, instruction and robot state, GIFT attaches three training objectives to the intermediate visual tokens of a policy: geometry alignment with a frozen geometric foundation model, object-centric affordance prediction, and instruction-conditioned goal-region prediction. The default no-injection design does not route auxiliary predictions into the action pathway; the auxiliary objectives shape the shared intermediate features only through their training gradients. We instantiate the same principle in a VLA (top) and two WAM action formulations (bottom).

Simulation Evaluation

We evaluate GIFT on three simulation benchmarks that expose complementary aspects of manipulation performance. LIBERO [ 50 ] spans four suites, Spatial, Object, Goal, and Long, which respectively emphasize spatial relations, object-centric knowledge, goal interpretation, and long-horizon composition. Each suite contains ten language-conditioned manipulation tasks, performed by a Franka Emika Panda manipulator.

Following prior work, we execute 50 rollouts per task and report both suite-wise and overall success rates.

Simulation Evaluation — GIFT: Guided Intermediate Feature Training via Action-Oriented Structural Supervision for Robotic Manipulation — Fig. 3: Injection and no-injection variants. With injection, model-predicted affordance and goal features are routed to the model-specific action head during both training and inference. Without injection, the same predictions provide auxiliary supervision only. Geometry remains an alignment-only objective because its scene-level structure is absorbed directly by the visual tokens already entering the model-specific action pathway, whereas the object-level affordance and goal features provide compact auxiliary conditions.

Real-World Experiments

To evaluate our method in real-world scenarios, we construct the two robotic manipulation systems shown in Fig. 8 . Figure 8 (a) shows a UFACTORY xArm7 arm observed by two Intel RealSense D435 cameras: one provides a third-person view and the other is mounted on the wrist, mirroring the camera configuration used in LIBERO.

Figure 8 (b) shows an ARX X5 dual-arm platform equipped with three Intel RealSense D405 cameras, including one head-mounted camera and one wrist-mounted camera for each arm.

Real-World Experiments — GIFT: Guided Intermediate Feature Training via Action-Oriented Structural Supervision for Robotic Manipulation — Fig. 4: Visualization of the structured intermediate guidance. Examples from GIFT-VLA (top) and GIFT-WAM-IDM (bottom) show observations together with geometry, affordance, and goal predictions.

Sources and Demonstrations

Figures are reproduced from Zheng et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.