HABILIS Brain 0: Geometry-Change Supervision for Vision-Language-Action and Residual Flow Recovery

Jinu Pahk, Jesoon Kang, Taegeon Park, Jisu An, Soo Min Kimm, Jaejoon Kim, Byoung-Tak Zhang · N/A · 2026

Summary

Vision-language-action policies benefit from geometric supervision, but current-frame geometry alone does not explicitly describe the changes associated with manipulation.

Abstract Summary

Vision-language-action policies benefit from geometric supervision, but current-frame geometry alone does not explicitly describe the changes associated with manipulation. This design is motivated by the goal of learning an embodiment-agnostic visual interface that can be pretrained across robot and egocentric video before robot-specific action alignment. We introduce Geometry-Change VLA (GC-VLA), which learns to predict multiview future-current geometry-change tokens from current observations. Offline frame pairs define a nominal 0.5-second prediction horizon; future observations are used only to construct training targets. Stage 1 trains a geometry-change vision-language model (GC-VLM). Stage 2 introduces a continuous ActionExpert and aligns it with robot actions while stopping action-flow gradients at the VLM interface. Stage 3 enables these gradients to update the trainable VLM components jointly with the ActionExpert. Stage 4 freezes GC-VLA and applies Geometry-Conditioned Residual Flow (GCRF), using a binary intervention router and a single bounded residual velocity policy learned from closed-loop feedback. GC-VLA achieves 95.20% success on LIBERO, and GC-VLA with GCRF achieves 99.55%. Inference uses current observations and the learned GC representation without executing the offline target encoders.

Key Points

Vision-language-action policies benefit from geometric supervision, but current-frame geometry alone does not explicitly describe the changes associated with manipulation.
This design is motivated by the goal of learning an embodiment-agnostic visual interface that can be pretrained across robot and egocentric video before robot-specific action alignment.
We introduce Geometry-Change VLA (GC-VLA), which learns to predict multiview future-current geometry-change tokens from current observations.
Offline frame pairs define a nominal 0.5-second prediction horizon; future observations are used only to construct training targets.
Stage 1 trains a geometry-change vision-language model (GC-VLM)

Method

Let the online input be where $\ell$ is the language instruction, $I_{t}^{(v)}$ is an available RGB view, and $q_{t}$ is the robot state.

An offline geometry encoder $\phi$ maps a camera observation and its pseudo-depth estimate to a compact spatial representation: The training label is the geometry change over the prediction horizon The $10\times 10$ spatial grid is quantized into 100 token positions per view, $y_{t,H_{\mathrm{GC}}}^{(v)}\in\{1,\ldots,K\}^{100}$ .

Method — HABILIS Brain 0: Geometry-Change Supervision for Vision-Language-Action and Residual Flow Recovery — Figure 1 : GC-VLA architecture and training stages. GC-VLM predicts multiview geometry-change tokens from current observations. The ActionExpert is introduced during action alignment and consumes the learned representations. The action-flow gradient is detached at the VLM interface in Stage 2 and enabled for trainable VLM components in Stage 3. Future observations are used only for offline target generation.

Experimental Setup

We use two frozen pretrained models only during offline target generation. For pseudo-depth estimation, we use depth-anything/DA3-BASE , a 0.12B-parameter checkpoint from Depth Anything 3 [ 17 ] . For geometry representation extraction, we use facebook/dinov2-small , corresponding to the 21M-parameter distilled DINOv2 ViT-S/14 model [ 25 ] .

Both checkpoints are used as preprocessing models and are not executed during GC-VLA inference. For each camera view, Depth Anything 3 first predicts a pseudo-depth map from the current RGB observation.

Experimental Setup — HABILIS Brain 0: Geometry-Change Supervision for Vision-Language-Action and Residual Flow Recovery — Figure 2 : Multiview geometry-change target construction on LIBERO. Rows correspond to head and wrist views. Columns show (a) current RGB, (b) future RGB at the selected geometry horizon, (c–d) their offline DA3 pseudo-depth estimates, (e) the magnitude of the DINO feature difference, (f) the quantized GC token IDs, and (g) the GC target overlay on the current RGB frame. The overlay visualizes the training target rather than model attention. Future observations are used only to construct offline supervision.

Results

We evaluate whether geometry-change representations encode action-relevant information using matched action-chunk and manipulation-phase probes. Table 3 compares target representations under a common split and state baseline. Table 4 evaluates information retained in frozen learned representations. Action-chunk gain: 95% CI $[+0.004,+0.037]$, one-sided sign-flip $p=0.027$.

Phase gain: positive in 9/10 folds, Holm-adjusted $p=0.0195$.

Results — HABILIS Brain 0: Geometry-Change Supervision for Vision-Language-Action and Residual Flow Recovery — Figure 3 : Examples of geometry-change supervision across training sources. Each example contains current RGB, future RGB at the source-normalized prediction horizon, and a GC target overlay on the current image. The overlay is the spatial magnitude of the DA3–DINO feature difference before quantization. Frame selection uses $H_{\mathrm{GC}}=\operatorname{round}(0.5f)$ for source frame rate $f$; the realized interval depends on frame discretization. Future RGB is used only to construct training targets.

Sources and Demonstrations

Figures are reproduced from Pahk et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.