SpatialHarness: Test-Time Spatial Scaffolding for Fine Robotic Manipulation

Summary

Frontier multimodal foundation models (e.g., GPT-6 Astra) have recently shown strong potential for direct robotic control, yet their performance on fine manipulation remains limited.

Abstract Summary

Frontier multimodal foundation models (e.g., GPT-6 Astra) have recently shown strong potential for direct robotic control, yet their performance on fine manipulation remains limited. We argue that an important source of failure is not necessarily insufficient policy capability, but insufficient spatial observability, where task-critical spatial relationships may be poorly revealed by the existing physical camera setup. We introduce SpatialHarness, a test-time embodied harness that provides test-time spatial scaffolding for fine robotic manipulation without policy fine-tuning or changes to the physical sensing setup. SpatialHarness maintains an online simulated scene synchronized with real-world execution, identifies task-critical spatial relationships, and renders complementary virtual views that expose them to a frozen multimodal policy. To keep the simulated scene aligned during interaction, we develop interaction-aware scene synchronization that distinguishes static, held, and transition modes. We evaluate SpatialHarness on four real-robot manipulation tasks spanning precise geometric alignment, object-relative placement, and articulated-object interaction. Using the same frozen GPT-6 Astra policy, SpatialHarness substantially improves task success, including from 26.7% to 66.7% on plug insertion and from 0% to 100% on Tower of Hanoi. These results indicate that improving spatial observability at test time can unlock fine-manipulation capabilities already present in strong multimodal foundation models. Project website: https://emilia113.github.io/SpatialHarness/.

Key Points

Frontier multimodal foundation models (e.g., GPT-6 Astra) have recently shown strong potential for direct robotic control, yet their performance on fine manipulation remains limited.
We argue that an important source of failure is not necessarily insufficient policy capability, but insufficient spatial observability, where task-critical spatial relationships may be poorly revealed by the existing physical camera setup.
We introduce SpatialHarness, a test-time embodied harness that provides test-time spatial scaffolding for fine robotic manipulation without policy fine-tuning or changes to the physical sensing setup.
SpatialHarness maintains an online simulated scene synchronized with real-world execution, identifies task-critical spatial relationships, and renders complementary virtual views that expose them to a frozen multimodal policy.
To keep the simulated scene aligned during interaction, we develop interaction-aware scene synchronization that distinguishes static, held, and transition modes.

Method

SpatialHarness is a test-time embodied harness that provides test-time spatial scaffolding for a frozen multimodal policy. Its spatial context comprises complementary virtual views and structured spatial information. At each decision step, the policy receives this spatial context together with the task instruction, real observations, robot states.

The policy integrates these inputs to assess the current scene and generate robot actions through tool calls.

Method — SpatialHarness: Test-Time Spatial Scaffolding for Fine Robotic Manipulation — Figure 1 : The top two rows show real and auxiliary virtual views, respectively, for the four tasks shown; cyan marks task-relevant regions. Below, baseline (green) and SpatialHarness (yellow) success rates are compared using the same GPT Astra Low-based manipulation policy and real camera configuration. Values above the bars show gains over the baseline in percentage points (pp).

Experiments

We evaluate SpatialHarness on four real robot tasks to assess its benefits for fine manipulation. With the manipulation model and real camera configuration held fixed, we compare task success rates and model query overhead with and without supplementary spatial observations. We use a Franka Research 3 arm and two Intel RealSense RGB-D cameras, one mounted on the wrist and one providing a fixed third-person view.

Sources and Demonstrations

Figures are reproduced from Wang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.