Know Your Body: A Harness for Direct and Self-Improving Robot Control with VLMs

Summary

A general-purpose vision-language model can understand a task goal without knowing how a particular robot's motion and functional parts produce the intended effect.

Abstract Summary

A general-purpose vision-language model can understand a task goal without knowing how a particular robot's motion and functional parts produce the intended effect. We introduce KnowBody, a harness that makes these action-relevant body relations explicit, queryable, and revisable while keeping the model weights frozen. Initialized from one off-task trajectory, a partial body model guides action selection and the interpretation of past interactions. New evidence refines the model, and knowledge dependent on revised body estimates is rechecked before reuse. Across 32 fixed-budget trials on four real-robot tasks, initialized KnowBody achieves 75% completion versus 25% for the native harness and requires fewer planner rounds on successful trials in tasks completed by both. With persistent updates enabled, planner rounds decrease by 29-53% from the first to the fifth recorded success.

Key Points

A general-purpose vision-language model can understand a task goal without knowing how a particular robot's motion and functional parts produce the intended effect.
We introduce KnowBody, a harness that makes these action-relevant body relations explicit, queryable, and revisable while keeping the model weights frozen.
Initialized from one off-task trajectory, a partial body model guides action selection and the interpretation of past interactions.
New evidence refines the model, and knowledge dependent on revised body estimates is rechecked before reuse.
Across 32 fixed-budget trials on four real-robot tasks, initialized KnowBody achieves 75% completion versus 25% for the native harness and requires fewer planner rounds on successful trials in tasks completed by both.

Problem and Learning Protocol

At model–tool decision $i$, a general-purpose VLM receives instruction $\ell$, current images and available depth $o_{t}$, measured pose $T_{t}=(R_{t},p_{t})$, gripper state, and available joint feedback. The index $t$ advances with fresh observations and physical transitions, not with every read-only query. Declared robot and motion limits are supplied as interface constraints.

The output $u_{i}$ is a query, observation request, motion proposal, or termination request.

Problem and Learning Protocol — Know Your Body: A Harness for Direct and Self-Improving Robot Control with VLMs — Figure 4: Real-robot evaluation setup. A Franka Research 3 with a Robotiq 2F-85 gripper operates over the worktable. A ZED Mini provides the wrist view, while two ZED 2i cameras provide third-person views. The same robot and camera arrangement is used in both evaluation protocols.

Method

KnowBody organizes robot interaction around a queryable, partial body model. Before acting, the model relates motion and functional-part geometry to the commands chosen by a frozen VLM. After acting, it supports checks of the observed response and supplies the geometric conditions under which experience is interpreted.

These two roles couple body estimation to experience reuse: correcting a body estimate can change an earlier interaction’s conditions without changing its observed outcome.

Method — Know Your Body: A Harness for Direct and Self-Improving Robot Control with VLMs — Figure 5: Task examples and two evaluation protocols. Left: illustrative development sequences for duck placement, apple pushing, writing, and pouring. Upper right: reasoning rounds for five successive successful completions per task under the continual-interaction protocol. Knowledge is retained across attempts and tasks; intervening failures contribute updates but are not plotted. The horizontal axis indexes successful trials, and vertical ranges differ across tasks. Lower right: all 32 fixed-budget trials, with four trials per method and task. Native Codex denotes the native Codex harness using the same frozen VLM as KnowBody .Numbers denote reasoning rounds for successful trials; F denotes failure. KnowBody starts each trial from the same initialization with cross-episode updates disabled. Objects are randomly repositioned before each attempt in both protocols.

Real-Robot Evaluation

We evaluate task completion under fixed reasoning budgets and reasoning effort during continual interaction. The first protocol tests the initialized harness without accumulated experience; the second tracks successful completions as knowledge accumulates from both successful and failed attempts. As shown in Fig. 4 , we use a Franka Research 3 with a Robotiq 2F-85 gripper.

A ZED Mini provides the wrist view, and two ZED 2i cameras provide third-person views of the worktable.

Sources and Demonstrations

Figures are reproduced from Lou et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.