WM-VS: Progress-Aligned World Models for Closed-Loop Visual Servoing

Guanzhong Sun, Junyi Ma, Yixuan Zhou, Yuxuan Wu, Yanzi Miao, Hesheng Wang · N/A · 2026

Summary

Closed-loop visual servoing requires predictions that indicate whether an action reduces task error, not only whether the action is plausible.

Abstract Summary

Closed-loop visual servoing requires predictions that indicate whether an action reduces task error, not only whether the action is plausible. We call this gap the prediction-control mismatch and introduce WM-VS, a target-centric progress-aligned world-model framework for closed-loop visual servoing. Offline target-region DINOv2 correspondences define a signed four-dimensional servo coordinate for translation, scale, and in-plane rotation. Stage 1 aligns action-conditioned latent transitions with this coordinate; Stage 2 freezes the world model and trains a reactive joint-velocity policy with action imitation, consequence supervision, and short imagined rollouts that favor error contraction. Deployment is RGB-only and reactive, without online trajectory optimization. On a real 7-DoF eye-to-hand system, WM-VS reaches a corner RMSE no larger than 10 percent of its initial value in 30/30 trials and retains this criterion at the final valid frame in 25/30 (83.33 percent). Removing future-error alignment reduces retention to 26.67 percent. The learned progress signal agrees with an external AprilTag corner error not used for training or control (mean Spearman rho = 0.8778). Without retraining, two unseen 3D targets achieve translation-error reductions of 86.48 percent and 90.27 percent and rotation-error reductions of 70.01 percent and 65.70 percent. These results link progress-aligned action consequences to repeated closed-loop correction and transfer. Code and data will be released as open source.

Key Points

Closed-loop visual servoing requires predictions that indicate whether an action reduces task error, not only whether the action is plausible.
We call this gap the prediction-control mismatch and introduce WM-VS, a target-centric progress-aligned world-model framework for closed-loop visual servoing.
Offline target-region DINOv2 correspondences define a signed four-dimensional servo coordinate for translation, scale, and in-plane rotation.
Stage 1 aligns action-conditioned latent transitions with this coordinate; Stage 2 freezes the world model and trains a reactive joint-velocity policy with action imitation, consequence supervision, and short imagined rollouts that favor error contraction.
Deployment is RGB-only and reactive, without online trajectory optimization

Method

We consider a fixed eye-to-hand camera observing a robot whose end-effector carries the task target. At time $t$, the controller receives an RGB image $I_{t}$, joint state $\mathbf{q}_{t}$, recent joint and action history, and a desired RGB image $I_{g}$, and outputs a seven-joint velocity command $\dot{\mathbf{q}}_{t}$.

At every control cycle WM-VS refreshes the observation and robot state and recomputes the command, so it operates as a reactive closed-loop servo controller rather than replaying an open-loop action sequence.

Method — WM-VS: Progress-Aligned World Models for Closed-Loop Visual Servoing — Fig. 2: Directional servo coordinate for action-consequence learning. The goal-conditioned policy predicts a joint action from the current and goal latents together with robot history. The aligned predictive representation maps the imagined consequence to the four-dimensional error coordinate. Action imitation constrains the policy near the data distribution, while the servo terms regularize local consequences and encourage short-horizon error contraction.

Experiments

We use the suffix E2H to denote the fixed-camera eye-to-hand adaptations of the corresponding methods. Experiments use a fixed eye-to-hand Gemini 2 RGB-D camera and a seven-axis RealMan RMC-DA robot. The RGB stream is $640\times 480$ (4:3), with horizontal, vertical, and diagonal fields of view of $63.55^{\circ}$, $49.82^{\circ}$, and $75.49^{\circ}$, respectively.

A custom-designed, 3D-printed end-effector is rigidly bolted to the robot flange and carries the task target.

Experiments — WM-VS: Progress-Aligned World Models for Closed-Loop Visual Servoing — Fig. 3: Best-observed real-robot precision. (a) WM-VS, (b) VSNet-E2H, (c) Moment-VS, and (d) ViT-VS-E2H under identical initial/desired configurations. Blue/red quadrilaterals mark initial/desired AprilTag corners.

Sources and Demonstrations

Figures are reproduced from Sun et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.