SWIM: Vision-Language-Grounded Soft Whole-Body Interactive Manipulation

Tingcong Liu, Aye Phyu Phyu Aung, Junjie Xiong, Siyi Ma, Bo An, Ke Wu, Senthilnath Jayavelu · N/A · 2026

Summary

Abstract Summary

Soft and continuum robots enable manipulation through distributed body deformation and contact, yet translating language and visual context into executable whole-body actuation remains a fundamental challenge. We present SWIM, a framework that maps an initial RGB observation and a language instruction to a complete actuation-command sequence. Its vision-language-action (VLA) policy, SWIM-VLA, combines a diffusion action head with Visual Soft Proprioception (VSP) through a shared representation of RGB observations, language instructions, and tendon states. The diffusion head models conditional distributions of expert command chunks, while VSP supervises ordered body-anchor predictions using simulation ground truth, encouraging the representation to retain body geometry when learning from limited demonstrations. Embodied mechanical intelligence supports physical execution of command sequences generated through iterative virtual rollout from evolving simulated observations, with intrinsic compliance providing local contact adaptation without online policy queries. We evaluate SWIM on packing, reaching, and grasping on a planar tendon-driven soft robot, with grasping targets anchored. In simulation, SWIM-VLA achieves success rates of 100\%, 96\%, and 88\%, respectively, outperforming an adapted OpenVLA-OFT baseline and controlled ablations. On hardware, SWIM achieves success rates of 100\%, 80\%, and 75\%, compared with 75\%, 40\%, and 25\% for direct online deployment of the same policy checkpoint.

Key Points

Soft and continuum robots enable manipulation through distributed body deformation and contact, yet translating language and visual context into executable whole-body actuation remains a fundamental challenge.
We present SWIM, a framework that maps an initial RGB observation and a language instruction to a complete actuation-command sequence.
Its vision-language-action (VLA) policy, SWIM-VLA, combines a diffusion action head with Visual Soft Proprioception (VSP) through a shared representation of RGB observations, language instructions, and tendon states.
The diffusion head models conditional distributions of expert command chunks, while VSP supervises ordered body-anchor predictions using simulation ground truth, encouraging the representation to retain body geometry when learning from limited demonstrations.
Embodied mechanical intelligence supports physical execution of command sequences generated through iterative virtual rollout from evolving simulated observations, with intrinsic compliance providing local contact adaptation without online policy queries.

Problem Statement

We study vision-language-grounded whole-body manipulation for soft robots. A task is specified by an initial RGB observation $\mathbf{I}$ of the robot and task scene, together with a language instruction $\ell$.

The desired output is a complete actuation-command sequence where $T$ is the number of command steps in the complete sequence, $N_{a}$ is the number of independently controlled actuation channels, and $\mathbf{a}_{t}\in\mathbb{R}^{N_{a}}$ is the command vector at step $t$.

Problem Statement — SWIM: Vision-Language-Grounded Soft Whole-Body Interactive Manipulation — Fig. 2: Overview of the SWIM framework. (A) SWIM-VLA is trained from simulated interaction trajectories using diffusion action learning with VSP supervision. (B) At inference, the trained policy performs virtual rollout to construct a complete actuation-command sequence, which is then executed open-loop on the physical robot.

Method

SWIM maps an initial RGB observation $\mathbf{I}$ and language instruction $\ell$ to a complete actuation-command sequence $\mathbf{U}$. As illustrated in Fig. 2 , the framework consists of two stages. During training, SWIM-VLA learns to predict command chunks through a diffusion action head from simulated whole-body interaction trajectories with VSP supervision.

During inference, SWIM uses the trained policy to construct $\mathbf{U}$ through virtual rollout, and the resulting sequence is executed open-loop on the physical robot.

Method — SWIM: Vision-Language-Grounded Soft Whole-Body Interactive Manipulation — Fig. 3: Qualitative evaluation of SWIM-VLA.

Experimental Validation

We evaluate the proposed framework from two aspects: (1) In simulation , we evaluate SWIM-VLA on packing, reaching, and grasping tasks, together with controlled ablations of VSP and the diffusion action head; (2) In physical deployment , we evaluate SWIM by comparing its virtual-rollout-based command-sequence execution with the direct online execution of the same trained SWIM-VLA policy.

Experiments are conducted using the SpiRob platform described in Sec. II-B .

Experimental Validation — SWIM: Vision-Language-Grounded Soft Whole-Body Interactive Manipulation — Fig. 4: Spatial distributions of simulation outcomes. Marker locations represent initial positions for packing and target-object positions for reaching and grasping. Green and red markers indicate successful and failed trials, respectively.

Sources and Demonstrations

Figures are reproduced from Liu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.