CLIPort: What and Where Pathways for Robotic Manipulation

Mohit Shridhar, Lucas Manuelli, Dieter Fox · University of Washington, NVIDIA · 2021

Summary

CLIPort fuses CLIP's semantic understanding with Transporter Networks' spatial precision to perform language-conditioned manipulation tasks like 'stack the red block on the blue block' with 2D pick-and-place affordances.

Abstract Summary

CLIPort is one of the earliest works to combine pretrained vision-language models with robot manipulation. The core idea is disentangling the "what" and "where" of manipulation: CLIP provides the semantic understanding ("red block" vs "blue block"), while a Transporter Network handles the spatial precision (exact pixel coordinates for picking and placing). This two-stream architecture is simple, interpretable, and surprisingly effective for tabletop manipulation. The model takes an RGB image and a language instruction as input, and outputs two heatmaps: a pick affordance (where to grasp) and a place affordance (where to release). Training uses 1000+ demonstrations per task in a simulated tabletop environment (PyBullet), with data augmentation (color jittering, camera noise) to improve sim-to-real transfer. The model is trained end-to-end with cross-entropy loss on the affordance maps. On the Ravens benchmark (15 language-conditioned tasks: packing, stacking, sorting, aligning), CLIPort achieves 85% success—outperforming pure CLIP-based baselines (which lack spatial precision) and pure Transporter baselines (which lack semantic grounding). The sim-to-real transfer is strong: the policy deploys directly on a real UR5 arm with a wrist-mounted camera, achieving 70% success on unseen object instances. The open-source release includes the full PyTorch implementation, the Ravens benchmark environments, training scripts, and pretrained weights for all 15 tasks. The codebase is compact (~2K lines) and well-documented, making it a popular starting point for researchers new to language-conditioned manipulation. The modular design also allows swapping CLIP for newer VLMs (e.g., SigLIP, OpenCLIP) with minimal code changes.

Key Points

Two-stream: CLIP for semantics, Transporter for spatial precision.
85% on Ravens benchmark; 70% sim-to-real transfer to UR5.
Outputs interpretable pick/place heatmaps from RGB + language.
Compact codebase (~2K lines), well-documented for beginners.
Swappable VLMs: CLIP, SigLIP, OpenCLIP supported.

CLIPort

CLIPort is an imitation-learning agent based on four key principles: (1) Manipulation through a two-step primitive where each action involves a start and final end-effector pose. (2) Visual representations of actions that are equivariant to translations and rotations [ 56 , 57 ] . (3) Two separate pathways for semantic and spatial information.

(4) Language-conditioned policies for specifying goals and also transferring concepts across tasks.

CLIPort — CLIPort: What and Where Pathways for Robotic Manipulation — Figure 2: CLIPort Two-Stream Architecture. An overview of the semantic and spatial streams. The semantic stream uses a frozen CLIP ResNet50 [ 1 ] to encode RGB input, and its decoder layers are conditioned with tiled language features from the CLIP sentence encoder. The spatial stream encodes RGB-D input, and its decoder layers are laterally fused with the semantic stream. The final output is a map of dense pixelwise features that is used for pick or place affordance predictions. This same two-stream architecture is used in all 3 Fully-Convolutional-Networks $f_{\text{pick}},\Phi_{\text{query}}$, and $\Phi_{\text{key}}$ with $f_{\text{pick}}$ is used to predict pick actions, and $\Phi_{\text{query}}$ and $\Phi_{\text{key}}$ are used to predict place actions. See Appendix C for the exact architecture.

Results

We perform experiments both in simulation and hardware aimed at answering the following questions: 1) How effective is the language-conditioned two-stream architecture for fine-grained manipulation compared to one-stream alternatives and other simpler baselines? 2) Is it possible to train a multi-task model for all tasks, and how well does it perform and generalize?

3) How well do these models generalize to seen and unseen semantic attributes like colors, shapes, and object categories? Environment.

Results — CLIPort: What and Where Pathways for Robotic Manipulation — Figure 4: Affordance predictions from CLIPort (multi) models in sim (left two) and real settings (right three). More examples in Appendix H .

Sources and Demonstrations

Figures are reproduced from Shridhar et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.