CLIPort: What and Where Pathways for Robotic Manipulation

CLIPort is one of the earliest works to combine pretrained vision-language models with robot manipulation. The core idea is disentangling the "what" and "where" of manipulation: CLIP provides the semantic understanding ("red block" vs "blue block"), while a Transporter Network handles the spatial precision (exact pixel coordinates for picking and placing). This two-stream architecture is simple, interpretable, and surprisingly effective for tabletop manipulation. The model takes an RGB image and a language instruction as input, and outputs two heatmaps: a pick affordance (where to grasp) and a place affordance (where to release). Training uses 1000+ demonstrations per task in a simulated tabletop environment (PyBullet), with data augmentation (color jittering, camera noise) to improve sim-to-real transfer. The model is trained end-to-end with cross-entropy loss on the affordance maps. On the Ravens benchmark (15 language-conditioned tasks: packing, stacking, sorting, aligning), CLIPort achieves 85% success—outperforming pure CLIP-based baselines (which lack spatial precision) and pure Transporter baselines (which lack semantic grounding). The sim-to-real transfer is strong: the policy deploys directly on a real UR5 arm with a wrist-mounted camera, achieving 70% success on unseen object instances. The open-source release includes the full PyTorch implementation, the Ravens benchmark environments, training scripts, and pretrained weights for all 15 tasks. The codebase is compact (~2K lines) and well-documented, making it a popular starting point for researchers new to language-conditioned manipulation. The modular design also allows swapping CLIP for newer VLMs (e.g., SigLIP, OpenCLIP) with minimal code changes.