OK-Robot: Open-Ended Object Manipulation with Pretrained Vision-Language Models

OK-Robot asks: can we skip robot learning entirely and just use pretrained vision-language models to manipulate objects in new environments? The answer, surprisingly, is yes—for simple pick-and-place tasks. OK-Robot chains three off-the-shelf models: an open-vocabulary object detector (OWL-ViT) to find objects, a grasping model (AnyGrasp) to pick them up, and a VLM (CLIP) to verify the goal. No training on robot data is required; the system works out-of-the-box in homes it has never seen before. The system architecture is a perception-action loop: a mobile robot (Hello Robot Stretch) scans the room with its head camera, OWL-ViT detects objects matching the language query ("find the yellow sponge"), the LLM plans a sequence of actions ("navigate to kitchen → pick sponge → navigate to sink → place sponge"), and a heuristic grasping policy executes the motions. The robot maintains a semantic map (object locations + room categories) built incrementally as it explores. On 40 real-world tasks in 10 unseen homes, OK-Robot achieves 58% success for pick-and-place and 72% for navigation-only tasks. The failures are illuminating: most come from grasping failures (slippery objects, tight spaces) or navigation errors (collisions with unseen obstacles), not from VLM hallucinations. This suggests that perception is solved enough for home manipulation; the bottleneck is now low-level control and hardware reliability. The open-source release includes the full pipeline (ROS2 nodes for perception, planning, and control), setup guides for the Stretch robot, and a simulator version that runs in iGibson for testing without hardware. The modular design makes it easy to swap components: replace OWL-ViT with Grounding DINO, or AnyGrasp with a learned policy like Diffusion Policy.