VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

VoxPoser tackles the long-standing bottleneck in LLM-driven robotics: while language models can reason and plan, most existing systems still rely on hard-coded motion primitives to execute physical actions. The authors propose a framework where an LLM grounds free-form language instructions directly into the robot’s observation space by composing *3D value maps*. These maps encode spatial constraints and affordances inferred from the language prompt, effectively acting as a cost landscape that guides trajectory optimization without any pre-programmed skills. The key insight is to leverage the LLM’s code-writing capability to call a vision-language model (VLM), generating programs that produce composable 3D value functions. For example, given the instruction "place the apple inside the bowl on the left," the LLM generates code that queries object segmentation, computes spatial relationships, and assembles an attraction field toward the target region and a repulsion field around obstacles. The resulting dense value map is then passed to an LQR-style model-based planner that synthesizes a closed-loop sequence of 6-DoF end-effector waypoints. Because the maps are recomputed online, the policy is robust to dynamic perturbations such as objects moving mid-trajectory. Beyond zero-shot synthesis, VoxPoser also demonstrates efficient online adaptation for contact-rich interactions. In scenes where precise force profiles matter—such as opening a drawer or inserting a peg—the authors learn a local dynamics model from online rollouts, which refines the value map and improves success rates with only a handful of real-world interactions. This hybrid approach marries the high-level semantic reasoning of LLMs with low-level physics modeling, yielding a system that handles both coarse reaching and fine-grained contact without switching controllers. Evaluation spans both simulated environments and real robotic hardware. The system performs over 30 everyday manipulation tasks specified in free-form language, often succeeding from entirely unseen initial configurations. Importantly, VoxPoser operates zero-shot: no task-specific pre-training or demonstration data is required. The authors release code, pretrained perception modules, and a simulated benchmark, providing a valuable resource for researchers exploring the intersection of foundation models and embodied control.