Scaling Online Exploration for Robot Agents

Runhan Huang, Xiangcheng Zhang, Haoyu Chen, Yitong Li, Yen-Jen Wang, Bohan Lyu, Haotian Ye, Chuye Hong, Yixuan Wang, Chloe Su, Haoyu Xiong, Yangzhen Wu, Enze Xie, Travis Manderson, Stefano Ermon, Pieter Abbeel, Antonio Torralba, Dawn Song, Weirui Ye, Yilun Du · N/A · 2026

Summary

Robot learning has advanced by scaling demonstration datasets and model capacity; however, deploying a new robot embodiment or task typically necessitates a dedicated teleoperation corpus.

Abstract Summary

Robot learning has advanced by scaling demonstration datasets and model capacity; however, deploying a new robot embodiment or task typically necessitates a dedicated teleoperation corpus. The resulting policies remain bounded by their imitation distributions, limiting generalization and continual self-improvement. We introduce RAX (Robot Agent eXploration), an open-source framework for robot agents centered on scaling exploration: enabling agents to improve by acquiring online experience and synthesizing executable programs. In RAX, a general-purpose coding agent generates robot control code, operating without action demonstration data, hand-engineered action primitives, or value-function guidance. Through online exploration, the agent identifies decision sequences that achieve task goals and compiles them into modular programs encapsulating the full perception-planning-control loop. Using RAX, we present a systematic multi-embodiment study on scaling online exploration across precision and long-horizon manipulation tasks using single-arm, bimanual, dexterous, mobile, and humanoid systems in simulation and on real robots. Generalization performance on unseen environments improves consistently as exploration scales. In simulation, RAX outperforms top baseline models by +37.4% on average. On real robots, RAX autonomously learns memory-dependent, articulated, high-precision, deformable, and long-horizon mobile manipulation tasks without human demonstrations, continually improving performance by composing verified skills from prior exploration sessions. As exploration scales, we observe emergent capabilities including active perception, error recovery, safety guardrails, and human-robot collaboration. We open-source RAX as a generalist robot-agent platform in which new robot embodiments are integrated through their APIs alone, without teleoperated data collection or policy training.

Key Points

Robot learning has advanced by scaling demonstration datasets and model capacity; however, deploying a new robot embodiment or task typically necessitates a dedicated teleoperation corpus.
The resulting policies remain bounded by their imitation distributions, limiting generalization and continual self-improvement.
We introduce RAX (Robot Agent eXploration), an open-source framework for robot agents centered on scaling exploration: enabling agents to improve by acquiring online experience and synthesizing executable programs.
In RAX, a general-purpose coding agent generates robot control code, operating without action demonstration data, hand-engineered action primitives, or value-function guidance.
Through online exploration, the agent identifies decision sequences that achieve task goals and compiles them into modular programs encapsulating the full perception-planning-control loop.

The RAX Cognitive Architecture

Inspired by Sutton’s OaK architecture ( Sutton et al., 2022 ) , RAX acquires all task-specific knowledge at runtime from the agent’s own experience. Building on task-agnostic robot control interfaces and low-level controllers, the agent autonomously develops task-specific perception, manipulation strategies, and executable skills through exploration.

The following modules surround the language-model core: perception and control , autonomous robot program , and memory for continual learning ( Figure 2 ).

The RAX Cognitive Architecture — Scaling Online Exploration for Robot Agents — Figure 2 : System architecture and self-improving loop of RAX. The coding agent, powered by a frozen LLM, interacts with physical hardware across an isolation boundary using multimodal perception and control SDK interfaces. By reflecting on execution outcomes, the agent continually updates a persistent memory system containing working context, task notes, and a versioned repository of verified skills. During exploration, the agent compiles its perception and planning logic into standalone Python control programs that execute high-frequency loops directly on the robot. When encountering perception or planning failures due to environment variations, the program raises an exception ( raise Unsure() ), delegating control back to the agent to make adjustments and refine the codebase.

The RAX System

RAX supports a wide range of embodiments in simulation and in the real world, including single-arm and bimanual manipulators, dexterous hands, mobile manipulators, and humanoids, with one agent architecture. To do so, it adopts a separation familiar from software engineering, in which a frontend is kept apart from the backend that serves it.

In RAX, the backend is the robot’s physical stack: the hardware drivers or the simulator, the low-level controllers, the sensor pipelines, and the safeguards around them.

The RAX System — Scaling Online Exploration for Robot Agents — Figure 3 : Agentic 3D perception and motion planning on a real robot. (a) The articulated air fryer model constructed programmatically by the agent from point-cloud measurements and fit to RGB-D observations, consisting of a fixed chassis and a drawer articulated via a prismatic joint along pull direction $d$ relative to central axis $c$. (b) Action trajectories planned on the instantiated model with the drawer opened by $13.1\,\text{cm}$. End-effector poses are visualized as pyramids whose bases face the approach vector and whose thick edges indicate the finger closure axis. The underlying geometric estimator maintains high fidelity, yielding only a $6.5\,\text{mm}$ displacement between the predicted basket rim and the post-opening physical measurement. Figure 27 illustrates the corresponding execution sequence on the physical robot.

Experiments

In this section we report our simulation benchmark results as well as case studies on both simulation and real robot experiments. We comprehensively evaluate RAX in both simulated environments and on real-world robotic hardware.

In simulation, the evaluation spans six benchmarks covering single-arm, bimanual, dexterous, mobile, and humanoid embodiments: RoboDojo, BEHAVIOR-1K, HumanoidArena, DexJoCo, LIBERO, and LIBERO-Pro. In each session, a coding agent starts in a localized workspace containing only the platform’s baseline skills.

Experiments — Scaling Online Exploration for Robot Agents — Figure 4 : Frontend–backend separation. The agent and its programs act only through the tool interface; the backend owns control, sensing, and safeguards on every platform.

Interface Contracts

On the real robots the agent acts only through tools that wrap each vendor’s control SDK, and its own programs call the same tools. Every command is checked before the robot moves; workspace, joint limit, or IK violations are refused, and motion works only while the operator has armed the robot. Table 11 summarizes the tools; both robots also offer status and request_operator .

Agent Instructions and Skills

Every session starts from the same workspace template: one instruction file and a folder of skills, written once and shared by all platforms ( auto_research_agent/ and agent_skills/ in our code). Robot-specific facts, such as tool names, cameras, and control rates, are filled in when the workspace is built, so the agent reads the same instructions on every robot.

The instruction file fixes the order of work rather than how to solve any task.

Agent Instructions and Skills — Scaling Online Exploration for Robot Agents — Figure 6 : The program RAX writes for boxing books on BEHAVIOR-1K ( Table 3 ). It detects the books, shelves and box in the head camera, tracks the robot’s motion by visual odometry, maps the scene, and plans paths with A*; the policy flow shows the camera view at each step.

Benchmarks and Tasks

This section gives the protocol that all simulation experiments share and then the setting of each benchmark and its tasks. The real-robot tasks are described in Section 14 .

Every experiment is a session of one coding agent in its own command-line harness: Claude Code with Claude Opus 5.5 or, on four RoboDojo tasks and one BEHAVIOR-1K task, Codex with a GPT-6 model, at the xhigh reasoning effort unless stated otherwise.

Benchmarks and Tasks — Scaling Online Exploration for Robot Agents — Figure 7 : From exploration to code. Inference time (left) and output tokens (right) of each successful exploration episode, on base-2 logarithmic vertical axes; for rehearsals, inference time is the program’s computation time as measured by the harness, without the overhead of running in isolation. The token axis is linear below one thousand tokens so that zero can be shown. Filled dots mark episodes in which the agent takes part, alone or together with its program; hollow circles mark rehearsals, in which the program runs alone. On loading the car, raw points are faint and the line is a running median over five successes; output tokens of parallel episodes are attributed to the environment an agent was assigned, or split evenly across the episodes active at that moment.

Simulation Case Studies

This section follows single sessions on the simulation benchmarks in detail: what the agent explored, the program it wrote, and how that program behaves on unseen scenes. The first task of the LIBERO-Long suite asks the robot to put both the alphabet soup and the tomato sauce in the basket, on a table that also holds a ketchup bottle, a carton of orange juice, milk, butter, and cream cheese.

Exploration Budget of ASPIRE on LIBERO-Pro

ASPIRE ( Lu et al., 2026 ) writes robot programs in CaP-X ( Fu et al., 2026 ) with Claude Code and Claude Opus 4.6. Its programs call the CaP-X interfaces for perception, including text-prompted segmentation and vision-language pointing, for grasp planning, including a top-down grasp selector, for inverse kinematics, and for joint-space motion, and they may not read the simulator’s ground-truth state.

Summary Results of 1

The two simulation charts of 1 summarize RoboDojo, BEHAVIOR-1K, HumanoidArena, DexJoCo, and LIBERO-Pro, and compare RAX with the strongest prior model, that is, with learned policies such as VLAs and world action models rather than with other agents.

Real-Robot Tasks

This section describes the setup of every real-robot experiment: the task, the robot and its sensors, what the agent was given, what the operator did, how many trials the agent ran, how success was judged, and the final outcome.

Five tasks use a Franka FR3 arm with a parallel gripper, two exterior RGB-D cameras (left and right of the table), and an RGB-D camera on the wrist; the sixth uses a Rainbow RB-Y1 mobile manipulator with an RGB-D camera on its head. The control interfaces are those of Section 4.2 .

Real-Robot Tasks — Scaling Online Exploration for Robot Agents — Figure 11 : Continual learning on the real-world RB-Y1 mobile manipulator. Top: Modular skills acquired during prior exploration sessions and stored in the skill repository, complete with descriptions from their respective skill files. Middle: The final execution of a complex tidy-up task—a single high-level instruction carried out across six sequential stages, annotated with completion timestamps and invoked skills. Bottom: The agent’s generated code for each stage (condensed from execution logs with passed arguments). Code invoking pre-stored skills is color-coded by skill ( chair , cups , bucket , and box correspond to each skill’s run.py ), while ad-hoc control logic written directly by the agent is shown in black.

Compute and Cost

Across all experiments in this paper, simulated and real, the agents’ model calls consumed over 100k US dollars of API credits. We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report. Our team has already identified the following issues .

Compute and Cost — Scaling Online Exploration for Robot Agents — Figure 12 : Adapting skills and programs on the real Franka. Each row is one program that already worked, with the task it learned (left image) and the operator’s change (green). Right: the program’s entry point after the change, condensed from the agent’s code, with new lines in green, modified lines in blue, and reused lines in black. Top: find the cube, where each image overlays frames from the left camera a quarter of a second apart, earlier ones fainter, to show how the object is hidden; the cube is first dropped into a cup and then replaced by a red wedge. Bottom: wiping with paper towels from a box; S and P are the towel program’s modules, imported unchanged.

Sources and Demonstrations

Figures are reproduced from Huang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.