Robotics Research Hub
Reinforcement Learning from Real-Robot Interaction without High-Quality State Estimates

Reinforcement Learning from Real-Robot Interaction without High-Quality State Estimates

Ryan Gao, Ashvin Anilkumar, Nicholas Corrado, William Cong, Josiah Hanna · N/A · 2026

vision reinforcement-learning learning-from-demonstration
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

A longstanding goal for robot reinforcement learning (RL) is that robots learn from their own physical experience, as humans and animals do.

Abstract Summary

A longstanding goal for robot reinforcement learning (RL) is that robots learn from their own physical experience, as humans and animals do. A key obstacle to learning directly from physical experience, especially outside instrumented lab environments, is that the state of the world is usually unobservable to the learner and must be estimated from on-board sensing. Errors in that estimate corrupt both the policy input and potentially the reward signal. We study this challenge using the contact-rich task of a bipedal robot learning to score soccer goals. We estimate all key state variables on-board with an unscented Kalman filter (UKF) and no external instrumentation (e.g., motion capture or overhead cameras). Because the UKF may incorrectly estimate state when the ball is unobserved, we learn a new state representation with a recurrent state-space model (RSSM) that, critically, only corrects the RSSM's latent state when the ball is observed. Building on RLPD, we introduce RSSM-RLPD and apply it to the task of a NAO robot learning to dribble a soccer ball into a goal. We show that the robot learns to score from multiple starting positions after 30 minutes of real-world interaction, reaching 100% success against 0% for a feed-forward policy under the same training budget. We complement our physical robot experiments with a simulation study and show that RSSM-RLPD learns policies that achieve a higher success rate than behavior cloning, a feed-forward, and GRU baseline. Taken together, our results show that the addition of a learned state representation to existing on-robot learning methods enables RL from real-robot data even with a high degree of partial observability.

Key Points

  • A longstanding goal for robot reinforcement learning (RL) is that robots learn from their own physical experience, as humans and animals do.
  • A key obstacle to learning directly from physical experience, especially outside instrumented lab environments, is that the state of the world is usually unobservable to the learner and must be estimated from on-board sensing.
  • Errors in that estimate corrupt both the policy input and potentially the reward signal
  • We study this challenge using the contact-rich task of a bipedal robot learning to score soccer goals.
  • We estimate all key state variables on-board with an unscented Kalman filter (UKF) and no external instrumentation (e.g., motion capture or overhead cameras).

Task and System

We formulate the task of dribbling a soccer ball into a goal as a partially observable Markov decision process (POMDP) [ 24 ] defined by the tuple $(\mathcal{S},\mathcal{A},\mathcal{O},T,\Omega,R,d_{0},\gamma)$ where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\mathcal{O}$ is the observation space, $T$ is the transition function, $\Omega$ is the observation function, $R$ is the reward function, $d_{0}$ is the initial state distribution, and $\gamma\in[0,1)$ is the discount factor.

Task and System — Reinforcement Learning from Real-Robot Interaction without High-Quality State Estimates
Fig. 1: Physical setup: a NAOv6 humanoid robot on the $7\times 4.8$ m half-field, learning to dribble the ball into the goal from real experience and on-board sensing. RL is challenged by on-board sensing, which often yields inaccurate estimates of key state information, such as the ball’s location when it is outside the field of view.

RSSM–RLPD for On-Robot Learning

This section introduces our main algorithmic contribution, RSSM–RLPD, which addresses partial ball visibility by augmenting the replay-mixture approach of RLPD [ 1 ] with an observation-gated recurrent state-space model. The actor and critics share a representation trained by prediction losses and the critic objective to retain information about ball motion relevant to the control task.

RSSM–RLPD for On-Robot Learning — Reinforcement Learning from Real-Robot Interaction without High-Quality State Estimates
Fig. 4: Outline of our real-world training environment setup

Experiments

We evaluate our approach on the task of learning to dribble a ball into a goal through physical interaction.

Our experiments combine hardware and simulation to leverage their complementary strengths: hardware captures real-world sensing noise and contact dynamics without simulation-induced approximations, while simulation enables multi-seed replication and systematic evaluation across a broader range of failure conditions. The physical task uses a $7\times 4.8$ m half-field.

Experiments — Reinforcement Learning from Real-Robot Interaction without High-Quality State Estimates
Fig. 5: Outline of our real-world validation environment setup

Sources and Demonstrations

Figures are reproduced from Gao et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Awomo-SimDataEngine: Agentic Simulation-ReadyWorld Generation
arXiv preprint

Awomo-SimDataEngine: Agentic Simulation-ReadyWorld Generation

PhysicalRSI Team, Danjiao Ma, Enhui Ma et al.

Generating useful robot-training data requires more than visually plausiblescenes: objects must support interaction, placements must remain physicallyvalid, and tasks must admit repeatable execution.

vision reinforcement-learning learning-from-demonstration benchmark
PDF Intermediate