Robotics Research Hub
Scouting the Dynamics Gap: Test-Time Policy Adaptation via Action-Outcome Feedback

Scouting the Dynamics Gap: Test-Time Policy Adaptation via Action-Outcome Feedback

Yishu Li, Liyuan Geng, Xinyi Mao, Amber Li, David Held · N/A · 2026

manipulation reinforcement-learning sim-to-real control benchmark
PDF arXiv Code (GitHub)

Framework

N/A

License

N/A

Stars

N/A

Summary

While pretrained robotic policies exhibit impressive capabilities in controlled environments, unobserved physical properties and dynamics require these policies to rapidly adapt during deployment.

Abstract Summary

While pretrained robotic policies exhibit impressive capabilities in controlled environments, unobserved physical properties and dynamics require these policies to rapidly adapt during deployment. Existing test-time adaptation methods typically rely on sparse scalar rewards, failing to exploit the rich geometric and dynamic feedback from the environment during physical interaction. To address this challenge, we propose SCOUT, a dynamics-aware meta-learning framework that enables manipulation policies to rapidly adapt by continuously revising their internal beliefs about environment dynamics. Our approach couples an action-prediction policy with a forward dynamics model via a shared belief latent space. During meta-training, an inner loop updates this shared belief latent by minimizing the dynamics prediction error against the observed action outcome, while the outer loop optimizes the network for action selection. At deployment, this structure allows the agent to infer and adapt to unknown physical dynamics on the fly. By updating its latent belief based on action-outcome mismatches, the policy automatically adapts without risking catastrophic forgetting. We demonstrate that SCOUT significantly accelerates online adaptation across simulated manipulation benchmarks and achieves robust sim-to-real transfer in the real world. Project webiste can be found here: https://liy1shu.github.io/SCOUT/

Key Points

  • While pretrained robotic policies exhibit impressive capabilities in controlled environments, unobserved physical properties and dynamics require these policies to rapidly adapt during deployment.
  • Existing test-time adaptation methods typically rely on sparse scalar rewards, failing to exploit the rich geometric and dynamic feedback from the environment during physical interaction.
  • To address this challenge, we propose SCOUT, a dynamics-aware meta-learning framework that enables manipulation policies to rapidly adapt by continuously revising their internal beliefs about environment dynamics.
  • Our approach couples an action-prediction policy with a forward dynamics model via a shared belief latent space.
  • During meta-training, an inner loop updates this shared belief latent by minimizing the dynamics prediction error against the observed action outcome, while the outer loop optimizes the network for action selection.

Problem Statement and Assumptions

We formulate the problem as policy learning under a partially observable Markov decision process (POMDP) subject to unknown or unobserved dynamics. In this setting, the environment’s behavior depends on latent physical properties that are not directly observable but are critical for accomplishing the task.

Formally, we consider a family of POMDPs indexed by a latent variable $z\in\mathcal{Z}$ that specifies the environment’s unobserved dynamics properties.

Problem Statement and Assumptions — Scouting the Dynamics Gap: Test-Time Policy Adaptation via Action-Outcome Feedback
Figure 2: Overview of our test-time adaptation framework. In the inner loop, the initial belief $q_{0}$ is adapted to $q_{k}$ using past interactions by minimizing the latent dynamics prediction error ($L_{\text{inner}}$) while keeping all network weights frozen. In the outer loop, the adapted belief $q_{k}$ modulates the policy head via FiLM to predict the action distribution. The entire framework, including $q_{0}$, is meta-trained end-to-end by backpropagating the policy loss ($L_{\text{outer}}$) through the inner loop updates.

Method

Based on the above motivation, we propose a test-time adaptation framework that couples a policy with a predictive dynamics model via a shared belief latent. By optimizing this shared representation through an inner loop of dynamics prediction within a meta-learning framework, we provide a principled mechanism to translate dynamics prediction errors into actionable policy updates.

An overview of our framework is shown in Fig. 2 . We denote the agent’s current observation at time step $t$ as $o_{t}$.

Method — Scouting the Dynamics Gap: Test-Time Policy Adaptation via Action-Outcome Feedback
Figure 3: Real world uneven bar pickup and push results . Left: The number of attempts to success with and without test-time training, reported with 95% confidence interval. Right: A visualization of the policy prediction weights across the x-axis of the uneven bar before and after adaptation. We can see that SCOUT efficiently adapts to different Center of Mass positions.

Experiment and Analysis

We evaluate our proposed method against baselines across multiple manipulation tasks. We follow the ISE evaluation task suite [ 40 ] , which includes the Slide Brick, Push Bar, Pick Bar, Open Box, and Turn Faucet, and also evaluate on an ambiguous door dataset [ 31 , 41 ] . We also conduct a sim-to-real experiment of pick bar and push bar on a real-world uneven bar.

Experiment and Analysis — Scouting the Dynamics Gap: Test-Time Policy Adaptation via Action-Outcome Feedback
Figure 4: Belief Latent Space Evolution. PCA projection of the belief latent $q_{k}$ across three stages of adaptation. Points are colored by their ground-truth motion mode, while background regions represent kNN-estimated decision boundaries. As test-time training progresses, the latents successfully migrate from the shared prior $q_{0}$ into distinct, well-separated kinematic clusters.

Sources and Demonstrations

Figures are reproduced from Li et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Dexterity from Supervised Play
arXiv preprint

Dexterity from Supervised Play

Botao He, Yiming Dou, Yiannis Aloimonos et al.

How can robots autonomously collect the contact-rich experience needed to learn dexterous manipulation?

manipulation reinforcement-learning sim-to-real planning control learning-from-demonstration benchmark
Code PDF Intermediate
GitHub ★ —