Robotics Research Hub
When Does Test-Time Physical Diagnosis Pay? A Frozen Policy Buys Evidence It Never Reads

When Does Test-Time Physical Diagnosis Pay? A Frozen Policy Buys Evidence It Never Reads

Zhengshu Zhang · N/A · 2026

reinforcement-learning control benchmark
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

When a robot faces unfamiliar physical conditions, a common approach is to collect evidence about what changed and adapt.

Abstract Summary

When a robot faces unfamiliar physical conditions, a common approach is to collect evidence about what changed and adapt. For such diagnosis to improve behavior, six ordered empirical conditions must hold: a meaningful reference, identifiability of the physical condition, use of the acquired evidence, decision value, selection value over a fixed alternative, and safe realization. We test this chain in controlled and public environments. It holds end to end in our controlled environments. After transfer to unseen mechanisms, however, it breaks at evidence use. On decisions requiring the full trace, the frozen decoder does not change its choice. A linear model using only trace increments recovers the correct choice on mechanisms excluded from fitting, showing that the trace is informative but unused. The failure is concentrated at the richest evidence level: those decisions fall to chance, while decisions settled with lower-cost evidence remain correct, a split hidden by aggregate accuracy. The same chain can fail at other links in public environments. Successful physical identification therefore guarantees neither evidence use nor useful adaptation; evaluation should identify where the chain breaks rather than rely on recovery accuracy or aggregate performance alone.

Key Points

  • When a robot faces unfamiliar physical conditions, a common approach is to collect evidence about what changed and adapt.
  • For such diagnosis to improve behavior, six ordered empirical conditions must hold: a meaningful reference, identifiability of the physical condition, use of the acquired evidence, decision value, selection value over a fixed alternative, and safe realization.
  • We test this chain in controlled and public environments
  • It holds end to end in our controlled environments
  • After transfer to unseen mechanisms, however, it breaks at evidence use

What Must Hold for Physical Diagnosis to Pay?

Diagnosis pays only if a chain of conditions holds end to end. Between “the system found out what changed” and “the system did better for knowing it” sit six conditions, ordered as a single decision passes through them (Figure 1 a); a break at any link ends the case for diagnosis however well the earlier ones went.

A failed condition ends the case for diagnosis and blocks downstream development, although diagnostics already available from the same panel may expose later failures.

What Must Hold for Physical Diagnosis to Pay? — When Does Test-Time Physical Diagnosis Pay? A Frozen Policy Buys Evidence It Never Reads
Figure 1: The chain, and where this record breaks it. (a) The six conditions one decision passes through; the mark after evidence use is where this record breaks. (b) Accuracy of the frozen decoder on three unseen mechanisms, split by the price paid. The 30 decisions that bought the full trace land at chance, and the aggregate is the average of the two groups. The hatched bar is a post-result diagnostic over those rows: it localizes a recoverable signal and does not repair the decoder.

Evaluation Protocol: Isolating the Failed Link

Every threshold below was set before the run it judges, so where a result lands close to its threshold the verdict is not a judgement made after seeing the number. A system that chooses how much evidence to buy can also beat a fixed rule by buying more, so the fixed rule receives the same purchases: whatever the adaptive system bought on a given decision, the comparator gets too, leaving only allocation to compare.

Evaluation Protocol: Isolating the Failed Link — When Does Test-Time Physical Diagnosis Pay? A Frozen Policy Buys Evidence It Never Reads
Figure 2: The frozen record. One row per setting, one column per condition of Figure 1 (a). Settings stop at different points in the chain, and no single condition accounts for them all. Glyph shape carries the status, so the panel survives greyscale; grey records absence of result, not negative evidence.

Findings

We first check that the evaluation registers success. In two simulators we built, every decision chooses between two action strategies, and which wins depends on a physical condition the system cannot see: whether contact slips, whether the actuator has margin, whether the button breaks.

Before acting it buys evidence about that condition at one of four prices, paid in bytes serialized: nothing, a single bit saying whether the situation is resolvable, a telemetry summary, or the full trace.

Sources and Demonstrations

Figures are reproduced from Zhang See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced