Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordination

Suyu Ye, Zheyuan Zhang, Vaishnav Tadiparthi, Hossein Nourkhiz Mahjoub, Ehsan Moradi Pari, Tianmin Shu, Homanga Bharadhwaj, Nakul Agarwal · N/A · 2026

Summary

Abstract Summary

Robots operating in the physical world will increasingly need to coordinate with other robots, particularly in manipulation tasks where an object may be too large or heavy for a single robot to carry alone. Physical limitations caused by hardware degradation or actuator faults can restrict the actions a robot can reliably execute, yet these limitations may be unknown to its partner. We study whether a helper can infer a robot partner's physical constraints from observing it coordinate with another robot, then use the inferred capability to coordinate with the same partner on a new task. This is difficult because a demonstration shows what the constrained robot did, but not what it could have done. In physically coupled tasks, the other robot may also compensate for its limitations, making those limitations difficult to identify from the constrained robot's behavior alone. Our key insight is that these constraints shape the joint behavior of the team, making the actions of both robots informative about the constrained partner's capability. We introduce Watch, Infer, Coordinate, a benchmark spanning three physically coupled manipulation settings, together with an inference approach that scores candidate constraints using observed joint behavior. Across all three settings, our method substantially improves constraint inference and zero-shot coordination, approaching an oracle with access to the true constraints.

Key Points

Robots operating in the physical world will increasingly need to coordinate with other robots, particularly in manipulation tasks where an object may be too large or heavy for a single robot to carry alone.
Physical limitations caused by hardware degradation or actuator faults can restrict the actions a robot can reliably execute, yet these limitations may be unknown to its partner.
We study whether a helper can infer a robot partner's physical constraints from observing it coordinate with another robot, then use the inferred capability to coordinate with the same partner on a new task.
This is difficult because a demonstration shows what the constrained robot did, but not what it could have done.
In physically coupled tasks, the other robot may also compensate for its limitations, making those limitations difficult to identify from the constrained robot's behavior alone.

Watch, Infer, Coordinate: Problem Setup and Benchmark

We consider a helper that must coordinate with a robot whose low-level physical capabilities are initially unknown. We refer to this robot as the constrained agent . Its physical actions are restricted by constraints, such as joint position limits, velocity limits, or allowable motion directions.

During the observation stage, the helper watches the constrained agent coordinate with another robot, which we call the demonstrator .

Watch, Infer, Coordinate: Problem Setup and Benchmark — Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordination — Figure 2: Overview of our physically coupled partner-adaptation benchmark. (A) During the observation stage, the helper watches a demonstrator coordinate with the constrained agent on an obstacle-free task. The labels show physical constraints on the action space of the constrained agent, which are dependent on the setting. (B) At test time, the helper replaces the demonstrator and coordinates zero-shot with the same constrained agent on a new task with obstacles. The benchmark includes three settings of increasing physical complexity: 2D rod carrying, fixed-base dual-UR5 carrying, and mobile dual-UR5 carrying. Across all settings, the constrained agent’s physical constraints remain fixed while the task and helper change.

Method

We infer the constrained agent’s capability by treating each candidate capability as a hypothesis about which actions the agent can execute. At each observed state, a candidate changes the constrained agent’s feasible actions and, in turn, how the demonstrator is expected to respond.

Using the boundedly rational behavioral model, we predict the joint behavior of the two agents under each candidate and score how well it matches the observed actions across demonstrations.

Method — Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordination — Figure 3: Qualitative examples of zero-shot coordination using our method. (A) Representative successful trajectories for 2D rod carrying, fixed-base dual-UR5 carrying, and mobile dual-UR5 carrying, shown from left to right over time. (B) Example of how an inferred physical constraint changes test-time planning. Ignoring the constrained agent’s limitation leads to a pose it cannot reach, while incorporating the inferred constraint causes the planner to select a feasible alternative.

Experiments and Results

We evaluate whether our method can accurately infer the constrained agent’s physical constraints from prior coordination and whether the inferred capability supports zero-shot coordination on a new task. We study both questions across the 2D rod, fixed-base dual-UR5, and mobile dual-UR5 settings.

For capability inference, we vary the number of observed demonstrations over $N\in\{1,2,4,8\}$ and measure normalized Hamming distance between the inferred and true constraints, where lower is better.

Sources and Demonstrations

Figures are reproduced from Ye et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.