OJOx: Specification-Conditioned Demonstrations for Embodied AI in Construction

Summary

Abstract Summary

Large-scale egocentric and whole-body human demonstrations are becoming a primary source of data for embodied intelligence. They record what people perceive and do, but rarely the external specification that gave an action its purpose. In construction that omission is consequential: skilled work is directed at project-specific configurations defined in a design model - configurations not yet present in the environment being observed. A mason's transferable competence is not the geometry of one wall but the ability to realise a new geometry from a specification. We introduce the specification-conditioned demonstration: a synchronised record of the physical state a demonstrator perceives, the intended state supplied to them by an external design, and the behaviour connecting the two. We present OJOx, a capture interface that realises this for construction - delivering design geometry to a headset, anchoring it in the physical workspace, rendering it into a demonstrator's stereo passthrough view, and recording that view synchronously with whole-body and hand motion. We report one fully instrumented session - a 33-component wall laid against a specification that changes while the work proceeds - and check the record against the physical scene through an external camera registered independently of the capture. Recorded sessions remain compatible with existing humanoid retargeting infrastructure and replay onto a Unitree G1 in simulation. The result is a data interface for testing whether embodied policies can learn not merely to imitate demonstrated actions, but to act toward specifications absent from their training experience.

Key Points

Large-scale egocentric and whole-body human demonstrations are becoming a primary source of data for embodied intelligence.
They record what people perceive and do, but rarely the external specification that gave an action its purpose.
In construction that omission is consequential: skilled work is directed at project-specific configurations defined in a design model - configurations not yet present in the environment being observed.
A mason's transferable competence is not the geometry of one wall but the ability to realise a new geometry from a specification.
We introduce the specification-conditioned demonstration: a synchronised record of the physical state a demonstrator perceives, the intended state supplied to them by an external design, and the behaviour connecting the two.

OJOx: Specification-Conditioned Capture

Augmented reality is the mechanism here, not the contribution. What the system produces is a data interface (Fig. 2 ): a way of getting an externally specified intended state into a demonstration at the moment the demonstration happens.

The specification is design geometry — in construction practice originating in BIM/CAD tooling — packaged with its task data and published to cloud storage, from which the headset retrieves it over HTTP and caches it on the device.

OJOx: Specification-Conditioned Capture — OJOx: Specification-Conditioned Demonstrations for Embodied AI in Construction — Fig. 4: The target state changes while the human acts. Below: the assembly-sequence state $g_{t}$ over the session clock, read from the session’s own assembly-event record at its recorded timestamps. The complete 33-component design is displayed for the first minute; the system then switches to sequential targeting and reveals components one at a time. Open circles mark the six transitions that step backwards . Above: four recorded frames at the marked instants. A goal recovered from the finished wall would contain none of this.

The Specification-Conditioned Demonstration

Write one session as with $o_{t}$ the observation of the physical state, $g_{t}$ the externally supplied intended state, and $h_{t}$ the demonstrator’s whole-body and hand configuration. In OJOx, $o_{t}$ and the visual form of $g_{t}$ arrive fused: the specification is composited into the same pixels the demonstrator perceives.

The symbolic record provides $g_{t}$ a second time, as component identity, 6-DoF pose, and current assembly-sequence state on the shared session clock.

The Specification-Conditioned Demonstration — OJOx: Specification-Conditioned Demonstrations for Embodied AI in Construction — Fig. 5: The record checked against the physical scene, at one moment. (a) an external camera that took no part in the capture, unmodified. (b) the same frame with the recorded specification, body and hands projected into it: every drawn element is read back from the session, and the only thing the video supplies is the viewpoint. (c) the same recorded motion retargeted onto a Unitree G1 in MuJoCo, seen from that same viewpoint, which is what allows (b) and (c) to be compared. The robot is replaying recorded human motion in simulation: it is not executing the task, no object is simulated, and nothing is learned.

System Validation

Whole-body Hands Egocentric video ● yes ◐ partial ○ no n/a not applicable. Rows group representative systems. Several columns lie outside a given system’s purpose rather than marking a deficiency, and where no goal representation is recorded at all the four goal columns are negative by entailment.

Toward Specification-Conditioned Embodied AI

A conventional imitation dataset supports one mapping, $o_{t}\rightarrow a_{t}$: act from what you see. A specification-conditioned demonstration makes a different one learnable in principle, which reads: given what exists and a statement of what should exist, how should I act? The shift it describes is from imitate this action to realise this specification .

None of what follows is demonstrated here; it is the programme the interface exists to enable.

Sources and Demonstrations

Figures are reproduced from Dawod See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.