PAC-MAN: Perception-Aware CBF-RL for Whole-Body Safety in Humanoid Dodgeball

Summary

We present PAC-MAN, a perception-aware CBF-RL framework that couples control-barrier safety with deployment-realistic onboard sensing for whole-body humanoid dodgeball.

Abstract Summary

We present PAC-MAN, a perception-aware CBF-RL framework that couples control-barrier safety with deployment-realistic onboard sensing for whole-body humanoid dodgeball. The deployed policy sees the ball only as segmentation-masked depth from a head-mounted camera, while training-time CBF guidance represents clearance to every body link, and an adversarial motion prior regularizes the resulting evasive reflexes. We evaluate on a controlled any-link contact benchmark with seeded throws in two regimes: single throws and a deployment loop in which the robot walks back to its station and recovers between throws. On this benchmark, the policy comes within a few points of a privileged state oracle: a fixed onboard camera alone is adequate for evasion. We find that usable barrier structure depends on perceptual observability: Joint-CBF gives the best performance with accurate ball states, degrades under fixed-camera observations when used only as training guidance, and recovers with a ball-tracking gimbal or privileged runtime filter. We therefore deploy a lightweight Link-CBF policy zero-shot on the Unitree G1 in the real world, where it tolerates imperfect perception, succeeds on 95% of throws, and uses semantic segmentation to dodge different balls.

Key Points

We present PAC-MAN, a perception-aware CBF-RL framework that couples control-barrier safety with deployment-realistic onboard sensing for whole-body humanoid dodgeball.
The deployed policy sees the ball only as segmentation-masked depth from a head-mounted camera, while training-time CBF guidance represents clearance to every body link, and an adversarial motion prior regularizes the resulting evasive reflexes.
We evaluate on a controlled any-link contact benchmark with seeded throws in two regimes: single throws and a deployment loop in which the robot walks back to its station and recovers between throws.
On this benchmark, the policy comes within a few points of a privileged state oracle: a fixed onboard camera alone is adequate for evasion.
We find that usable barrier structure depends on perceptual observability: Joint-CBF gives the best performance with accurate ball states, degrades under fixed-camera observations when used only as training guidance, and recovers with a ball-tracking gimbal or privileged runtime filter.

Safety-Critical Perception-aware Dodgeball

We formulate perception-aware dodgeball as a safety-critical whole-body reaction problem under partial observation. The robot must avoid contact with an incoming object while preserving balance, yet its deployed policy has only limited onboard sensing. This information constraint affects both which safety mechanisms we can use and how we measure success.

A ball is launched toward a standing humanoid, which must avoid being struck while preserving balance. We model the task as a partially observed Markov decision process.

Safety-Critical Perception-aware Dodgeball — PAC-MAN: Perception-Aware CBF-RL for Whole-Body Safety in Humanoid Dodgeball — Fig. 3 : Whole-body barrier geometry across representative motion-prior clips; the outlines show the union of per-link keep-out shells as the humanoid crouches, leaps, and sidesteps. The reference set contains 1 minute and 43 seconds of human motion retargeted to the Unitree G1 using [ 35 ] .

Methods

PAC-MAN trains a perception-aware policy to map partial onboard observations to whole-body actions shaped by barrier information available only during training. Privileged ball and link states instantiate the barrier conditions, while the policy receives only the onboard observation $o_{t}$ of Eq. ( 1 ).

We use CBF-RL to inject violations of these conditions into policy optimization [ 11 ] , so the policy internalizes the safety structure during training and needs no privileged geometry at deployment.

Methods — PAC-MAN: Perception-Aware CBF-RL for Whole-Body Safety in Humanoid Dodgeball — Fig. 4 : Emergent whole-body evasion modes in simulation (left) and hardware (right). Crouching, leaning, and sidestepping coordinate the legs, torso, and arms to move threatened links rather than only translating the base. Semantic segmentation allows the same policy to dodge different balls, shown with a soccer ball (top row) and a foam ball (lower two rows).

Results

We first evaluate the PAC-MAN framework in simulation to quantify the impact of perception and safety structure; then we verify the full stack on the physical robot. Figure 4 shows emergent evasion modes from both.

In simulation, we compare the onboard fixed-camera policy against a gimbal-mounted camera and a privileged state oracle while varying the safety structure at each perception level, from no barrier to the full joint-space CBF with its runtime filter.

Results — PAC-MAN: Perception-Aware CBF-RL for Whole-Body Safety in Humanoid Dodgeball — TABLE II : Whole-body success (no link contact or fall) and fall rates (%) by throw regime, perception, and safety-reward mode (defined in § IV-A ) at a matched $20$k-iteration checkpoint. Bold marks the deployed fixed-camera Link-CBF cells. Top: deployment success versus ball observability, one line per safety mode; Joint-CBF improves sharply with observability, Link-CBF remains consistent, and +filter requires runtime ball states.

Sources and Demonstrations

Figures are reproduced from Yang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.