SocialVLA: A Social Perception Gateway for Human-Reaction-Based Failure Detection and Recovery in VLA Manipulation

Sofya Konstantinova, Miguel Altamirano Cabrera, Artem Lykov, Dzmitry Tsetserukou · N/A · 2026

Summary

Abstract Summary

Vision-language-action (VLA) policies enable diverse robotic manipulation but can fail during execution without recognizing their own errors. Human observers provide complementary signals, as unexpected robot behavior can trigger rapid vocal, facial, or verbal reactions before failure is completed. We introduce SocialVLA, a local, policy-agnostic social perception gateway that converts spontaneous human reactions into runtime intervention signals for VLA manipulation. SocialVLA combines causal paralinguistic audio detection, visual reaction recognition, explicit stop phrases, and robot-relevance estimation. An asynchronous first-event fusion mechanism triggers a VLA hold from the earliest sufficiently confident signal, while a separate speech channel captures verbal corrections for participant-directed continuation, restart, or instruction revision. We evaluate SocialVLA on physical Unitree G1 manipulation using 15 participants, with 238 annotated intervention-worthy episodes and 1.038 h of non-intervention behavior. Frozen offline replay achieves 54.6% recall and 69.5% precision, while unfiltered audio-video fusion reaches 64.3% recall. Relevance estimation reduces false-stop episodes from 100 to 57 and increases precision from 60.5% to 69.8%. In prospective deployment on an unseen 16th participant, the frozen system achieves 59.5% recall and 91.7% precision. Median detector-to-fusion latency is 47.9 ms, VLA-gate-to-physical-hold latency is 336 ms, and reaction-onset-to-hold latency is 1.021 s. These results demonstrate a complete local pathway from spontaneous social reaction to physical VLA interruption and participant-directed recovery.

Key Points

Vision-language-action (VLA) policies enable diverse robotic manipulation but can fail during execution without recognizing their own errors.
Human observers provide complementary signals, as unexpected robot behavior can trigger rapid vocal, facial, or verbal reactions before failure is completed.
We introduce SocialVLA, a local, policy-agnostic social perception gateway that converts spontaneous human reactions into runtime intervention signals for VLA manipulation.
SocialVLA combines causal paralinguistic audio detection, visual reaction recognition, explicit stop phrases, and robot-relevance estimation.
An asynchronous first-event fusion mechanism triggers a VLA hold from the earliest sufficiently confident signal, while a separate speech channel captures verbal corrections for participant-directed continuation, restart, or instruction revision.

SYSTEM ARCHITECTURE

Let $x_{a}(t)$ and $x_{v}(t)$ denote participant-facing audio and video streams. A detector produces: where $m_{i}$ is the modality, $c_{i}$ the event class, $s_{i}$ the score, $[t_{i}^{0},t_{i}^{1}]$ the source interval supporting the decision, and $t_{i}^{e}$ the publication time. The system distinguishes fast reaction events, extended-context reaction events, and direct lexical events.

Attention-target estimates are diagnostic; only the binary robot-relevance probability enters the candidate fusion rule. Audio is recorded at 16 kHz in mono.

SYSTEM ARCHITECTURE — SocialVLA: A Social Perception Gateway for Human-Reaction-Based Failure Detection and Recovery in VLA Manipulation — Fig. 2: SocialVLA architecture. Independent audio, lexical, and visual reaction channels produce asynchronous evidence. The selected soft fusion rule combines reaction confidence with bounded robot-relevance context. The first authorized event requests a VLA hold, while subsequent evidence is attached to the open reaction episode. A post-stop speech log records the participant’s correction without affecting the robot hold state.

EXPERIMENTAL METHODOLOGY

Experiments use a physical Unitree G1 humanoid controlled by GR00T-WholeBodyControl and a GR00T policy trained for a pick-and-place task. The robot must grasp a ball and place it in a specified target region. Participants observed both successful executions and naturally occurring VLA failures. Failure occurrence was not announced, and participants were not instructed to produce specific reactions.

The prediction target in this study is the observer’s intervention-worthy response rather than a predefined taxonomy of physical failure modes.

EVALUATION RESULTS

Table I reports the complete frozen multimodal cascade replayed over all annotated sessions. The evaluation contains 238 manually annotated robot-related intervention episodes and 1.038 h of explicitly annotated non-intervention or robot-unrelated behavior. All audio, video, relevance, lexical, and temporal-fusion components use the deployed weights, thresholds, and 700-ms cross-modal association window.

The complete cascade detects 130 of 238 intervention episodes, corresponding to 54.6% event recall and 69.5% precision.

Sources and Demonstrations

Figures are reproduced from Konstantinova et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.