A Deployment Study of Identity-Gated Drone Gesture Control

Summary

Vision-based gesture control accepts commands from any hand in the camera field of view, which is unsafe in shared indoor spaces.

Abstract Summary

Vision-based gesture control accepts commands from any hand in the camera field of view, which is unsafe in shared indoor spaces. This paper presents IGate, an identity-gated control stack that includes gesture control and face tracking, in which commands are admitted only when an enrolled operator is verified. The system performs few-shot user enrolment from 20 initial face frames, without prior user-specific training: verification compares an embedding of the current face crop against the enrolled template by cosine similarity, while face tracking uses proportional correction. Gesture control is achieved by classifying extracted hand landmarks using an RBF-SVM trained on a custom dataset. Additionally, a hierarchical finite-state machine handles mode selection, default, and fallback behaviours. The approach is tested on a DJI Tello EDU, each component evaluated offline and in-flight across 270 trials (149 flown). Face verification yields a 0.32% offline equal error rate versus 19.3% in-flight. Under hover-locked conditions, the RBF-SVM gesture classifier outperforms the geometric rule (0.850 vs. 0.651 accuracy), with 82% of this gap stemming from the depth channel. All logs and reproduction scripts will be released.

Key Points

Vision-based gesture control accepts commands from any hand in the camera field of view, which is unsafe in shared indoor spaces.
This paper presents IGate, an identity-gated control stack that includes gesture control and face tracking, in which commands are admitted only when an enrolled operator is verified.
The system performs few-shot user enrolment from 20 initial face frames, without prior user-specific training: verification compares an embedding of the current face crop against the enrolled template by cosine similarity, while face tracking uses proportional correction.
Gesture control is achieved by classifying extracted hand landmarks using an RBF-SVM trained on a custom dataset.
Additionally, a hierarchical finite-state machine handles mode selection, default, and fallback behaviours.

Method

DJI Tello EDU camera IMU, barometer motors aircraft host latest-frame buffer thread, depth 1 MediaPipe Hands every 2nd frame MediaPipe face every 4th frame gesture classifier rule / RBF-SVM identity gate $\tau_{\mathrm{on}}{=}0.55$ , $\Delta_{\max}{=}0.35$ s arbitration priority order RC command 10 Hz reason-only LLM local Qwen2.5-0.5B video UDP 11111 telemetry UDP 8890 command UDP 8889 Fig. 1: IGate .

The aircraft carries no computation: perception, authorization and arbitration run on the host, and every stage is logged per frame.

Method — A Deployment Study of Identity-Gated Drone Gesture Control — Fig. 2: The 21-point hand skeleton IGate classifies (left) and the runtime overlay the operator sees (right). The numbered joints are the indices used throughout: 0 the wrist, 4 the thumb tip, 5 and 8 the index metacarpophalangeal joint and tip that carry the rule’s directional vector, and 20 the little-finger tip that separates LEFT from BACK (Section V-D ).

Experimental Setup

Each launch writes a timestamped run directory containing telemetry, decision, per-frame performance and trial logs, with a manifest recording configuration, Git commit, library versions and host. Two conventions make the latency figures meaningful: a throttled stage contributes a sample only on frames where it executed, and frames are deduplicated by sequence number before any rate is computed.

Results

The classifier is scored three ways, each stricter than the last: a within-session split; a held-out session split, trained on three sessions and tested on the fourth; and an operational measurement on frames the aircraft itself encoded. The metric is per-frame accuracy over the seven classes.

Evaluated within each session under a stratified random split, the SVM reaches an accuracy of 0.997, 0.988, 1.000 and 1.000 on sessions A, B, C and D.

Sources and Demonstrations

Figures are reproduced from Salih et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.