REACT: A Fully Spiking State-Space Model for Real-Time Event-Driven Temporal Perception

Summary

Robotic systems operating in dynamic environments require visual perception that evolves continuously with the incoming sensory stream.

Abstract Summary

Robotic systems operating in dynamic environments require visual perception that evolves continuously with the incoming sensory stream. Event cameras provide microsecond temporal resolution and asynchronous sensing, but most learning-based methods accumulate events into frames or temporal bins, introducing an integration delay that can limit fast reaction. Here we propose REACT, a fully spiking state-space model for event-driven temporal perception that processes raw events one by one, without temporal accumulation. REACT uses a complex-valued spiking neuron, C-SiLIF, whose continuous-time dynamics are driven by the physical inter-event interval, allowing its internal state to evolve at the temporal resolution of individual events. We evaluate REACT on gesture recognition and time-to-collision (TTC) estimation from full-field event streams, without a target bounding box or localization input. On EvTTC, REACT achieves a 9.59% relative TTC error with 4.6 ms end-to-end inference latency, within 0.15 percentage points of the best learned method while requiring no target prior. At the dataset's mean approach speed, this latency corresponds to only 4 cm of vehicle motion, compared with 1 m for the fastest competing learned method. REACT further supports anytime TTC prediction, zero-shot transfer to a different driving sequence, and INT8 quantization, reducing the estimated energy consumption from 18.5 to 2.8 mJ per 32,768 events. These results show that event-driven spiking state-space dynamics can provide low-latency, continuously updated temporal perception for reactive robotic systems.

Key Points

Robotic systems operating in dynamic environments require visual perception that evolves continuously with the incoming sensory stream.
Event cameras provide microsecond temporal resolution and asynchronous sensing, but most learning-based methods accumulate events into frames or temporal bins, introducing an integration delay that can limit fast reaction.
Here we propose REACT, a fully spiking state-space model for event-driven temporal perception that processes raw events one by one, without temporal accumulation.
REACT uses a complex-valued spiking neuron, C-SiLIF, whose continuous-time dynamics are driven by the physical inter-event interval, allowing its internal state to evolve at the temporal resolution of individual events.
We evaluate REACT on gesture recognition and time-to-collision (TTC) estimation from full-field event streams, without a target bounding box or localization input.

Methods

An event camera outputs an asynchronous stream of $L$ tuples $\mathcal{E}=\{(t_{k},x_{k},y_{k},p_{k})\}_{k=1}^{L}$, where $t_{k}$ is the timestamp, $(x_{k},y_{k})$ the pixel coordinate on an $H\times W$ sensor, and $p_{k}\in\{0,1\}$ the polarity of the brightness change. Most learning-based pipelines first accumulate this stream into frames, voxel grids, or fixed time bins.

REACT consumes it directly: no event representation is constructed at any stage, and the model state is updated once per event.

Methods — REACT: A Fully Spiking State-Space Model for Real-Time Event-Driven Temporal Perception — Fig. 2: Qualitative TTC predictions over complete sequences. ( A ) In-domain evaluation on the CCR1 split of the medium-overlap-50 set of EvTTC [ 28 ] . ( B ) Zero-shot evaluation on Sequence 1 of FCWD [ 20 ] . For each sequence, four event frames are shown at decreasing time to contact. The values above each frame indicate the ground-truth TTCs, while the colored labels in the lower-left corners show the corresponding REACT predictions (in seconds). Predictions progressively converge toward the ground truth as the time to contact decreases, with reliable estimates obtained within approximately 3 s of impact (see the critical zone in orange). REACT exhibits similar temporal prediction behavior on the unseen FCWD sequence under the domain shift, without fine-tuning. In this cases, the average vehicle speed is 9.22 m/s.

Experiments

We first evaluate the architecture on a standard neuromorphic classification benchmark, before turning to the continuous temporal prediction task of Sec. IV-B . DVS128-Gesture is the benchmark on which Event-SSM and its compute-in-memory counterpart both report results, so it measures whether a fully spiking SSM reaches the accuracy of comparable event-by-event models at a much smaller cost.

Sources and Demonstrations

Figures are reproduced from Keime et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.