ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception

Ruoxuan Feng, Yutong Chen, Ruihua Song, Huan Yang, Zhongyuan Wang, Guocai Yao, Di Hu · N/A · 2026

Summary

Abstract Summary

Humans inherently understand the physical world through an active process. When sensory evidence is insufficient to infer physical properties, we naturally interact with the environment by deciding what information is missing, how to acquire it, and when sufficient evidence has been obtained. In stark contrast, existing multi-sensory robot systems mainly integrate sensory inputs rather than actively acquiring missing evidence through interactions. In this work, we introduce ROMA, an LLM-based system for Real-World Object-Centric Multi-Sensory Active Perception. ROMA integrates vision, audio, tactile, and force sensing into a reasoning-interaction-feedback loop. The model identifies missing evidence and determines the target objects, interactions, and modalities, while a physical interface executes the selected interactions and collects the multi-sensory feedback. To support this capability, we construct ROMI-2K, a large-scale real-world multi-sensory object interaction dataset covering nearly 2,000 objects and 6 atomic interactions with synchronized sensory feedback. Building on these data, we develop a two-stage training framework that aligns sensory modalities and equips the LLM to assess evidence sufficiency, select informative interactions, and reason over the multi-sensory feedback. We further characterize active perception as perception chains, where acquired evidence guides subsequent interactions and reasoning, and establish ROMA Bench to evaluate single-attribute, long-horizon multi-attribute, and intent-driven active perception. Experiments show that ROMA can actively acquire missing evidence and solve complex, long-chain multi-sensory perception tasks that existing methods struggle to handle, laying a strong perceptual foundation for active multi-sensory embodied agents.

Key Points

Humans inherently understand the physical world through an active process
When sensory evidence is insufficient to infer physical properties, we naturally interact with th...
In stark contrast, existing multi-sensory robot systems mainly integrate sensory inputs rather th...
In this work, we introduce ROMA, an LLM-based system for Real-World Object-Centric Multi-Sensory ...
ROMA integrates vision, audio, tactile, and force sensing into a reasoning-interaction-feedback loop

ROMI-2K: Object-Centric Multi-Sensory Interaction Dataset

Existing multi-sensory interaction datasets face a fundamental trade-off between object diversity and interaction diversity . Interaction-centric datasets collect diverse trajectories over a limited set of objects ( e.g. , 381 objects in Hoi! [ 39 ] ). In contrast, object-centric datasets cover numerous objects with simple interactions like hitting [ 28 ] .

Consequently, existing resources provide limited coverage of how diverse objects respond to different physical interactions and their multi-sensory feedback.

ROMI-2K: Object-Centric Multi-Sensory Interaction Dataset — ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception — Figure 2 : Overview of ROMI-2K and the formulation of multi-sensory active perception tasks . ROMI-2K combines two complementary real-world data subsets: a large-scale handheld object collection for diverse single-object interactions and a robotic-arm tabletop collection for multi-object scenes, with synchronized visual, audio, tactile, and force feedback. We organize the active perception tasks into perception chains and construct three task types: (1) Single-Chain tasks for an individual attribute, (2) Multi-Chain tasks for multiple attributes requiring interaction planning, and (3) Intent-Driven tasks for inferring and perceiving target attributes from implicit user instructions. Multiple perception chains may also involve reusing feedback across chains.

REAL-WORLD Object Multi-Sensory Active Perception System

Building on the rich object and scene interaction data provided by ROMI-2K, we introduce ROMA , an LLM-Based System for Real-World Object-Centric Multi-Sensory Active Perception, as shown in Fig. 3 . We describe its physical interface for executing interactions and collecting multi-sensory feedback (Sec. 4.1 ) and its multi-sensory LLM for active perception (Sec. 4.2 ) in this section.

Experiments

In this section, we first introduce ROMA Bench and the evaluation protocol for multi-sensory active perception (Sec. 5.1 ). We then evaluate ROMA against existing MLLMs on both the offline benchmark and real-world tabletop scenes (Sec. 5.2 ), and analyze how different methods allocate interaction effort for evidence acquisition (Sec. 5.3 ). More details are shown in Appendix 14 .

We construct ROMA Bench from the test split of the tabletop subset in ROMI-2K, comprising 2,100 scene-level active perception tasks.

Experiments — ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception — Figure 4 : Multi-Sensory Alignment and dynamic scene sampling for training the Multi-Sensory LLM . We illustrate the alignment process using tactile-visual-text alignment as an example. We align tactile representations with visual and textual semantics, enabling the model to better leverage complementary cues across modalities. For dynamic scene sampling, we use Beta-distribution-based sampling to smoothly shift the training from interaction-level supervision toward real-world scene training.

Handheld Data Collection

Before collecting structured tabletop scenes with the robotic arm, we first build a handheld data collection platform to efficiently acquire large-scale object-interaction trajectories. The handheld platform allows us to rapidly explore a diverse set of everyday objects under different grasping and interaction configurations.

These data provide broad coverage of object-level physical interactions and serve as an important component of ROMI-2K for training ROMA’s active perception capabilities.

Handheld Data Collection — ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception — Figure 5 : A case study of MLLM baselines and ROMA-7B on active perception tasks in ROMA Bench. GPT-5.4 tends to terminate with fewer active interactions, occasionally skipping critical objects and sensory evidence. Both GPT-5.4 and Gemini 3.5 Flash also produce inappropriate action-modality combinations, such as attempting to observe gravity force during vigorous shaking. In contrast, ROMA-7B more often continues interaction when the available evidence is insufficient and selects complementary actions and modalities to acquire task-relevant information.

Tabletop Scene Collection and Robotic System Details

In this section, we describe the robotic system developed for ROMA. The system is designed for large-scale, multi-sensory data acquisition through physical object interaction.

Given the name of an object and a corresponding visual prompt, the system autonomously detects the target from multi-view RGB-D observations, estimates its 3D geometry, plans and executes a feasible 6-DoF grasp, and performs a sequence of pre-defined physical interactions while synchronously recording visual, audio, tactile, and force feedback.

Tabletop Scene Collection and Robotic System Details — ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception — Figure 6 : The illustration of the handheld object interaction data collection platform.

Dataset Statistics

In this section, we first provide a detailed statistical analysis of tabletop scene subset (including both the training set and ROMA Bench) of ROMI-2K. We provide statistics on the object materials, the presence of contents within objects, and the task statistics of ROMA Bench for the training set and ROMA Bench of the tabletop scene subset, as shown in Fig. 16 , 17 , and 18 , respectively.

Implementation Details

We initialize the ROMA-7B LLM from the weights of Qwen 2.5-Omni 7B [ 42 ] and incorporate the 2-frame version of the AnyTouch 2 [ 48 ] tactile encoder. For stage 1 alignment, we construct modality-specific QA pairs from publicly available datasets and combine them with the handheld object subset of ROMI-2K. For audio alignment, we use AudioCaps [ 49 ] and AudioSet [ 50 ] .

Specifically, we use AudioCaps and select AudioSet samples that contain object-related sounds to construct audio QA pairs.

Implementation Details — ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception — Figure 8 : An example of a interaction-conditioned QA sample constructed using the handheld object subset of ROMI-2K.

Interaction and Ablation Study

In this section, we first analyze the interaction patterns of different models on completed active perception tasks, followed by component-level ablations and missing-modality experiments.

To analyze how different models explore the physical world and select interactions to acquire additional sensory evidence, we compare the relative distributions of the six interaction primitives performed by GPT-5.4, Gemini 3.5 Flash, and ROMA-7B on ROMA Bench, as shown in Fig. 23 . The results reveal clear differences in their interaction preferences.

Interaction and Ablation Study — ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception — Figure 9 : An example of a single-chain active perception QA sample constructed using the handheld object subset of ROMI-2K.

Sources and Demonstrations

Figures are reproduced from Feng et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.