Training-Free Multi-Agent Cooperative Embodied Search from Uncertain Vision-Language Priors

Fan Pu, Keshu Wu, Jiwan Jiang, Shiyu Shen, Zhiwen Fan, Zhengzhong Tu, Jiachen Li, Yang Zhou · N/A · 2026

Summary

Abstract Summary

Real-world Vision-Language Navigation (VLN) requires agents to act under partial observability, where map structure, semantic cues, and target locations may be incomplete or incorrectly inferred by foundation models. This paper presents a lightweight, training-free, and uncertainty-aware framework for cooperative multi-agent multi-target search in mapless environments. Instead of learning an end-to-end navigation policy, the framework separates high-level semantic reasoning from low-level execution through a scalable graph-optimization backbone. Multimodal Large Language Models (MLLMs) serve only as hypothesis generators, proposing semantic zones, target-location cues, and possible navigable connections. These outputs are treated as uncertain priors and further refined using online observations and graph-context evidence. Cooperative navigation is then formulated as a rolling-horizon multi-agent mixed-integer linear program with a lexicographic objective that first maximizes target-directed reward and then reduces travel distance, uncertain-edge use, and repeated visits. Experiments derived from 100 Matterport3D test cases show that the proposed framework improves target coverage and search efficiency over representative baseline methods, while shared information and joint planning enable more balanced workloads across agents. By operating on an abstract graph without relying on embodiment-specific policy parameters, the framework scales across varying numbers of agents and targets, generalizes to unseen environments, and accommodates different perception modules and low-level controllers without task-specific retraining.

Key Points

Real-world Vision-Language Navigation (VLN) requires agents to act under partial observability, where map structure, semantic cues, and target locations may be incomplete or incorrectly inferred by foundation models.
This paper presents a lightweight, training-free, and uncertainty-aware framework for cooperative multi-agent multi-target search in mapless environments.
Instead of learning an end-to-end navigation policy, the framework separates high-level semantic reasoning from low-level execution through a scalable graph-optimization backbone.
Multimodal Large Language Models (MLLMs) serve only as hypothesis generators, proposing semantic zones, target-location cues, and possible navigable connections.
These outputs are treated as uncertain priors and further refined using online observations and graph-context evidence.

Problem Formulation

We consider cooperative embodied search in partially observed, mapless environments, where a team of robots $\mathbf{Q}$ must locate multiple targets $\mathbf{K}$ without prior knowledge of the environment topology, semantic regions, traversable connections, or target locations. As illustrated in Figure 1 , the proposed framework follows a closed-loop hypothesize–verify–optimize–execute process.

Fig. 1: Overall architecture of the proposed framework.

Experiments

We evaluate the proposed framework and comparison methods in the Matterport3D Simulator [ 6 ] . The evaluation consists of 100 test cases drawn from five indoor scans in the Matterport3D dataset [ 55 ] . For each scan, target objects are manually selected and annotated with a textual description and the indices of viewpoints whose panoramas contain the target.

These visibility annotations are used exclusively by the offline Oracle and for performance evaluation.

Experiments — Training-Free Multi-Agent Cooperative Embodied Search from Uncertain Vision-Language Priors — Fig. 2: Panoramic observation at an agent’s current viewpoint. The red numbers denote the IDs of neighboring viewpoints returned by the navigation system and constitute the agent’s feasible one-hop action space.

Sources and Demonstrations

Figures are reproduced from Pu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.