Robotics Research Hub
Demo: Vision-Language Model-Guided Online Calibration of an Electromagnetic Digital Twin

Demo: Vision-Language Model-Guided Online Calibration of an Electromagnetic Digital Twin

Zerui Kang, Yishen Lim, Zhouyou Gu, Seungnyun Kim, Seung-Woo Ko, Tony Q. S. Quek, Jihong Park · N/A · 2026

vision vla planning
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

An electromagnetic (EM) digital twin gives mobile robots wireless situational awareness but depends on material conductivities that change with the environment. Online calibration faces initialization sensitivity and measurement travel costs.

Abstract Summary

An electromagnetic (EM) digital twin gives mobile robots wireless situational awareness but depends on material conductivities that change with the environment. Online calibration faces initialization sensitivity and measurement travel costs. We demonstrate a vision-language model (VLM)-guided framework using a Unitree G1 robot and NVIDIA Sionna, with two VLM calls: material classification maps visible materials through ITU-R P.2040 to conductivity priors for Sionna's gradient descent on accumulated received signal strength (RSS) measurements; waypoint planning selects the next measurement location online using residual RSS calibration error and image coverage. In a real indoor scenario, the framework achieves a normalized mean absolute conductivity error of $1.74\times10^{-4}$ within 20 m of travel; random initialization never converges, while random waypoints require over twice the travel.

Key Points

  • An electromagnetic (EM) digital twin gives mobile robots wireless situational awareness but depends on material conductivities that change with the environment.
  • Online calibration faces initialization sensitivity and measurement travel costs
  • We demonstrate a vision-language model (VLM)-guided framework using a Unitree G1 robot and NVIDIA Sionna, with two VLM calls: material classification maps visible materials through ITU-R P.2040 to conductivity priors for Sionna's gradient descent on accumulated received signal strength (RSS) measurements; waypoint planning selects the next measurement location online using residual RSS calibration error and image coverage.
  • In a real indoor scenario, the framework achieves a normalized mean absolute conductivity error of $1.74\times10^{-4}$ within 20 m of travel; random initialization never converges, while random waypoints require over twice the travel.

System Design

We demonstrate the framework in the indoor scenario of Fig. 1 (a) with a fixed 5G NR base station and a Unitree G1 humanoid robot carrying a 5G dongle and an RGB camera. A desktop with an NVIDIA RTX Pro 6000 GPU hosts the EM digital twin in Sionna and GPT-5.4 as the VLM.

The dongle records RSS measurements from the fixed transmitter, while the Sionna configuration uses the same carrier frequency and antenna locations as the physical setup. The following three stages run once per round at each waypoint.

System Design — Demo: Vision-Language Model-Guided Online Calibration of an Electromagnetic Digital Twin
(a) NMAE vs. traveled distance.

Observations

We ablate each VLM call with a random baseline that draws initial conductivities at random or picks waypoints uniformly in the room. Fig. 2 (a) plots the normalized mean absolute error (NMAE) of the estimated conductivity against traveled distance.

For evaluation, the ground-truth conductivity of each material is predefined and used by Sionna to generate the radio map, while material parameters remain unknown to the digital twin throughout the calibration. Initialization decides convergence.

Observations — Demo: Vision-Language Model-Guided Online Calibration of an Electromagnetic Digital Twin
(b) Calibrated radio maps.

Demonstration

The demonstration runs the full pipeline in a photorealistic NVIDIA Isaac Sim replica of our meeting room, reproducing its layout, object placement, robot viewpoints, and measurement locations; real-world sensing noise and radio-hardware effects require separate validation. The EM digital twin and VLM run remotely in our laboratory.

A live dashboard displays each round’s material labels, robot movement to the selected waypoint, and calibrated radio map alongside the ground truth.

Sources and Demonstrations

Figures are reproduced from Kang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers