GS-Agent: Creating 4D Physical Worlds With Generative Simulation

Hongxin Zhang, Chunru Lin, Junyan Li, Zhou Xian, Tsun-Hsuan Wang, Chuang Gan · N/A · 2026

Summary

Abstract Summary

Creating dynamic and physically realistic 4D worlds from natural language descriptions is both fascinating and challenging. Traditional computer graphics methods rely on manual creation, requiring extensive human effort to fine-tune materials, motions, and visual fidelity. Recent advances in generative foundation models have sparked interest in learning to generate such 4D worlds from large-scale data; however, existing methods still struggle to ensure physical plausibility and controllability. In this work, we take a different path by leveraging foundation models to construct an agentic system that emulates how humans traditionally create 4D worlds, yet automates the entire process. We present GS-Agent, an end-to-end multi-agent framework that integrates physics engines in the loop to generate realistic, dynamic, and controllable 4D physical worlds from natural language. Inspired by how humans build 4D worlds, GS-Agent decomposes the task into entity management, covering 3D asset curation, material tuning, placement, and motion control, and rendering configuration, including camera and lighting manipulation. Multiple agents with distinct expertise interact with the physics engine via code, seek multimodal feedback, and collaborate to iteratively construct 4D worlds that align with the given descriptions. Experimental results show that GS-Agent effectively converts natural language into diverse and physically plausible 4D worlds exhibiting rich interactions among liquids, deformable objects, and rigid bodies, while achieving cinematic camera and lighting control. We envision GS-Agent as a foundation for a new paradigm in 4D world generation, empowering creative content creation and physical AI. Project page at https://umass-embodied-agi.github.io/gs-agent/

Key Points

Creating dynamic and physically realistic 4D worlds from natural language descriptions is both fa...
Traditional computer graphics methods rely on manual creation, requiring extensive human effort t...
Recent advances in generative foundation models have sparked interest in learning to generate suc...
In this work, we take a different path by leveraging foundation models to construct an agentic sy...
We present GS-Agent, an end-to-end multi-agent framework that integrates physics engines in the l...

GS-Agent: Creating 4D Physical Worlds With Generative Simulation

|Authors: Hongxin Zhang, Chunru Lin, Junyan Li, Zhou Xian, Tsun-Hsuan Wang, Chuang Gan

|Venue: arXiv preprint | Year: 2026

|arXiv: 2607.21522v1

Abstract

Key Contributions

Creating dynamic and physically realistic 4D worlds from natural language descriptions is both fa…
Traditional computer graphics methods rely on manual creation, requiring extensive human effort t…
Recent advances in generative foundation models have sparked interest in learning to generate suc…
In this work, we take a different path by leveraging foundation models to construct an agentic sy…
We present GS-Agent, an end-to-end multi-agent framework that integrates physics engines in the l…

Topics

manipulation
vision
reinforcement-learning
control

Code & Data

GitHub repository: https://github.com/nerfies/nerfies.github.io

BibTeX

@article{Zhang2026_260721522v1,
  title     = {GS-Agent: Creating 4D Physical Worlds With Generative Simulation},
  author    = {Hongxin Zhang and Chunru Lin and Junyan Li and Zhou Xian and Tsun-Hsuan Wang and Chuang Gan},
  year      = {2026},
  eprint    = {2607.21522v1},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url       = {https://arxiv.org/abs/2607.21522v1}
}