GS-Agent: Creating 4D Physical Worlds With Generative Simulation

Hongxin Zhang, Chunru Lin, Junyan Li, Zhou Xian, Tsun-Hsuan Wang, Chuang Gan · N/A · 2026

Summary

Abstract Summary

Creating dynamic and physically realistic 4D worlds from natural language descriptions is both fascinating and challenging. Traditional computer graphics methods rely on manual creation, requiring extensive human effort to fine-tune materials, motions, and visual fidelity. Recent advances in generative foundation models have sparked interest in learning to generate such 4D worlds from large-scale data; however, existing methods still struggle to ensure physical plausibility and controllability. In this work, we take a different path by leveraging foundation models to construct an agentic system that emulates how humans traditionally create 4D worlds, yet automates the entire process. We present GS-Agent, an end-to-end multi-agent framework that integrates physics engines in the loop to generate realistic, dynamic, and controllable 4D physical worlds from natural language. Inspired by how humans build 4D worlds, GS-Agent decomposes the task into entity management, covering 3D asset curation, material tuning, placement, and motion control, and rendering configuration, including camera and lighting manipulation. Multiple agents with distinct expertise interact with the physics engine via code, seek multimodal feedback, and collaborate to iteratively construct 4D worlds that align with the given descriptions. Experimental results show that GS-Agent effectively converts natural language into diverse and physically plausible 4D worlds exhibiting rich interactions among liquids, deformable objects, and rigid bodies, while achieving cinematic camera and lighting control. We envision GS-Agent as a foundation for a new paradigm in 4D world generation, empowering creative content creation and physical AI. Project page at https://umass-embodied-agi.github.io/gs-agent/

Key Points

Creating dynamic and physically realistic 4D worlds from natural language descriptions is both fascinating and challenging.
Traditional computer graphics methods rely on manual creation, requiring extensive human effort to fine-tune materials, motions, and visual fidelity.
Recent advances in generative foundation models have sparked interest in learning to generate such 4D worlds from large-scale data; however, existing methods still struggle to ensure physical plausibility and controllability.
In this work, we take a different path by leveraging foundation models to construct an agentic system that emulates how humans traditionally create 4D worlds, yet automates the entire process.
We present GS-Agent, an end-to-end multi-agent framework that integrates physics engines in the loop to generate realistic, dynamic, and controllable 4D physical worlds from natural language.

GS-Agent: A Multi-Agent Framework with Generative Simulation in the Loop

Creating a 4D world is a complex, multi-stage process that typically requires coordinated effort across different areas of expertise: entity modeling, material design, spatial arrangement, motion specification, lighting setup, and camera planning.

In human production pipelines, such as visual effects or game development, a director translates high-level requirements into actionable plans, delegates tasks to specialists, evaluates intermediate outputs, and coordinates iterative refinement through natural language communication.

GS-Agent: A Multi-Agent Framework with Generative Simulation in the Loop — GS-Agent: Creating 4D Physical Worlds With Generative Simulation — Figure 3 : Qualitative comparisons between GS-Agent and the baseline methods. More cases are presented in the Appendix C .

Experiments

To systematically evaluate GS-Agent’s ability to construct 4D physical worlds, we conduct experiments across two evaluation suites. The first comprises 24 scenes from the NewtonGen benchmark Yuan et al. (2025) , specifically designed to test 12 distinct physical laws (e.g., parabolic motion, deformation).

To assess the novel capabilities unlocked by our method, we collected a second suite of 30 complex scenes featuring non-continuous dynamics in multi-object interactions (e.g., collisions) and dynamic camera controls. Metrics.

Experiments — GS-Agent: Creating 4D Physical Worlds With Generative Simulation — Figure 5 : Fine-grained controllability. Users can interact with GS-Agent via natural language to adjust rendering effects ( e.g., slow motion ), create entities ( e.g., add more billiard balls ), and control entity motions ( e.g., increase velocity for a break ).

Sources and Demonstrations

Figures are reproduced from Zhang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.