STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration

Maitrey Gramopadhye, Prakash Baskaran, Xiao Liu, Songpo Li, Soshi Iba · N/A · 2026

Summary

Effective human-robot collaboration in industrial settings requires robots to understand human intentions and assist with task planning, reducing workload.

Abstract Summary

Effective human-robot collaboration in industrial settings requires robots to understand human intentions and assist with task planning, reducing workload. Recent works have explored the use of Multi-modal Large Language Models (MM-LLMs) for task planning in such data-scarce scenarios, leveraging in-context learning to interpret user actions and generate long-horizon action plans in natural language. However, MM-LLMs inherently lack an understanding of system states and do not track state transitions, often leading to hallucinated actions that deviate from the intended goal. Additionally, generating action plans in natural language tends to limit the generated plans to a high level, introducing ambiguity in action execution. To address these limitations, we propose the State-aware Task Estimator and Planner (STEP), which prompts a MM-LLM to explicitly estimate the state of the system and predict the state transitions resulting from executed actions. By forecasting future states alongside actions, STEP ensures task-convergent planning while also providing additional assistance parameters necessary for executing the predicted actions. We evaluate STEP in a simulated environment using a robot assembly task. Our approach outperforms the state-of-the-art by 32.8% in action executability and 14.8% in final-state error.

Key Points

Effective human-robot collaboration in industrial settings requires robots to understand human intentions and assist with task planning, reducing workload.
Recent works have explored the use of Multi-modal Large Language Models (MM-LLMs) for task planning in such data-scarce scenarios, leveraging in-context learning to interpret user actions and generate long-horizon action plans in natural language.
However, MM-LLMs inherently lack an understanding of system states and do not track state transitions, often leading to hallucinated actions that deviate from the intended goal.
Additionally, generating action plans in natural language tends to limit the generated plans to a high level, introducing ambiguity in action execution.
To address these limitations, we propose the State-aware Task Estimator and Planner (STEP), which prompts a MM-LLM to explicitly estimate the state of the system and predict the state transitions resulting from executed actions.

Approach

STEP takes as input the actions performed by an operator and an image of the workstation, along with a set of example interactions to prompt an MM-LLM. As illustrated in Fig. 2 , our pipeline predicts the overall task the operator is trying to accomplish. Simultaneously, it also estimates a structured state representation of the current workspace from its image.

It then uses the estimated task and action history to predict future actions required to complete the task.

Approach — STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration — Fig. 3 : For each of the blocks our structured state representation records the block location (table/another block with its number), block orientation (stand/lie on face/lie on side/NA), top face orientation (long-side parallel/perpendicular to camera/NA), specific location (left/right/center if the block is on another block, otherwise NA). The modular state design also allows extension to other scenarios by adding descriptors - ‘nearby objects’, ‘facing’, ‘containing objects’ etc.

Evaluation

We compared our method with a state-of-the-art LTA baseline inspired by the top-down approach proposed by AntGPT [ 9 ] . The baseline consisted of using the same inputs as our method, for task estimation (§ III-A ) followed by a single rollout of action prediction (§ III-C ), with no state information.

We selected this comparison to directly examine the impact of explicitly estimating and tracking the system state on action prediction accuracy.

Evaluation — STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration — Fig. 4 : The eight block assembly tasks.

Sources and Demonstrations

Figures are reproduced from Gramopadhye et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.