TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns

Guangxin Zhao, Yiran Hu, Yuan Cao, Chenxi Jiang, Jianfei Yang, Yegang Du, Yasuyuki Taki, Yoshifumi Kitamura, Lin Gu, Zhi Zheng · N/A · 2026

Summary

Existing LLM-driven robot task planners rely on a taken-for-granted assumption of an ideal user whose instructions are clear, complete, and task-focused.

Abstract Summary

Existing LLM-driven robot task planners rely on a taken-for-granted assumption of an ideal user whose instructions are clear, complete, and task-focused. However, when interacting with real-world users, especially those experiencing cognitive impairments, such as people living with dementia (PLWD), the planners often make mistakes and even pose physical safety risks. We proposed TALK-Dem (Talking Attributes and Linguistic Knowledge in Dementia), the first benchmark for evaluating LLM-driven robot task planning under dementia-associated verbal communication. TALK-Dem contains 4,800 instructions and covers five typical communication patterns, including Referential Imprecision, Object Substitution, Empty Speech, Topic Drift, and Intrusion, at three intensity levels. Experiments across six open-weight LLMs reveal a substantial robustness gap. Across communication patterns, open-weight models exhibited performance drops of up to 22.3 percentage points compared to ideal instructions. This revealed a critical gap and even danger for real-world applications, especially in assistive robotics, where locally deployable models are necessary due to privacy concerns and connectivity constraints. To mitigate this issue, we proposed the Context-Aware Retrieval from Experience (CARE) method, which retrieves relevant previously resolved tasks to provide task-specific interpretation and planning context. CARE generally outperformed standard prompting baselines across the six open-weight models, improving average task success by 18.1 percentage points over the vanilla prompt. These results highlighted the importance of both evaluating communication robustness and developing effective adaptation strategies for locally deployable assistive robots. The TALK-Dem dataset is publicly available at https://anonymous.4open.science/r/TALK-Dem-A6B3/.

Key Points

Existing LLM-driven robot task planners rely on a taken-for-granted assumption of an ideal user whose instructions are clear, complete, and task-focused.
However, when interacting with real-world users, especially those experiencing cognitive impairments, such as people living with dementia (PLWD), the planners often make mistakes and even pose physical safety risks.
We proposed TALK-Dem (Talking Attributes and Linguistic Knowledge in Dementia), the first benchmark for evaluating LLM-driven robot task planning under dementia-associated verbal communication.
TALK-Dem contains 4,800 instructions and covers five typical communication patterns, including Referential Imprecision, Object Substitution, Empty Speech, Topic Drift, and Intrusion, at three intensity levels.
Experiments across six open-weight LLMs reveal a substantial robustness gap

TALK-Dem Benchmark

PLWD may express the same task intent in ways that differ from a direct and concise instruction. The altered communication patterns can make the intended task more difficult for an embodied task planner to infer, leading to planning failures.

To study this effect, we modeled the process of the transformation from a clean instruction to an instruction that preserves the task intent while exhibiting a dementia-associated communication pattern. Let $x$ denote the original instruction for task $\tau$.

TALK-Dem Benchmark — TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns — Figure 2: Examples of TALK-Dem transformations for a single seed instruction. Each row shows one dementia-associated communication pattern, and columns correspond to increasing pattern-specific intensity from Level 1 to Level 3.

Experiment Setup and Measurements

We evaluated models using the SayCan task planner in the AI2-THOR environment.

Our evaluation contained six popular open-weight LLMs spanning different model families and scales, including Llama-3.1-8B-Instruct and Llama-3.1-70B-Instruct ( Grattafiori et al., 2024 ) , Qwen2.5-7B-Instruct and Qwen2.5-72B-Instruct ( Qwen Team, 2024 ) , gemma-2-9b-it ( Gemma Team, 2024 ) , and Ministral-8B-Instruct-2410 ( Mistral AI Team, 2024 ) .

Experiment Setup and Measurements — TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns — Figure 3: TALK-Dem construction pipeline. Each seed instruction was conversationalized into a clean control and independently transformed using five communication patterns at three intensity levels, yielding 15 patterned variants per task.

Quantitative Results

Figure 4 summarizes LLM-based planning performance under the five communication patterns, averaged across the three intensity levels. Overall, dementia-associated communication patterns reduced task success for most models, but the degradation varied substantially across patterns and model families.

This indicated that good performance on clean instructions does not necessarily translate into robustness to communication variation.

Quantitative Results — TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns — Figure 4: Task success rate (%) of Vanilla prompt on clean instructions and under five dementia-associated communication patterns, averaged across the three intensity levels. *: statistically significant difference compared to the clean condition ($p<0.05$).

Qualitative Analyses

We conducted error analyses on cases where the model succeeded on the clean instruction but failed on the patterned variants. Randomly sampling three instances from each open-weight model under each pattern-intensity condition yielded 270 instances.

Manual examination of clean and patterned instructions, the reference plan, the model outputs, and the simulator feedback was conducted to categorize failures into primary types.

Qualitative Analyses — TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns — Figure 5: Effect of communication-pattern intensity on clean-conditioned retention (%). Gray line: individual open-weight model. Black line: mean values. Colored lines: gemini-3.8-flash and claude-opus-5. Dashed line: clean baseline.

Sources and Demonstrations

Figures are reproduced from Zhao et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.