Skill-SLM: Agent Skill-driven Small Language Models for Reliable Robot Operation

Summary

Abstract Summary

Small language models (SLMs) have been increasingly adopted for onboard robot operation because they enable intelligent decision-making. However, existing approaches are mainly distillation-oriented and rely on enumerating representative task-solution pairs. This makes dataset construction difficult and limits generalization to diverse robot tasks whose possible forms grow rapidly. This paper proposes Skill-SLM, a framework that reformulates SLM-driven robot operation as a task-decomposition and skill-composition problem. Given a natural language task instruction, Skill-SLM decomposes the task into subtasks, selects appropriate skills from the skill library, and orchestrates the selected skills into executable robot operations. First, to support the skill-driven workflow, we propose a novel robot operational skill aware context-free grammar (CFG) to extract the skills required to accomplish tasks and build the skill library accordingly. Then, we configure LLM teachers to induce and synthesize training datasets for the SLMs, enabling SLMs to decompose tasks and orchestrate skills reliably. Additionally, we employ a progressive skill orchestration strategy to improve the reliability of skill implementation and overall robot operation. Experiments on UAV operation tasks indicate that Skill-SLM substantially outperforms distillation-oriented baselines, especially on unseen tasks that require generalization of capabilities. Additional experiments on ground vehicle tasks further demonstrate that Skill-SLM can be applied to different robot platforms.

Key Points

Small language models (SLMs) have been increasingly adopted for onboard robot operation because they enable intelligent decision-making.
However, existing approaches are mainly distillation-oriented and rely on enumerating representative task-solution pairs.
This makes dataset construction difficult and limits generalization to diverse robot tasks whose possible forms grow rapidly.
This paper proposes Skill-SLM, a framework that reformulates SLM-driven robot operation as a task-decomposition and skill-composition problem.
Given a natural language task instruction, Skill-SLM decomposes the task into subtasks, selects appropriate skills from the skill library, and orchestrates the selected skills into executable robot operations.

Method

Fig. 2 presents the workflow of our Skill-SLM framework. Given a task $T$, the goal of Skill-SLM deployed on the robot is to analyze the task and generate the corresponding robot operation code $C$ for the robot to complete the task.

Experiments

The robots used in our experiment include a Holybro X500 V2 UAV with a Pixhawk 6X flight controller and a ROSMASTER X3 ground vehicle with ROS, as presented in Fig. 4 . Both robots are equipped with Jetson Orin NX (16 GB) for the deployment of Skill-SLM. We evaluate four SLMs: Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, Qwen3-4B, and Qwen3-8B [ 29 , 16 ] .

Model LoRA [ 27 ] training is conducted on a server equipped with four NVIDIA H200 GPUs.

Experiments — Skill-SLM: Agent Skill-driven Small Language Models for Reliable Robot Operation — Fig. 2: Overall design of Skill-SLM with an illustrative example.

Sources and Demonstrations

Figures are reproduced from Wang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.