Fly-by-Code: Embodied Coding Agents for Aerial Manipulation with Active Visual and Physical Feedback

Jaewoo Lee, Jeongyeon Seo, Sihyun Cho, Gyeongrak Choe, Yutong Wang, Bavin Saravanan, Jia-Bin Huang, Furong Huang, Sebastian Scherer, Guanya Shi, H. Jin Kim, Seungjae Lee, Dongjae Lee · N/A · 2026

Summary

Embodied coding agents construct robot policies from natural-language instructions by generating executable programs and refining them from execution feedback.

Abstract Summary

Embodied coding agents construct robot policies from natural-language instructions by generating executable programs and refining them from execution feedback. This refinement assumes that execution feedback reveals whether the task succeeded and, after a failure, provides sufficient evidence to guide program revision. Aerial manipulation violates this assumption in two ways. Payload constraints restrict onboard camera coverage and leave task-critical outcomes outside the robot's view (partial observability), while strong dynamic coupling and limited control margins make physical interactions fragile (limited interaction robustness) and leave failure causes unresolved. Re-examining the same feedback cannot recover observations that were never collected or separate failure causes that the available evidence does not distinguish. To acquire this missing evidence, we introduce Active Feedback (AF), an active "View and Probe" framework. When execution feedback leaves unresolved questions, the agent synthesizes a separate diagnostic program that runs in simulation to inspect unseen outcomes from informative viewpoints (View) or test hypotheses about failure causes through targeted physical interactions (Probe). Across six simulated aerial manipulation tasks with 60 trials per condition, AF reduces false completion declarations from 25 to 5 and raises final task success from 31.7% to 75% compared with terminal feedback under the same coding-agent configuration and task-execution limit. Using shared robot APIs, task programs refined with AF in simulation run on a physical aerial manipulator without task-program source-code modification and complete multi-stage manipulation tasks in the real world. These results show that executable diagnosis fills gaps in execution feedback for embodied coding agents, improving both completion assessment and program refinement in aerial manipulation.

Key Points

Embodied coding agents construct robot policies from natural-language instructions by generating executable programs and refining them from execution feedback.
This refinement assumes that execution feedback reveals whether the task succeeded and, after a failure, provides sufficient evidence to guide program revision.
Aerial manipulation violates this assumption in two ways
Payload constraints restrict onboard camera coverage and leave task-critical outcomes outside the robot's view (partial observability), while strong dynamic coupling and limited control margins make physical interactions fragile (limited interaction robustness) and leave failure causes unresolved.
Re-examining the same feedback cannot recover observations that were never collected or separate failure causes that the available evidence does not distinguish.

Active Feedback for Program Synthesis in Aerial Manipulation

Active Feedback (AF) distinguishes Fly-by-Code from existing embodied coding agents that rely solely on passive execution feedback for completion assessment and program refinement. Such feedback is limited to evidence collected during task execution.

AF inserts an active diagnostic stage before the decision to finish or revise the task program: the agent writes a separate program that runs in simulation to acquire additional evidence for unresolved questions (Figure 2 ).

Active Feedback for Program Synthesis in Aerial Manipulation — Fly-by-Code: Embodied Coding Agents for Aerial Manipulation with Active Visual and Physical Feedback — Figure 1: Fly-by-Code uses Active Feedback (AF) to address missing outcome observations and unresolved failure causes arising from partial observability and limited interaction robustness. Rather than relying only on feedback passively collected during task execution, Fly-by-Code generates an executable diagnostic program to acquire the evidence needed for outcome assessment and program revision. The program either navigates to an informative viewpoint ( View ) to inspect an unobserved outcome or performs a targeted physical interaction ( Probe ) to distinguish between possible failure causes. Using shared robot APIs, task programs refined in simulation run on a physical aerial manipulator without source-code modification and complete multi-stage tasks such as the illustrated four-step tool-manipulation sequence.

Experiments

We evaluate Active Feedback (AF) through three questions: (Q1) Does AF improve task success and completion assessment over passive execution feedback? (Q2) How does diagnostic evidence inform program refinement? (Q3) How does AF perform across models and agent interfaces? Section 4.5 then evaluates real-world deployment of the simulation-synthesized programs.

We evaluate Fly-by-Code in six tasks extending AM-Bench ( Wang et al., 2026 ) , implemented in Isaac Lab ( Mittal et al., 2025 ) .

Experiments — Fly-by-Code: Embodied Coding Agents for Aerial Manipulation with Active Visual and Physical Feedback — Figure 2: Overview of Fly-by-Code. Bottom: The program refinement loop. The coding agent translates a task prompt into an executable task program using shared robot APIs. Following execution, it enters the Active Feedback (AF) stage to acquire diagnostic evidence, iteratively refining the program until it decides to terminate the revision process. Top: During AF, the agent synthesizes a diagnostic program to resolve execution uncertainties. View actively navigates to informative viewpoints (e.g., inspecting a basket from above to verify placement). Probe conducts diagnostic physical tests (e.g., testing door articulation with an arc motion after a straight pull fails).

Sources and Demonstrations

Figures are reproduced from Lee et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.