Eureka: Human-Level Reward Design via Coding Large Language Models

Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, Anima Anandkumar · NVIDIA, UT Austin, Caltech · 2024

Summary

Eureka uses GPT-4 to write reward functions for RL environments, achieving human-level reward design on 29 tasks and enabling zero-shot sim-to-real transfer on Shadow Hand dexterous manipulation.

Abstract Summary

Eureka automates one of the most tedious parts of reinforcement learning: reward engineering. Instead of manually tuning reward functions for days, the authors prompt GPT-4 with the environment source code and task description, and GPT-4 outputs a Python reward function. Eureka then evaluates the reward in the RL environment (IsaacGym), iteratively refining it based on training curves—just like a human RL researcher would. On 29 diverse tasks across 10 robot morphologies (quadruped, humanoid, dexterous hand, UAV, etc.), Eureka-designed rewards outperform expert human-written rewards in 83% of cases. The key insight is that LLMs excel at code-level reasoning: given the physics simulation code, GPT-4 can infer what behaviors should be rewarded (e.g., "reward the hand for maintaining contact with the cube while lifting it"). Eureka uses a evolutionary selection mechanism: it generates multiple reward candidates, trains policies with each, and selects the top performers for the next iteration. The most impressive result is zero-shot sim-to-real transfer: policies trained with Eureka rewards in IsaacGym deploy directly on a real Shadow Hand robot for dexterous pen spinning—without any real-world fine-tuning. This beats prior SOTA by a large margin and suggests that LLM-generated rewards capture physically realistic behavior better than hand-tuned alternatives. The open-source release includes the full prompting pipeline, IsaacGym integration, and 29 benchmark environments. The codebase also supports custom environments: drop in your MuJoCo/IsaacGym XML, write a task description, and Eureka will generate and tune rewards automatically. It's one of the first systems to show LLMs can replace a key human-in-the-loop step in the RL pipeline.

Key Points

GPT-4 writes and iteratively refines RL reward functions.
Outperforms human rewards on 83% of 29 tasks across 10 robot types.
Zero-shot sim-to-real transfer to Shadow Hand dexterous manipulation.
Evolutionary selection: generate → train → select → iterate.
Open-source with IsaacGym integration and 29 benchmark tasks.

Problem Setting and Definitions

The goal of reward design is to return a shaped reward function for a ground-truth reward function that may be difficult to optimize directly (e.g., sparse rewards); this ground-truth reward function may only be accessed via queries by the designer. We first introduce the formal definition from Singh et al. (2010) , which we then adapt to the program synthesis setting, which we call reward generation .

Method

Eureka consists of three algorithmic components: 1) environment as context that enables zero-shot generation of executable rewards, 2) evolutionary search that iteratively proposes and refines reward candidates, and 3) reward reflection that enables fine-grained reward improvement. See Alg. 1 for pseudocode; all prompts are included in App. A .

Reward design requires the environment specification to be provided to the LLM. We propose directly feeding the raw environment source code (without the reward code, if exists) as context.

Method — Eureka: Human-Level Reward Design via Coding Large Language Models — Figure 4: Eureka outperforms Human and L2R across all tasks. In particular, Eureka realizes much greater gains on high-dimensional dexterity environments.

Experiments

We thoroughly evaluate Eureka on a diverse suite of robot embodiments and tasks, testing its ability to generate reward functions, solve new tasks, and incorporate various forms of human input. We use GPT-4 ( OpenAI, 2023 ) , in particular the gpt-4-0314 variant, as the backbone LLM for all LLM-based reward-design algorithms unless specified otherwise. Environments.

Our environments consist of 10 distinct robots and 29 tasks implemented using the IsaacGym simulator ( Makoviychuk et al., 2021 ) .

Experiments — Eureka: Human-Level Reward Design via Coding Large Language Models — Figure 5: Eureka progressively produces better rewards via in-context evolutionary reward search.

Sources and Demonstrations

Figures are reproduced from Ma et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.