Eureka: Human-Level Reward Design via Coding Large Language Models

Eureka automates one of the most tedious parts of reinforcement learning: reward engineering. Instead of manually tuning reward functions for days, the authors prompt GPT-4 with the environment source code and task description, and GPT-4 outputs a Python reward function. Eureka then evaluates the reward in the RL environment (IsaacGym), iteratively refining it based on training curves—just like a human RL researcher would. On 29 diverse tasks across 10 robot morphologies (quadruped, humanoid, dexterous hand, UAV, etc.), Eureka-designed rewards outperform expert human-written rewards in 83% of cases. The key insight is that LLMs excel at code-level reasoning: given the physics simulation code, GPT-4 can infer what behaviors should be rewarded (e.g., "reward the hand for maintaining contact with the cube while lifting it"). Eureka uses a evolutionary selection mechanism: it generates multiple reward candidates, trains policies with each, and selects the top performers for the next iteration. The most impressive result is zero-shot sim-to-real transfer: policies trained with Eureka rewards in IsaacGym deploy directly on a real Shadow Hand robot for dexterous pen spinning—without any real-world fine-tuning. This beats prior SOTA by a large margin and suggests that LLM-generated rewards capture physically realistic behavior better than hand-tuned alternatives. The open-source release includes the full prompting pipeline, IsaacGym integration, and 29 benchmark environments. The codebase also supports custom environments: drop in your MuJoCo/IsaacGym XML, write a task description, and Eureka will generate and tune rewards automatically. It's one of the first systems to show LLMs can replace a key human-in-the-loop step in the RL pipeline.