Q: How should I evaluate a coding agent?
If your coding agent handles a wide variety of tasks, start by using public benchmarks much as you would a foundation model. For an agent that handles a narrow workflow, product-specific evals are a better fit. The evals FAQ explains this distinction.
Popular coding benchmarks include SWE-bench, Terminal-Bench, Aider Polyglot, and HumanEval.
In addition to public benchmarks, you can also build a private benchmark of difficult tasks from your organization. OpenAI described using real internal software engineering tasks to evaluate Codex at launch. Each task needs a working environment and code-based tests that establish whether the agent completed it successfully.
To decide which tasks to include, look at how people use your agent and where it fails. Review runs with engineers, group recurring problems, and turn useful examples into tests. This is error analysis, and it applies to coding products too. If existing tests already identify failures, use those results to choose runs to investigate.
Anthropic’s Clio research illustrates a related approach that clusters chat conversations by topic. You can apply that idea to coding sessions to identify the kinds of work your benchmark should cover.
Anthropic’s coding-agent eval guidance recommends starting with clearly specified tasks and a stable environment where unit tests can verify results. After you have these unit tests, they recommend adding checks for things those tests don’t capture, such as code quality or how the agent interacts with users. Claude Code’s team, for example, added evals for file edits and later for over-engineering. There are many approaches to measure file edits and over-engineering but you can start with metrics like net new lines of code added and cyclomatic complexity.
John Berryman and Shawn Simister’s Copilot talk provides additional examples of coding-agent evals. For code completions, the team removed function implementations from repositories, had the model regenerate them, and ran the existing tests. For chat, they used LLM judges with specific criteria and separate checks for whether the assistant called the right tool. They also ran A/B tests, tracking whether users accepted suggestions and kept the code afterward. These product metrics complemented the offline evals.
This article is part of our AI Evals FAQ, a collection of common questions (and answers) about LLM evaluation. View all FAQs or return to the homepage.