Q: How many examples do I need for an eval?
Building evals is a pipeline, and each stage needs a different amount of data. We describe these stages below:
| Stage | What to do |
|---|---|
| 1. Review the application | Read traces and write down the ways your application fails. This process is called error discovery. Start with 100 diverse traces and annotate at least the first 30 yourself. |
| 2. Create and validate evaluators | Choose between two evaluator types. Use a code-based eval when an objective rule can identify the failure. Include Pass and Fail examples for every condition and important edge case. Use an LLM judge when the failure requires human judgment. Label 100 to 200 examples for each failure mode. |
| 3. Build a repeatable eval set | Collect examples that represent important workflows and confirmed failures. Run this set when you change your application. These sets often grow to 100 or more examples. |
Stage 1: Review traces to find failures
A trace is a complete record of one user session with your application. Ask a coding agent to help you sample the initial pool so it covers different users and workflows. Our evals plugin can help with sampling and build an annotation interface for your traces.
Review at least 30 traces yourself
We recommend annotating at least 30 traces with a process called error discovery yourself before asking the agent to suggest failures. Write free-text notes about anything that seems wrong from the user’s perspective. These examples give the agent a concrete record of your judgment.
Keep this first pass manual. If the agent starts suggesting problems too early, its guesses can bias your judgment. You may miss failures that depend on product context or your definition of a good user experience.
After 30 traces, ask the agent to search the remaining pool for similar examples. Review every suggestion yourself. Accept or reject each one and correct the agent when it misunderstands your criteria.
When to stop
Continue until new traces stop revealing failure modes or changing existing ones. Qualitative researchers call this theoretical saturation. We recommend reviewing at least 100 traces. Continue past 100 while you are still learning.
If you want to see the process done live, watch the live walkthrough. The video shows how Shreya Shankar uses an agent to review traces quickly while keeping a human in charge of the failure criteria.
This review produces a failure taxonomy, which is a list of the specific ways your application fails. Use that taxonomy to decide which evaluators to build.
Stage 2: Create and validate evaluators
Choose an evaluator for each important failure mode. The evaluator type determines how many labeled examples you need. Code-based evals work for objective rules. LLM judges work for failures that require human judgment.
Code-based evals need coverage
Use a code-based eval when a deterministic rule can identify the failure. Examples include checking whether JSON parses or whether a tool call uses the correct arguments.
The number of examples depends on the scenarios the check covers. At minimum, include examples that should Pass and Fail for every condition. Add important edge cases you found during error discovery. A check with one rule may need only a few examples that Pass and a few that Fail.
LLM judges need labeled examples
Use an LLM judge when the failure requires subjective or domain-specific judgment. Plan to label 100 to 200 examples for each failure mode. Reuse labeled traces from error discovery when they match the failure mode, then collect more until you reach that range. The labels should come from a trusted domain expert and contain enough Pass and Fail examples to evaluate both classes.
Split these examples into train, dev, and test sets. Use 10 to 20 percent for train examples that may appear in the prompt. Use 40 to 45 percent for dev while refining the judge. Reserve the remaining 40 to 45 percent for one final test. When possible, include 30 to 50 Pass examples and 30 to 50 Fail examples in both the dev and test sets.
The validation guide explains the full process. The judge-validation flashcard is a good visual reference as well.
After validating the evaluators, assemble the examples you will run repeatedly during development.
Stage 3: Build the repeatable eval set
Start with examples from error discovery that capture important failure modes. Add confirmed failures as you find them.
A purpose-built eval set often grows to 100 or more examples. Coverage determines the final size. Each important workflow and known failure should be represented, and the set should remain cheap enough to run often. Code-based checks and LLM judges can run over the same examples. The CI evals FAQ explains how to use this set during development.
This article is part of our AI Evals FAQ, a collection of common questions (and answers) about LLM evaluation. View all FAQs or return to the homepage.