AI Evals

Build, evaluate, and improve AI agents that work in production.

AI evals show where an AI product fails so you can improve how it works in production.

Start with the workflow

Your AI Product Needs Evals explains how to construct an evaluation system for an AI product.
A Field Guide to Rapidly Improving AI Products covers error analysis and the wider process for improving an AI product.
How Engineers and PMs Should Collaborate on Evals covers how engineers and PMs can connect evals to business metrics. It also explains how to make the case for evals.

Intro to Error Analysis With Just Spreadsheets walks through error analysis from scratch with a spreadsheet.
How to Sample Production Traces for Review combines a random baseline with targeted review. It links the sampling flashcard and explains an active learning workflow.
The Revenge of the Data Scientist covers five eval mistakes and the data science practices that fix them. The page includes an annotated transcript of the talk. It also links to the video and slides.
“It’s Hard to Eval” Is a Product Smell uses three product examples to show how product design can make AI output easier to verify.

Using LLM-as-a-Judge for Evaluation is a step-by-step guide to building and checking an LLM judge against human labels.
Do Automated Evals Work? compares automated eval results with labels from people on 100 traces. Parlance Labs hosts the original study.

AI Evals for Engineers and PMs gives you a structured, hands-on way to practice evals with feedback.

The LLM Evals FAQ answers specific questions about evaluating AI products.
Evals Flashcards review the main ideas from the course and can be downloaded as a PDF.
Evals Memes make fun of familiar eval mistakes.