AI evals resources

My writing and teaching materials on AI evals, organized by format.

For a guide to what to read based on your situation, start here.

Blog posts

Resource	What you’ll find
Your AI Product Needs Evals	An evaluation system for a real estate AI assistant.
A Field Guide to Rapidly Improving AI Products	How teams use error analysis and experiments to improve AI products.
“It’s Hard to Eval” Is a Product Smell	Product designs that make AI outputs easier to verify.
Using LLM-as-a-Judge for Evaluation	How to build an LLM judge and validate it against human labels.
Selecting the Right AI Evals Tool	A comparison of three eval tools on the same assignment.

Resource	What you’ll find
Intro to Error Analysis With Just Spreadsheets	A video walkthrough of error analysis.
How to Automate AI Evals (Correctly)	Shreya demonstrates error analysis with the Error Discovery skill.
From Noob to Automated Evals In A Week (as a PM)	Teresa Torres walks through getting started with automated evals.
Building Custom Eval Tools With Coding Agents	Isaac Flath demonstrates building custom eval tools.
How to Eval an AI Product That Seems Impossible to Eval	How to connect evaluation criteria with what users need to verify.
Evals for AI Writing	Evaluating writing beyond factual correctness.
Reasoning Models & LLM-as-a-Judge	A discussion of reasoning models as evaluators.
Intro to Error Analysis: Creating Custom Data Annotation Apps	A walkthrough of building an annotation app for error analysis.
How Engineers and PMs Should Collaborate on Evals	A discussion of teamwork and connecting evals to business metrics.
Case Study: Putting Evals Into Production	How Nova Escola built evals and ran them on production traffic.
The Revenge of the Data Scientist	An annotated talk about common eval mistakes, with video and slides.
Do Automated Evals Work?	A Parlance Labs study comparing automated evals with human labels.

The LLM Evals FAQ. Short answers to specific questions about evaluating AI products.

Evals skills. Skills that help your coding agent review traces and audit evals.

Resource	What you’ll find
Evals Flashcards	Review the main eval concepts, with a downloadable PDF.
Evals Memes	Familiar eval mistakes illustrated with humor.

AI Evals for Engineers and PMs. Hands-on practice building evals, with feedback.