Automating Error Analysis

Build an error taxonomy from human review instead of asking an LLM to invent one.

This note covers Shreya Shankar’s session in the AI Product Engineering series.

Before creating any evals its important to perform data analysis of your logs so you know which errors occur. A common mistake is writing a rubric up front before looking at any data.

This process is called error analysis. It is the most human-intensive part of evals because it requires you to read and annotate traces.

The Error Discovery skill removes much of this friction. It updates a failure-mode taxonomy as you annotate and checks earlier records for each new pattern. It uses active learning to choose the next examples based on what you have already labeled.

The Error Discovery skill workflow. A human reviewer annotates in a local review app while an AI agent updates the failure-mode taxonomy and picks the next samples to look at.

Here is a link to the talk, which is a walkthrough of how to use the skill properly.

---

Want to learn more about AI Evals? Check out our AI Evals course. It’s a live cohort with hands on exercises and office hours.