Evals: Getting started

For anyone wondering what evals are and whether they need them.

Overview

Start by reading Your AI Product Needs Evals to see what an evaluation system looks like in a real product.

For a video walkthrough, watch Teresa Torres’s From Noob to Automated Evals In A Week (as a PM).

Common questions

Getting started

• What are AI Evals?
• What’s a minimum viable evaluation setup?
• What is a trace?
• Why is "error analysis" so important in AI evals, and how is it performed?
• Should I build automated evaluators for every failure mode I find?

Working with your team

• What should an internal eval platform standardize across teams?
• Can I help with evals if I’m not a domain expert?
• Should product managers and engineers collaborate on error analysis? How?
• How do I make the case for investing in evaluations to my team?
• What does a good eval dashboard look like for sharing with stakeholders?

Planning the investment

• How much of my development budget should I allocate to evals?
• Will today’s evaluation methods still be relevant in 5-10 years given how fast AI is changing?

← See all the guides