Evals: Getting started

For anyone wondering what evals are and whether they need them.

Overview

Start by reading Your AI Product Needs Evals to see what an evaluation system looks like in a real product.

For a video walkthrough, watch Teresa Torres’s From Noob to Automated Evals In A Week (as a PM).

Common questions

Getting started

• What are AI evals?
• What’s a minimum viable evaluation setup?
• What is a trace?
• How to do error analysis
• Should I build automated evaluators for every failure?

Working with your team

• Can I help if I’m not a domain expert?
• How should PMs and engineers collaborate?
• How do I make the case for evals to my team?

Planning the investment

• How much development budget should go to evals?
• Will these methods still be relevant as AI changes?

← See all the guides