Evals: Hard-to-evaluate products
For products whose outputs are difficult to judge, even for an expert.
Overview
Start by reading “It’s Hard to Eval” Is a Product Smell to see how product design can make difficult outputs easier to verify.
Watch How to Eval an AI Product That Seems Impossible to Eval for a discussion of what users need to verify. Evals for AI Writing covers a subjective use case.
Common questions
Making human review easier
- How can I make AI outputs easier for people to evaluate?
- How do you review a trace that is really large?
- What makes a good custom interface for reviewing LLM outputs?
- Can I help with evals if I’m not a domain expert?
- How do I evaluate outputs in a language I don’t speak?
- How can I do evals when traces contain sensitive data?