Evals: Reducing time and cost

For teams spending more time or money on evals than they can sustain.

Overview

Start by reading “It’s Hard to Eval” Is a Product Smell to see how product changes can reduce the effort of checking outputs.

For help choosing tools, read Selecting the Right AI Evals Tool. Evals skills can help you review traces and audit your setup.

Read Do Automated Evals Work? to see what automated error analysis caught and missed in real production traces. Watch Shreya’s How to Automate AI Evals (Correctly) for a demo of assisted error analysis, or Isaac’s Building Custom Eval Tools With Coding Agents for help building review tools.

Common questions

Review interfaces and tools

• What should an internal eval platform standardize across teams?
• What makes a good custom interface for reviewing LLM outputs?
• Should I build a custom annotation tool or use something off-the-shelf?
• What gaps in eval tooling should I be prepared to fill myself?
• What’s your favorite eval vendor?

Eval maintenance and review workload

• Should I build automated evaluators for every failure mode I find?
• How much of my development budget should I allocate to evals?
• What should I do when my "gold" eval dataset becomes stale?
• How can I efficiently sample production traces for review?
• How many examples do I need for an eval?
• How often should I re-run error analysis on my production system?
• How many people should annotate my LLM outputs?

Automation, outsourcing, and model selection

• What model or LLM should I use to build automated evals?
• Can I use Jev for evals?
• What parts of evals can be automated with LLMs?
• Should I outsource annotation & labeling to a third party?
• Should I stop writing prompts manually in favor of automated tools?
• How much time should I spend on model selection?

← See all the guides