Evals: Can I trust my scores?
For teams whose eval scores don’t match what they see in practice.
Overview
If you use an LLM judge, start by reading Using LLM-as-a-Judge for Evaluation for the process of checking its decisions against human labels.
Do Automated Evals Work? shows why automated review still needs human judgment and product context.
For a production example, read Case Study: Putting Evals Into Production. The Revenge of the Data Scientist covers common eval mistakes, and Evals skills can help you audit your setup.
Common questions
Metrics and judge decisions
- How do I know if I can trust my automated eval?
- What should I do when I can’t get my LLM judge to agree with human reviewers?
- How do I combine my evals into a single metric?
- What model or LLM should I use to build automated evals?
- Can I use Jev for evals?
- Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales)?
- Should I use "ready-to-use" evaluation metrics?
- Are similarity metrics (BERTScore, ROUGE, etc.) useful for evaluating LLM outputs?
- Can I use the same model for both the main task and evaluation?
- How much context should I give a LLM judge?
Labels and eval data
Production monitoring and changes
- How are evaluations used differently in CI/CD vs. monitoring production?
- How do I surface problematic traces for review beyond user feedback?
- What’s the difference between guardrails & evaluators?
- Can my evaluators also be used to automatically fix or correct outputs in production?
- How should I version and manage prompts?