Hamel’s Blog

This page is supposed to be secret!

A listing of all my blog posts can be found here

Frequently Asked Questions (And Answers) About AI Evals

LLMs
evals

FAQ from our course on AI Evals.

Aug 17, 2025
Hamel Husain, Shreya Shankar

Q: How should I version and manage prompts?

LLMs
evals
faq
faq-individual
There is an unavoidable tension between keeping prompts close to the code vs. an environment that non-technical stakeholders can access.
Aug 17, 2025

Q: How do we evaluate a model’s ability to express uncertainty or “know what it doesn’t know”?

LLMs
evals
faq
faq-individual
Many applications require a model that can refuse to answer a question when it lacks sufficient information. To evaluate whether this refusal behavior is well-calibrated…
Aug 17, 2025

Q: How do I make the case for investing in evaluations to my team?

LLMs
evals
faq
faq-individual
Don’t try to sell your team on “evals”. Instead, show them what you find when you look at the data.
Aug 16, 2025

Q: Should product managers and engineers collaborate on error analysis? How?

LLMs
evals
faq
faq-individual
At the outset, collaborate to establish shared context. Engineers catch technical issues like retrieval issues and tool errors. PMs identify product failures like unmet user…
Aug 4, 2025

Q: Should I build automated evaluators for every failure mode I find?

LLMs
evals
faq
faq-individual
Focus automated evaluators on failures that persist after fixing your prompts. Many teams discover their LLM doesn’t meet preferences they never actually specified - like…
Aug 4, 2025

Q: Why is “error analysis” so important in LLM evals, and how is it performed?

LLMs
evals
faq
faq-individual
Error analysis is the most important activity in evals. Error analysis helps you decide what evals to write in the first place. It allows you to identify failure modes…
Aug 4, 2025

Q: What’s a minimum viable evaluation setup?

LLMs
evals
faq
faq-individual
Start with error analysis, not infrastructure. Spend 30 minutes manually reviewing 20-50 LLM outputs whenever you make significant changes. Use one domain expert who…
Aug 4, 2025

Q: How do I approach evaluation when my system handles diverse user queries?

LLMs
evals
faq
faq-individual
Complex applications often support vastly different query patterns—from “What’s the return policy?” to “Compare pricing trends across regions for products matching these…
Aug 4, 2025

Q: What are LLM Evals?

LLMs
evals
faq
faq-individual
If you are completely new to product-specific LLM evals (not foundation model benchmarks), see these posts: part 1, part 2 and part 3. Otherwise, keep reading.
Aug 4, 2025

Q: How are evaluations used differently in CI/CD vs. monitoring production?

LLMs
evals
faq
faq-individual
The most important difference between CI vs. production evaluation is the data used for testing.
Aug 4, 2025

Q: How do I evaluate complex multi-step workflows?

LLMs
evals
faq
faq-individual
Log the entire workflow from initial trigger to final business outcome. Include LLM calls, tool usage, human approvals, and database writes in your traces. You will need…
Aug 4, 2025

Q: How often should I re-run error analysis on my production system?

LLMs
evals
faq
faq-individual
Re-run error analysis when making significant changes: new features, prompt updates, model switches, or major bug fixes. A useful heuristic is to set a goal for reviewing at…
Aug 4, 2025

Q: Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales)?

LLMs
evals
faq
faq-individual
Engineers often believe that Likert scales (1-5 ratings) provide more information than binary evaluations, allowing them to track gradual improvements. However, this added…
Aug 4, 2025

Q: Seriously Hamel. Stop the bullshit. What’s your favorite eval vendor?

LLMs
evals
faq
faq-individual
Eval tools are in an intensely competitive space. It would be futile to compare their features. If I tried to do such an analysis, it would be invalidated in a week! Vendors…
Aug 4, 2025

Q: Are there scenarios where synthetic data may not be reliable?

LLMs
evals
faq
faq-individual
Yes: synthetic data can mislead or mask issues. For guidance on generating synthetic data when appropriate, see What is the best approach for generating synthetic data?
Aug 4, 2025

Q: Should I use “ready-to-use” evaluation metrics?

LLMs
evals
faq
faq-individual
No. Generic evaluations waste time and create false confidence. (Unless you’re using them for exploration).
Aug 4, 2025

Q: What gaps in eval tooling should I be prepared to fill myself?

LLMs
evals
faq
faq-individual
Most eval tools handle the basics well: logging complete traces, tracking metrics, prompt playgrounds, and annotation queues. These are table stakes. Here are four areas…
Aug 4, 2025

Q: How do I justify evaluation time and budget to management?

LLMs
evals
faq
faq-individual
Keep a log of every error you catch. Document learnings found during error analysis, the fix implemented, and the potential impact avoided. Present this log weekly or…
Aug 4, 2025

Q: Should I outsource annotation & labeling to a third party?

LLMs
evals
faq
faq-individual
Outsourcing error analysis is usually a big mistake (with some exceptions). The core of evaluation is building the product intuition that only comes from systematically…
Aug 4, 2025

Q: How do I choose the right chunk size for my document processing tasks?

LLMs
evals
faq
faq-individual
Unlike RAG, where chunks are optimized for retrieval, document processing assumes the model will see every chunk. The goal is to split text so the model can reason…
Aug 4, 2025

Q: Are similarity metrics (BERTScore, ROUGE, etc.) useful for evaluating LLM outputs?

LLMs
evals
faq
faq-individual
Generic metrics like BERTScore, ROUGE, cosine similarity, etc. are not useful for evaluating LLM outputs in most AI applications. Instead, we recommend using error analysis t…
Aug 4, 2025

Q: What’s the difference between guardrails & evaluators?

LLMs
evals
faq
faq-individual
Guardrails are inline safety checks that sit directly in the request/response path. They validate inputs or outputs before anything reaches a user, so they typically are:
Aug 4, 2025

Q: Can I use the same model for both the main task and evaluation?

LLMs
evals
faq
faq-individual
For LLM-as-Judge selection, using the same model is usually fine because the judge is doing a different task than your main LLM pipeline. The judges we recommend building do …
Aug 4, 2025

Q: What should go in the system prompt vs. the user prompt?

LLMs
evals
faq
faq-individual
Nothing beats experimentation. Test both approaches (ideally with evals) with your specific model and use case. Models handle system and user prompts differently, and these…
Aug 4, 2025

Q: What is the best approach for generating synthetic data?

LLMs
evals
faq
faq-individual
A common mistake is prompting an LLM to "give me test queries" without structure, resulting in generic, repetitive outputs. A structured approach using dimensions produces…
Aug 4, 2025

Q: How much of my development budget should I allocate to evals?

LLMs
evals
faq
faq-individual
It’s important to recognize that evaluation is part of the development process rather than a distinct line item, similar to how debugging is part of software development.
Aug 4, 2025

Q: What makes a good custom interface for reviewing LLM outputs?

LLMs
evals
faq
faq-individual
Great interfaces make human review fast, clear, and motivating. We recommend building your own annotation tool customized to your domain. The following features are possible…
Aug 4, 2025

Q: How many people should annotate my LLM outputs?

LLMs
evals
faq
faq-individual
For most small to medium-sized companies, appointing a single domain expert as a “benevolent dictator” is the most effective approach. This person—whether it’s a…
Aug 4, 2025

Q: Is RAG dead?

LLMs
evals
faq
faq-individual
Question: Should I avoid using RAG for my AI application after reading that “RAG is dead” for coding agents?
Aug 4, 2025

Q: What is a trace?

LLMs
evals
faq
faq-individual
A trace is the complete record of all actions, messages, tool calls, and data retrievals from a single initial user query through to the final response. It includes every…
Aug 4, 2025

Q: How can I efficiently sample production traces for review?

LLMs
evals
faq
faq-individual
It can be cumbersome to review traces randomly, especially when most traces don’t have an error. These sampling strategies help you find traces more likely to reveal problems:
Aug 4, 2025

Q: How should I approach evaluating my RAG system?

LLMs
evals
faq
faq-individual
RAG systems have two distinct components that require different evaluation approaches: retrieval and generation.
Aug 4, 2025

Q: How do I debug multi-turn conversation traces?

LLMs
evals
faq
faq-individual
Start simple. Check if the whole conversation met the user’s goal with a pass/fail judgment. Look at the entire trace and focus on the first upstream failure. Read the…
Aug 4, 2025

Q: Can my evaluators also be used to automatically fix or correct outputs in production?

LLMs
evals
faq
faq-individual
Yes, but only a specific subset of them. This is the distinction between an evaluator and a guardrail that we previously discussed. As a reminder:
Aug 4, 2025

Q: How do I evaluate agentic workflows?

LLMs
evals
faq
faq-individual
We recommend evaluating agentic workflows in two phases:
Aug 4, 2025

Q: How do I surface problematic traces for review beyond user feedback?

LLMs
evals
faq
faq-individual
While user feedback is a good way to narrow in on problematic traces, other methods are also useful. Here are three complementary approaches:
Aug 4, 2025

Q: What parts of evals can be automated with LLMs?

LLMs
evals
faq
faq-individual
LLMs can speed up parts of your eval workflow, but they can’t replace human judgment where your expertise is essential. For example, if you let an LLM handle all of error…
Aug 4, 2025

Q: Should I practice eval-driven development?

LLMs
evals
faq
faq-individual
Generally no. Eval-driven development (writing evaluators before implementing features) sounds appealing but creates more problems than it solves. Unlike traditional…
Aug 4, 2025

Q: Will today’s evaluation methods still be relevant in 5-10 years given how fast AI is changing?

LLMs
evals
faq
faq-individual
Yes. Even with perfect models, you still need to verify they’re solving the right problem. The need for systematic error analysis, domain-specific testing, and monitoring…
Aug 4, 2025

Q: How much time should I spend on model selection?

LLMs
evals
faq
faq-individual
Many developers fixate on model selection as the primary way to improve their LLM applications. Start with error analysis to understand your failure modes before considering…
Aug 4, 2025

Q: Should I build a custom annotation tool or use something off-the-shelf?

LLMs
evals
faq
faq-individual
Build a custom annotation tool. This is the single most impactful investment you can make for your AI evaluation workflow. With AI-assisted development tools like Cursor or…
Aug 4, 2025

Q: Should I stop writing prompts manually in favor of automated tools?

LLMs
evals
faq
faq-individual
Automating prompt engineering can be tempting, but you should be skeptical of tools that promise to optimize prompts for you, especially in early stages of development. When…
Aug 4, 2025

Q: How do I evaluate sessions with human handoffs?

LLMs
evals
faq
faq-individual
Capture the complete user journey in your traces, including human handoffs. The trace continues until the user’s need is resolved or the session ends, not when AI hands off…
Aug 4, 2025

A Field Guide to Rapidly Improving AI Products

LLMs
AI

Evaluation methods, data-driven improvement, and experimentation techniques from 30+ production implementations.

Mar 24, 2025
Hamel Husain

Building an Audience Through Technical Writing: Strategies and Mistakes

Writing

What I’ve seen work and what doesn’t.

Nov 30, 2024
Hamel Husain

Creating a LLM-as-a-Judge That Drives Business Results

LLMs
evals

A step-by-step guide with my learnings from 30+ AI implementations.

Oct 29, 2024
Hamel Husain

An Open Course on LLMs, Led by Practitioners

llms
courses

A free survey course on LLMs, taught by practitioners.

Jul 29, 2024
Hamel Husain

Debugging AI With Adversarial Validation

LLMs

Quickly detect a common bug in AI products using an automated technique.

Apr 12, 2024
Hamel Husain

Your AI Product Needs Evals

LLMs
evals

How to construct domain-specific LLM evaluation systems.

Mar 29, 2024
Hamel Husain
 

Is Fine-Tuning Still Valuable?

A reaction to a recent trend of disillusionment with fine-tuning.

Mar 27, 2024
Hamel Husain

Fuck You, Show Me The Prompt.

llms
ml

Quickly understand inscrutable LLM frameworks by intercepting API calls.

Feb 14, 2024
Hamel Husain

How To Debug Axolotl

LLMs
fine-tuning
axolotl

Best practices for debugging axolotl with an example VSCode config.

Jan 11, 2024
Hamel Husain

Dokku: my favorite personal serverless platform

infra
severless

Like Heroku, but you own it.

Jan 9, 2024
Hamel Husain

Why Should ML Engineers Learn Kubernetes?

K8s

Learning K8s can give you an unreasonable advantage as an MLE and unblock your team.

Jan 16, 2023
Hamel Husain
No matching items