Hamel’s Blog

Eval tools are in an intensely competitive space. It would be futile to compare their features. If I tried to do such an analysis, it would be invalidated in a week! Vendors…

Q: Seriously Hamel. Stop the bullshit. What’s your favorite eval vendor?

LLMs

evals

faq

faq-individual

For LLM-as-Judge selection, using the same model is usually fine because the judge is doing a different task than your main LLM pipeline. While research has shown that…

Q: Can I use the same model for both the main task and evaluation?

LLMs

evals

faq

faq-individual

Many applications require a model that can refuse to answer a question when it lacks sufficient information. To evaluate whether this refusal behavior is well-calibrated…

Q: How do we evaluate a model’s ability to express uncertainty or “know what it doesn’t know”?

LLMs

evals

faq

faq-individual

Don’t try to sell your team on “evals”. Instead, show them what you find when you look at the data.

Q: How do I make the case for investing in evaluations to my team?

LLMs

evals

faq

faq-individual

Selecting The Right AI Evals Tool

Evals

An example of how to assess AI eval tools like Langsmith, Braintrust, and Arize Phoenix.

Oct 1, 2025

At the outset, collaborate to establish shared context. Engineers catch technical issues like retrieval issues and tool errors. PMs identify product failures like unmet user…

Q: Should product managers and engineers collaborate on error analysis? How?

LLMs

evals

faq

faq-individual

Focus automated evaluators on failures that persist after fixing your prompts. Many teams discover their LLM doesn’t meet preferences they never actually specified - like…

Q: Should I build automated evaluators for every failure mode I find?

LLMs

evals

faq

faq-individual

Error analysis is the most important activity in evals. Error analysis helps you decide what evals to write in the first place. It allows you to identify failure modes…

Q: Why is “error analysis” so important in LLM evals, and how is it performed?

LLMs

evals

faq

faq-individual

Start with error analysis, not infrastructure. Spend 30 minutes manually reviewing 20-50 LLM outputs whenever you make significant changes. Use one domain expert who…

Q: What’s a minimum viable evaluation setup?

LLMs

evals

faq

faq-individual

Complex applications often support vastly different query patterns—from “What’s the return policy?” to “Compare pricing trends across regions for products matching these…

Q: How do I approach evaluation when my system handles diverse user queries?

LLMs

evals

faq

faq-individual

If you are completely new to product-specific LLM evals (not foundation model benchmarks), see these posts: part 1, part 2 and part 3. Otherwise, keep reading.

Q: What are LLM Evals?

LLMs

evals

faq

faq-individual

The most important difference between CI vs. production evaluation is the data used for testing.

Q: How are evaluations used differently in CI/CD vs. monitoring production?

LLMs

evals

faq

faq-individual

Log the entire workflow from initial trigger to final business outcome. Include LLM calls, tool usage, human approvals, and database writes in your traces. You will need…

Q: How do I evaluate complex multi-step workflows?

LLMs

evals

faq

faq-individual

Re-run error analysis when making significant changes: new features, prompt updates, model switches, or major bug fixes. A useful heuristic is to set a goal for reviewing at…

Q: How often should I re-run error analysis on my production system?

LLMs

evals

faq

faq-individual

Engineers often believe that Likert scales (1-5 ratings) provide more information than binary evaluations, allowing them to track gradual improvements. However, this added…

Q: Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales)?

LLMs

evals

faq

faq-individual

Yes: synthetic data can mislead or mask issues. For guidance on generating synthetic data when appropriate, see What is the best approach for generating synthetic data?

Q: Are there scenarios where synthetic data may not be reliable?

LLMs

evals

faq

faq-individual

No. Generic evaluations waste time and create false confidence. (Unless you’re using them for exploration).

Q: Should I use “ready-to-use” evaluation metrics?

LLMs

evals

faq

faq-individual

Most eval tools handle the basics well: logging complete traces, tracking metrics, prompt playgrounds, and annotation queues. These are table stakes. Here are four areas…

Q: What gaps in eval tooling should I be prepared to fill myself?

LLMs

evals

faq

faq-individual

Keep a log of every error you catch. Document learnings found during error analysis, the fix implemented, and the potential impact avoided. Present this log weekly or…

Q: How do I justify evaluation time and budget to management?

LLMs

evals

faq

faq-individual

Outsourcing error analysis is usually a big mistake (with some exceptions). The core of evaluation is building the product intuition that only comes from systematically…

Q: Should I outsource annotation & labeling to a third party?

LLMs

evals

faq

faq-individual

Unlike RAG, where chunks are optimized for retrieval, document processing assumes the model will see every chunk. The goal is to split text so the model can reason…

Q: How do I choose the right chunk size for my document processing tasks?

LLMs

evals

faq

faq-individual

Generic metrics like BERTScore, ROUGE, cosine similarity, etc. are not useful for evaluating LLM outputs in most AI applications. Instead, we recommend using error analysis t…

Q: Are similarity metrics (BERTScore, ROUGE, etc.) useful for evaluating LLM outputs?

LLMs

evals

faq

faq-individual

Guardrails are inline safety checks that sit directly in the request/response path. They validate inputs or outputs before anything reaches a user, so they typically are:

Q: What’s the difference between guardrails & evaluators?

LLMs

evals

faq

faq-individual

Nothing beats experimentation. Test both approaches (ideally with evals) with your specific model and use case. Models handle system and user prompts differently, and these…

Q: What should go in the system prompt vs. the user prompt?

LLMs

evals

faq

faq-individual

A common mistake is prompting an LLM to "give me test queries" without structure, resulting in generic, repetitive outputs. A structured approach using dimensions produces…

Q: What is the best approach for generating synthetic data?

LLMs

evals

faq

faq-individual

It’s important to recognize that evaluation is part of the development process rather than a distinct line item, similar to how debugging is part of software development.

Q: How much of my development budget should I allocate to evals?

LLMs

evals

faq

faq-individual

Great interfaces make human review fast, clear, and motivating. We recommend building your own annotation tool customized to your domain. The following features are possible…

Q: What makes a good custom interface for reviewing LLM outputs?

LLMs

evals

faq

faq-individual

For most small to medium-sized companies, appointing a single domain expert as a “benevolent dictator” is the most effective approach. This person—whether it’s a…

Q: How many people should annotate my LLM outputs?

LLMs

evals

faq

faq-individual

Question: Should I avoid using RAG for my AI application after reading that “RAG is dead” for coding agents?

Q: Is RAG dead?

LLMs

evals

faq

faq-individual

A trace is the complete record of all actions, messages, tool calls, and data retrievals from a single initial user query through to the final response. It includes every…

Q: What is a trace?

LLMs

evals

faq

faq-individual

It can be cumbersome to review traces randomly, especially when most traces don’t have an error. These sampling strategies help you find traces more likely to reveal problems:

Q: How can I efficiently sample production traces for review?

LLMs

evals

faq

faq-individual

RAG systems have two distinct components that require different evaluation approaches: retrieval and generation.

Q: How should I approach evaluating my RAG system?

LLMs

evals

faq

faq-individual

Start simple. Check if the whole conversation met the user’s goal with a pass/fail judgment. Look at the entire trace and focus on the first upstream failure. Read the…

Q: How do I debug multi-turn conversation traces?

LLMs

evals

faq

faq-individual

Yes, but only a specific subset of them. This is the distinction between an evaluator and a guardrail that we previously discussed. As a reminder:

Q: Can my evaluators also be used to automatically fix or correct outputs in production?

LLMs

evals

faq

faq-individual

We recommend evaluating agentic workflows in two phases:

Q: How do I evaluate agentic workflows?

LLMs

evals

faq

faq-individual

While user feedback is a good way to narrow in on problematic traces, other methods are also useful. Here are three complementary approaches:

Q: How do I surface problematic traces for review beyond user feedback?

LLMs

evals

faq

faq-individual

LLMs can speed up parts of your eval workflow, but they can’t replace human judgment where your expertise is essential. For example, if you let an LLM handle all of error…

Q: What parts of evals can be automated with LLMs?

LLMs

evals

faq

faq-individual

Generally no. Eval-driven development (writing evaluators before implementing features) sounds appealing but creates more problems than it solves. Unlike traditional…

Q: Should I practice eval-driven development?

LLMs

evals

faq

faq-individual

Yes. Even with perfect models, you still need to verify they’re solving the right problem. The need for systematic error analysis, domain-specific testing, and monitoring…

Q: Will today’s evaluation methods still be relevant in 5-10 years given how fast AI is changing?

LLMs

evals

faq

faq-individual

Many developers fixate on model selection as the primary way to improve their LLM applications. Start with error analysis to understand your failure modes before considering…

Q: How much time should I spend on model selection?

LLMs

evals

faq

faq-individual

Build a custom annotation tool. This is the single most impactful investment you can make for your AI evaluation workflow. With AI-assisted development tools like Cursor or…

Q: Should I build a custom annotation tool or use something off-the-shelf?

LLMs

evals

faq

faq-individual

Automating prompt engineering can be tempting, but you should be skeptical of tools that promise to optimize prompts for you, especially in early stages of development. When…

Q: Should I stop writing prompts manually in favor of automated tools?

LLMs

evals

faq

faq-individual

Capture the complete user journey in your traces, including human handoffs. The trace continues until the user’s need is resolved or the session ends, not when AI hands off…

Q: How do I evaluate sessions with human handoffs?

LLMs

evals

faq

faq-individual

A Field Guide to Rapidly Improving AI Products

LLMs

Evaluation methods, data-driven improvement, and experimentation techniques from 30+ production implementations.

Mar 24, 2025

Building an Audience Through Technical Writing: Strategies and Mistakes

Writing

What I’ve seen work and what doesn’t.

Nov 30, 2024

Using LLM-as-a-Judge For Evaluation: A Complete Guide

LLMs

evals

A step-by-step guide with my learnings from 30+ AI implementations.

Oct 29, 2024

An Open Course on LLMs, Led by Practitioners

llms

courses

A free survey course on LLMs, taught by practitioners.

Jul 29, 2024

Debugging AI With Adversarial Validation

LLMs

Quickly detect a common bug in AI products using an automated technique.

Apr 12, 2024

Your AI Product Needs Evals

LLMs

evals

How to construct domain-specific LLM evaluation systems.

Mar 29, 2024

Is Fine-Tuning Still Valuable?

A reaction to a recent trend of disillusionment with fine-tuning.

Mar 27, 2024

Fuck You, Show Me The Prompt.

llms

Quickly understand inscrutable LLM frameworks by intercepting API calls.

Feb 14, 2024

How To Debug Axolotl

LLMs

fine-tuning

axolotl

Best practices for debugging axolotl with an example VSCode config.

Jan 11, 2024

Dokku: my favorite personal serverless platform

infra

severless

Like Heroku, but you own it.

Jan 9, 2024

Why Should ML Engineers Learn Kubernetes?

K8s

Learning K8s can give you an unreasonable advantage as an MLE and unblock your team.

Jan 16, 2023