Hamel’s Blog

WarningThis page is supposed to be secret!

A listing of all my blog posts can be found here

Claude’s new auto eval tool

AI
Evals

Reflections on trying Anthropic’s new eval plugin for Claude.

Sep 30, 2026
Hamel Husain

AI Evals: Everything You Need to Know

LLMs
evals

A guide to AI and LLM evals for engineers and product managers. Learn how to test AI systems, analyze failures, and improve retrieval, agents, and AI products.

Sep 18, 2026
Hamel Husain, Shreya Shankar

Q: How many examples do I need for an eval?

LLMs
evals
faq
faq-individual

Start with 100 traces for error discovery. Code-based evals need coverage of their logic, while LLM judges need 100 to 200 labeled examples per failure mode.

Sep 1, 2026
Hamel Husain, Shreya Shankar

“It’s Hard to Eval” Is a Product Smell

LLMs
evals

Your product should make it easy to verify AI outputs.

Jun 29, 2026
Hamel Husain

The Revenge of the Data Scientist

LLMs
evals

AI eval mistakes and the data science practices that fix them.

Mar 26, 2026
Hamel Husain

Evals Skills for Coding Agents

AI
Evals

Teach your coding agent how to run product evals.

Mar 2, 2026
Hamel Husain, Shreya Shankar

Why I Stopped Using nbdev

AI
nbdev
Tools

What I learned about choosing tools in the age of AI.

Jan 18, 2026
Hamel Husain

Selecting The Right AI Evals Tool

AI
Evals

An example of how to assess AI eval tools like Langsmith, Braintrust, and Arize Phoenix.

Oct 1, 2025
Hamel Husain

Q: Can my evaluators also be used to automatically fix or correct outputs in production?

LLMs
evals
faq
faq-individual

Latency and error costs determine whether an evaluator is suitable for the request path.

Jul 16, 2025
Hamel Husain, Shreya Shankar

Q: Should I stop writing prompts manually in favor of automated tools?

LLMs
evals
faq
faq-individual

Manual prompt writing exposes requirements before automated optimization becomes useful.

Jul 12, 2025
Hamel Husain, Shreya Shankar

Q: How can I efficiently sample production traces for review?

LLMs
evals
faq
faq-individual

Use a random baseline and targeted signals to build a production trace review batch.

Jul 6, 2025
Hamel Husain, Shreya Shankar

Q: Should I use “ready-to-use” evaluation metrics?

LLMs
evals
faq
faq-individual

Generic evaluation metrics can create false confidence, but they can still help explore traces.

Jul 6, 2025
Hamel Husain, Shreya Shankar

Q: Are similarity metrics (BERTScore, ROUGE, etc.) useful for evaluating LLM outputs?

LLMs
evals
faq
faq-individual

Generic similarity metrics often miss application-specific failures, but they can still help evaluate retrieval.

Jul 4, 2025
Hamel Husain, Shreya Shankar

Q: What are AI Evals?

LLMs
evals
faq
faq-individual

AI evals measure whether an AI system works for its users on realistic tasks and data.

Jul 3, 2025
Hamel Husain, Shreya Shankar

Q: How are evaluations used differently in CI/CD vs. monitoring production?

LLMs
evals
faq
faq-individual

CI evals use small test datasets, while production evals sample live traces.

Jun 29, 2025
Hamel Husain, Shreya Shankar

Q: How do I evaluate agentic workflows?

LLMs
evals
faq
faq-individual

Check task success before using step-level diagnostics and transition failure matrices.

Jun 29, 2025
Hamel Husain, Shreya Shankar

Q: What’s a minimum viable evaluation setup?

LLMs
evals
faq
faq-individual

Start evaluation with a small trace review led by one domain expert.

Jun 29, 2025
Hamel Husain, Shreya Shankar

Q: What’s the difference between guardrails & evaluators?

LLMs
evals
faq
faq-individual

Guardrails block clear failures before users see them, while evaluators measure quality after a response.

Jun 29, 2025
Hamel Husain, Shreya Shankar

Q: What’s your favorite eval vendor?

LLMs
evals
faq
faq-individual

Vendor support and fit matter more than comparing features that change quickly.

Jun 29, 2025
Hamel Husain, Shreya Shankar

Q: Why is “error analysis” so important in AI evals, and how is it performed?

LLMs
evals
faq
faq-individual

Open coding and axial coding turn reviewed traces into a failure taxonomy.

Jun 27, 2025
Hamel Husain, Shreya Shankar

Q: How much of my development budget should I allocate to evals?

LLMs
evals
faq
faq-individual

Evaluation belongs in development work, and error analysis should guide spending on automated checks.

Jun 23, 2025
Hamel Husain, Shreya Shankar

Q: What makes a good custom interface for reviewing LLM outputs?

LLMs
evals
faq
faq-individual

A custom review interface should present traces clearly and help reviewers find recurring failures.

Jun 18, 2025
Hamel Husain, Shreya Shankar

Q: How should I approach evaluating my RAG system?

LLMs
evals
faq
faq-individual

Evaluate retrieval with search metrics and generation with validated application-specific judges.

Jun 10, 2025
Hamel Husain, Shreya Shankar

Q: How do I choose the right chunk size for my document processing tasks?

LLMs
evals
faq
faq-individual

Output shape and context limits determine chunk size for document processing.

Jun 6, 2025
Hamel Husain, Shreya Shankar

Q: How do I approach evaluation when my system handles diverse user queries?

LLMs
evals
faq
faq-individual

Observed failure patterns should guide eval criteria for systems that handle different query types.

Jun 5, 2025
Hamel Husain, Shreya Shankar

Q: What gaps in eval tooling should I be prepared to fill myself?

LLMs
evals
faq
faq-individual

Eval tools often need custom support for pattern discovery and application-specific review workflows.

Jun 1, 2025
Hamel Husain, Shreya Shankar

Q: What is the best approach for generating synthetic data?

LLMs
evals
faq
faq-individual

Handwritten dimension tuples can produce synthetic queries for testing full traces.

Jun 1, 2025
Hamel Husain, Shreya Shankar

Q: How many people should annotate my LLM outputs?

LLMs
evals
faq
faq-individual

One domain expert can set quality standards, while multiple annotators need agreement checks.

May 31, 2025
Hamel Husain, Shreya Shankar

Q: Should I build automated evaluators for every failure mode I find?

LLMs
evals
faq
faq-individual

Persistent failure modes may justify automated evaluators, while simpler failures can be fixed directly or checked with code.

May 30, 2025
Hamel Husain, Shreya Shankar

Q: How do I debug multi-turn conversation traces?

LLMs
evals
faq
faq-individual

Find the first failure in a conversation trace and reduce it to the simplest useful test.

May 29, 2025
Hamel Husain, Shreya Shankar

Q: Should I build a custom annotation tool or use something off-the-shelf?

LLMs
evals
faq
faq-individual

A custom annotation tool can fit product data and review workflows better than a generic tool.

May 29, 2025
Hamel Husain, Shreya Shankar

Q: Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales)?

LLMs
evals
faq
faq-individual

Binary labels produce clearer judgments than rating scales, while separate checks can track gradual progress.

May 29, 2025
Hamel Husain, Shreya Shankar

Q: Can I use the same model for both the main task and evaluation?

LLMs
evals
faq
faq-individual

Compare a judge with human labels before deciding whether to use a different model.

May 28, 2025
Hamel Husain, Shreya Shankar

Q: How much time should I spend on model selection?

LLMs
evals
faq
faq-individual

Error analysis should come before switching models.

May 28, 2025
Hamel Husain, Shreya Shankar

Q: Is RAG dead?

LLMs
evals
faq
faq-individual

RAG includes more than vector search, so test retrieval methods for the specific application.

May 28, 2025
Hamel Husain, Shreya Shankar

A Field Guide to Rapidly Improving AI Products

LLMs
AI

Evaluation methods, data-driven improvement, and experimentation techniques from 30+ production implementations.

Mar 24, 2025
Hamel Husain

Building an Audience Through Technical Writing: Strategies and Mistakes

Writing

What I’ve seen work and what doesn’t.

Nov 30, 2024
Hamel Husain

Using LLM-as-a-Judge For Evaluation: A Complete Guide

LLMs
evals

A step-by-step guide to building an LLM judge and validating it against human labels.

Oct 29, 2024
Hamel Husain

An Open Course on LLMs, Led by Practitioners

llms
courses

A free survey course on LLMs, taught by practitioners.

Jul 29, 2024
Hamel Husain

Debugging AI With Adversarial Validation

LLMs

Quickly detect a common bug in AI products using an automated technique.

Apr 12, 2024
Hamel Husain

Your AI Product Needs Evals

LLMs
evals

How to construct domain-specific LLM evaluation systems.

Mar 29, 2024
Hamel Husain
 

Is Fine-Tuning Still Valuable?

A reaction to a recent trend of disillusionment with fine-tuning.

Mar 27, 2024
Hamel Husain

Fuck You, Show Me The Prompt.

llms
ml

Quickly understand inscrutable LLM frameworks by intercepting API calls.

Feb 14, 2024
Hamel Husain

How To Debug Axolotl

LLMs
fine-tuning
axolotl

Best practices for debugging axolotl with an example VSCode config.

Jan 11, 2024
Hamel Husain

Dokku: my favorite personal serverless platform

infra
severless

Like Heroku, but you own it.

Jan 9, 2024
Hamel Husain

On commercializing nbdev

Jupyter
nbdev

Why I decided not to commercialize nbdev.

May 30, 2023
Hamel Husain

Why Should ML Engineers Learn Kubernetes?

K8s

Learning K8s can give you an unreasonable advantage as an MLE and unblock your team.

Jan 16, 2023
Hamel Husain
No matching items