Hamel’s Blog

The Revenge of the Data Scientist

LLMs

evals

Is Data Science in decline?

Mar 26, 2026

Evals Skills for Coding Agents

Evals

Teach your coding agent evals.

Mar 2, 2026

Why I Stopped Using nbdev

nbdev

Tools

What I learned about choosing tools in the age of AI.

Jan 18, 2026

Selecting The Right AI Evals Tool

Evals

An example of how to assess AI eval tools like Langsmith, Braintrust, and Arize Phoenix.

Oct 1, 2025

Q: Can my evaluators also be used to automatically fix or correct outputs in production?

LLMs

evals

faq

faq-individual

Latency and error costs determine whether an evaluator is suitable for the request path.

Jul 16, 2025

Q: Should I stop writing prompts manually in favor of automated tools?

LLMs

evals

faq

faq-individual

Manual prompt writing exposes requirements before automated optimization becomes useful.

Jul 12, 2025

Q: How can I efficiently sample production traces for review?

LLMs

evals

faq

faq-individual

Use a random baseline and targeted signals to build a production trace review batch.

Jul 6, 2025

Q: Should I use “ready-to-use” evaluation metrics?

LLMs

evals

faq

faq-individual

Generic evaluation metrics can create false confidence, but they can still help explore traces.

Jul 6, 2025

Q: Are similarity metrics (BERTScore, ROUGE, etc.) useful for evaluating LLM outputs?

LLMs

evals

faq

faq-individual

Generic similarity metrics often miss application-specific failures, but they can still help evaluate retrieval.

Jul 4, 2025

Q: What are LLM Evals?

LLMs

evals

faq

faq-individual

New readers can start with the main guides for product evaluation systems and LLM judges.

Jul 3, 2025

Q: How are evaluations used differently in CI/CD vs. monitoring production?

LLMs

evals

faq

faq-individual

CI evals use small test datasets, while production evals sample live traces.

Q: How do I evaluate agentic workflows?

LLMs

evals

faq

faq-individual

Check task success before using step-level diagnostics and transition failure matrices.

Q: What’s a minimum viable evaluation setup?

LLMs

evals

faq

faq-individual

Start evaluation with a small trace review led by one domain expert.

Q: What’s the difference between guardrails & evaluators?

LLMs

evals

faq

faq-individual

Guardrails block clear failures before users see them, while evaluators measure quality after a response.

Q: What’s your favorite eval vendor?

LLMs

evals

faq

faq-individual

Vendor support and fit matter more than comparing features that change quickly.

Q: Why is “error analysis” so important in LLM evals, and how is it performed?

LLMs

evals

faq

faq-individual

Open coding and axial coding turn reviewed traces into a failure taxonomy.

Jun 27, 2025

Q: How much of my development budget should I allocate to evals?

LLMs

evals

faq

faq-individual

Evaluation belongs in development work, and error analysis should guide spending on automated checks.

Jun 23, 2025

Q: What makes a good custom interface for reviewing LLM outputs?

LLMs

evals

faq

faq-individual

A custom review interface should present traces clearly and help reviewers find recurring failures.

Jun 18, 2025

Q: How should I approach evaluating my RAG system?

LLMs

evals

faq

faq-individual

Evaluate retrieval with search metrics and generation with validated application-specific judges.

Jun 10, 2025

Q: How do I choose the right chunk size for my document processing tasks?

LLMs

evals

faq

faq-individual

Output shape and context limits determine chunk size for document processing.

Jun 6, 2025

Q: How do I approach evaluation when my system handles diverse user queries?

LLMs

evals

faq

faq-individual

Observed failure patterns should guide eval criteria for systems that handle different query types.

Jun 5, 2025

Q: What gaps in eval tooling should I be prepared to fill myself?

LLMs

evals

faq

faq-individual

Eval tools often need custom support for pattern discovery and application-specific review workflows.

Jun 1, 2025

Q: What is the best approach for generating synthetic data?

LLMs

evals

faq

faq-individual

Handwritten dimension tuples can produce synthetic queries for testing full traces.

Jun 1, 2025

Q: How many people should annotate my LLM outputs?

LLMs

evals

faq

faq-individual

One domain expert can set quality standards, while multiple annotators need agreement checks.

May 31, 2025

Q: Should I build automated evaluators for every failure mode I find?

LLMs

evals

faq

faq-individual

Persistent failure modes may justify automated evaluators, while simpler failures can be fixed directly or checked with code.

May 30, 2025

Q: How do I debug multi-turn conversation traces?

LLMs

evals

faq

faq-individual

Find the first failure in a conversation trace and reduce it to the simplest useful test.

May 29, 2025

Q: Should I build a custom annotation tool or use something off-the-shelf?

LLMs

evals

faq

faq-individual

A custom annotation tool can fit product data and review workflows better than a generic tool.

May 29, 2025

Q: Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales)?

LLMs

evals

faq

faq-individual

Binary labels produce clearer judgments than rating scales, while separate checks can track gradual progress.

May 29, 2025

Q: Can I use the same model for both the main task and evaluation?

LLMs

evals

faq

faq-individual

Compare a judge with human labels before deciding whether to use a different model.

Q: How much time should I spend on model selection?

LLMs

evals

faq

faq-individual

Error analysis should come before switching models.

LLM Evals: Everything You Need to Know

LLMs

evals

A comprehensive guide to LLM evals, drawn from questions asked in our popular course on AI Evals. Covers everything from basic to advanced topics.

Q: Is RAG dead?

LLMs

evals

faq

faq-individual

RAG includes more than vector search, so test retrieval methods for the specific application.

A Field Guide to Rapidly Improving AI Products

LLMs

Evaluation methods, data-driven improvement, and experimentation techniques from 30+ production implementations.

Mar 24, 2025

Building an Audience Through Technical Writing: Strategies and Mistakes

Writing

What I’ve seen work and what doesn’t.

Nov 30, 2024

Using LLM-as-a-Judge For Evaluation: A Complete Guide

LLMs

evals

A step-by-step guide with my learnings from 30+ AI implementations.

Oct 29, 2024

An Open Course on LLMs, Led by Practitioners

llms

courses

A free survey course on LLMs, taught by practitioners.

Jul 29, 2024

Debugging AI With Adversarial Validation

LLMs

Quickly detect a common bug in AI products using an automated technique.

Apr 12, 2024

Your AI Product Needs Evals

LLMs

evals

How to construct domain-specific LLM evaluation systems.

Mar 29, 2024

Is Fine-Tuning Still Valuable?

A reaction to a recent trend of disillusionment with fine-tuning.

Mar 27, 2024

Fuck You, Show Me The Prompt.

llms

Quickly understand inscrutable LLM frameworks by intercepting API calls.

Feb 14, 2024

How To Debug Axolotl

LLMs

fine-tuning

axolotl

Best practices for debugging axolotl with an example VSCode config.

Jan 11, 2024

Dokku: my favorite personal serverless platform

infra

severless

Like Heroku, but you own it.

Jan 9, 2024

On commercializing nbdev

Jupyter

nbdev

Why I decided not to commercialize nbdev.

May 30, 2023

Why Should ML Engineers Learn Kubernetes?

K8s

Learning K8s can give you an unreasonable advantage as an MLE and unblock your team.

Jan 16, 2023