AI Product Engineering Notes
For the last three years, my work has centered on evals. While evals set the foundation for improving an AI product, there are many knobs to turn, each with different trade-offs.
The lowest-hanging fruit is to optimize retrieval and context. Once that’s done, you should improve your systems and harness. Finally, consider post-training your own model after other approaches are exhausted.
I hosted an AI Product Engineering series to explore some of these items. I’ve summarized all 13 sessions organized by theme, with links to the source materials. Warning: I’ve tried to pull the most important ideas from each talk, so some notes are short (9.5 hours of sessions comes out to about 20 minutes of reading).
Notes
| Category | | Title | Notes |
|---|---|---|---|
| Evals |
|
Evals for Data Agents | A data-agent benchmark that recreates messy data warehouses. The concepts explored can be applied to your own data agents. |
| Evals |
|
Cut Classification Costs With a Model Cascade | A technique to route classification tasks to a small model when it’s confident and to a large model when it’s not, while preserving the large model’s accuracy. |
| Evals |
|
Automating Error Analysis | Use AI to help automate error analysis and quickly find production issues, borrowing ideas from active learning. |
| Evals |
|
Case Study: Putting Evals Into Production | A case study on how an education startup deployed evals to find issues with their lesson-planning assistant. |
| Context |
|
An Intro to Multivector Retrieval | Learn about multi-vector embeddings and how they can be used to improve retrieval quality. |
| Context |
|
How to Improve Search Agents | Eval strategies for search agents along with analytical tools you can use to identify inefficiencies. |
| Context |
|
How to Optimize Retrieval Embeddings | How to choose an embedding model for your use case. Additionally, when and how to fine-tune an embedding model. |
| Context |
|
How to Choose an OCR Model | Surprising ways OCR fails on documents, and how to make the trade-offs between accuracy and cost. |
| Context |
|
Steer AI Writing With Footnotes for Agents | Subtext is a special kind of footnote for agents that can steer AI-assisted writing effectively. |
| Systems |
|
Debugging Inference Latency | Learn what drives inference latency and how to find bottlenecks. |
| Systems |
|
Open Weight Model Economics | An overview of open-weight model economics, and when you should self-host vs. use an API. |
| Systems |
|
The Case for Agent Sandboxes | Why you probably should provide your agent with a sandbox, and how to design them for developer productivity. |
| Systems |
|
Turn Eval Results Into a Better Model | When you should leverage your evals to post-train your own model, and tips for doing so. |
Nearly every improvement in these notes starts with good evals. If you want to work through evals with a live group, the AI Evals course has hands-on exercises and office hours.