AI Product Engineering Notes
AI product engineering is the work of turning a model into a useful product. While evals set the foundation for improving an AI product, there are many knobs to turn, each with different trade-offs.
The lowest hanging fruit is to optimize retrieval and context. Once that’s done, you should improve your systems and harness. Finally, consider post-training only after the other approaches are exhausted.
I hosted an AI Product Engineering series to explore some of these items. I’ve summarized all 13 sessions organized by theme, with links to the source materials. Warning: I’ve tried to pull the most important ideas from each talk, so some notes are short (9.5 hours of sessions comes out to about 20 minutes of reading).
Notes
| Category | | Title | Notes |
|---|---|---|---|
| Evals |
|
Evals for Data Agents | A data-agent benchmark that recreates messy data warehouses. The concepts explored can be applied to your own data agents. |
| Evals |
|
Cut Classification Costs With a Model Cascade | A technique to route classification tasks to a small model when it’s confident and to a large model when it’s not, while preserving the large model’s accuracy. |
| Evals |
|
Automating Error Analysis | Use AI to help automate error analysis and quickly find production issues, borrowing ideas from active learning. |
| Evals |
|
Case Study: Putting Evals Into Production | A case study on how an education startup deployed evals to find issues with their lesson-planning assistant. |
| Context |
|
An Intro to Multivector Retrieval | Learn about multi-vector embeddings and how they can be used to improve retrieval quality. |
| Context |
|
How to Improve Search Agents | Eval strategies for search agents along with analytical tools you can use to identify inefficiencies. |
| Context |
|
How to Optimize Retrieval Embeddings | How to choose an embedding model for your use case. Additionally, when and how to fine-tune an embedding model. |
| Context |
|
How to Choose an OCR Model | Surprising ways OCR fails on documents, and how to make the trade-offs between accuracy and cost. |
| Context |
|
Steer AI Writing With Footnotes for Agents | Subtext is a special kind of footnote for agents that can steer AI-assisted writing effectively. |
| Systems |
|
Debugging Inference Latency | Learn what drives inference latency and how to find bottlenecks. |
| Systems |
|
Open Weight Model Economics | An overview of open-weight model economics, and when you should self-host vs. use an API. |
| Systems |
|
The Case for Agent Sandboxes | Why you probably should provide your agent with a sandbox, and how to design them for developer productivity. |
| Systems |
|
Turn Eval Results Into a Better Model | When you should leverage your evals to post-train your own model, and tips for doing so. |
Nearly every improvement in these notes starts with good evals. If you want to work through evals with a live group, the AI Evals course has hands-on exercises and office hours.