Case Study: Putting Evals Into Production
This note covers Lucas Machado Rocha’s session in the AI Product Engineering series.
Lucas used evals to optimize a lesson planner for Nova Escola, a Brazilian nonprofit education platform with ~1M monthly users and 200,000 teachers. His talk covers the full process. Two mistakes his team made:
- He wrote a rubric before doing error analysis. Since most of his criteria never appeared as failures, his team spent time labeling data they did not need.
- He built an eval for a problem that needed a simple fix. The assistant sometimes emitted two learning goals when it should emit one. His team changed the prompt and the problem stopped. We discuss this tradeoff in the FAQ on when to automate an evaluator.
The labeling process exposed another problem. Lucas used two annotators, and they agreed less often than chance. The team had not defined what good looked like clearly enough, so they rewrote the rubric with their pedagogical experts and relabeled the data.
After his team identified the failure modes through error analysis, Lucas used the eval skills to automate parts of the work that followed, including writing judges and validating them against human labels.
Lucas’s end-to-end example shows his entire process. He catches regressions early by running a suite of evals daily against 2% of production traffic.
Want to learn more about AI Evals? Check out our AI Evals course. It’s a live cohort with hands on exercises and office hours.