Evals for Data Agents
This note covers Shreya Shankar’s session in the AI Product Engineering series.
Data agents answer business questions like “which cohort had the highest churn?” that a data analyst would normally answer. Even though these questions are a large share of knowledge work, good benchmarks for them are scarce.
Shreya Shankar’s talk introduces the Data Agent Benchmark (DAB). The authors recreate the mess of a real data warehouse. Each task spreads data across at least two database systems, with inconsistent join keys, key values buried in free text, and ambiguous or ill-defined schemas. The table below shows how DAB compares to related benchmark categories.
While this general benchmark answers a different question than product evals, the failure analysis is useful if you are building a data agent.
Surprisingly, data agents failed the most for incorrect plans and implementations rather than wrong data selection. The study found that agents often wrote a plan first and stuck to it even after the data contradicted it. These failure modes are worth keeping in mind while debugging your own data agents.
The benchmark is open and accepts submissions to its leaderboard. The team refreshes it as new models come out, so it is worth watching.
Want to learn more about AI Evals? Check out our AI Evals course. It’s a live cohort with hands on exercises and office hours.