Q: What model or LLM should I use to build automated evals?
First check whether you can test the condition with code assertions. For example, suppose an AI assistant manages your contacts, and you want to test whether it creates a contact when asked. To test this functionality, you can give it a new contact to create, then query the database to check that exactly one matching record exists with the requested details. Using code assertions avoids the need for human labels.
When a check requires judgment, use an LLM or another machine learning classifier. When using an LLM judge, we recommend using it as a classifier that returns Pass or Fail for the error you want to catch. Whichever model you use, validate it against human labels before trusting its decisions.
For example, you could try Jev from TypeSafe, BERT, or logistic regression. A different model may be cheaper or faster, and it may agree more or less closely with human labels. Measure these differences on your data to find the model that meets your application’s needs. For example, you might accept slower evaluations if they catch costly failures, or prefer a faster model when you need immediate feedback.
When using an LLM, starting with a powerful model can make it easier to develop the judge’s prompt. Once it works well, try smaller, cheaper models and measure how much accuracy you lose. You can also use the same model as your application.
An agent can help optimize the judge’s prompt once you have defined the task and labeled examples. Give it a specific failure to detect and a way to measure progress against your labels. “Find all errors and keep improving” is too vague. The agent needs to know what counts as an error and how to tell whether a change helped. Keep a separate test set outside the optimization process to check if the judge generalizes to examples it was not tuned against.
This article is part of our AI Evals FAQ, a collection of common questions (and answers) about LLM evaluation. View all FAQs or return to the homepage.