Q: How often should I run my evals?
There are three dimensions to consider:
- The cost to run and maintain the eval. The more expensive the eval, the greater benefit it needs to provide to justify running it frequently. For example, LLM-as-a-judge is more expensive to run than a unit test.
- How saturated the eval is on the dataset. If the eval passes all examples, its giving you no new information. You should consider retiring the eval or running it less frequently if its saturated. However, you should first try to make the eval more difficult so its not saturated to begin with.
- The business value of catching this error. For critical errors, the busines value of catching it may be high enough that you should run it more frequently, despite its cost. One caveat here is not to get carried away with hypothetical errors. At the very least, you should prove that you can trigger the error at least once by red-teaming your application before implementing the eval (which will also help you make a better eval)
There are no bright-line rules. This decision often requires judgement as opposed to something formulaic. Here’s a visual that can help you think through the tradeoffs:
Examples
Below are concrete examples to help you understand the factors involved. Note that these are illustrative:
| Eval | Test examples passing | Business cost of failure | Suggested schedule |
|---|---|---|---|
| An answer-quality judge with GPT-6 Astra on max reasoning. | All pass | Medium | Retire or run infrequently (e.g. every 2 weeks) |
| A code assertion which checks that a contact was saved correctly. | All pass | Medium | You can run this on every change b/c its incredibly cheap. |
| A judge with Fable 5 checks whether a support agent follows a new refund policy. | None pass | Medium | Even though expensive, the eval is providing useful feedback b/c nothing is passing, and the business value of catching the error is high enough. I would run this as frequently as possible. |
| A judge with GPT-6 Luna checks whether a support agent resolves the customer’s problem. | Some pass | Medium | Not a terribly expensive judge b/c model is smaller and the eval is still catching errors, so I would run this somewhat frequently (e.g. nightly). |
| A judge with GPT-6 Astra on max reasoning checks for improper disclosure of confidential information on a legal assistant. | All pass | High | Even though expensive and eval is saturated, the business value of catching the error is high enough that I would run this prior to each release. Given the importance of the error, I would also try to make the eval more difficult so that it’s more useful. |
| A judge with GPT-6 Astra on medium reasoning checks a minor formatting preference. | Some pass | Low | Occasionally or retire; use code instead if possible. |
It’s always worth exploring cheaper evaluators to see if you can find one that provides similar or better alignment with human labels for less cost.
Offline vs. Online Evals
The discussion here focused on offline evals. Online evals involve similar considerations, with an additional decision about how many production traces to sample. For example, you might run cheap checks on every trace and an expensive judge on a nightly sample. For more discussion on how these approaches work together, see How are evaluations used differently in CI/CD vs. monitoring production?
This article is part of our AI Evals FAQ, a collection of common questions (and answers) about LLM evaluation. View all FAQs or return to the homepage.