Q: How can I make AI outputs easier for people to evaluate?
Start by scrutinizing your product design. It’s often helpful to surface intermediate outputs users can check before a final result. For example, suppose you have an agent that writes a medical report by synthesizing a patient’s medical history. Instead of asking a doctor to provide feedback on the report, show the extracted facts with links to the source material and let doctors correct a fact or resolve conflicting evidence before generating the report. This also keeps the doctor involved and helps them build trust by checking the work as they go. This is a sketch of how such an interface might look:
For more discussion on designing for verification, see “It’s Hard to Eval” Is a Product Smell. The post expands on this example and discusses several others with before-and-after mockups.
After you have designed for verification, make sure the review interface removes friction from reviewing data. See the advice on building a review interface. Some common tips include:
- Display outputs in a familiar format. Render generated emails as emails, and use syntax highlighting for code.
- Keep the context reviewers need on the same screen. Put less important details in sections they can expand when needed.
- Add keyboard shortcuts for moving between examples and recording judgments. Make it easy to save notes without reaching for the mouse.
- Show progress, such as “45 of 100 examples reviewed,” so reviewers know how much work remains.
Next, debug the review process. First, try fewer examples so reviewers have time to inspect each one carefully. Have people review the same examples independently and discuss disagreements. You can also review examples together to see where people get stuck. Disagreement can reveal unclear instructions or missing information.
This article is part of our AI Evals FAQ, a collection of common questions (and answers) about LLM evaluation. View all FAQs or return to the homepage.