Q: Can I help with evals if I’m not a domain expert?
Yes, especially when you’re beginning with evals. I’m often surprised by the number of low-hanging fruit I find while reviewing data that don’t require domain knowledge. For example, I’ve found issues like this in specialized domains as an outsider:
- Text message chatbots getting confused by the conversational flow of lots of short, broken-up messages people tend to write in text versus chat.
- Lack of query disambiguation or follow-up when users’ requests are obviously vague.
- Not having proper instrumentation, logging or traces to begin with.
- Lack of widgets, UI elements or other affordances that help users complete tasks versus over-reliance on text responses.
Furthermore, ask a domain expert to walk through an example and explain why it is good or bad. Watch what they check and which evidence they need. Use what you learn to build a better annotation interface that makes reviewing easier.
You can also help the team collect interactions and review them regularly. For example, see how product managers and engineers can collaborate on error analysis to get an idea of how to structure cross-functional collaboration.
Lastly, make sure you leave judgments that require specialized knowledge to the expert. However, don’t assume you need domain expertise to start being useful!
This article is part of our AI Evals FAQ, a collection of common questions (and answers) about LLM evaluation. View all FAQs or return to the homepage.