Q: How do I evaluate outputs in a language I don’t speak?
Even though you can translate interactions in a foreign language with an LLM, be cautious about relying on it. Translation often loses meaning as some words and expressions have no direct equivalent. Moreover, what “good” means often depends on culture and social norms. For example, understanding the literal meaning of an answer is not enough to judge whether its tone is appropriate in a different cultural frame.
Because of these limitations, we recommend involving a reviewer who understands both the language and the cultural context of your users. You can still contribute to evals by helping organize the review and investigating problems you can identify yourself. That reviewer should set the standard for error analysis.
If you cannot find a reviewer who understands both the language and cultural context, be aware that your assessment will be limited.
This article is part of our AI Evals FAQ, a collection of common questions (and answers) about LLM evaluation. View all FAQs or return to the homepage.