How to Improve Search Agents
This note covers Nandan Thakur’s session in the AI Product Engineering series.
Nandan Thakur’s talk covers three projects that are useful if you are building a search agent.
The first is ORBIT, a synthetic data pipeline you can use to bootstrap data for retrieval evals. Traditional synthetic data approaches generate questions from documents naively which generate questions that a single search can easily answer. ORBIT inverts the question instead. It describes an entity’s properties without naming it, then checks that each question is difficult enough to require multiple search hops to get right.
The ORBIT approach also verifies every question by requiring the search agent to confirm each claim against a source document. Additionally, an independent judge must reproduce the answer from those documents alone. The approach is completely synthetic, which makes it promising for anyone without labeled data. The ORBIT pipeline generated 20,000 verified questions without spending anything on search APIs or paid LLMs, and you can reuse the recipe on your own corpus.
The second is Hawkeye, a visual analytics interface for understanding agent trajectories (under review, no public artifact yet). It helps you catch errors in your search agents by showing how each run unfolds. In their analysis, correct runs needed far fewer search rounds than unresolved runs. A view like this lets you find the right threshold for max search rounds for your own agent. It’s a good idea to have your favorite coding agent build something similar for you.
He also introduced BrowseComp-Plus, a reproducible version of OpenAI’s BrowseComp, a benchmark of hard fact-finding questions where the answer is short and easy to check but takes many searches to find. An important finding from this work is that the retriever limits accuracy more than the model does. When the authors gave a model the documents that contain the answer, it answered almost every question. GPT-4.1 scored 14.6% when it had to find those documents with BM25 search, and 93.5% when it was handed them directly.
You can watch Nandan’s full talk here.
To learn how to build the evals that reveal failures like these, see the AI Evals course, a live cohort with hands-on exercises and office hours.