An Intro to Multivector Retrieval
This note covers Marek Galovic’s session in the AI Product Engineering series.
Most retrieval systems compress each document into a single vector, which becomes an information bottleneck. Multivector retrieval avoids this bottleneck (but incurs a cost) by storing one vector per token for both the query and the document. Each query token is matched against its best document token, and those token scores are summed to score the document. This scoring operator is called MaxSim. My notes on late interaction cover why this approach is something you should try.
This slide from Marek’s talk shows the idea. Each cell compares one query token to one document token, and the shaded cells are the best matches that MaxSim sums.
However, there is a tradeoff. Since you are storing and comparing vectors for every token in both the query and the document, compute and storage costs grow fast. Marek estimates MaxSim costs roughly 2,000x the arithmetic of a single dot product and 10 to 100x the storage, which is why it stayed in research papers.
Marek discusses a set of optimizations that bring those costs down enough to run at scale. The main one is SMVE. SMVE projects each token onto a large set of random directions and keeps only the top eight values. Summing those per document produces one mostly empty vector, which goes into an inverted index. Documents that share no entries with the query are never scored, and exact MaxSim runs only on the survivors. Marek reported p99 latency under 100 ms at one billion documents.
His team also released Iso-ModernColBERT, a corrected version of GTE-ModernColBERT-v1 whose embedding geometry works with SMVE’s random projections. It also runs about 3x faster at bf16 precision with almost no loss in ranking quality.
You can watch Marek’s full talk here.
To learn how to measure retrieval quality on your own data, see the AI Evals course, a live cohort with hands-on exercises and office hours.