Choose and Fine-Tune an Embedding Model
This note covers Radu Gheorghe’s session in the AI Product Engineering series.
Radu Gheorghe walks through how to pick an embedding model for retrieval. He starts on the MTEB leaderboard, then warns that it ranks quality and says little about costs. He then walks through several trade-offs worth considering. Here are important ones:
- Model quantization (the precision of the model weights): on a CPU, an INT8 model runs about 3x faster than FP32 while keeping most of the quality. On a GPU, use FP16 instead.
- Vector precision (how you store the output vectors): going from FP32 to binary cuts storage by 32x, and bfloat16 gives you a 2x cut with no measurable quality loss.
- Matryoshka models (embedders you can truncate): these models let you drop later dimensions and keep most of the quality, which saves storage and speeds up search.
- Performance on your data (what the leaderboard cannot show): test on your own data, because rankings shuffle on specialized domains. The common gaps are multilingual support and context length.
When a model still falls short on your domain, an overlooked optimization is to fine-tune it. Fine-tuning an embedder is far easier than fine-tuning an LLM, and it often has a bigger impact. My favorite part of the talk was VespaEmbed, Radu’s open source fine-tuning tool. VespaEmbed is Apache 2.0 and needs no code. Pick a base model from Hugging Face, upload your pairs, choose a loss, hit train. The setup screen below shows those steps.
You can try VespaEmbed here.
You can watch Radu’s full talk here and his deck is available here.
To learn how to build the evals that guide choices like this, see the AI Evals course, a live cohort with hands-on exercises and office hours.