Find Where Inference Latency Lives
This note covers Abi Aryan’s session in the AI Product Engineering series.
Abi Aryan’s session examines two stages of inference: prefill (reading the input) and decoding (generating each output token). Her Colab notebook runs three request shapes on the same model and prints the timing side by side.
Decoding is sequential: the model emits one token at a time. Prefill processes the whole input in parallel, so per-token it runs an order of magnitude faster. There are three common inference shapes:
- Long input, short output (RAG, classification, extraction): the friendliest shape. Prefill dominates and parallelizes well.
- Short input, long output (agent trajectories, code generation, long-form writing): the worst shape because latency is driven by decoding.
- Short input, short output (chat turns, single-turn queries): decode still dominates the wall clock, and prefill starves the GPU because 25 tokens are too few to saturate it. Inference engines like vLLM use continuous batching to pack many small requests into the same forward pass.
Abi’s advice: If output dominates, cut response length first. After that, try speculative decoding, a smaller model, or better hardware. If prefill dominates, batch requests and shorten context.
The notebook is built for tinkering. Here are some experiments worth trying:
- Batch size: run the same shapes at batch 4 and 8 to see how throughput per request improves and where memory caps you.
- Inference engine: swap the Hugging Face pipeline for llama.cpp or vLLM, which changes decode speed the most.
- Quantization: drop in an int8 or int4 model and watch weight memory shrink while throughput usually goes up.
You can watch Abi’s session here and open her notebook here.
To learn how to measure what matters in your AI product, see the AI Evals course, a live cohort with hands-on exercises and office hours.