A production RAG response is five or six sequential network hops. If you do not assign milliseconds to each one up front, you find out about them from users. Here is the budget I start from.
Most RAG systems are not slow because of one bad component. They are slow because nobody wrote down what "fast" meant, so every stage took as long as it wanted and the total landed at four seconds.
I now start every retrieval project with a latency budget on one page, before any code. It changes the architecture decisions more than any benchmark does.
A starting budget for a 1.5s first token
| Stage | Budget |
|---|---|
| Request handling, auth, validation | 20 ms |
| Query embedding | 40 ms |
| Vector search (top-k) | 60 ms |
| Rerank (cross-encoder, top-20) | 150 ms |
| Prompt assembly | 20 ms |
| LLM time-to-first-token | 600 ms |
| Network and streaming overhead | 100 ms |
| Headroom | 510 ms |
The headroom is the point. Without it, one slow p99 in any stage blows the whole response. If a stage cannot fit its line, that is a design decision to make now — a smaller reranker, a faster embedding model, a colocated vector store — not a production incident to discover later.
Things that quietly break the budget
Cold embedding endpoints. A managed embedding API with no warm pool can spike to several hundred milliseconds. Either keep it warm or run the model yourself.
Cross-region hops. Vector store in one region, model in another, app in a third. Three round trips of 80 ms each is 240 ms you will never get back. Colocate.
Reranking too many candidates. Retrieving 100 and reranking all of them is a common default and rarely worth it over reranking 20. Measure the recall difference before paying for it.
Serial retrieval when parallel would do. Hybrid search (dense plus keyword) should fire both queries at once. Many implementations do them in sequence out of habit.
Streaming as an afterthought. Time to first token is what users feel. A pipeline that streams from the first generated token feels twice as fast as one that buffers the full answer, at identical total cost.
Measure the stages, not the endpoint
Instrument each stage separately and alert on p95 per stage. A single end-to-end latency metric tells you something got slower; per-stage metrics tell you what, and that difference is usually the whole debugging session.
Set the budget first. Then every later optimization argument has a number to settle it.
Informed by the technovice.net latency-budget writeup, GMI Cloud's notes on RAG infrastructure, and the GROUNDED framework guide for production RAG architecture.