Agent loops multiply retrieval calls, and the bill grows faster than the answer quality. How I cap loops, cache aggressively, and decide when a single-shot retriever is the better engineering call.
Agentic RAG looks like a straight upgrade over classic retrieve-then-generate. The agent reasons about what it is missing, retrieves again, critiques, retrieves again. Answer quality usually does improve. What is easy to miss in a demo is that every one of those turns is a full embedding call, a vector search, a rerank, and a generation pass over a context window that keeps growing.
The failure mode I see most often in production is not a wrong answer. It is a support assistant that quietly costs eight times what the finance model assumed, because the median question triggers one retrieval and the tail triggers eleven.
Where the money actually goes
Three things compound at once:
- Token growth per turn. Each retrieval appends documents to context. Turn five is not five times the cost of turn one — it is worse, because every prior turn's retrieved text is still in the prompt.
- Reranking. A cross-encoder rerank over 50 candidates is cheap once and expensive twenty times.
- Unbounded loops. Without a hard stop, a model that cannot find an answer will keep looking. The questions your retriever cannot answer are exactly the ones that cost the most.
What I do about it
Cap the loop, and make the cap visible. Two to three retrieval rounds, then the agent must answer with what it has or say it does not know. "I could not find this" is a cheap, honest answer and a good signal for what to fix in your index.
Route by difficulty, not by default. Most queries do not need an agent. A classifier or a simple confidence check on the first retrieval decides whether to escalate. Adaptive routing between a small and a large model for the generation step is the same idea applied to the other half of the bill.
Cache at the semantic layer. Exact-match caching barely helps in natural language. Embedding the query and matching against recent queries above a similarity threshold does, especially for internal tools where the same twenty questions dominate.
Trim context between turns. Carry forward a summary of what was already retrieved, not the raw chunks. This alone often halves cost on longer loops.
Budget per request, not per month. Attach a token ceiling to each request and make exceeding it a handled outcome rather than a surprise on the invoice.
The measurement that matters
Track cost per resolved question, not cost per call. A single-shot retriever that answers 70% of questions at one unit of cost and escalates the rest to a human can easily beat an agent that answers 85% at nine units. Which one wins is an arithmetic question about your support economics, not an architecture preference.
Build the cheap path first, measure where it fails, and spend loops only on the queries that earn them.
Background reading that informed this post: the dev.to writeup on agentic RAG retrieval-loop cost, Trust Insights' per-million-token pricing report, and the Towards Data Science piece on adaptive model routing in multi-agent systems.