Per-million-token pricing keeps falling while AI bills keep rising. The reason is that token price is the wrong unit — here is the model I use to forecast and control real inference spend.
Token prices have dropped hard over the past two years, and plenty of teams have watched their AI bill go up anyway. That is not a contradiction. Cheaper tokens made it reasonable to use more of them: longer contexts, reasoning traces, multi-agent handoffs, retries, evaluation runs.
Per-token price is an input cost. It is not a unit anyone in the business can act on.
The unit that works
Cost per completed outcome: a resolved support ticket, an extracted document, a generated report, a qualified lead. Compute it as:
cost_per_outcome =
(prompt_tokens + completion_tokens) x price
x average_attempts_per_outcome
/ success_rateTwo terms in there usually dominate and almost never appear in cost dashboards: attempts and success rate. A pipeline with a 60% success rate and two retries costs more than three times its naive per-call price. Improving the prompt or the retriever is often a bigger cost lever than switching models.
Where spend actually concentrates
- Reasoning and agent traces. Invisible in the response, fully billed. Cap them.
- Long contexts held across turns. Cost grows with conversation length unless you summarize and truncate.
- Evaluation and regression runs. Legitimate spend, but it belongs in its own budget line, not mixed with production traffic.
- Self-hosted GPU idle time. If you run your own inference, utilization is the whole economic story. A dedicated GPU at 15% utilization is more expensive than an API for the same work — the break-even is usually somewhere north of 50-60% sustained utilization, and you should compute yours before committing to hardware.
Controls worth putting in place
- Tag every request with a feature and a tenant, so cost maps to something a product owner recognizes.
- Route by difficulty: small model first, escalate only on low confidence. Most traffic does not need your best model.
- Cap tokens per request and handle the cap as a normal outcome.
- Cache semantically, not exactly.
- Set a monthly ceiling per feature and alert at 70%, not at 100%.
Make the tradeoff explicit
Every cost decision is a quality decision. The right conversation is not "can we make this cheaper" but "what does a percentage point of accuracy cost us, and is it worth it for this feature". With cost per outcome in hand, that becomes a normal business decision instead of an engineering argument.
Drawing on Trust Insights' 2026 per-million-token cost report and Eyestech's AI inference and hardware TCO statistics.