More on the topic...
Generating detailed summary...
Failed to generate summary. Please try again.
In April 2026, Uber’s AI budget evaporated faster than expected. After rolling out Claude Code to all 5,000 engineers, monthly API bills hit $500–$2,000 per engineer. By month’s end, their entire annual AI budget was gone. That spike happened because agentic AI workflows aren’t single-shot chatbot prompts. Instead, each user task can trigger 10–20 model calls, each one reprocessing the same system prompts, tool definitions, state history and context. Gartner finds these loops use 5–30× more tokens per task than simple chat queries, so pilot costs never forecast the real production spend.
OpenAI’s CEO Sam Altman echoed this problem on CNBC in June 2026. Customers told him they’d already burned through their 2026 budgets. He called the ROI question “fair,” signaling that unchecked agentic AI spending is a problem even for deep-pocketed users. Token prices have dropped 98% since early 2024, but enterprise bills keep rising. That’s because cheaper tokens don’t matter when overall token consumption doubles or triples as agents plan, call tools, self-correct and loop repeatedly.
Teams also forget two hidden costs: re-sent context and context rot. Stanford’s 2025 research credits 62% of agent inference bills to re-sending the same prompts and history to the model on each call. On proprietary APIs, that shows up as token fees; on self-hosted models it appears as GPU memory pressure and slower throughput. Then there’s context rot—each additional token in the prompt degrades output quality. Chroma’s tests on 18 top models confirmed every one slows down and loses accuracy as context grows, forcing more retries and inflating costs further.
Finally, routing work to the cheapest suitable model can cut bills dramatically. One team trimmed its API spend from $40,000 to $24,000 a month just by sending simpler queries to smaller models. Gartner predicts inference on trillion-parameter models will be 90% cheaper by 2030. Yet firms shouldn’t expect those savings to translate directly into lower bills. Consumption growth and limited pass-through of price cuts mean unit-cost deflation won’t tame runaway agentic workloads.
Questions about this article
No questions yet.