More on the topic...
Generating detailed summary...
Failed to generate summary. Please try again.
In April 2026, Uber’s CTO Praveen Neppalli Naga admitted their annual AI budget vanished after ramping Claude Code use from 32% to 84% of 5,000 engineers in three months. Engineers were burning $500–$2,000 per head each month on API calls. Agentic workflows—where a model plans, fetches data, invokes tools and validates results—use five to thirty times more tokens per task than a simple chatbot query. That multiplier turns per-token pricing on its head: production costs explode compared to pilot estimates.
OpenAI CEO Sam Altman has heard the same story from customers who blew through their 2026 budgets already. Even though token prices have dropped 98% since early 2024, total bills keep climbing. Gartner’s data shows an agentic loop can fire off 10–20 inferences for one user request, so the real unit of cost isn’t “per prompt” anymore but “per completed task.” Lower token prices, they warn, won’t lower your final invoice if you’re running multi-step agents thousands of times a day.
Beyond raw inference fees, four hidden layers drive costs higher. First is re-sent context: Stanford’s Digital Economy Lab found 62% of agent inference spend comes from models re-reading the same system prompts, tool definitions and state histories across steps. Second is context rot—Chroma’s 2025 tests revealed every leading transformer model degrades in accuracy as input length grows, forcing teams to trim or segment context more often. Third and fourth layers—tool orchestration and state management—bring their own API calls, validation loops and data-storage bills, all multiplying complexity and spend.
Gartner predicts one-trillion-parameter model inference will be 90% cheaper by 2030, but falling unit costs won’t tame runaway consumption. As Gartner analyst Will Sommer puts it, cheaper tokens don’t equal frontier reasoning for every user. Teams that audit token use, route simple subtasks to smaller models, and eliminate redundant context slashed one client’s monthly spend from $40,000 to $24,000—no product changes required. Without that discipline, every AI program risks a budget blowout once it scales.
Questions about this article
No questions yet.