The Cloud Bill Nobody Forecast: Why AI Turned FinOps Upside Down

AzeniQ Engineering Team· October 2, 2026· 8 min read

For several years the cloud waste figure moved in the right direction. Estimated wasted spend on IaaS and PaaS fell from 28% in 2023 to 27% in 2024, then held flat through 2025. Teams that had spent those years building tagging policies and rightsizing pipelines could point at the trend as evidence that the discipline was working.

The Flexera 2026 State of the Cloud Report, published in March 2026 and based on 753 practitioner responses, shows a reversal: wasted spend on IaaS and PaaS has reached 29%, the first rise in five years. Flexera attributes the increase to the surge in cloud based AI workloads and to the cost complexity of the newer platform services that surround them.

This article looks at how the shape of cloud waste has changed, why published GPU utilization benchmarks disagree, what continuous batching does and does not achieve, the telemetry gap that leaves inference cost unattributed, and the controls that reduce the exposure.

Estimated wasted IaaS and PaaS spend · Flexera, 2023 to 2026

Spend concentration

The reversal does not mean FinOps practice declined. 63% of organizations now run a dedicated FinOps team, the highest share Flexera has recorded, and 71% operate a cloud center of excellence. What changed is the shape of the thing being managed.

Cloud waste used to be spread thin across individually cheap resources: oversized instances running at 20% of CPU, Kubernetes nodes idling with no obvious owner, development environments left running over the weekend, unattached storage volumes outliving the servers they were created for. None of these demanded urgent action on its own, and FinOps tooling grew up to find and trim thousands of small inefficiencies.

AI inverted that distribution. An idle general purpose instance costs a few dollars a week. An idle H100 instance costs five figures a year, attached to a project nobody wants executive attention on. In the State of FinOps 2026 data, GPUs account for roughly 18% of spend at AI focused companies, up from about 4% in 2023. The waste did not shrink; it concentrated.

Utilization layers

Published utilization benchmarks disagree, and the disagreement confuses teams trying to benchmark their own estates. Cast AI's 2026 analysis reported average GPU utilization of roughly 5% across the non optimized Kubernetes clusters in its dataset. Weights & Biases found nearly a third of training workloads below 15%. A Microsoft study of a multi tenant DNN training cluster measured about 47.7% of in use GPU cycles as wasted. Guidance for a well tuned training job targets 70% to 85%.

These figures are not contradictory. They measure three different layers, each with a different owner and a different remedy.

Layer 1 · Fleet
Is a workload scheduled on it at all?
Capacity commitment and scheduling policy. The lowest published figures come from this layer; it is a procurement and scheduling problem, not an engineering one.
Owner: Platform and procurement
Layer 2 · Job
Is the run actually making progress?
Queueing, failed runs, checkpoint restarts, and stragglers decide how much wall clock time is productive.
Owner: ML platform
Layer 3 · Within job
Is the device genuinely saturated?
Batching, kernel efficiency, and memory layout decide how much of the silicon does useful work while a kernel runs.
Owner: Model engineering

At the within job layer the familiar GPU-Util metric misleads: it reports whether a kernel was running, not whether the kernel filled the device. A card can show 95% activity and still deliver throughput far below the hardware ceiling. Model FLOPs utilization is the right measure, and it is rarely instrumented.

Remediation follows the layers in order. A fleet problem cannot be fixed by within job tuning.

Telemetry limits

Cost Explorer and the other native billing tools attribute spend to instances, and they do that accurately. What they cannot report is token consumption, cost per inference, or which model version caused a step change in the bill, because the provider has no visibility inside the serving process.

That blind spot is growing in importance. Specialist GPU providers currently report roughly 70% of revenue from training and 30% from inference, and the ratio is expected to invert as enterprise inference scales. Training is a project cost with an endpoint. Inference is a recurring cost, and recurring costs are the ones that outrun forecasts.

The instrumentation has to live in the serving layer. It is not a procurement decision or a dashboard setting.

Continuous batching

Static batching collects a fixed group of requests, runs them together, and releases them together. In a batch of sixteen where one request generates 512 tokens and the rest generate 64, the short requests hold GPU slots for hundreds of tokens they never produce. The device reports busy while much of its capacity is spent on padding.

Continuous batching, also called iteration level or in flight batching, recomposes the batch at every forward pass: a finished request leaves immediately and a queued request takes its slot.

The published numbers are large and need context. The Orca paper reported up to a 36.9x throughput improvement over FasterTransformer at equivalent latency targets. Anyscale's widely cited benchmark, Llama 13B on a single A100, moved from about 7 sustainable requests per second under static batching to about 165 under vLLM, the source of the much quoted 23x. vLLM's own release measured 24x over Hugging Face Transformers and 3.5x over TGI.

Those comparisons are against naive baselines. The figures relevant to a production migration are lower:

Against a stack already doing continuous batching, the remaining gain is mostly memory efficiency, roughly 2x.
On mixed production traffic with variable output lengths, 4x to 8x is the realistic range.
On prefill bound workloads with long prompts and short completions, 3x to 5x.
Where output lengths are uniform, the gain approaches zero, because there is no padding to reclaim.

The mechanism depends entirely on variance. Production traffic has heavy tailed length distributions, and static batching charges the tail against every request in the batch.

Protective measures

The controls run in this order, because each depends on the one before it.

Establish allocation first. Tag every GPU instance and every job by team, project, model, and environment. Until allocation is reliable, cost reviews turn into attribution disputes instead of remediation.
Match pricing to the workload profile. Bursty, unpredictable demand is usually better served by per token APIs. Steady, high volume demand suits self hosted models on reserved or spot capacity, where spot typically runs 60% to 90% below on demand. The common failure is committing reserved capacity to a workload that swings between 0% and 100% of a GPU, which describes most training work.
Instrument the serving layer. Emit tokens, latency, and cost per model and environment from the inference server itself. No billing tool can produce these.
Fill provisioned capacity before adding more. Work the utilization layers in order: fleet, then job, then within job. Continuous batching is the largest single lever for inference workloads.
Automate idle termination. Policy driven shutdown for GPU instances below a defined utilization threshold over a sustained window, and enforced schedules on development environments. Low technical risk, immediate saving.

The AzeniQ approach

Classic rightsizing does not catch this class of waste. Rightsizing finds machines that are too big; these machines are the right size and simply not being fed. The risk lives in the scheduling layer, the serving layer, and the attribution model that ties both back to a team and a budget.

We build the allocation layer first, because every later control depends on it: instrumented serving, cost and latency per model, and budget alerts denominated in tokens rather than instance hours. Then we work the utilization layers in dependency order, so remediation effort lands where the spend actually sits. Where the economics support it, we design on premises GPU deployments that run open models on hardware you own, which removes the per token bill entirely for steady, high volume inference.

Conclusion

Wasted cloud spend reached 29% in 2026, ending a run of falling and flat figures, and the stated reason is AI workloads. The waste moved from thousands of cheap instances to a handful of GPU instances costing five figures a year each, which should make it easier to find. What became harder is seeing it, because the tooling most organizations inherited measures instance hours, not tokens.

Teams that treat inference telemetry as a reporting nicety will keep paying for capacity they cannot see. Instrument the serving layer, attribute cost per model and environment, and work the utilization layers in order. That is the change that closes the gap.