AI Infrastructure & Inference Platforms
Inference, not training, is where most enterprise AI money and risk now sit: it runs continuously and grows with every user and agent step. Platforms pay off when they are sized and measured on cost per successful outcome, with consumption capped by design, because performance per dollar moves faster than any multi-year commitment.
Part of Cloud, Infrastructure & Applications, in the Build mode. 9 sources, checked .
- Share of worldwide AI-optimized IaaS spending forecast to support inference rather than training in 2026, rising to 59% in 2027 (Gartner)
- 55%
- Share of worldwide AI-optimized IaaS spending forecast to support inference rather than training in 2026, rising to 59% in 2027 (Gartner)
- Gartner, 2026(opens in a new tab)
- Fall in the inference cost of a system performing at GPT-3.5 level, November 2022 to October 2024 (Stanford AI Index 2025)
- 280x
- Fall in the inference cost of a system performing at GPT-3.5 level, November 2022 to October 2024 (Stanford AI Index 2025)
- Stanford Institute for Human-Centered Artificial Intelligence, 2025(opens in a new tab)
- Gain in the best per-accelerator DeepSeek R1 server result in MLPerf Inference v6.1 (2026) versus a year earlier
- 5.7x
- Gain in the best per-accelerator DeepSeek R1 server result in MLPerf Inference v6.1 (2026) versus a year earlier
- MLCommons, 2026(opens in a new tab)
The problem
Enterprise AI spending is shifting to inference. Gartner forecasts worldwide AI-optimized IaaS spending of $42 billion in 2026, with inference ($23.3 billion) overtaking training ($19 billion) and reaching 59% of that spend in 2027.1 The CNCF's 2025 survey found 52% of organizations do not build or train models at all; they consume them. As the CNCF puts it, training needs expensive GPUs for hours or days, while inference runs continuously, which makes cost management critical.2
The price of a given capability falls fast. Stanford's 2025 AI Index found the inference cost of a system performing at GPT-3.5 level fell more than 280-fold between November 2022 and October 2024, while hardware costs fell 30% a year.3 In MLPerf Inference v6.1, the best per-accelerator result on the DeepSeek R1 server test was 5.7 times better than a year earlier.4 Software matters as much as silicon: the vLLM serving system and its PagedAttention method raised LLM throughput 2 to 4 times at the same latency.5
Uncapped inference is both a security and a cost exposure. OWASP's 2026 Top 10 for LLM Applications moved unbounded consumption up four places to LLM06, noting that attackers can trigger expensive computation at negligible cost to themselves. It now says request-rate limiting alone is no longer sufficient and calls for token-aware cost controls, hard spending caps and agent-level circuit breakers.6 The FinOps Foundation advises tracking cost per inference, per user and per successful outcome for each use case, reported as distributions because AI costs vary from run to run.7
Good looks like a shared serving platform, not a GPU cluster per project: a model gateway with quotas and routing, autoscaled serving, retrieval and vector services close to the data, and isolation for sensitive workloads. Capacity is a mix of managed APIs, reserved cloud GPUs and, where volume and data rules justify it, owned or colocated hardware. NSA, CISA and Five Eyes partners have published joint guidance on deploying and operating externally developed AI systems securely.8
What you get
- 01
Inference demand and capacity model
Expected load per use case (users, tokens, agent steps, peak and steady state) translated into a mix of managed APIs, reserved cloud GPUs and owned or colocated capacity, with the cost per outcome of each option.
- 02
Model serving platform
Autoscaled serving on an efficient inference engine with batching and caching, a model gateway for routing, quotas and fallback across providers, and isolated environments for sensitive workloads.
- 03
Retrieval and vector services
Vector and search services, embedding pipelines and access controls that respect the permissions of the source systems, placed close to the data they index.
- 04
Consumption and cost controls
Token-based rate limits, per-user and per-application quotas, hard spending caps and cost allocation by use case, so spend is visible and capped before it reaches the invoice.
- 05
Security and operations runbooks
Hardened deployment environments, protection for models and data, logging and monitoring, and incident procedures aligned with joint NSA and CISA guidance for deploying AI systems.
Where it goes wrong
The failure modes we design against from the first week, and the control for each.
Locking in capacity on today's benchmarks
On one MLPerf test, the best per-accelerator inference result improved 5.7 times in a year.4 Control: keep long commitments to the steady base load, rent the rest, and re-benchmark on your own models and prompts before each renewal.
Measuring price per token, not cost per outcome
Token prices can fall while total spend rises, because usage grows faster. Control: track cost per successful outcome for each use case, reported as a range, as the FinOps Foundation recommends,7 and give each use case owner a target.
No ceiling on consumption
Without limits, a bug, an agent loop or an attacker can run up inference cost and degrade service for everyone, and OWASP warns that alerts alone can be outpaced by fast-accumulating workloads.6 Control: enforce token-based rate limits and hard spending caps that halt inference at the gateway, and set step, time and cost limits on every agent run.
Hardware the facility cannot power or cool
Uptime Institute reports that soaring AI demand is straining existing data center designs, especially power and cooling.9 Control: confirm power density, cooling and delivery lead times with your facility or colocation provider before buying GPUs, and keep a cloud option for overflow.
Standards and rules
OWASP Top 10 for LLM Applications 2026, LLM06:2026 Unbounded Consumption
Primary text on genai.owasp.org(opens in a new tab)Uncontrolled inference is a top-ten LLM risk, and the 2026 edition expects token-based rate limits, hard spending caps that halt inference, agent circuit breakers, and patched, authenticated inference endpoints.
Deploying AI Systems Securely (NSA AISC, CISA, FBI and Five Eyes partners, 2024)
Primary text on cisa.gov(opens in a new tab)Organizations deploying externally developed AI systems should protect their confidentiality, integrity and availability, mitigate known vulnerabilities, and be able to detect and respond to malicious activity.
FinOps Framework: FinOps for AI
Primary text on finops.org(opens in a new tab)Allocate AI spend first, then right-size models, use tokens efficiently and commit only once usage is stable, measuring cost per inference, per user and per successful outcome.
How to measure it
- Cost per successful outcome for each AI use case, reported as a range
- GPU utilization, and reserved capacity used versus paid for
- Latency at the 95th percentile (time to first token and total) against each use case's target
- Requests throttled by quotas, and spend anomalies caught before the invoice
- Share of AI workloads served through the shared platform rather than one-off stacks
Questions buyers ask
Should we buy GPUs or rent them?
Rent first, and buy only for the base load you can prove. Inference performance per accelerator has been improving several-fold a year, so hardware bought on today's numbers ages quickly. Owning makes sense when demand is high and steady, when data cannot leave your control, and when your facility can supply the power and cooling. Most enterprises end up with a mix of managed APIs, reserved cloud capacity and some owned hardware.
What is a model gateway and do we need one?
A model gateway sits between your applications and the models they call. It handles authentication, routing between providers and models, quotas, rate limits, caching, logging and cost allocation. Once more than a couple of teams use AI, you need one: without it each team rebuilds the same controls, spend is hard to attribute, and nobody can enforce a ceiling on consumption.
How do we make inference cheaper without hurting quality?
Start by measuring cost per successful outcome for each use case. Then route simple requests to smaller models, cache repeated answers, batch where latency allows, and use an efficient serving engine. Quantization and right-sized context windows help further. Test each change against a fixed evaluation set, so the savings do not come from quietly worse answers.
Sources
Every figure on this page comes from the sources below. We checked each link on .
- 1.Gartner Forecasts Worldwide AI-Optimized IaaS Spending to Grow 96% Through 2026(opens in a new tab)Gartner, August 10, 2026 · gartner.com
- 2.The CNCF Annual Cloud Native Survey: The Infrastructure of AI's Future(opens in a new tab)Cloud Native Computing Foundation, January 20, 2026 · cncf.io
- 3.The 2025 AI Index Report(opens in a new tab)Stanford Institute for Human-Centered Artificial Intelligence, April 2025 · hai.stanford.edu
- 4.MLCommons Sets Participation Record with New MLPerf Inference v6.1 Benchmark Results(opens in a new tab)MLCommons, September 16, 2026 · mlcommons.org
- 5.Efficient Memory Management for Large Language Model Serving with PagedAttention(opens in a new tab)arXiv (SOSP 2023), September 12, 2023 · arxiv.org
- 6.OWASP Top 10 for LLM Applications 2026 (LLM06:2026 Unbounded Consumption)(opens in a new tab)OWASP Gen AI Security Project, August 3, 2026 · genai.owasp.org
- 7.FinOps for AI Overview(opens in a new tab)FinOps Foundation, October 28, 2026 · finops.org
- 8.Joint Guidance on Deploying AI Systems Securely(opens in a new tab)Cybersecurity and Infrastructure Security Agency, April 15, 2024 · cisa.gov
- 9.Uptime Announces Annual Outage Analysis Report 2025(opens in a new tab)Uptime Institute, May 6, 2025 · uptimeinstitute.com