Diffusion CapitalDiffusion Capital
Back to insights

Research · August 7, 2026

AI Inference Supply & Demand: Sizing the Compute Balance

Compute prices are rising for the labs while token prices fall. We size the supply of inference capacity against the demand for it, estimate where the two cross, and set out how that crossing could resolve.

Maximilian Ruess

In a recent blog post, Dwarkesh compared the labs' revenue rise against their compute growth. He argued that revenue at the leading labs has been growing roughly 10x a year while their compute grows roughly 3x, and since chip supply is physically limited in how fast it can respond, compute prices will inevitably rise. We have investigated this before in our internal models, so we thought we would share our methodology here.

The fundamental question is what happens if demand is higher than supply. The logical conclusion is that the price rises — to be clear, the price of compute, not the token price. The price of an H100 has risen over the last couple of quarters, as excellently tracked by SemiAnalysis's rental price index: a one-year rental bottomed at $1.70 an hour in October 2025 and reached about $2.35 by March 2026, up roughly 40% for the same chip. At the same time, each new generation makes a unit of compute cheaper: a B200 rents at about 1.3x the H100's hourly rate but delivers over twice the compute, so per H100-equivalent the newer chip is roughly 35 to 65% cheaper, depending on precision.

This is largely in line with expectations: if a good with limited supply has excess demand, the price will rise. Token prices make this dynamic more complicated. More advanced chips deliver more FLOPs and more memory bandwidth, which means more tokens per second. Inference frameworks like vLLM, SGLang and Dynamo keep getting better at serving models on the same hardware — we are not going into the technical details here, and SemiAnalysis's InferenceX work is a great resource. Algorithmic improvements and internal distillation let the labs serve intelligent models at a lower price. And the current competitive landscape entices the labs to subsidise token prices heavily.

This leads to a weird dynamic: compute prices are rising for the labs while token prices are falling. Epoch measures the price of a fixed level of capability falling between 9x and 900x a year depending on the task — though the fall is per unit of intelligence. Falling token prices also add Jevons paradox to the calculation: as a good gets cheaper, people use so much more of it that total spending on it rises. Current token demand growth certainly supports this.

Over the last few years the industry has gotten more and more out of the chips it already has: better serving software, caching, quantisation. But demand nevertheless is growing much faster than the physical supply of chips.

We start on the supply side, with how much compute is available and how far the stretching goes, then move toward demand, estimate where the two cross, and offer a few possibilities for how the crossing could be resolved. The supply side is easier to model than the demand side, but there are large uncertainties baked into both, and any number here should be read with wide error bars. We do our best to point out every assumption and where it stems from.

1. Supply: The Installed Base and Buildout

We start this analysis by estimating how much computing power is available today and how much is coming online. The unit we are using throughout is H100-equivalent (H100e): every accelerator is weighted by its dense 8-bit compute relative to one Nvidia H100. For example, a B300 counts as roughly 2.5 H100e.

Exhibit 1. Cumulative capacity at the sites Epoch tracks, outside China. Everything to the right of the dashed line is announced rather than built.

Epoch AI maintains the best publicly available datasets here: a chip-sales ledger and a data-center facilities database. The chip-sales ledger counts accelerators sold or shipped, since revenue is recognised at delivery — which means those chips are delivered but not necessarily energised and ready to use. The dataset puts the ex-China stock at 25.6 million H100e sold by 2026Q2, growing 23% a quarter.

The facilities database counts capacity at sites Epoch has identified through satellite imagery, permits and reporting. Epoch estimates its coverage at about 27% of globally delivered compute in April 2026; our own estimates put it closer to 60% of US chip sales. The facilities database is most useful for estimating the timing of the buildout. Epoch grades every construction phase of every tracked site and attaches a completion probability to each. We are using this dataset to create our buildout schedule.

Exhibit 2. Quarterly energisation at tracked sites, announced against probability-weighted. The year-end clustering is an artefact of how undated projects are recorded, not a build pattern — see the note beneath the chart.

We also checked both datasets against sources outside Epoch: Nvidia's own shipment statements (~4 million lifetime Hopper units, 6 million Blackwell dies by October 2025), analyst unit counts from Omdia, Morgan Stanley, JP Morgan and TechInsights, TPU production surveys, and AMD and Trainium volumes. Based on these sources, the estimates are between 22 and 26 million ex-China H100e by mid-2026; Epoch's figure of 25.6 million sits well inside that range. No independent source publishes a cumulative installed base, so triangulating across these sources is the best estimate we can provide.

2. Supply: Fleet Composition and Turnover

As new accelerators come online, the composition of the fleet changes. This composition determines the peak FLOPs and the memory bandwidth the fleet can reach.

Exhibit 3. Bandwidth per accelerator, coloured by manufacturer. Hollow markers are announced or modelled rather than shipped.
Exhibit 4. Compute per accelerator, dense 8-bit — the same basis as the H100e weighting used in section 1.

Compute per chip doubles every 1.4 years; memory bandwidth every 2.5 years. Serving depends on bandwidth, because every generated token has to read the model's weights from memory. So chips are getting better at training faster than they are getting better at serving.

Epoch's chip-sales data gives us only a baseline for the accelerator mix. We use vendor roadmaps and sales projections to forecast the accelerator mix going forward. The further out we go, the more uncertain the projections become. For example, from 2029 most hardware has no published performance figures yet, so we assume a 2.2x uplift per generation; the observed range across past transitions was 1.9x to 6.8x.

Exhibit 5. Share of the US fleet's H100-equivalents by generation. The hatched region to the right of the dashed line is projected from disclosure rather than measured.

As more custom and inference-optimised chips come online, we are likely undercounting supply at the long end of this chart. Most of these designs are still early and will take time to ramp. Our focus is the more immediate future, where they play a very small role.

We also don't retire any hardware in our calculations. The most likely candidates for retirement are A100-class chips and some H100s, and given their small share of the fleet's total compute the effect would be negligible. As long as a chip's output can be sold for more than the cost of running it, there is no incentive to retire it.

Because compute ratings are misleading for serving, we also measure what each generation actually serves using benchmark runs on matched settings. Measured this way, the B300 serves 3.5x an H100, and the H200 serves 43% more than an H100, even though both carry the same rating.

Exhibit 6. Measured serving against Epoch's rating, from SemiAnalysis InferenceX runs. A ratio above 1.00x means the part serves more than its rating implies; the two orange labels serve less.

3. Supply: Serving-Stack Efficiency Gains

One of the biggest stories of the last 18 months is how much better we can serve models on the same hardware. Inference engineering has emerged as a field of its own, which has produced serving frameworks with better quantisation, speculative decoding, attention sparsity and smarter batching.

To measure this, we track benchmark settings where the hardware never changes: holding the chip, model, precision, request shape and chip count fixed, the median setting gains about 87% a year. This is based on InferenceX, a great benchmarking tool built by the SemiAnalysis team.

Exhibit 7. One grey line per benchmark setting, indexed to its own first run; the dark line is the median. Most of the movement lands in the first two months, and a minority of settings sit below 1x.

Settings on chips launched in 2025 gain about 144% a year, while settings on mature chips gain about 3%. Most of the measured gain happens in a chip's first year, while the software catches up with the new silicon. Whether the pooled rate continues depends on whether each new generation arrives as unoptimised as the last one did, which is uncertain.

4. Demand: Global Token Consumption and Growth

On the demand side we made a conscious decision: this note measures supply, and takes demand from published estimates. Demand forecasting is a separate exercise, and others have done it carefully and well.

Exponential View puts global inference near 34 quadrillion tokens a month, counting API, subscription and internal use, with every token counted as input or output, growing about 14x a year; the more conservative series imply around 10x. Subtracting China using its measured national aggregate leaves 837 to 977 trillion tokens a day outside China. Exponential View has produced a great report and we highly encourage reading it.

Exhibit 8. The dark bar is the 10x-a-year path and the full bar height the 14x path, so the band between them is the range used here. It extrapolates the current rate rather than forecasting one.

The demand estimate counts every token processed. Our supply numbers, however, come from benchmarks where every token is computed from scratch. Most input tokens are cache hits, and a repeated token is served from cache at about 5% of the cost of a fresh one. Chat traffic reuses about 45% of its input (measured 40 to 60%); agent traffic reuses about 95% (coding agents measure 95%+).

InferenceX also uses real agent sessions with 100,000-token contexts. On that traffic a chip delivers less than half of its benchmark rate, mostly because the long context fills the chip's memory. Putting both chat and agentic workload types together, the average demanded token takes about half the compute the benchmarks assume.

We also adjust the numbers for internal use: 15 to 40% of demand is the labs' own work, such as reinforcement learning and synthetic data. These tokens do not need to arrive at chat speed, so a chip can serve them in big batches and produce up to 2.5x more tokens a second. They still count as demand; each one just uses less of a chip.

Token demand projections are arguably the most important measure in AI today: they drive the AI buildout, and the buildout has become a major driver of the economy. We are still early in diffusing this technology into the wider economy, and today's demand is mostly language. Voice, image and video (world) models are still ramping, and they are far more compute- and memory-intensive. As these modalities scale, they will likely push token demand further up. Agentic workflows are also likely becoming a larger share of token demand, which will mean each query uses more tokens than traditional chat interactions. In this note we use the 10 to 14x growth band, but this could well be underforecasting demand.

5. The Workload: How Demand Splits Across Models

Which models serve this demand is the next important factor: the same accelerator can serve 5 to 8 times more tokens on a small model than on a frontier one. The frontier labs only periodically publish token numbers, and while routers like OpenRouter publish theirs in great detail, they are not yet representative of the wider ecosystem. We therefore assembled the split from an array of different sources.

Google reportedly serves about 107 trillion tokens a day, much of it defaulting to its Flash models (the Gemini app, AI Overviews, Gmail, Docs); its Pro-class flagship carries perhaps a tenth of the volume. OpenAI's router sends roughly two thirds of ChatGPT traffic to its fast Instant models, with reasoning at 7% of free and 24% of paid requests in the last disclosure. The open-weights hosting market serves about 110 trillion tokens a day, of which Fireworks alone disclosed 40.

Putting all this together, we think that small proprietary models are about half of global tokens, open-weights models about a quarter, the frontier tier 13% (range 10 to 18), and fine-tuned open-weights models 12%.

Exhibit 9. Token share by model class against each class's measured serving rate. The two 5,621 figures match because both classes are priced at the same open-weights stand-in, and the frontier tier is the only share carrying a published range.

To price each class we use measured serving rates from open-weights stand-ins, because closed models are never benchmarked from outside. The frontier tier is priced at Kimi K3, a 2.8-trillion-parameter model comparable in scale to Opus or GPT 5.6 Sol. The small proprietary tier is priced at Qwen 3.5, the fastest model still being benchmarked. Stock open-weights models are priced at a blend of the models that make up the majority of the traffic (DeepSeek V4, MiniMax M3 and Qwen 3.5), weighted by their router volumes. Fine-tuned open-weights models are also priced at Qwen 3.5, since nearly all fine-tuned serving is small models.

There is also an inherent tradeoff between throughput and speed. A chip can always serve more tokens by batching more users together, which slows every user down, so “tokens per second per chip” means nothing until you fix how fast each user gets their answer. That speed differs by class: measured across 25 commercial services, frontier models deliver a median of 61 tokens a second per user, mid-tier models 102, and fast small models 179. We hold the frontier class at 60 tokens a second per user and every other class at 100, the mid-tier median, since the small proprietary class spans everything from Sonnet-class models to Flash-class ones and most of its volume sits at the mid rate.

6. The Balance: Serving Share and Crossing Points

The fleet outside China is 25.6 million H100e sold by 2026Q2. Chips are powered and working about 66% of the time, based on cluster traces and datacenter energy studies. We make no assumption about how the fleet splits between serving and training. We compute what the whole fleet could serve and let demand tell us the share it needs.

At these speed floors, weighted by the model mix from section 5, the fleet could serve about 2,000 trillion tokens a day. We assume 70% of Blackwell chips are installed in full rack systems. Separately, we add a gain for speculative decoding, which the benchmark runs leave out: 1.4x on rack systems and 1.3x on single machines. Both are within measured ranges: racks measure 1.2 to 1.8, busy single machines 0.9 to 2.0.

Serving all demand outside China takes 45 to 55% of the fleet, and 16 to 26% after the two adjustments from section 4: the cache-and-delivery conversion, and the labs' internal use served at the full-batch rate. That would leave 74 to 84% of the fleet for training, research and everything else.

Peak demand for inference will run somewhere between 1.7x and 2.8x the average. Because of timezone differences the global peak is likely around 1.4x, which would result in 23 to 37% of global compute. OpenAI and Anthropic will likely sit well above these thresholds because they don't have core business functions that require compute, like Meta's ranking systems or Alphabet's search, ads and YouTube recommendation models.

Exhibit 10. Both curves extrapolate current rates rather than forecasting them. The shaded band is where demand overtakes supply; the dashed line is the same supply path with the software term cut to its mature rate.

Supply grows 46% a quarter: silicon +23%, the mix shift +2%, serving software +17%. Demand grows 78 to 93% a quarter. Demand doubles every three months; supply doubles every five. At these rates demand passes gross supply within 2.5 to 4.6 quarters. If the software gains fade to their mature rate, within 1.7 to 2.6.

These lines will cross at the peak hours first. Anthropic has already run off-peak promotions, and more off-peak rates and discounts are likely to follow. The labs and hyperscalers will also be pushed to move their own internal workloads into the quiet hours, where the work allows it — synthetic data generation, for example.

7. Levers: How a Shortfall Gets Addressed

We have identified that sooner or later, depending on demand, there will be a supply shortfall. However, this shortfall will not be an outage or a blackout. There are a number of levers the labs and hyperscalers can use to ration supply or curb demand. We outline a few here — by no means a comprehensive list — deliberately covering both the product and pricing side and the technical side.

Quotas and prices. The frontier labs' subscription tiers have been very generous so far. That is unlikely to last: expect higher prices or smaller quotas, and the return of off-peak offers. The current race to the bottom on price benefits the consumer more than anyone, reminiscent of the early Uber and Lyft years. The best open-weights models (Kimi K3, GLM 5.2, Qwen 3.8) are too large for a user's own hardware and need a GPU cluster to run well. Inference providers can offer cheaper intelligence than OpenAI or Anthropic, because they have no training run to amortise. But compute prices are rising for everyone, and in the short term that either eats into margin or raises token prices — for open and closed models alike. The question is who can burn the most cash to win market share.

Routing. Model routers are the talk of the town at the moment, and that will continue — we have written about this on a few occasions before. A token answered by a small model instead of a frontier one frees about 5x the compute. ChatGPT has already hidden much of its model choice behind the UI. Subscription users tend to pick the biggest and most advanced model regardless of the task, which comes back to the previous point: as long as usage is heavily subsidised, there is no incentive to choose otherwise.

Responsiveness. A chip can always serve more tokens by slowing each user down. Letting every user's speed drop to 30 tokens a second buys up to 3x more serving capacity in the best case. This will likely affect usage though: users are highly sensitive to how fast the first token arrives and to speed in general. Nailing that balance down will be a difficult task for the labs.

Exhibit 11. The throughput-against-latency frontier for one model on three accelerators; each point is a concurrency setting. The shaded band is where commercial services actually operate, so only the right-hand stretch of each curve is reachable in practice.

Engineering. Serving at four-bit precision where the silicon supports it increases serving capacity by 1.4 to 1.9x. Raising the fleet's duty cycle from 66% to 75% adds 14%. The biggest opportunity is the long-context penalty from section 4: agent traffic gets less than half a chip's benchmark rate, and the industry has been investing heavily here — sparse attention, cheaper context storage, new model architectures, cluster-wide context pools, speculation tuned to agents. These could bring efficiency gains of 2x to several-fold on agent workloads.

Sources

Every figure in this note traces to one of the entries below. The tag on each says what kind of evidence it is: primary for a first-party disclosure or filing, measured for a published measurement or dataset, secondary for reporting on a primary, vendor for a seller's own numbers, and judgment where we state an assumption no source settles.

Compute stock and buildout

  • Epoch AI Data Centers Database — the chip-sales ledger and facilities tracker behind the 25.6m H100e stock and the buildout schedule. Exhibits 1, 2 and 5. measured
  • Nvidia shipment statements — ~4m lifetime Hopper units and 6m Blackwell dies to October 2025; 1.3m Hopper to the top four cloud providers in 2024. primary, via secondary carriers
  • Analyst unit counts — TechInsights (3.76m Nvidia data-centre GPUs in 2023), Omdia buyer tables, JP Morgan (5.2m Blackwell units in 2025, falling to 1.8m in 2026 as Rubin ramps), Morgan Stanley (27,300 NVL72 racks in 2025, rising to 70,000–80,000 in 2026), and TPU volumes from Mizuho (4.3m in 2026, ~35m by 2028) and BofA (about 2m in 2025, over 3m in 2026). analyst estimate, via secondary carriers
  • Nvidia Q3 FY2026 earnings call — A100s still fully utilised roughly six years after shipping, which is why nothing retires inside our window. primary transcript
  • AI 2027 compute forecast — an independent trajectory interpolating ~27m H100e globally at mid-2026, close to our own. independent model
  • Die-versus-package counting — Nvidia counts dies where analysts count packages, a factor of two on Blackwell. methodological note

Fleet composition and serving rates

Compute prices

Token demand

The model mix

Peak, duty cycle and the levers

  • Peak-to-mean ratios — DynamoLLM on Azure traces at 1.7x conversation and 2.8x coding, with BurstGPT and ServeGen diurnal shapes. measured
  • Mooncake — cluster-pooled KV cache lifting request capacity by roughly 75% on production workloads, the strongest published number on long-context serving. primary, measured in production
  • DeepSeek Sparse Attention — sparse attention in production, with an API price cut of more than half at launch. primary
  • FP8 KV cache — a flat halving of context memory at any length, now a production default. measured
  • Kimi Delta Attention — constant-size state on three layers in four, up to 75% less KV cache. primary
  • SGLang HiCache — cache hit rate from 40% to 80% on a coding-agent workload. vendor, measured
  • MiniMax on abandoning efficient attention — the contested case, documented on multi-hop reasoning quality. primary
  • Suffix speculation for agents — amortising the whole-context read across drafted tokens, 1.8 to 4.5x end-to-end on agentic tasks. vendor
  • DeepSeek V4 sparsity — 27% of single-token FLOPs and 10% of the KV cache at 1m context against V3.2. measured
  • Aegaeon (SOSP 2025) — token-level multiplexing cutting a long-tail serving fleet from 1,192 to 213 GPUs, though that tail is a small slice of the whole. primary
  • RouteLLM — cost cuts at held quality, the published basis for routing gains. measured
  • Priority and batch tiers — Anthropic service tiers, OpenAI priority processing, and Google pricing. primary
  • Trough-filling — draining inference nodes to research overnight; the mechanism is published, the recovered hours are not. primary, qualitative
  • Anthropic usage limits — the quota and off-peak sequence referenced in section 7. primary
  • Effort controls and the effort API — matching a prior model's best score at 76% fewer output tokens. vendor