Production Capacity Planning for AI Inference Fleets

“Nameplate throughput × instance count” is a fiction

The simplest way to size an inference fleet is to take the vendor-quoted throughput for the accelerator, multiply by the number of instances, and call it the fleet’s capacity. The number that comes out has no relationship to what the fleet will actually serve under production conditions. The vendor number is a peak measurement at conditions that maximize the metric. The production deployment runs at a different operating point — bounded by the latency SLO, exposed to bursty traffic, sharing memory and IO with the host — and the throughput available at that operating point is substantially lower, and a different number, than the nameplate.

A fleet sized on nameplate arithmetic is over-counted, sometimes by a large factor. In our experience, the over-count ratio between vendor-quoted throughput on a synthetic harness and SLO-bounded throughput on the production AI Executor commonly sits in the range of roughly 2× to 5× (observed pattern across engagements; not a benchmarked rate, and the right number for a given fleet must be measured rather than borrowed). The number of instances actually required to meet an SLO at production traffic is determined by the executor’s saturation behavior under the production workload, not by the throughput vendors quote at conditions that don’t apply.

The implication for procurement is sharper than the planning implication. A faster-on-paper GPU can end up more expensive per unit of sustained AI work because the gap between its peak and its SLO-bounded throughput is wider than the cheaper alternative’s gap — and the cost-per-served-request metric lives on the SLO-bounded side, not on the peak side. We discuss the mechanics of that inversion in steady-state performance and capacity planning, which this article extends into the operational sizing problem.

How is AI inference capacity planning different from web-service capacity planning?

Web service capacity planning has well-understood mechanics. A request consumes a small, bounded slice of CPU and memory; the host can run many concurrent requests; capacity scales near-linearly with instance count until the next-tier resource (database, downstream service, network) saturates. The arithmetic is roughly “request budget per instance × instance count.”

AI inference capacity planning has different mechanics:

The unit of work is large and variable. A single inference can occupy the accelerator for hundreds of milliseconds; a token-generative request occupies it for the duration of generation. Concurrency on a single instance is bounded by batch policy and memory, not by lightweight request multiplexing.
Throughput is not linear with batch. Larger batches raise throughput sublinearly and raise per-request latency. The operating point that maximizes throughput is rarely the operating point that meets the SLO.
The bottleneck is the executor, not the host. Adding hosts behind a load balancer scales capacity; adding requests to a single host beyond its saturation point degrades latency without raising throughput.
Saturation behavior has a knee. Below the knee, latency rises slowly with load; above the knee, it rises sharply. Sizing for “average load” without accounting for the knee produces a fleet that meets the SLO at the average and fails it during normal demand variation.
Headroom must absorb bursts and partial failures. A fleet sized at 100% of measured capacity has no margin for traffic spikes or for one instance going offline.

The arithmetic that fits these mechanics is not “throughput × count.” It is saturation-curve fitting to projected demand under SLO and headroom constraints — which is a different kind of calculation entirely. It also pulls the inference runtime stack — TensorRT, vLLM, Triton, the choice of FlashAttention kernels, the precision regime selected by the framework — into scope, because the curve shifts when any of those shift.

The inputs an inference capacity plan actually needs

The inputs to a defensible inference capacity plan come from measurement, not from spec sheets:

Per-workload throughput-vs-latency curve on the production AI Executor (accelerator + driver + runtime + framework + inference runtime + precision regime). The curve is traced by sweeping batch size and concurrency. CUDA driver version, cuDNN build, and torch.compile or TensorRT-LLM settings are all part of the executor identity here; changing any of them invalidates the curve.
The SLO operating point — typically expressed as p99 (or p99.9) latency below a budget. This selects a point on the curve below which the executor’s effective capacity is bounded.
Per-workload demand forecast — projected request rate over the planning horizon, including expected diurnal and weekly patterns, expected growth, and expected bursts above the average.
Headroom policy — the fraction of per-instance capacity reserved for traffic spikes, scaling latency, and partial failures. A typical convention is roughly 25–40% headroom (planning heuristic, observed across engagements; not a benchmarked rate). The right number depends on burst behavior and recovery time.
Recovery margin — capacity to absorb the load redistribution when one or more instances go offline (n+1, n+2 redundancy depending on availability target).

The first input is the one most often missing. Vendor numbers are not a substitute. Synthetic benchmarks at vendor-published configurations are not a substitute. The required input is the throughput-vs-latency curve on the production executor running the production workload, because both axes shift when either changes.

A capacity-sizing checklist

A capacity plan that survives production should satisfy the following:

A plan that satisfies this list produces a fleet sizing that survives the conditions production imposes. A plan that elides items produces a sizing that holds only as long as no condition shifts — which is a stronger assumption than AI workloads usually justify.

Why a single point estimate of throughput is insufficient

The reason point-estimate throughput cannot ground capacity is that the operational risk an inference fleet exists to manage is not in the average; it is in the latency tails and burst response. A fleet sized at “average demand × headroom factor” using a single throughput number can be:

Right on average but unable to absorb a 2× burst because per-instance saturation behavior produces a sharp latency knee the average couldn’t see.
Right on the median but failing p99 SLOs because the throughput number was a mean-latency throughput rather than an SLO-bounded throughput.
Right at the moment of measurement but wrong six months later because the workload mix shifted and the per-workload throughput on the same hardware shifted with it.

The latency-budget-conditioned throughput distribution is what bounds the operational risk. A capacity plan that uses the distribution as its input — choosing fleet size to keep the SLO-bounded throughput above projected demand at the chosen percentile — is robust to the conditions a point-estimate plan is fragile to.

The same shift in input is what makes capacity planning an economic exercise rather than only an engineering one. Cost efficiency in GPU infrastructure does not emerge from picking the GPU with the highest peak; it emerges from the executor whose SLO-bounded sustained throughput, multiplied across the fleet, meets demand with the smallest total cost. That is a function of the same throughput-vs-latency curve, not of nameplate arithmetic.

What a benchmark has to report to support an economic decision

A benchmark that exists only to rank accelerators on a peak metric cannot ground a sizing decision and cannot ground a procurement decision. To support both, a benchmark report needs to carry the data that the capacity plan above consumes:

The full throughput-vs-latency curve, not a single peak point.
The executor identity behind each curve — accelerator, driver, runtime, framework, inference runtime, precision regime — so the reader knows what the curve is the curve of.
The SLO operating points the curve was read at, including the p99 or p99.9 latency budget that bounded the throughput.
The workload definition — model, model size, precision regime, input distribution — that the curve is a function of.
Stability data: variance across runs, behavior under thermal load, behavior across the diurnal cycle if measured long enough to see it.

A report that contains these elements lets a planner project a fleet sizing, a cost-per-served-request, and a procurement comparison from the same source data. A report that contains only a peak throughput number, even an honestly measured one, supports an engineering bragging right and not much else. ## The framing that helps

Production AI inference capacity planning anchors to saturation-curve measurements on the production AI Executor under the production workload, selects the SLO operating point, applies headroom and redundancy policy explicitly, and re-measures when the executor or workload changes. Nameplate-throughput arithmetic produces fleet sizings that don’t survive contact with production. Latency-budget-conditioned throughput distributions produce fleet sizings that do — and, because they live on the same axis as cost, they also produce procurement decisions that survive the spreadsheet.

LynxBench AI treats throughput-vs-latency curves on the production AI Executor as the required input for capacity planning — because the SLO-bounded operating point on those curves is what determines real per-instance capacity, and a fleet sized on that input is robust to the conditions a fleet sized on nameplate throughput is exposed to. For the fleet sizing in front of you, does the input curve carry the SLO operating point at which sustained throughput is the binding constraint — measured on the production AI Executor under the production workload — or is the projection still inheriting a nameplate figure the saturation curve never validated for your latency budget?