Production Capacity Planning for AI Inference Fleets

“Nameplate throughput × instance count” is a fiction

Fleet sizing often reduces to multiplying vendor-published accelerator throughput by instance count. The number that comes out has no relationship to what the fleet will actually serve under production conditions. The vendor number is a peak measurement at conditions that maximize the metric. The production deployment runs at a different operating point — bounded by the latency SLO, exposed to bursty traffic, sharing memory and IO with the host — and the throughput available at that operating point is substantially lower, and a different number, than the nameplate.

A fleet sized on nameplate arithmetic is over-counted, sometimes by a large factor. In our experience, the over-count ratio between vendor-quoted throughput on a synthetic harness and SLO-bounded throughput on the production AI Executor commonly sits in the range of roughly 2× to 5× (observed pattern across engagements; not a benchmarked rate, and the right number for a given fleet must be measured rather than borrowed). The number of instances actually required to meet an SLO at production traffic is determined by the executor’s saturation behavior under the production workload, not by the throughput vendors quote at conditions that don’t apply.

The implication for procurement is sharper than the planning implication. A faster-on-paper GPU can end up more expensive per unit of sustained AI work because the gap between its peak and its SLO-bounded throughput is wider than the cheaper alternative’s gap — and the cost-per-served-request metric lives on the SLO-bounded side, not on the peak side. We discuss the mechanics of that inversion in steady-state performance and capacity planning, which this article extends into the operational sizing problem.

How is AI inference capacity planning different from web-service capacity planning?

Capacity planning for web services follows established patterns with known mechanics. A request consumes a small, bounded slice of CPU and memory; the host can run many concurrent requests; capacity scales near-linearly with instance count until the next-tier resource (database, downstream service, network) saturates. The arithmetic is roughly “request budget per instance × instance count.”

AI inference capacity planning has different mechanics:

The unit of work is large and variable. A single inference can occupy the accelerator for hundreds of milliseconds; a token-generative request occupies it for the duration of generation. Concurrency on a single instance is bounded by batch policy and memory, not by lightweight request multiplexing.
Throughput is not linear with batch. Larger batches raise throughput sublinearly and raise per-request latency. The operating point that maximizes throughput is rarely the operating point that meets the SLO.
The bottleneck is the executor, not the host. Adding hosts behind a load balancer scales capacity; adding requests to a single host beyond its saturation point degrades latency without raising throughput.
Saturation behavior has a knee. Below the knee, latency rises slowly with load; above the knee, it rises sharply. Sizing for “average load” without accounting for the knee produces a fleet that meets the SLO at the average and fails it during normal demand variation.
Headroom must absorb bursts and partial failures. A fleet sized at 100% of measured capacity (by definition, no headroom, a rule of thumb) has no margin for traffic spikes or for one instance going offline.

The arithmetic that fits these mechanics is not “throughput × count.” It is saturation-curve fitting to projected demand under SLO and headroom constraints — which is a different kind of calculation entirely. It also pulls the inference runtime stack — TensorRT, vLLM, Triton, the choice of FlashAttention kernels, the precision regime selected by the framework — into scope, because the curve shifts when any of those shift.

The inputs an inference capacity plan actually needs

The inputs to a defensible inference capacity plan come from measurement, not from spec sheets:

Per-workload throughput-vs-latency curve on the production AI Executor (accelerator + driver + runtime + framework + inference runtime + precision regime). The curve is traced by sweeping batch size and concurrency. CUDA driver version, cuDNN build, and torch.compile or TensorRT-LLM settings are all part of the executor identity here; changing any of them invalidates the curve.
The SLO operating point — typically expressed as p99 (or p99.9) latency below a budget. This selects a point on the curve below which the executor’s effective capacity is bounded.
Per-workload demand forecast — projected request rate over the planning horizon, including expected diurnal and weekly patterns, expected growth, and expected bursts above the average.
Headroom policy — the fraction of per-instance capacity reserved for traffic spikes, scaling latency, and partial failures. A typical convention is roughly 25–40% headroom (planning heuristic, observed across engagements; not a benchmarked rate). The right number depends on burst behavior and recovery time.
Recovery margin — capacity to absorb the load redistribution when one or more instances go offline (n+1, n+2 redundancy depending on availability target).

The first input is the one most often missing. Vendor numbers are not a substitute. Synthetic benchmarks at vendor-published configurations are not a substitute. The required input is the throughput-vs-latency curve on the production executor running the production workload, because both axes shift when either changes.

A capacity-sizing checklist

A capacity plan that survives production should satisfy the following:

A plan that satisfies this list produces a fleet sizing that survives the conditions production imposes. A plan that elides items produces a sizing that holds only as long as no condition shifts — which is a stronger assumption than AI workloads usually justify.

Why a single point estimate of throughput is insufficient

Inference fleet capacity cannot rest on average throughput because operational risk concentrates in tail latencies and burst handling, not mean performance. A fleet sized at “average demand × headroom factor” using a single throughput number can be:

Right on average but unable to absorb a 2× burst (illustrative threshold) because per-instance saturation behavior produces a sharp latency knee the average couldn’t see.
Right on the median but failing p99 SLOs because the throughput number was a mean-latency throughput rather than an SLO-bounded throughput.
Right at the moment of measurement but wrong six months later because the workload mix shifted and the per-workload throughput on the same hardware shifted with it.

The latency-budget-conditioned throughput distribution is what bounds the operational risk. A capacity plan that uses the distribution as its input — choosing fleet size to keep the SLO-bounded throughput above projected demand at the chosen percentile — is robust to the conditions a point-estimate plan is fragile to.

The same shift in input is what makes capacity planning an economic exercise rather than only an engineering one. Cost efficiency in GPU infrastructure does not emerge from picking the GPU with the highest peak; it emerges from the executor whose SLO-bounded sustained throughput, multiplied across the fleet, meets demand with the smallest total cost. That is a function of the same throughput-vs-latency curve, not of nameplate arithmetic.

Which score does a fleet get sized against?

Moving from sustained measurement to sizing input requires an explicit category decision. A fleet is sized for training, or for inference, or for compute — and a device that is strong at one of those is routinely weak at another. Reading an aggregate score and dividing it by a price collapses that distinction: the aggregate is an unbounded ordinal figure with no reference-device normalisation, so the quotient is not a cost coefficient and does not carry units anyone can defend. Size against the category score that matches the work.

Planning question	Read this	Do not read this
How much of my inference SLO can one device hold?	the inference-category sustained figure at the batch size where throughput plateaus	the aggregate score, or a peak-FLOPS number
Which of two accelerator classes is cheaper per unit of sustained work?	category score ÷ delivered cost, both from the same release name	aggregate ÷ list price
Is this device’s figure trustworthy as a planning input?	the population of runs for that device on the public leaderboard	a single run, ours or anyone’s
Does the number still hold after a stack upgrade?	a re-measured curve on the changed executor	the prior release’s figure carried forward

Two rules keep the arithmetic honest, and both are boring. Use the category that matches the work. Keep every number inside one release name — a 26Q3 figure and a 27Q1 figure are not comparable, and a cost model that mixes them produces a per-dollar ranking that reverses for reasons that have nothing to do with hardware.

Energy belongs in the same discipline. Rated TDP is a design ceiling, not a consumption figure, so it is a poor input to a cost comparison; what enters the model is energy actually drawn across the timed window divided by the work completed inside it. That is a measured quantity or it is nothing.

What a benchmark has to report to support an economic decision

Peak-metric accelerator rankings cannot inform either sizing or procurement choices. To support both, a benchmark report needs to carry the data that the capacity plan above consumes:

The full throughput-vs-latency curve, not a single peak point.
The executor identity behind each curve — accelerator, driver, runtime, framework, inference runtime, precision regime — so the reader knows what the curve is the curve of.
The SLO operating points the curve was read at, including the p99 or p99.9 latency budget that bounded the throughput.
The workload definition — model, model size, precision regime, input distribution — that the curve is a function of.
Stability data: variance across runs, behavior under thermal load, behavior across the diurnal cycle if measured long enough to see it.

A report that contains these elements lets a planner project a fleet sizing, a cost-per-served-request, and a procurement comparison from the same source data. A report that contains only a peak throughput number, even an honestly measured one, supports an engineering bragging right and not much else.

One caution about scope. A published benchmark score covers a fixed model catalogue at fixed precisions; it is not a throughput figure for your workload and it does not substitute for measuring your workload. What it gives a planner is a population — a device sitting among runs other people produced with the same instrument — against which a local measurement can be sanity-checked before it becomes a purchase order.

The framing that helps

Valid inference capacity planning measures saturation curves on production executors running production workloads, identifies the SLO operating point, explicitly applies headroom and redundancy policies, and remeasures after executor or workload modifications. Nameplate-throughput arithmetic produces fleet sizings that don’t survive contact with production. Latency-budget-conditioned throughput distributions produce fleet sizings that do — and, because they live on the same axis as cost, they also produce procurement decisions that survive the spreadsheet.

LynxBenchAI treats throughput-vs-latency curves on the production AI Executor as the required input for capacity planning — because the SLO-bounded operating point on those curves is what determines real per-instance capacity, and a fleet sized on that input is robust to the conditions a fleet sized on nameplate throughput is exposed to. For the fleet sizing in front of you, does the input curve carry the SLO operating point at which sustained throughput is the binding constraint — measured on the production AI Executor under the production workload, at one release name, against the category the fleet is actually being sized for — or is the projection still inheriting a nameplate figure the saturation curve never validated for your latency budget?

Frequently Asked Questions

How many instances do I need to meet an SLO at production traffic?

The instance count is read off the per-workload throughput-vs-latency curve on the production AI Executor, not from vendor arithmetic. Select the SLO operating point (typically a p99 or p99.9 latency budget), read the effective per-instance throughput at that point, divide projected demand by it, then add headroom and redundancy margin. Nameplate “throughput × count” sizing over-counts capacity, often by a factor of roughly 2× to 5× in our experience.

What headroom should I reserve when sizing an inference fleet?

A fleet sized at 100% of measured capacity has no margin for traffic spikes or for an instance going offline. A typical convention is roughly 25–40% headroom (a planning heuristic observed across engagements, not a benchmarked rate), with the right number depending on burst behavior and recovery time. Add a separate redundancy margin (n+1, n+2) to absorb load redistribution when instances fail.

When should I re-measure the throughput-vs-latency curve?

The curve is tied to the full executor identity — accelerator, driver, runtime, framework, inference runtime, and precision regime — so any change to those invalidates it. Re-measure after model version changes, precision changes, framework upgrades, or thermal regime shifts. The workload mix can also drift over months, which shifts per-workload throughput on the same hardware even when nothing in the stack changes.

Why does sizing for average load fail during normal demand variation?

Saturation behavior has a knee: below it latency rises slowly with load, above it latency rises sharply. A fleet sized for the average sits comfortably at the mean but crosses the knee during routine bursts, so it meets the SLO on average and fails it under variation. Sizing against the latency-budget-conditioned throughput distribution at the chosen percentile is what makes the plan robust to that variation.

Can I mix benchmark figures from different releases in one cost model?

No. A figure carries the release name it was produced under, and two release names are not points on the same axis — the model catalogue, precisions, and scoring can all move between them. Pin every number in a per-dollar comparison to a single release; if a device only has a figure from an older release, re-measure it rather than carrying the old number forward.

Headroom calculations for unpredictable query mixes

Workloads with high prompt-length and token-count variance need 35 percent overhead reserves; tighter distributions permit safe operation above 75 percent utilization. Is that executor close enough to yours for the result to mean anything?