A YOLO detector that clocks a fast per-image forward pass in a notebook and then delivers a fraction of the expected frames-per-second in production is almost never limited by the model kernels. The GPU is idle most of the time. It sits waiting for one image to be decoded, resized, and normalised on the CPU, runs a forward pass on a batch of one, then stalls again while non-maximum suppression (NMS) filters boxes back on the host. The convolutions are fast. The pipeline around them starves the device.
This is the trap in “slow YOLO inference.” The instinct is to reach for kernel-level tuning — better occupancy, a faster NMS implementation, lower precision. Those help at the margin. But when a pipeline feeds the GPU one frame at a time, no kernel tuning recovers the throughput lost to an under-fed device. The largest gains live in the shape of the pipeline, not the shape of the kernels.
How does YOLO inference work in practice?
YOLO architectures perform detection in one forward pass: a backbone and neck emit a dense prediction grid, which a post-processing step filters into final boxes and class scores. That “single-stage” property is exactly why people underestimate the pipeline. Because the model itself is one pass, teams assume inference is one thing. It is three.
Every YOLO inference request moves through three distinct stages, and each runs on different hardware unless you force otherwise:
- Pre-processing — decode the image (often JPEG), resize to the model’s input resolution (commonly 640×640, a typical model default), convert layout (HWC to CHW), normalise to float, and move the tensor to the GPU. By default much of this runs on the CPU with libraries like OpenCV or Pillow.
- Model forward pass — the convolutional backbone and detection head run on the GPU. This is the part everyone profiles, and it is usually the fastest of the three per image once the data has arrived.
- Post-processing (NMS) — the model emits thousands of raw boxes per image; NMS removes overlapping duplicates and applies a confidence threshold. In many stock implementations this happens on the CPU after copying results back from the device.
The forward pass is GPU work. Pre-processing and NMS, in a naive setup, are CPU work with a device round-trip on each side. That structure means the GPU is genuinely busy for only a slice of wall-clock time per request — which is why single-image serving so often shows single-digit or low-double-digit GPU utilisation while the CPU pegs a core.
Why does batching change YOLO throughput more than tuning kernels?
Thousands of CUDA cores and terabytes-per-second memory bandwidth define the execution model of a contemporary datacenter GPU. A single 640×640 image (as a sizing example) does not contain enough parallel work to saturate it. The convolution kernels launch, finish quickly, and the scheduler has nothing else queued. You are paying for a wide machine and running it narrow.
In configurations we have worked through, moving a stock YOLO pipeline from batch-of-one to dynamic batching lifts GPU utilisation from a single-digit or low-double-digit percentage to well above 70% (operational measurement in our project engagements), which translates into several-fold throughput gains at a fixed batch latency budget (observed across TechnoLynx GPU pipeline engagements; not a published benchmark).
The reason kernel tuning cannot match this is arithmetic. If the GPU is idle 85% of the time because it is starved, making the busy 15% run 20% faster buys you a 3% end-to-end improvement (illustrative figures). Feeding it properly turns the 85% idle into useful work. This is the practical face of the claim that the unit of performance is the workload feeding the device, not the device in isolation — the same principle that separates peak-spec throughput from what you actually measure in a serving loop.
The catch is latency. Batching trades per-request latency for throughput: a frame that arrives just after a batch closes waits for the next one to fill. Dynamic batching — where the server collects requests up to either a max batch size or a max wait window, whichever comes first — is how production servers like NVIDIA Triton Inference Server resolve this. You cap the wait (say, a few milliseconds) so tail latency stays bounded while still amortising kernel launches across whatever arrived in that window.
Where do the bottlenecks actually sit?
Split inference into preprocessing, neural-network forward pass, and NMS, then ask which of the three limits throughput and where each stage runs. Here is where the time goes in a typical unoptimised versus reworked YOLO serving pipeline, and what moves the needle in each stage.
YOLO pipeline stage diagnostic
| Stage | Naive setup | Where it stalls | The rework |
|---|---|---|---|
| Pre-processing | CPU decode/resize/normalise, then H2D copy | CPU-bound; blocks the GPU per frame | On-GPU decode (NVIDIA DALI, nvJPEG) + batched resize; overlap copy with compute via CUDA streams |
| Forward pass | Batch-of-one, FP32 | GPU under-occupied | Dynamic batching; FP16/INT8 via TensorRT if accuracy allows |
| Post-processing (NMS) | CPU NMS after D2H copy | Device round-trip + serial host work | On-GPU / fused NMS (TensorRT EfficientNMS plugin, torchvision GPU ops) |
| Orchestration | Synchronous, one request at a time | No overlap between stages | Pipeline overlap so stage N+1 of batch k runs while stage N of batch k+1 is in flight |
The pattern is consistent: the biggest single win is usually eliminating the device round-trips and the batch-of-one, not optimising any individual kernel. When pre-processing runs on the GPU with a library like NVIDIA DALI and NMS runs on-device via a TensorRT plugin, the data stops bouncing across PCIe on every request, and the CPU stops being the pacing item.
How do batch size and GPU occupancy interact for real-time detection?
Batch size governs how many execution slots remain busy; occupancy rises steeply at first, then flattens once the model’s arithmetic intensity saturates available parallelism. Past that plateau, larger batches add latency without adding throughput because you are already memory- or compute-bound. The useful batch size is the smallest one that reaches the plateau, because anything larger just inflates latency.
Finding it is empirical, not theoretical. A worked example makes the trade-off concrete:
Illustrative sizing exercise. Suppose a YOLO model runs a forward pass in roughly 4 ms at batch-of-one on a given GPU, and 12 ms at batch-of-32. Per-image cost has dropped from 4 ms to under 0.4 ms — roughly a 10× throughput gain — while the per-batch latency rose only 3×. If your latency budget is 20 ms, batch-32 fits comfortably. If the model reaches its occupancy plateau at batch-16 (say, 9 ms), batch-32 buys little extra throughput and costs latency, so batch-16 is the better operating point. These are illustrative numbers; the actual curve must be measured on your executor.
Real-time workloads (live video, on-camera detection) add a wrinkle: frames arrive at a fixed rate, so you cannot always wait to fill a large batch. Here the dynamic-batching wait window is the lever — it bounds how long a frame waits before dispatch. Streaming detection also benefits from where the model runs; for constrained or edge targets, a lighter model or a more efficient segmentation backbone like MobileSAM on constrained GPUs changes the batching maths entirely because the per-image cost is lower to begin with.
When should NMS and pre-processing run on the GPU?
For maximum throughput, run both the convolution layers and non-maximum suppression on the GPU rather than shuttling intermediate tensors back to the host. The device round-trip — copying decoded images up and detections back down over PCIe on every request — is pure overhead that dynamic batching alone does not remove. Keeping the whole pipeline resident on the device removes it.
The exceptions are real, though:
- Pre-processing stays on CPU when the CPU is idle anyway (batch offline jobs where the GPU is the scarce resource and CPU decode overlaps cleanly), or when input decoding needs codecs that lack a GPU path.
- NMS stays on CPU when detection counts per image are tiny and the host NMS is genuinely negligible, or when a custom NMS variant has no GPU implementation and the port cost outweighs the gain.
- Both move to GPU for any sustained real-time or high-QPS serving workload, which is the common case that motivates the question in the first place.
For decode specifically, on-GPU JPEG decoding via nvJPEG and batched augmentation via NVIDIA DALI keep the tensor on the device from the start. For NMS, TensorRT’s fused NMS plugin runs suppression as part of the engine, so the model output never leaves the GPU before the final boxes are ready.
How do I tell whether my pipeline is limited by structure or by kernels?
Profile first: knowing whether you are bound by host-device transfer, kernel compute, or memory bandwidth tells you whether to redesign the pipeline or tune low-level operations. Run this before writing a single CUDA kernel.
Structure-vs-kernel diagnostic checklist
- Check GPU utilisation under load. If it sits below ~40% (as a diagnostic threshold and rule of thumb) while requests queue, the limit is structural (starvation), not kernel efficiency. This is the single most informative reading, and it connects directly to how you read GPU utilisation across the three pillars of observability.
- Check the batch size in production. If it is effectively one, you have not started the real optimisation yet.
- Profile the wall-clock split across the three stages. If pre-processing plus NMS dwarf the forward pass, the bottleneck is the pipeline, not the model.
- Watch a CPU core. A pegged single core during inference is the classic signature of CPU-bound pre/post-processing.
- Measure PCIe traffic. Heavy per-request host-device transfer means you are paying the round-trip tax that on-GPU pre/post-processing removes.
- Only if utilisation is already high and the forward pass dominates does kernel-level work — occupancy tuning, kernel fusion, precision changes — become the right lever.
The metrics worth tracking to know an optimisation paid off are narrow and operational: frames-per-second per GPU, p99 inference latency, and cost per million detections. These are exactly the levers a GPU Performance Audit quantifies before recommending any kernel-level work, and they are the same throughput-and-tail-latency framing that separates ML metrics that explain GPU bottlenecks from metrics that mislead. A rise in mean throughput that comes with a p99 latency blowout is not a win; the metrics have to be read together.
FAQ
How should you think about yolo inference in practice?
Batching multiple frames together changes memory access patterns and kernel occupancy, amplifying throughput beyond what serial frame-by-frame inference achieves. In practice, inference is three stages — pre-processing (decode, resize, normalise), the GPU forward pass, and NMS post-processing — that by default run across CPU and GPU with a device round-trip on each side. The forward pass is fast; the surrounding pipeline is usually what limits throughput.
Why does batching change YOLO throughput more than tuning individual kernels?
A single 640×640 image (illustrative dimensions) does not contain enough parallel work to saturate a wide GPU, so the device sits idle most of the time. Batching fills that width, amortises kernel-launch overhead, and raises occupancy. If the GPU is idle 85% of the time from starvation (as a sizing estimate), speeding up the busy 15% barely helps end to end, whereas feeding it properly turns idle time into useful work — which is why batching dominates kernel tuning.
What are the main stages of a YOLO inference pipeline and where do the bottlenecks sit?
The three stages are pre-processing, the model forward pass, and NMS post-processing. In a naive setup pre-processing and NMS run on the CPU with a device round-trip each, so the bottleneck usually sits there rather than in the model kernels. The biggest wins come from eliminating the round-trips and the batch-of-one, not from optimising individual kernels.
How do batch size and GPU occupancy interact for real-time detection workloads?
Occupancy rises with batch size until the model saturates the GPU, then plateaus; past the plateau, larger batches add latency without adding throughput. The useful batch size is the smallest one that reaches the plateau. For real-time streams, a dynamic-batching wait window bounds how long a frame waits before dispatch, keeping tail latency in check while still amortising kernel launches.
When should NMS and pre-processing run on the GPU instead of the CPU?
For any sustained real-time or high-QPS serving workload, both should run on the GPU to remove the per-request PCIe round-trip. Pre-processing can stay on CPU for offline batch jobs where the CPU is idle and decode overlaps cleanly, and NMS can stay on CPU when detection counts are tiny or no GPU implementation exists. The common serving case that motivates the question almost always wants both on-device.
How do I tell whether my YOLO pipeline is limited by algorithmic structure or by kernel-level inefficiency?
Check GPU utilisation under load first: below roughly 40% while requests queue means the limit is structural starvation, not kernel efficiency. Corroborate by checking the production batch size, profiling the wall-clock split across the three stages, watching for a pegged CPU core, and measuring PCIe traffic. Only when utilisation is already high and the forward pass dominates does kernel-level work become the right lever.
What throughput and latency metrics should I track to know a YOLO inference optimisation actually paid off?
Track frames-per-second per GPU, p99 inference latency, and cost per million detections. Read them together — a rise in mean throughput that comes with a p99 latency blowout is not a win. These are the same operational levers a GPU Performance Audit quantifies before recommending any kernel-level work.
Feed starvation kills GPU utilization faster than slow kernels
Measure end-to-end latency with a representative workload—preprocessing overhead often dominates before the first tensor reaches the network. Classify the bottleneck as algorithmic (batching, layout, pipeline overlap) or micro-level before you touch a kernel; that classification is exactly what a GPU Performance Audit produces, and getting it wrong is how teams spend weeks tuning the fast part of a starved pipeline.
