GPUs don’t do the same parallelism as CPUs
CPU multithreading—8, 16, or 64 threads running independent tasks—forms the mental model most engineers bring to parallel computing discussions. GPU parallelism is fundamentally different in architecture, granularity, and the class of problems it accelerates. A modern GPU has thousands of cores (an NVIDIA A100 has 6,912 CUDA cores), but each core is far simpler than a CPU core. The design trades per-thread sophistication for massive throughput on data-parallel workloads — problems where the same operation applies to thousands or millions of data elements simultaneously.
Understanding this distinction is prerequisite to GPU optimisation. In our experience, engineers who approach GPU programming with a CPU threading mental model write code that underutilises the hardware by an observed range of 10–100× (observed pattern across our GPU audit engagements, not a benchmarked rate).
The execution model: warps, threads, and occupancy
SIMT (Single Instruction, Multiple Threads) organises GPU execution at the architectural level. Threads are grouped into warps of 32 (NVIDIA) that execute the same instruction simultaneously on different data. A GPU schedules thousands of warps concurrently, switching between them to hide memory latency — when one warp stalls waiting for data, another warp executes immediately. This is the foundational mechanism that lets a GPU keep its arithmetic units busy despite the long latency of global memory access.
| Dimension | CPU parallelism | GPU parallelism |
|---|---|---|
| Core count | 8–128 complex cores | 1,000–16,000 simple cores |
| Thread independence | Fully independent threads | Threads in warps execute same instruction |
| Context switching | Expensive (OS-level) | Near-free (hardware warp scheduling) |
| Memory model | Large caches per core (MB) | Small shared memory per SM (KB), high-bandwidth global memory |
| Ideal workload | Task-parallel, complex branching | Data-parallel, uniform operations on large arrays |
| Latency strategy | Minimise latency per thread | Hide latency through massive thread occupancy |
When does GPU parallelism win — and when does it not?
Thousands of lightweight cores tackle data-parallel problems on GPUs; CPUs deploy fewer, more sophisticated cores for task-parallel workloads with branching logic. The performance gap between serial and parallel execution grows non-linearly with problem size — Amdahl’s law governs the ceiling.
Where GPUs dominate:
- Matrix multiplication (the foundation of neural network computation) — scales near-linearly with core count
- Element-wise operations on large tensors (activation functions, normalisation)
- Convolution operations across spatial dimensions
- Batch processing where the same model processes thousands of inputs simultaneously
Where CPUs remain faster:
- Sequential algorithms with data dependencies between steps
- Workloads with heavy branching (if/else paths that diverge within a warp cause serialisation)
- Small problem sizes where GPU launch overhead exceeds the computation time
- Irregular memory access patterns that prevent coalesced reads
Amdahl’s law: the parallel ceiling
Serial bottlenecks bound the speedup achievable through parallelisation, as Amdahl’s law formalises. If, illustratively, 10% of a computation is inherently sequential, the maximum theoretical speedup from infinite parallel resources is 10× — regardless of how many GPU cores you add.
In practice, this means profiling to identify the serial fraction is the first step in any GPU parallelisation effort. The serial fraction is not always where engineers assume: data loading, preprocessing, host-device memory transfer (over PCIe or NVLink), and synchronisation points all contribute serial time that limits the parallel speedup. We see this pattern regularly when teams report “the kernel is fast but the end-to-end pipeline is not.”
Practical implications for AI workloads
PyTorch, TensorFlow, and JAX hide GPU parallelism behind high-level APIs, yet performance remains dictated by the execution model beneath. Understanding that a matrix multiplication fully utilises data parallelism while a custom loss function with conditional logic may serialise across warps explains why some model operations run an order of magnitude or more faster on GPU while others show no speedup. The same observation drives the API decision covered in CUDA vs OpenCL vs SYCL: choosing a GPU compute API for your workload — because the SIMT model is what each API ultimately exposes, the choice of API does not change the underlying execution rules.
The gap between “runs on GPU” and “runs efficiently on GPU” is the gap between launching kernels and launching kernels that match the hardware’s execution model. Closing that gap on a specific codebase is the work a GPU Performance Audit is scoped to do. Data layout, memory access patterns (coalesced reads against global memory, bank-conflict-free access to shared memory), and operation structure all determine whether the thousands of available cores are working or waiting.
FAQ
How is GPU parallelism different from CPU parallelism?
CPU parallelism uses a small number of complex, independent cores optimised to minimise latency per thread; GPU parallelism uses thousands of simple cores grouped into warps that execute the same instruction on different data. The GPU hides memory latency by oversubscribing warps rather than by minimising it.
What is the SIMT execution model?
Single Instruction, Multiple Threads. Threads are scheduled in fixed-size warps (32 on NVIDIA hardware) and the warp issues one instruction per cycle across all its lanes. Divergent control flow inside a warp serialises the divergent branches, which is the structural reason GPUs penalise heavy branching.
Why does branching hurt GPU performance?
Because threads inside a warp share an instruction pointer. When some lanes take the if branch and others take the else, the warp executes both branches sequentially with the inactive lanes masked off. That converts what should be parallel work into serial work for the duration of the divergence.
What workloads benefit most from GPU acceleration?
Data-parallel workloads where the same operation applies to many elements: dense matrix multiplication, convolutions, element-wise tensor operations, and large-batch inference. Workloads with irregular memory access, heavy branching, or small problem sizes typically do not benefit and may even regress once kernel-launch overhead is counted.
