Two ways to distribute a model across GPUs

Model size and throughput requirements frequently exceed single-GPU capacity, forcing distribution across multiple devices. The two primary strategies split the model differently, and choosing between them is rarely about preference. It is about which interconnect tier you actually have, and whether your workload is latency-sensitive or throughput-sensitive.

Tensor parallelism (TP) splits individual operations across GPUs. A single matrix multiplication is divided so each GPU computes a portion, then results are combined via all-reduce communication. Every GPU participates in every layer’s computation.

Pipeline parallelism (PP) splits model layers across GPUs. GPU 0 runs layers 1–10, GPU 1 runs layers 11–20, and so on. Each GPU runs complete operations but only for its assigned layers. Data flows through the pipeline sequentially.

In short: tensor parallelism splits individual operations across GPUs (low latency, high bandwidth requirement); pipeline parallelism splits model layers across GPUs (tolerates lower bandwidth, adds pipeline bubble overhead). The two are not interchangeable knobs — they sit on different points of the bandwidth-versus-latency frontier, and the cluster’s interconnect topology decides which one is even feasible.

How tensor and pipeline parallelism differ in practice

Dimension	Tensor parallelism	Pipeline parallelism
Communication pattern	All-reduce after every operation	Point-to-point between adjacent stages
Bandwidth requirement	Very high (NVLink-class, on the order of 600+ GB/s)	Moderate (PCIe or InfiniBand sufficient)
Latency per token	Low (all GPUs compute simultaneously)	Higher (sequential stage execution)
GPU utilisation	High (all GPUs always active)	Reduced by pipeline bubble (idle time between micro-batches)
Scaling limit	Typically 4–8 GPUs per TP group; communication overhead grows beyond	Limited by bubble fraction and memory per stage
Memory efficiency	Each GPU holds partitioned layer weights	Each GPU holds only its assigned layers’ full weights
Typical framework support	Megatron-LM, DeepSpeed, vLLM, TensorRT-LLM	DeepSpeed, PipeDream-style schedulers, Megatron-LM

These structural tradeoffs emerge from deployment observations across many systems rather than isolated benchmark measurements. The bandwidth numbers are framing references, not promises — the actual achievable communication rate on any specific cluster depends on the NIC, switch fabric, and topology.

When does tensor parallelism win?

TP is optimal when:

• GPUs are connected via high-bandwidth interconnect (NVLink within a node, typically around 900 GB/s on H100 per NVIDIA’s published specifications)
• Latency matters more than throughput (real-time inference, interactive applications, chat-style serving)
• The model fits across a small number of GPUs (2–8) with TP alone
• Every GPU should contribute to every token’s computation

The constraint is structural. TP requires all-reduce communication after each tensor operation. On NVLink, this adds microseconds. On PCIe (roughly 64 GB/s for Gen4 x16) or on cross-node InfiniBand (commonly 200–400 Gb/s per port), the communication time dominates computation time, and TP collapses into something that runs but no longer accelerates. In our experience, teams who try to extend TP across node boundaries usually discover this the hard way — by watching utilisation drop without throughput rising.

When does pipeline parallelism win?

PP is optimal when:

• GPUs are connected via lower-bandwidth links (cross-node InfiniBand, or PCIe within a node without NVSwitch)
• The model is large enough to require many GPUs (16+)
• Throughput matters more than per-request latency
• The pipeline bubble can be amortised by filling it with micro-batches

The pipeline bubble is the unavoidable cost. When a pipeline starts, only GPU 0 is active; the others wait their turn. At the end of a batch, GPUs drain sequentially. The idle fraction is approximately (p − 1) / (p − 1 + m), where p is pipeline stages and m is micro-batches. With 8 stages and 32 micro-batches, bubble overhead is on the order of 18%. With fewer micro-batches — which is exactly what inference traffic often produces — the bubble grows quickly. This is one of several reasons inference deployments cannot simply inherit a training cluster’s parallelism plan; the broader pattern is covered in our analysis of why training and inference are fundamentally different workloads.

Is pipeline parallelism worth the bubble cost?

That depends on whether the alternative is feasible at all. If the model does not fit in a single TP group’s memory budget, pipeline parallelism is not a choice — it is the only path to running the model. The right question is not “TP or PP” but “what is the minimum bubble fraction I can tolerate given my latency target and my interconnect tier?”

The hybrid reality: TP within nodes, PP across nodes

Production deployments of large models almost always use both strategies simultaneously:

• TP within a node, leveraging NVLink’s high bandwidth for low-latency intra-operation communication
• PP across nodes, tolerating InfiniBand’s lower bandwidth for inter-stage communication

A 32-GPU deployment across 4 nodes commonly runs TP=8 (within each 8-GPU node) and PP=4 (across the 4 nodes). This maps the parallelism dimensions onto the interconnect tiers that can actually sustain them. Megatron-LM and DeepSpeed both ship with this kind of 2D/3D parallelism as the default for very large models, and frameworks like vLLM and TensorRT-LLM expose TP-degree as a first-class deployment parameter for inference.

The optimal strategy depends on interconnect bandwidth and model architecture — not just GPU count. A model that achieves X tokens per second with TP=4 on NVLink-connected A100s will reach a very different number with TP=4 on PCIe-connected A100s, because the all-reduce that was free on NVLink now sits on the critical path. Performance numbers do not transfer across topologies. This is one of the reasons benchmark consumers should treat “tokens/sec on N GPUs” as incomplete information until the parallelism plan and interconnect tier are named.

Data parallelism: the third dimension

Identical model replicas run on separate GPUs under data parallelism, with each device consuming distinct input batches. DP is the simplest form: each GPU holds a complete model copy, processes a different batch, and synchronises gradients via NCCL all-reduce at the end of the step. It scales throughput close to linearly with GPU count when communication is well-overlapped with computation, but it requires each GPU to hold the full model in memory — which is exactly the constraint TP and PP exist to break.

The full 3D combination is:

• TP within nodes for latency-bound intra-operation work
• PP across node groups for fitting the model
• DP across replica groups for throughput

The specific combination for your deployment depends on model size, available GPUs, interconnect topology, and whether you optimise for latency or throughput. Inference-heavy deployments usually push TP and DP and minimise PP (because bubbles hurt tail latency); training-heavy deployments often accept a larger PP degree because they can amortise the bubble with hundreds of micro-batches and they care about steady-state throughput.

That divergence is why we keep the two workload classes apart in measurement rather than trusting a single figure of merit. A LynxBenchAI run emits Training, Inference, and Compute as three independently meaningful scores, each readable on its own as how that executor handles that class of work, with GT sitting above them as an ordinal aggregate rather than a replacement. The precision sets differ by class because the workloads do — fp32 and bf16 on training, fp16, int8, and fp8 on inference, fp64 through bf16 on compute — and each precision is reported separately behind a correctness threshold. Each class is also measured at its own saturation point: batch size is raised until throughput stops improving, so a laptop GPU and a data-centre part are each read where they actually plateau instead of at one arbitrary shared batch size.

How do you actually pick a strategy?

A practical decision rubric, in order:

1. Does the model fit on one GPU? If yes, start with DP and stop. Parallelism within the model is overhead you do not need.
2. Does it fit in a single node’s aggregate memory? If yes, TP within the node is almost always the right starting point.
3. Does it require multiple nodes? Then TP within each node, PP across nodes. Choose the TP degree to match the NVLink/NVSwitch domain.
4. Is throughput the goal? Add DP across replica groups once a single TP×PP group is saturated.
5. Is latency the goal? Maximise TP degree within the bandwidth domain, minimise PP stages, and accept a smaller deployment footprint per replica.

Step 3 is where most procurement-driven mistakes happen. Teams buy a cluster optimised for one parallelism plan and discover their workload needs another — most often, a training cluster (PP-heavy, throughput-tuned) being asked to serve interactive inference (TP-heavy, latency-tuned). A device that trains poorly and infers well should appear in the evidence as exactly that; collapse the two into one overall rating and you get a number that matches nobody’s workload, and the procurement mistake above becomes invisible until the cluster is already racked. LynxBenchAI treats the parallelism strategy — tensor, pipeline, hybrid — together with the interconnect topology as part of the AI Executor specification, because the same model on the same GPUs reaches different throughput under different partitionings. Before accepting any large-model performance claim as evidence, ask: do the parallelism strategy, interconnect bandwidth, and runtime scheduler match the deployment’s hardware-software stack, or was the published number achieved on a topology the procurement cannot reproduce?

Frequently Asked Questions

How does batch size interact differently with throughput and latency under tensor versus pipeline parallelism?

Raising batch size fills a pipeline’s micro-batch queue and shrinks the bubble fraction — (p − 1) / (p − 1 + m) improves as m grows — so throughput-oriented training runs benefit from it directly. Under tensor parallelism the batch mostly buys arithmetic efficiency while all-reduce stays on the critical path, so per-token latency degrades before throughput plateaus. That is why benchmark scoring raises batch size until throughput stops improving and reads each class at its own saturation point, rather than fixing one batch size across devices.

Why does a device sometimes look strong on training-style parallelism and weak on inference-style parallelism?

The two plans stress different tiers of the machine: pipeline-heavy training work leans on memory per stage and steady-state throughput, while tensor-parallel inference leans on interconnect bandwidth and tail latency. A part with generous memory but modest NVLink-class bandwidth can post good training numbers and poor interactive-serving numbers on the same model. Reporting Training, Inference, and Compute as three separate scores keeps that shape visible; averaging them into one rating hides the fact that matters most for the buying decision.

Can I reuse a published “tokens/sec on N GPUs” number for my own cluster?

Not without the parallelism plan and the interconnect tier attached. The same model on the same GPU model reaches a very different figure with TP=4 on NVLink than with TP=4 on PCIe, because the all-reduce that was nearly free moves onto the critical path. Treat any throughput claim as incomplete until the TP/PP/DP degrees, the fabric, and the runtime scheduler are named, and check that they are reproducible on the hardware you actually have.

Does minimising the pipeline bubble matter as much for training as for inference?

Much less. Training steps can be fed hundreds of micro-batches, which drives the bubble fraction down to a few percent and leaves steady-state throughput as the thing being optimised. Inference traffic arrives in small, irregular batches, so m stays low, the bubble stays large, and it lands squarely on tail latency — which is why latency-bound deployments minimise pipeline depth and push tensor parallelism inside the bandwidth domain instead.

Split the model or split the batch

Individual weight matrices are split horizontally across GPUs in tensor parallelism, while pipeline parallelism partitions the network vertically by assigning consecutive layers to different devices. Is that executor close enough to yours for the result to mean anything?