Throughput Optimization › Parallelism Strategies

When a single GPU cannot hold the model or cannot achieve sufficient throughput, you must distribute the work across multiple GPUs. Several parallelism strategies exist, each with distinct tradeoffs. Choosing the right strategy, or the right combination, depends on model size, hardware topology, and whether you are optimizing for latency or throughput.

Tensor parallelism splits individual weight matrices across multiple GPUs. For a linear layer $y = Wx$, you can split $W$ along either the row or column dimension and assign each partition to a different GPU. After the split computation, an all-reduce operation aggregates results.

To be concrete: a feed-forward layer in a transformer with hidden dimension 8192 has weight matrices of shape $[8192, 32768]$ (for the expansion) and $[32768, 8192]$ (for the projection). With tensor parallelism degree $T = 4$, each GPU holds $[8192, 8192]$ of the expansion matrix. All four GPUs receive the same input $x$ (broadcast), compute their partial output in parallel, then sum the results via all-reduce.

The critical property of tensor parallelism is that it operates within a single layer. Every token in a batch passes through every GPU at every layer, requiring a synchronization (all-reduce) after each attention and feed-forward layer. For a model with $N$ layers, you perform $2N$ all-reduce operations per forward pass.

The communication cost of all-reduce scales with model dimension and the number of GPUs. On NVLink-connected GPUs (within a single node), all-reduce bandwidth is roughly 600 GB/s, making tensor parallelism efficient up to about 8 GPUs per node. Across nodes (using InfiniBand), all-reduce is 10-30x slower and tensor parallelism becomes communication-bound.

Tensor parallelism reduces peak memory per GPU by the parallelism degree $T$ and provides roughly $T$x throughput improvement for compute-bound operations, partially offset by communication overhead. The memory reduction is multiplicative with model size: a 70B model that does not fit on one 80GB A100 can fit on 4 A100s with $T = 4$. This is the primary use case for tensor parallelism on large models.

Pipeline parallelism assigns different layers to different GPUs. GPU 0 runs layers 1-8, GPU 1 runs layers 9-16, and so on. A micro-batch moves through GPUs sequentially, like an assembly line. Unlike tensor parallelism, the communication between pipeline stages is the activation tensors passed from one stage to the next, which is much smaller than the all-reduce traffic of tensor parallelism.

The key challenge is pipeline bubbles: the idle time that occurs while waiting for the previous GPU to finish its stage. With $K$ pipeline stages and micro-batch size $m$, the fraction of time wasted on bubbles is approximately:

$$\text{bubble fraction} = \frac{K - 1}{K - 1 + m}$$

where:

• $K$: the number of pipeline stages (GPUs in the pipeline)
• $m$: the number of micro-batches in flight simultaneously

To minimize bubble time, you need many micro-batches flowing through the pipeline simultaneously. This requires a large overall batch, which may conflict with latency requirements. Pipeline parallelism is best suited for large-batch, throughput-focused serving.

For example, with $K = 4$ pipeline stages and $m = 16$ micro-batches, the bubble fraction is $(4 - 1) / (4 - 1 + 16) = 3 / 19 \approx 16\%$. With $m = 4$, the bubble fraction rises to $3 / 7 \approx 43\%$. This shows why pipeline parallelism needs large batches to be efficient: small batch sizes waste nearly half the GPU time on bubbles.

Pipeline parallelism requires inter-GPU communication only at layer boundaries (to pass activations from one stage to the next). This communication is much lighter than the all-reduce in tensor parallelism, making pipeline parallelism viable across nodes connected by InfiniBand. The activation tensor at a layer boundary has size $B \times d_{\text{model}} \times 2$ bytes, which for a batch of 32 and dimension 4096 is about 500 KB per step, easily handled by 100 Gb/s InfiniBand.

Data parallelism is the simplest strategy: run full copies of the model on multiple GPUs and route different requests to different replicas. There is no communication between replicas during inference. A load balancer distributes incoming requests across replicas, and each replica operates independently.

Data parallelism provides linear throughput scaling: $N$ replicas handle $N$x more requests. Its limitation is that each replica must hold the full model in memory, so it requires GPUs large enough to fit the model. For a 7B model in FP16, each replica needs at least 14 GB plus KV cache, so an A100 80GB can host one replica with plenty of memory for batch data.

Data parallelism is the go-to strategy for medium-sized models (under roughly 10B parameters) on modern 80GB GPUs. It provides maximum throughput per dollar because there is no communication overhead between replicas. It also provides natural fault tolerance: if one GPU fails, the remaining replicas continue serving. This is a significant operational advantage over tensor and pipeline parallelism, where a single GPU failure takes down the entire inference group.

The trade-off is cost: you need enough hardware to fit one model copy per replica, and replicas do not share parameters. For a 70B model requiring 4 GPUs for tensor parallelism, running 4 data-parallel replicas requires 16 GPUs total. Whether this is better than a single 4-GPU tensor-parallel group depends on your throughput requirements and whether the 70B model's quality advantage justifies the cost.

Large models (70B+) require combining strategies. A common pattern is:

• Tensor parallelism within a node: Use NVLink bandwidth for low-latency all-reduce across 4-8 GPUs on one machine.
• Pipeline parallelism across nodes: Divide layers across multiple machines, using InfiniBand to pass activations between stages.
• Data parallelism for replicas: Run multiple instances of the tensor+pipeline-parallel model to handle additional load.

For example, a 70B model might use tensor parallelism degree 4 (4 GPUs per node), pipeline parallelism degree 2 (2 nodes), and data parallelism degree 4 (4 replicas of the 2-node setup), for a total of 32 GPUs. This configuration balances the communication overhead of each strategy against its memory reduction benefits.

The selection process for combining strategies is not purely mathematical; it involves empirical benchmarking. The relative importance of NVLink vs. InfiniBand bandwidth, the specific layer dimensions of the model, and the memory bandwidth of the hardware all interact in ways that are difficult to predict analytically. Most teams run a grid search over (tensor_parallel_degree, pipeline_parallel_degree) configurations and select the one with the best sustained throughput at their target latency.