Pipeline Parallelism › Summary

Pipeline parallelism splits a deep model vertically along its layer dimension, assigning contiguous groups of layers to separate devices. Each device holds a fraction of the total parameters, enabling models that far exceed a single GPU's memory capacity to be trained in practice.

The core challenge is the pipeline bubble: the idle time that accumulates because later stages must wait for earlier stages to produce activations, and because gradients must drain backward through all stages before the next optimizer step. The bubble fraction is $(D-1)/(M+D-1)$, where $D$ is the number of stages and $M$ is the number of micro-batches. Increasing $M$ is the primary strategy for driving the bubble fraction toward zero, at the cost of a larger effective batch size and more time between weight updates.

GPipe addresses the bubble by processing all $M$ micro-batches through the complete forward pass before beginning any backward passes. This achieves good device utilization for large $M$ but requires storing all micro-batch activations simultaneously, leading to activation memory that scales as $M \times D$. GPipe uses gradient checkpointing to manage this memory at the cost of roughly 33% additional compute.

The 1F1B schedule interleaves forward and backward passes to bound activation memory at $D$ stages' worth of data regardless of $M$. The bubble fraction is identical to GPipe, but the memory efficiency of 1F1B makes it the standard choice for large model training: it allows much larger $M$ values without running out of activation memory, which in turn allows smaller micro-batches and better tensor core utilization.

The interleaved 1F1B variant assigns non-contiguous layer chunks to each device, reducing the bubble fraction by a factor equal to the number of chunks per device. This further reduces idle time at the cost of proportionally more inter-device communication volume. Whether the tradeoff is favorable depends on the interconnect bandwidth relative to compute throughput.

Key takeaways:

• Pipeline parallelism assigns consecutive layer groups to separate devices, allowing parameter count to scale linearly with device count while keeping per-device memory fixed
• The pipeline bubble is unavoidable but shrinks as the ratio $M/D$ grows; practical training runs use $M \gg D$ to keep bubble fractions below 10%
• GPipe and 1F1B achieve the same bubble fraction; 1F1B uses $M$ times less activation memory, making it the preferred schedule for large models
• The interleaved schedule reduces the bubble fraction by factor $1/V$ at the cost of $V$ times more communication
• In production systems, pipeline parallelism is combined with tensor parallelism and data parallelism into 3D parallelism, allowing training of models with hundreds of billions of parameters across thousands of GPUs