Pipeline Parallelism › Pipeline Parallelism: Stages, Micro-Batching, GPipe, and 1F1B

Training large language models requires distributing computation across multiple devices, and you have already seen two complementary strategies for doing this. Data parallelism splits the training dataset across workers so each device processes a different mini-batch while sharing gradient updates. Tensor parallelism splits individual weight matrices across devices so each GPU holds a slice of a single layer's parameters. Both strategies are powerful, but neither directly addresses a fundamental bottleneck: when a model is so large that a single layer or a modest contiguous block of layers cannot fit in one GPU's memory, and when the sequential depth of the network creates idle time because each layer must wait for the previous layer to finish before it can begin.

Pipeline parallelism takes a different approach. Instead of splitting data or splitting weight matrices, it splits the model itself along its depth, assigning contiguous groups of layers to different devices. The first device holds layers 1 through 8, the second holds layers 9 through 16, the third holds layers 17 through 24, and so on. This lets the model's total parameter count scale linearly with the number of pipeline stages, independent of any single device's memory capacity. The challenge, as we will see, is that a naive implementation leaves most devices idle most of the time, waiting for activations to arrive from the previous stage. Understanding how to minimize this idle time is the central engineering problem of pipeline parallelism, and the history of solutions to it has shaped how the largest models in existence are trained today.

The intuition behind pipeline parallelism is borrowed from industrial assembly lines. A car factory does not wait for one vehicle to be fully assembled before starting the next. Instead, different vehicles are in different stages of assembly simultaneously: the welding station, the painting station, and the final inspection station are all busy at the same time, each working on a different car. Pipeline parallelism applies exactly this logic to deep neural networks. While one micro-batch is being processed by the second stage of the model, the first stage can already be working on the next micro-batch. The goal is to keep every stage busy simultaneously, maximizing the fraction of time all devices are performing useful computation rather than sitting idle waiting for their neighbor to finish.

The history of pipeline parallelism in deep learning spans several important papers. The GPipe paper from Google Brain (2019) introduced a clean flush-based schedule that proved pipeline parallelism was practical at scale. The PipeDream paper from Microsoft Research (2019) demonstrated asynchronous variants with weight stashing. Megatron-LM from NVIDIA refined the approach with the 1F1B schedule and later the interleaved schedule, which became the standard for training hundreds-of-billions-parameter models. Understanding the progression from GPipe through 1F1B to interleaved scheduling is not just historical curiosity: each step was motivated by a concrete engineering constraint, and knowing those constraints helps you understand which schedule to choose for your own training runs.