Pipeline Parallelism › Micro-Batching: Filling the Bubble

The solution to the bubble problem is to process multiple micro-batches simultaneously. A micro-batch is a small chunk of the full mini-batch. Instead of waiting for one full mini-batch to complete its forward pass before starting the next one, we split the mini-batch into $M$ equal micro-batches and inject them into the pipeline as fast as the first stage can produce them.

The pipeline schedule looks quite different with multiple micro-batches in flight. While S1 is processing micro-batch $B_2$, S2 is simultaneously processing micro-batch $B_1$. While S1 processes $B_3$, S2 processes $B_2$ and S3 processes $B_1$. The stages are kept busy by overlapping the processing of different micro-batches on different stages. The bubble still exists at the beginning and end of the sequence (there is no way to avoid the startup ramp for a strictly sequential pipeline), but the bubble fraction decreases as $M$ increases because the useful work time grows while the ramp time stays constant.

The key insight is that all $M$ micro-batches share the same model weights and contribute to the same gradient update. At the end of the pipeline run, all stages have produced their partial gradients, these are accumulated across micro-batches, and the optimizer takes a single parameter update step. From the optimizer's perspective, the pipeline processed one mini-batch of size $B_{\text{total}} = M \times B_{\text{micro}}$. The decomposition into micro-batches is an implementation detail for managing pipeline utilization. It is transparent to the loss function and the optimizer.

One important subtlety is the choice of micro-batch size. Smaller micro-batches mean more micro-batches for the same total mini-batch size, which reduces the bubble fraction. But very small micro-batches can underutilize the GPU's tensor cores. Modern GPU tensor cores achieve peak efficiency when matrix dimensions are large multiples of 8 or 16 (and ideally 64 or 128 for the best hardware utilization on A100s and H100s). A micro-batch with too few sequences may result in matrix multiplications with tiny leading dimensions, causing the GPU to spend most of its time on memory access rather than computation. There is a sweet spot where micro-batches are large enough to keep the GPU's arithmetic units busy but small enough that $M$ is large enough to suppress the bubble. Finding this sweet spot requires profiling on the specific hardware configuration, though a rule of thumb is that the micro-batch size should be at least 8 sequences for standard transformer models.

The relationship between micro-batches, mini-batches, and gradient accumulation is worth understanding precisely, because it affects both memory usage and training stability.

During the forward pass for micro-batch $m$, each stage $k$ computes the activations $\mathbf{a}_{k,m}$ and passes them to stage $k+1$. Each stage also stores $\mathbf{a}_{k,m}$ for use during the backward pass. During the backward pass for micro-batch $m$, stage $k$ receives gradients $\nabla \mathbf{a}_{k+1,m}$ from stage $k+1$, computes gradients with respect to its own parameters, and passes $\nabla \mathbf{a}_{k,m}$ to stage $k-1$.

The parameter gradients are accumulated across micro-batches. After all $M$ micro-batches have completed their backward passes, stage $k$ holds the accumulated gradient:

where:

• $\nabla W_k$: the accumulated gradient used for the parameter update at stage $k$
• $\nabla W_{k,m}$: the contribution to the gradient from micro-batch $m$ at stage $k$
• $M$: the number of micro-batches per optimizer step

This accumulated gradient is mathematically equivalent to computing the gradient over the full mini-batch simultaneously, assuming the micro-batch splits are uniform and all micro-batches use the same parameter values. The optimizer step then proceeds exactly as it would in single-device training. This equivalence is what makes synchronous pipeline parallelism a valid distributed training strategy: the model sees the same gradient signal it would see in non-pipelined training, just computed in parallel across devices.

One practical consideration is that different micro-batches could in principle encounter different parameter values if weights are updated between micro-batches. The standard synchronous pipeline parallelism approach avoids this by processing all $M$ micro-batches before taking any weight update step. This ensures gradient coherence: all micro-batches see the same model weights, so their gradients can be averaged. Asynchronous variants that update weights between micro-batches introduce a form of weight staleness, where later micro-batches in the same optimizer step observe slightly different weights than earlier ones. Asynchronous schedules can potentially achieve better device utilization, but the staleness introduces gradient inconsistency that tends to harm convergence for large models. For this reason, synchronous pipeline parallelism is the standard approach in practice.

Understanding the relationship between micro-batch size, the number of micro-batches, and the total effective batch size is essential for correctly configuring pipeline parallelism training runs.

If each micro-batch contains $B_{\mu}$ sequences and there are $M$ micro-batches per optimizer step, the total batch size processed per step per pipeline replica is:

When this pipeline is also combined with data parallelism, where $R$ replicas of the full pipeline are run in parallel, the global effective batch size is:

Large-scale training runs often target a specific global batch size for training stability reasons (the learning rate and its schedule are typically tuned for a specific batch size). Pipeline parallelism and data parallelism together give you multiple levers to hit the target batch size: you can increase $M$ to reduce the bubble fraction, increase $R$ to add more data-parallel replicas, and adjust $B_{\mu}$ to satisfy hardware utilization constraints. The flexibility to tune these independently, rather than treating the batch size as a single monolithic parameter, is one of the practical benefits of the 3D parallelism approach used by modern large model training frameworks.