Pipeline Parallelism › The Pipeline Bubble Problem

The most important challenge in pipeline parallelism is not communication or memory but time: the problem of stages sitting idle while waiting for work. To understand why this happens, it is worth tracing through exactly what happens when you run a model through a naive pipeline.

Imagine four pipeline stages labeled S1 through S4. A single input mini-batch $B_1$ enters S1 at time step 1. S2 cannot begin until S1 finishes, S3 cannot begin until S2 finishes, and so on. The result is a startup pattern where:

• At time step 1: S1 is working, S2, S3, and S4 are idle.
• At time step 2: S2 is working on the result from S1, while S1 is idle (finished its pass and waiting), and S3 and S4 are still idle.
• At time step 3: S3 is working, while S1, S2, and S4 are all idle.
• At time step 4: S4 is working, while S1, S2, and S3 are all idle.

After the forward pass completes, the backward pass flows in reverse: S4 computes gradients first, then passes them to S3, and so on back to S1. This also creates a sequential chain of idle time, now in reverse order. The combined result is that the startup and teardown phases of the pipeline are almost entirely wasted: at any given moment, only one stage is doing useful work.

To quantify the bubble fraction precisely, consider $D$ pipeline stages processing $M$ micro-batches. The startup ramp contributes $D - 1$ idle time steps before all stages can participate (the first stage begins immediately, but the $k$-th stage cannot begin until $k - 1$ time steps have passed because it must wait for activations from earlier stages). The drain phase at the end is symmetric, contributing another $D - 1$ idle time steps as gradients move backward through the pipeline. Each stage performs $2M$ useful operations—$M$ forward and $M$ backward—and the full schedule lasts $2M + 2(D - 1)$ time steps. The bubble fraction is therefore:

where:

• $D$: number of pipeline stages (equal to the number of devices)
• $M$: number of micro-batches processed in each pipeline run
• $2(D-1)$: total idle time slots from the startup ramp and teardown drain

For large $M$, the denominator is dominated by $M$, so the bubble fraction approaches $(D-1)/M$, which converges to zero as $M$ grows. This asymptotic behavior is the key insight: the bubble overhead becomes negligible when $M$ is much larger than $D$.

To see this concretely with numbers: for $D = 4$ stages and $M = 1$ micro-batch, the bubble fraction is $3/4 = 0.75$. You are paying for four GPUs but using only about 25% of their combined compute capacity. Increasing $M$ to 8 micro-batches brings the bubble fraction to $3/11 \approx 0.27$. Increasing to 32 micro-batches brings it to $3/35 \approx 0.086$. The improvement continues, but each additional doubling of $M$ produces diminishing returns: going from $M = 32$ to $M = 64$ reduces the bubble from about 9% to about 4.5%.

It is worth understanding why the bubble cannot be eliminated entirely with any finite number of micro-batches. The startup ramp is unavoidable: the first micro-batch must travel through all $D$ stages sequentially before the pipeline can be considered "full." During this ramp-up, early stages are idle after finishing their portion of the first micro-batch and before the second micro-batch arrives. Similarly, the drain ramp at the end is unavoidable: after the last forward pass, backward passes flow back through the pipeline, and late stages finish first while early stages wait for gradients to arrive from later stages. As $M$ grows, these fixed startup and drain costs become a smaller fraction of the schedule and approach zero asymptotically.