Pipeline Parallelism › Worked Example: Four-Stage 1F1B Pipeline

To make the scheduling mechanics concrete, let us trace a 4-stage pipeline with $M = 4$ micro-batches through a standard 1F1B schedule. Label the stages S1 through S4 and the micro-batches B1 through B4. Use "F$k$" to denote the forward pass of micro-batch $k$ and "B$k$" to denote the backward pass of micro-batch $k$.

The timeline proceeds as follows.

During the warmup phase, S1 runs F1. After S1 completes F1, S2 can start F1 while S1 starts F2. After S2 completes F1, S3 can start F1; and so on. The warmup expands the pipeline: after $D - 1 = 3$ time steps, all stages are simultaneously occupied with forward passes. At this point, S4 completes F1 and can begin B1, which marks the start of the steady-state phase.

During the steady state, S4 alternates: after completing B1, it runs F2, then B2, then F3, then B3, then F4, then B4. S3 runs its forward passes one step behind S4 and its backward passes one step after S4. S1 and S2 follow the same pattern at their respective offsets.

During the drain phase, after S4 completes F4 (the last forward pass), the pipeline empties as backward passes drain from S4 back to S1. No new forward passes enter, and stages become idle one by one as their remaining backward passes complete.

The total number of time steps for the complete run is:

$$t_{\text{total}} = 2M + 2(D - 1) = 2(4) + 2(4 - 1) = 8 + 6 = 14$$

where:

• $2M = 8$: forward and backward work time steps for four micro-batches
• $2(D-1) = 6$: bubble time steps from the startup ramp and drain

Of the 14 total time steps, 8 represent useful forward or backward work and 6 are pipeline overhead, giving a bubble fraction of $6/14 \approx 0.43$. The example is deliberately small to make the schedule traceable; in practice $M \gg D$, which makes the bubble fraction much smaller.

The activation memory at any stage in this 1F1B schedule is bounded by the maximum number of concurrent forward-pass activations. In standard 1F1B, this maximum is $D = 4$ micro-batches' worth of activations per stage, regardless of $M$. At the peak of the warmup phase, S1 has processed F1, F2, F3, and F4 in the forward direction and is holding all four sets of activations. It then frees each set as the corresponding backward pass returns from the later stages. The peak is $D$ activations, confirming the $D \times A_{\text{layer}}$ memory bound.