Data Parallelism › Data Parallelism

Training a large language model on a single GPU is impractical. The datasets are enormous, the models are massive, and a single forward-backward pass can take minutes. The first and most natural response to this problem is to ask: what if we simply used more GPUs running the same computation at the same time?

This is the core idea behind data parallelism. Instead of changing the model or splitting it across devices, you keep a complete copy of the model on every GPU and divide the training data among them. Each GPU independently computes gradients on its assigned batch, and then all GPUs exchange those gradients so that every copy of the model stays synchronized. The result is that you can process $N$ times as much data per second with $N$ GPUs, cutting wall-clock training time roughly by a factor of $N$.

Data parallelism is the oldest and most widely used form of distributed training. Before ZeRO, tensor parallelism, and pipeline parallelism became mainstream (topics we will cover in the next several chapters), data parallelism was the default strategy for scaling language model training. It remains the backbone of distributed training even when combined with other parallelism strategies. Systems like GPT-3, PaLM, LLaMA, and virtually every production language model training run in recent years have relied on data parallelism as their primary scaling axis.

The appeal of data parallelism is its simplicity. Conceptually, you are just running the same training code on multiple GPUs with different data. You do not need to redesign your model architecture, change your optimizer, or reason about complicated data flow between different parts of the model. The distribution is transparent: from the model's perspective, it is receiving gradients and applying updates just like single-GPU training. The parallelism lives entirely in the training harness.

This chapter covers the full lifecycle of data parallelism: the DDP algorithm that makes it work, the gradient synchronization protocol that keeps model replicas in sync, the all-reduce operation that powers that synchronization, the scaling behaviour you can realistically expect as you add more GPUs, and the practical configuration decisions you need to make when setting up a real distributed training job. By the end, you will understand how to use PyTorch DDP and why each of its design decisions exists.