GPU Architecture › Why GPUs Dominate Deep Learning

Deep learning workloads are dominated by one operation: matrix multiplication. Training a transformer involves thousands of matrix multiplications per forward and backward pass. Every attention computation, every linear projection in the feed-forward network, every embedding lookup that touches a learned weight matrix, all reduce to general matrix multiplications. A naive CPU can perform these multiplications, but CPUs are optimized for sequential, latency-sensitive workloads rather than massive parallelism.

The key insight is that matrix multiplication is embarrassingly parallel. To compute the product $\mathbf{C} = \mathbf{A} \mathbf{B}$ where $\mathbf{A} \in \mathbb{R}^{m \times k}$ and $\mathbf{B} \in \mathbb{R}^{k \times n}$, each output element $C_{ij}$ is:

$$C_{ij} = \sum_{l=1}^{k} A_{il} \cdot B_{lj}$$

where:

• $C_{ij}$: the element at row $i$, column $j$ of the output matrix
• $A_{il}$: element from row $i$ of the left matrix, at column position $l$
• $B_{lj}$: element from column $j$ of the right matrix, at row position $l$
• $k$: the shared inner dimension, also called the contraction dimension

Every output element $C_{ij}$ can be computed independently of every other output element. There are $m \times n$ such elements, each requiring $k$ multiply-accumulate operations. A CPU with 16 cores can compute 16 of these simultaneously. A modern GPU can compute tens of thousands simultaneously.

This difference in parallelism is not just faster: it is a different computational regime entirely. A 1000x speedup on matrix multiplication changes which models are trainable at all. Architectures that were theoretically interesting but practically infeasible on CPUs became the workhorses of modern AI on GPUs. The transformer itself, with its quadratic attention complexity and enormous weight matrices, would never have become dominant if training had to happen on CPUs.

The parallel nature of matrix multiplication also maps cleanly onto the physical structure of a GPU. Rather than one powerful processing unit, a GPU is a collection of thousands of simpler units. Each unit handles one small piece of the computation, and all pieces proceed simultaneously. The result is that the total work finishes in roughly the same time a single CPU core would spend on one element, but all elements are done at once.

The design philosophies of CPUs and GPUs reflect fundamentally different engineering priorities, shaped by the types of workloads each was designed to handle.

CPUs optimize for latency: executing a single complex instruction stream as fast as possible. They dedicate most of their silicon area to caches, branch predictors, out-of-order execution units, and speculative execution hardware. These mechanisms minimize the time each individual instruction takes to complete. A modern CPU core can execute a single thread with extraordinary speed and flexibility, handling complex control flow, irregular memory access patterns, and computations with data-dependent branches. This is exactly what general-purpose software needs: web servers, compilers, operating systems, and databases all rely on the CPU's ability to handle varied, unpredictable instruction sequences efficiently.

GPUs optimize for throughput: executing many simple instruction streams simultaneously, even at the cost of each individual stream being slower. A GPU core, called a CUDA core in NVIDIA terminology, is much simpler than a CPU core. It cannot perform branch prediction or out-of-order execution. It executes instructions in lockstep with thousands of sibling cores. This simplicity means you can fit thousands of them on the same silicon die that holds just a few dozen CPU cores.

The tradeoff is stark and consequential. If you need to sort a list of 1000 items, a CPU does it faster because the sorting algorithm is sequential and latency-sensitive. If you need to multiply two 4096x4096 matrices, a GPU wins by an enormous margin because every output element can be computed in parallel. Deep learning training is almost entirely the second kind of problem: massive, regular, predictable computations that happen to be independent of each other.

This fit is no accident of history. The transformer architecture was co-designed with GPU hardware in mind. The attention mechanism, the linear projections, the feed-forward expansion, all of these are matrix operations that saturate GPU parallelism. Understanding the hardware helps you understand why the architecture looks the way it does.