PaLM (2022) › The Solution
History of Language AIAll chapters PaLM (2022) › The SolutionOne section · Sections in this chapter3 / 7 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Google combined a decoder-only transformer with distributed training through Pathways. The software coordinated a model whose parameters and computation exceeded the capacity of any single TPU pod.
The Pathways System
Pathways provided the software layer for training across multiple TPU pods. It coordinated distributed computation and parameter updates while handling failures in a large accelerator fleet.
Pathways coordinated thousands of TPU v4 chips across multiple pods for one training run. This infrastructure made the 540-billion-parameter experiment feasible, though it did not make such training inexpensive or broadly accessible.
Architecture and Training Optimizations
PaLM was a dense decoder-only transformer. Its paper described architectural choices intended to improve training quality and serving efficiency, while Pathways handled distribution across the accelerator fleet.
The training stack partitioned model state and computation across devices and used lower-precision arithmetic where appropriate. These implementation choices reduced memory and communication costs but did not remove the large resource requirement.
Training Data and Procedures
PaLM was trained with next-token prediction on a mixture of text and code. The corpus drew from books, websites, and other text sources, plus code in multiple programming languages.
Filtering changed the composition of the training mixture but could not guarantee clean or unbiased data. The paper also evaluated toxicity and bias, documenting risks that remained after training.
Scaling to 540 Billion Parameters
The 540-billion-parameter run reflected a choice among model size, token count, and available compute. Later scaling work would examine whether different allocations of the same budget could train smaller models on more data.
PaLM's results therefore captured one point on the scaling curve, not a proof that parameter count alone was the optimal use of compute.