PaLM (2022) › The Problem

GPT-3 and similar models produced fluent text but remained inconsistent on arithmetic, logical puzzles, and multi-step problems. Scaling experiments asked whether a larger pretrained model would improve few-shot performance on these evaluations without task-specific parameter updates.

Performance outside English was another concern. PaLM's training mixture included many languages, allowing the paper to measure cross-lingual and multilingual few-shot behavior in one model.

Code provided a separate test of transfer from a mixed text-and-code corpus. Generating a short solution from a prompt differed substantially from maintaining a multi-file software project, so benchmark gains did not establish general software-development competence.

Training 540 billion parameters exceeded the capacity of one accelerator pod. Pathways coordinated work across multiple TPU v4 pods. The training system also had to partition the model, manage communication, and recover from failures across thousands of chips.

Memory and communication constrained how parameters, optimizer state, and activations could be distributed. The engineering problem was therefore not simply to add chips, but to keep them supplied with work while coordinating model updates.

The question of how to best utilize computational resources also remained open. Previous models had demonstrated that larger models could perform better, but the optimal balance between model size, training data amount, and training procedures was not well understood. Researchers needed systematic approaches to training at unprecedented scales while ensuring that computational resources were used efficiently.