PaLM (2022) › Legacy and Looking Forward

PaLM demonstrated that Google could train a dense 540-billion-parameter transformer across thousands of TPU chips and obtain strong few-shot results on many reported tasks. It also made the resource concentration behind frontier scaling unusually clear.

Pathways was central to the result: the experiment depended on distributed systems that could keep a large TPU fleet coordinated through one run. That systems contribution was as important as the final parameter count.

The reasoning evaluations encouraged further work on arithmetic and multi-step tasks, while PaLM's own errors showed that scale had not made such behavior systematic.

Its multilingual results supplied evidence that a shared model could transfer across languages without a separate architecture for each one. Uneven data coverage and evaluation quality remained open problems.

The code results added to evidence that mixed text-and-code pretraining could support code understanding, generation, and debugging benchmarks. Later coding systems added tool use, execution feedback, and task-specific training.

PaLM is best read as a scaling and infrastructure result. It paired a very large dense transformer with Pathways and a broad evaluation suite. Questions about compute allocation and reproducibility remained, as did deployment reliability.

The chapter's main lesson is narrower than “bigger is always better.” Scaling improved many measured tasks, but the cost, data mixture, training design, and evaluation setup determined what those gains meant.