PaLM (2022) › 2022: PaLM
History of Language AIAll chapters PaLM (2022) › 2022: PaLMOne section · Sections in this chapter1 / 7 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
By 2022, language-model research was testing how performance changed with parameter count, training data, and compute. GPT-3 had reached 175 billion parameters in 2020. Google used its Pathways software and TPU infrastructure to train a substantially larger dense model across multiple TPU pods.
The resulting Pathways Language Model (PaLM) had 540 billion parameters, more than three times GPT-3's count. Google evaluated it on natural-language tasks, multilingual benchmarks, reasoning problems, and code generation.
PaLM improved over smaller comparison models on many reported evaluations. The paper highlighted few-shot results, multilingual transfer, arithmetic and reasoning tasks, and code generation. Performance still varied across tasks, so the results supported scaling trends rather than a general claim that the model reasoned reliably.
PaLM became a prominent data point in scaling research: a dense 540-billion-parameter model could be trained across thousands of accelerators and evaluated on a broad benchmark suite. Its cost and infrastructure requirements also showed how concentrated such experiments had become.