PaLM (2022) › Limitations

PaLM's training and inference costs created a substantial access barrier. Reproducing the 540-billion-parameter run required infrastructure available to only a small number of organizations.

The model's performance varied across tasks and domains. While it demonstrated strong capabilities in many areas, performance on specific tasks could be inconsistent, and the model sometimes struggled with tasks that required very specialized knowledge or reasoning patterns. This variability meant that deploying the model in production applications required careful evaluation and potentially fine-tuning for specific use cases.

The model could generate harmful, biased, or incorrect text. A deployment therefore required evaluation on its intended use, output controls, and human review where errors carried material consequences.

The model's size and computational requirements made it difficult to deploy in resource-constrained environments. Applications requiring real-time responses, deployment on edge devices, or operation with limited computational budgets faced significant challenges. The efficiency improvements from techniques like sparse attention helped but did not eliminate these constraints.

The benchmark suite did not capture every deployment concern. Factual accuracy, behavior on edge cases, and reliability over longer interactions required additional testing.