Foundation Models Report (2021) › Defining Foundation Models

The report's solution was to formally define "foundation models" as "models that are trained on broad data at scale and can be adapted to a wide range of downstream tasks." This definition captured three essential characteristics that distinguished these systems from previous AI approaches. First, foundation models are trained on "broad data".diverse, large-scale datasets covering multiple domains, languages, or modalities. Second, they are trained "at scale," meaning they use substantial computational resources and massive datasets. Third, they can be "adapted to a wide range of downstream tasks," meaning the same model can perform well across many different applications with minimal task-specific training.

This definition was deliberately inclusive. This included systems that might differ in architecture, training procedure, or specific capabilities, but shared these core characteristics. GPT-3, BERT, CLIP, and GPT-4 all qualified as foundation models, despite their differences. The term provided a conceptual umbrella under which diverse systems could be analyzed together, allowing researchers to identify patterns and principles that applied across the category.

The report's formalization of this concept provided immediate benefits. It gave researchers and decision-makers a common vocabulary for discussing these systems. When someone mentioned a "foundation model," listeners understood they were discussing systems with these three characteristics. This shared language supported more coherent conversations about the models' capabilities and risks. The definition also highlighted what distinguished the paradigm: beyond larger models and better training procedures, it described models trained on broad data that could be adapted across diverse applications.

The framework helped organize the rapidly evolving field. Instead of treating each new system as completely distinct, researchers could build on shared principles. Studies of GPT-3's few-shot learning capabilities could inform understanding of other foundation models. Analysis of BERT's transfer learning properties could generalize to other systems. The definition created a category for studying shared characteristics and limitations across foundation models.