Model Cards › Model Cards at Scale: Foundation Models

The model card framework was originally designed for relatively narrow, task-specific models. Applying it to modern foundation models that exhibit broad capabilities across thousands of tasks creates novel challenges. A general-purpose language model cannot have its intended use documented as specifically as a sentiment classifier; its evaluation cannot cover every task it might be applied to; its limitations cannot be fully enumerated without an effectively unlimited evaluation budget.

Several approaches have emerged to address this scale mismatch. One is the use of capability evaluations that characterize what a foundation model can and cannot do across a structured taxonomy of tasks, rather than listing specific intended uses. Instead of saying "intended for question answering about cooking," a foundation model card might document performance across a taxonomy of task types, allowing deployers to reason about where their specific use case falls.

Another approach is layered documentation, where a base model card documents the pre-trained model's characteristics and separate fine-tuning cards document specific deployment versions. This layering reflects the development process: a foundation model is first trained, then fine-tuned for specific tasks, and each stage can introduce or remove capabilities and limitations. A fine-tuned model's behavior can differ materially from the base model, and documentation that treats them as a single artifact will be inaccurate for at least one of them.

Red-teaming reports have emerged as a complement to traditional model cards for frontier models. Red-teaming involves deliberately trying to elicit harmful behaviors from a model before release, documenting what was found, and reporting what mitigations were applied. These reports address dual-use concerns and safety properties in more depth than traditional model card ethical considerations sections, and some developers now publish red-teaming reports alongside or as part of their model cards.

The challenge of documenting emergent capabilities is particularly acute. Large language models exhibit behaviors that emerge at scale and were not present in smaller versions of the same architecture. Some of these emergent behaviors are beneficial (in-context learning, chain-of-thought reasoning); others are concerning (increased capability for generating convincing disinformation at scale). Documenting emergent capabilities requires evaluation of the actual deployed model rather than extrapolation from smaller scale results, and the pace of capability development often makes documentation lag behind the actual capability state.