Model Cards › Documenting Limitations Effectively

Limitation documentation is where model cards most often fall short of their potential. The challenge is converting technical findings into guidance that a non-specialist deployer can act on. Performance metrics mean different things in different contexts, and the gap between what a metric measures and what a deployer needs to know is often larger than it appears.

Every performance metric implies something about deployment risk, but that implication is not self-evident. A 5% error rate on a medical coding task sounds small until you consider that a large hospital processes thousands of claims per day: at that volume, a 5% error rate means hundreds of errors daily. A model card that reports the 5% error rate without contextualizing its implications at realistic deployment scales gives deployers an incomplete picture.

Better documentation translates metrics into deployment implications:

• "The model achieves 95% accuracy on our evaluation set. At a deployment volume of 1,000 queries per day, this implies approximately 50 errors daily. For high-stakes decisions, we recommend human review for cases where confidence scores fall below 0.85, which reduces expected errors to approximately 10 per day."

This kind of quantitative translation requires knowing something about intended deployment scale, which is another reason the intended use section and the limitations section should be written together rather than independently. The intended use section establishes the deployment context; the limitations section should be calibrated to that context. Error rates that are acceptable for a low-stakes recommendation system may be completely unacceptable for an automated administrative decision that affects people's benefits, employment, or legal status.

The deployment volume calculation also reveals something important about model selection. Organizations choosing between models often focus on a few tenths of a percentage point in accuracy. But if Model A achieves 95.0% accuracy and Model B achieves 95.3% accuracy, and both are deployed at scale, the practical difference is small. If Model A has well-documented subgroup performance and Model B does not, the documentation gap is a much more important consideration than the accuracy gap. This reasoning argues for treating documentation quality as a material factor in model selection rather than an administrative checkbox.

Disaggregated performance data is one of the most valuable things a model card can provide, and one of the most commonly omitted. When overall performance looks adequate but subgroup performance is substantially degraded for some populations, aggregate metrics hide a deployment risk.

The challenge with disaggregated evaluation is that it requires demographic annotations in evaluation data, which are expensive to collect, raise their own privacy considerations, and may not be available for all relevant subgroups. There is no simple solution to this challenge. Some relevant subgroups can be identified through linguistic markers (dialect, writing style, vocabulary) rather than demographic annotations. Proxy variables can sometimes capture relevant variation when direct demographic data is unavailable. Adversarial evaluation using inputs constructed to represent specific groups can surface disparities without requiring annotated demographic data.

A model card should be honest about which disaggregated analyses were performed and which were not, rather than presenting aggregate numbers as if they were the complete story. When disaggregated data is unavailable, a responsible card acknowledges this gap explicitly: "We were unable to evaluate performance by demographic subgroup due to the absence of demographic annotations in our evaluation set. Users deploying this model in contexts where demographic equity matters should conduct their own disaggregated evaluation before deployment."

This acknowledgment of uncertainty is not a weakness in the model card; it is information. A deployer who reads this acknowledgment knows that they need to conduct their own evaluation before deploying in equity-sensitive contexts. A deployer who reads a model card with no mention of disaggregated evaluation might incorrectly assume the analysis was done and found no problems. Explicit documentation of what was not measured is as important as documentation of what was measured.

Some failure modes are structural and predictable rather than statistical. A model trained on formal text will systematically underperform on informal text. A model with a knowledge cutoff will systematically fail on questions about events after that date. A classification model trained on balanced classes will have poorly calibrated confidence when deployed on a naturally imbalanced distribution.

These structural failure modes deserve their own documentation distinct from aggregate accuracy numbers. They are not random errors distributed uniformly across inputs; they are predictable degradations for specific input types that a careful deployer can identify and mitigate if they know to look for them. A model card that documents structural failures enables deployers to build targeted mitigations: preprocessing to normalize informal text before it reaches a model trained on formal text, explicit fallback behavior for questions that may relate to post-cutoff events, confidence recalibration for deployment on imbalanced distributions.

Adversarial inputs represent a category of known failure mode that is increasingly important to document as language models are deployed in security-sensitive contexts. Prompt injection, jailbreaking, and adversarial suffixes can cause models to behave in ways far outside their documented capabilities. Testing for these failure modes and documenting both the vulnerabilities found and the mitigations applied gives deployers critical information for designing system-level defenses.

The structural nature of many failure modes also implies something about how models should be monitored after deployment. Aggregate accuracy monitoring will not detect systematic degradation on a specific subgroup unless that subgroup is large enough to affect aggregate numbers. Monitoring strategies designed around known structural failure modes, where specific input types or subpopulations are tracked separately, provide much earlier warning of deployment problems than generic aggregate monitoring.