Bias Measurement › Summary

Bias measurement in language models spans four major methodological categories, each probing a different layer of the system.

Embedding association tests (WEAT, SEAT) measure whether a model's internal representations encode stereotypical associations between demographic groups and attribute concepts. They work by computing cosine similarity between target word embeddings and attribute embeddings, quantifying the differential association with an effect size statistic. SEAT extends WEAT to contextualized sentence encoders by wrapping target words in neutral sentence templates. These tests are fast and interpretable, but they measure associations in representation space, not behavior. A model can show large WEAT effects without causing downstream harm, and can cause downstream harm without showing large WEAT effects.

Generation bias metrics assess what a model says rather than what it encodes internally. The stereotype score measures whether a model assigns higher probability to stereotypical completions over anti-stereotypical ones. The regard score uses a classifier to measure the social sentiment expressed in generated text about different demographic groups. Toxicity metrics quantify whether harmful content is generated more frequently for certain identity groups. Generation metrics are more directly connected to user experience than embedding tests, but they depend on the quality of the classifiers used to evaluate model outputs, which may themselves encode biases.

Classification bias metrics apply when a language model is used as a classifier. Demographic parity difference measures whether positive prediction rates are equal across groups. Equalized odds difference checks whether true positive and false positive rates are equal. Disparate impact ratio uses a multiplicative comparison, with the 80% rule as a commonly applied threshold. These metrics formalize fairness mathematically, but different definitions are mutually incompatible when base rates differ across groups. The impossibility theorem means every fairness-aware deployment must choose which definition to prioritize, and that choice reflects value judgments, not just technical constraints.

Bias benchmarks (StereoSet, CrowS-Pairs, WinoBias, BBQ, BOLD) provide standardized test sets for reproducible comparison. StereoSet measures stereotype preference using fill-in-the-blank sentences and a language model score that guards against trivial gaming. CrowS-Pairs uses minimal-contrast sentence pairs scored by pseudo-log-likelihood. WinoBias targets coreference resolution in occupational contexts. BBQ evaluates whether models use stereotypes when questions are ambiguous. BOLD measures sentiment and toxicity disparities in open-ended generation. Each benchmark captures different aspects of bias and each has known limitations: coverage gaps, annotation errors, and susceptibility to gaming all affect the reliability of benchmark scores as proxies for real-world fairness.

The honest conclusion is that no single measurement method is sufficient. Embedding association tests can miss generation-level bias. Generation metrics miss classification bias. Benchmarks can be gamed. The best practice is to apply multiple measurement techniques, document the specific word lists and evaluation sets used, and interpret results in the context of the deployment domain where harm would occur. In the next chapter, we will examine bias mitigation: once you know where the bias is, how do you reduce it?