Representation Harms › Limitations and Real-World Impact

Measuring representation harms is significantly harder than measuring allocative harms, and the limitations of existing measurement approaches are themselves part of the problem.

Template-based probes can only capture the stereotypes that researchers think to measure. They are effective at documenting known biases but poor at discovering new ones. A probe that asks about gender and occupations can't surface biases related to disability or sexual orientation unless it was specifically designed to test those dimensions. Real representation harms are intersectional and context-dependent in ways that template grids can't fully cover. The universe of possible representation harms is vast, and any fixed benchmark captures only a small slice of it.

Embedding-based tests have their own limitations. WEAT and its relatives measure associations in static embedding spaces, but modern language models use contextualized representations that vary with input. Association tests developed for word2vec and GloVe may not transfer cleanly to BERT-style encoders or decoder-only models like the GPT series, where representations depend on full context. Researchers have developed contextualized versions of WEAT (SEAT and CEAT), but these require choosing a sentence template to provide context, which reintroduces the coverage problem.

The framing of representation harms as a measurement problem can also create a false sense that they are solved once a metric is improved. Representation harms operate through accumulation: no single model output is harmful, but the aggregate pattern across millions of interactions shapes how users understand what is normal, who matters, and whose experience is central. Metrics computed on benchmark datasets can improve while the harms continue in deployment, if the benchmark doesn't capture the actual distribution of real-world uses.

There is also a deeper conceptual limitation. Representation harms are ultimately normative: they're harmful because we hold certain values about equity, inclusion, and human dignity. The measurement approaches described in this chapter operationalize those values into quantitative scores, but every operationalization reflects choices about what to measure and what to count. A benchmark that measures occupational gender balance implicitly accepts that binary gender is the relevant dimension. A baseline that compares to population proportions implicitly accepts that proportionality is the right standard. Neither choice is neutral, and the communities most affected by representation harms have often had no input into these choices.

Despite these limitations, the field has produced real progress. Large language models released since roughly 2021 show less stereotyping on standard benchmarks compared to their predecessors. Toxicity classifier bias against AAVE has been documented and addressed in some production systems. Gender-neutral pronoun handling has improved dramatically. The challenge going forward is to ensure that these improvements generalize beyond the benchmarks on which they were measured, and that the communities most affected by representation harms have ongoing input into what gets measured and how.

The next chapter on Hallucination Types examines a different failure mode, factual inaccuracy, and representation harms and hallucinations intersect in an important way: models that hallucinate contributions from historically marginalized groups both misrepresent history and actively harm the people who are misrepresented.