Representation Harms › Measuring Representation Harms

Each category of representation harm requires different measurement approaches. The measurement techniques from the Bias Measurement chapter apply here, but they need to be adapted to the specific phenomena. Stereotyping requires probes that reveal distributional patterns across groups in model completions or internal representations. Erasure requires counting and frequency analysis at the corpus level. Demeaning associations require both automated classifiers and human judgment, because the harm is often contextual and resistant to automated detection. No single tool covers all three categories, which is why representation harm auditing typically involves a battery of methods rather than a single score.

Template-based probing is the most widely used approach for measuring stereotyping. You construct a set of templates with demographic group slots and measure the distribution of model outputs across groups.

The basic pattern:

1. Define a template: "The [group] [occupation] was known for being [adjective]."
2. Instantiate with different group labels: "Black", "white", "Asian", etc.
3. Collect model completions or measure conditional probabilities
4. Compute disparity metrics across groups

Well-known benchmarks built on this approach include:

• WinoBias: Coreference resolution templates designed to test gender bias in occupational contexts
• StereoSet: Sentence completion templates measuring stereotyped versus anti-stereotyped completions across race, gender, religion, and profession
• CrowS-Pairs: Sentence pairs with minimal edits swapping demographic group references, measuring which version the model prefers
• BBQ (Bias Benchmark for QA): Question-answering templates that probe stereotyping through ambiguous contexts requiring the model to make inferences about individuals

The strength of template-based probes is their interpretability and reproducibility. Given a template and a group set, different researchers can run the same probe on different models and compare results directly. The weakness is coverage: templates can only measure stereotypes that researchers anticipated when designing them. Emerging stereotypes, context-dependent patterns, and intersectional combinations that weren't built into the template set are invisible to the measurement.

A less obvious weakness is that template probes test model behavior on artificial, carefully constructed inputs that may not resemble the inputs models receive in deployment. A model could perform well on WinoBias while still encoding occupational stereotypes in subtler ways that the template format doesn't expose.

For models that produce dense vector representations, embedding geometry reveals stereotyping patterns through the relative positions of group-associated words in the representation space. The WEAT framework measures the differential association between group word sets and attribute word sets using cosine similarities.

For an embedding space, WEAT computes the differential association score $s$, which measures how much more strongly group $X$ words are associated with attribute $A$ than with attribute $B$, compared to group $Y$:

$$s(X, Y, A, B) = \sum_{x \in X} \left[m(x, A) - m(x, B)\right] - \sum_{y \in Y} \left[m(y, A) - m(y, B)\right]$$

where:

• $X$ and $Y$ are sets of target words for two demographic groups (e.g., European-American and African-American names)
• $A$ and $B$ are attribute word sets representing contrasting concepts (e.g., pleasant and unpleasant adjectives)
• $m(x, A)$ is the mean cosine similarity between target word $x$ and the attribute set $A$

The mean cosine similarity function is defined as:

$$m(x, A) = \frac{1}{|A|} \sum_{a \in A} \cos(\vec{x}, \vec{a})$$

where:

• $|A|$ is the number of words in attribute set $A$
• $\cos(\vec{x}, \vec{a})$ is the cosine similarity between the embedding vectors of words $x$ and $a$
• The sum averages the similarity of a target word to all members of the attribute set

A large positive value of $s$ means group $X$ is more associated with attribute $A$ (and group $Y$ more with $B$) than expected by chance. A value near zero indicates the groups are associated equally with both attributes. Statistical significance is assessed by permutation testing: randomly reassigning words to groups thousands of times and measuring how often the permuted score exceeds the observed score.

The WEAT effect size $d$ normalizes the raw association score by the pooled standard deviation of association scores across both target groups:

$$d = \frac{\bar{s}(X, A, B) - \bar{s}(Y, A, B)}{\text{std}(s(X \cup Y, A, B))}$$

where $\bar{s}(X, A, B) = \frac{1}{|X|} \sum_{x \in X} [m(x, A) - m(x, B)]$ is the mean association score for group $X$.

This normalization makes the effect size comparable to Cohen's $d$ from classical statistics, with $d > 0.2$ conventionally treated as small, $d > 0.5$ as medium, $d > 0.8$ as large, and $d > 1.0$ as very large. The Caliskan et al. (2017) paper reported effect sizes in the range $d \in [1.06, 1.81]$ for their ten canonical tests, all above the large threshold.

The WEAT paper by Caliskan et al. (2017) reported effect sizes for ten canonical tests in GloVe embeddings trained on Common Crawl. The results showed large effects across multiple group-attribute pairings, with the name-pleasantness test (European-American vs. African-American names, pleasant vs. unpleasant attributes) producing one of the highest effect sizes.

The gender-career test (test 6) produces the largest effect size, meaning the association between male terms and career words versus female terms and family words is the strongest single-axis bias documented in that embedding space. The race-based tests (3-5) cluster around $d = 1.3\text{–}1.5$, all well above the conventional threshold for large effects.

Measuring representation harms often requires counterfactual comparisons: what would the model have generated if the only thing that changed was the demographic group referenced? This is the idea behind counterfactual data augmentation (CDA), where you take a real text, swap group references (pronouns, names, demographic labels), and measure whether model behavior changes.

For stereotyping measurement:

1. Start with a sentence containing a demographic reference: "The female engineer struggled with the proof."
2. Swap the demographic reference: "The male engineer struggled with the proof."
3. Measure the model's likelihood or quality score for each version
4. A systematic difference reveals a representational bias

The challenge is that some "neutral" substitutions are not neutral. Swapping "Black" for "white" in a sentence about poverty doesn't produce an equivalent sentence because the socioeconomic contexts differ. CDA interpretations need to account for these complicating factors.

A more sophisticated version of counterfactual testing uses minimal pairs: sentences where only the demographic attribute changes and all other content is held constant. The difference in model perplexity, likelihood, or generation behavior between the two versions gives a direct estimate of the attributable bias for that attribute. When done at scale across many minimal pairs, this provides a distributional picture of how much demographic attributes influence model behavior across different contexts.

Erasure is most naturally measured at the corpus level: across a large set of generated texts, how often do members of different groups appear, and in what roles?

For a generated corpus $\mathcal{G}$ and a group $g$, the representation rate is:

$$r(g, \mathcal{G}) = \frac{\text{count}(g, \mathcal{G})}{|\mathcal{G}|}$$

where $\text{count}(g, \mathcal{G})$ counts occurrences of group-associated tokens (names, pronouns, explicit labels) in $\mathcal{G}$ and $|\mathcal{G}|$ is the total number of texts or tokens.

Comparing $r(g, \mathcal{G})$ across groups and against a baseline (population rates, representation in training data, or ground truth data) reveals erasure. More sophisticated analyses condition on context: does group membership affect appearance rates differently in professional versus social contexts?

The choice of baseline matters considerably for the interpretation. If you compare model output representation to population proportions, you're asking whether the model accurately mirrors demographic reality. If you compare to training data representation, you're asking whether the model amplifies or attenuates the biases in its source data. If you compare to ground truth data (e.g., actual representation of women in tech companies), you're asking whether the model accurately represents a specific domain. Each comparison answers a different question, and researchers need to be explicit about which baseline they're using and why.

For demeaning associations, toxicity classifiers trained on labeled data provide automated measurement, but they have known biases. Many classifiers themselves exhibit higher false positive rates for AAVE and other non-dominant linguistic varieties. Human evaluation remains necessary for calibrating classifier outputs and for catching subtle demeaning patterns that don't surface in automated scores.

The key evaluation dimensions for human raters:

• Sentiment: Is the language used to describe the group positive, neutral, or negative?
• Agency: Are group members portrayed as active agents or passive objects?
• Competence framing: Is the group portrayed as capable, intelligent, and professional?
• Contextual appropriateness: Does the model apply consistent framing across groups in equivalent contexts?

Human evaluation introduces its own biases, since annotators bring their own cultural assumptions and demographic perspectives to the task. Best practice involves using diverse annotator panels, giving clear annotation guidelines with examples, measuring inter-annotator agreement, and analyzing whether annotator demographics correlate with annotation decisions. When annotators from the affected group disagree with annotators from other groups about whether a piece of text is demeaning, that disagreement is itself informative: it often indicates that the harm is real but only visible to people with the relevant lived experience.