Representation Harms › Stereotyping

Stereotyping occurs when a model systematically associates a demographic group with a fixed set of traits, roles, or behaviors that overgeneralize from some members of the group to all members, or that encode historically loaded assumptions rather than current reality.

The distinction between a stereotype and a statistical correlation is not always obvious, but it centers on whether the generalization erases individual variation and whether it tracks historical power structures rather than neutral distributions. The association between "nurse" and femininity reflects a real occupational imbalance: women currently make up a large majority of nurses in many countries. But a model that generates "she" far more often than "he" in nurse-related contexts encodes this imbalance as a natural or necessary feature of the role, rather than as a contingent outcome of labor market history. The same imbalance in a model trained on data from a society where men dominate nursing would simply run in the other direction.

Occupational stereotyping is among the most extensively studied forms of representation harm in language models. It manifests in two main ways: gender associations with specific occupations, and race associations with economic roles.

In template-based probes, researchers construct sentences like:

"The [occupation] said that [pronoun] had finished the report."

and measure how often models assign masculine versus feminine pronouns to different occupations. Models trained on large internet corpora consistently show:

• Higher feminine pronoun rates for nurse, receptionist, teacher, and librarian
• Higher masculine pronoun rates for engineer, CEO, pilot, and programmer
• These associations hold even when the model is given no explicit gender information

What makes this stereotyping rather than accurate reflection is that the associations are stronger than the underlying occupational distributions warrant, they resist updating when contradicting evidence is provided in context, and they cluster in ways that map directly onto traditional gender role hierarchies rather than being randomly distributed.

The resistance to contextual updating is particularly revealing. A well-calibrated model should assign higher feminine pronoun probability to "engineer" when the preceding context says "she graduated with a degree in electrical engineering." Models often fail to do this: the contextual signal is present and should theoretically resolve the ambiguity, but the prior association with masculine pronouns is strong enough to override it. This reveals that the stereotyping pattern goes beyond a simple reflection of data distributions; it is encoded in the model's representational geometry in a way that resists easy correction through context.

Beyond occupations, language models encode associations between demographic groups and personality traits. These associations are often revealed through embedding geometry: the relative distances and directions between word vectors in the model's representation space.

As discussed in the Bias Measurement chapter, the Word Embedding Association Test (WEAT) quantifies these associations by measuring the differential similarity between attribute word sets and target group word sets. Models consistently assign more pleasant associations to names typical of European-American groups and more unpleasant associations to names typical of African-American groups. This pattern emerges even when the words themselves carry no explicit demographic content: the association is built into the geometric structure of the representation space.

Large language models replicate and often amplify these patterns. When prompted to describe fictional characters with names drawn from different demographic groups, models generate descriptions that are systematically more positive, more competent-sounding, and more professionally focused for characters with names associated with high-status demographic groups. The gap isn't large on any individual generation, but it's consistent enough to be statistically reliable across thousands of samples.

Trait stereotyping also operates along axes that don't map neatly onto a single dimension. Warmth and competence are the two most studied dimensions in social psychology, and language models replicate the warmth-competence tradeoff documented in human social cognition: groups stereotyped as warm tend to be stereotyped as less competent, and vice versa. This means that even apparently positive stereotypes carry implicit costs. A model that consistently describes an immigrant character as warm and family-oriented while describing a similar native-born character as ambitious and professionally driven is encoding the same cultural hierarchy through opposite valence associations.

Representation harms don't operate along a single demographic dimension at a time. A person who is Black and a woman faces stereotyping patterns that differ from the sum of race-based and gender-based associations. The intersection creates new patterns: a Black woman described as a "professional" in generated text is more likely to have her credentials questioned than either a white woman or a Black man in the same scenario.

Measuring intersectional stereotyping requires going beyond single-axis analyses. Template probes need to systematically vary multiple demographic attributes simultaneously and look for interaction effects. Models show larger stereotyping effects at intersections, meaning the harms compound rather than average out.

The statistical reason for this compounding is that language model training data reflects real-world representation, and real-world representation of multiply marginalized groups is even more uneven than representation of each group individually. If women appear rarely in tech contexts in training data, and Black professionals appear rarely in leadership contexts, then Black women in tech leadership contexts appear extremely rarely. The model has almost no examples from which to learn anything other than the dominant pattern. This data sparsity at intersections is both a cause of intersectional bias and a reason why fixing single-axis biases doesn't automatically fix intersectional ones: you could achieve gender balance in leadership portrayals and racial balance in tech portrayals without producing any intersectionally balanced examples at all.

The implication for measurement is that fairness benchmarks built on single-axis comparisons can produce false confidence. A model can pass gender parity tests and racial parity tests while still systematically erasing Black women from professional roles. Complete representation harm auditing needs to treat intersectionality as a first-class concern, not an afterthought.

A more subtle form of stereotyping occurs through adjective selection rather than pronoun assignment. When generating descriptive passages about characters from different demographic groups in equivalent roles, models apply different adjective sets. Characters from one group are described as "driven" and "strategic"; characters from another group doing the same thing are described as "enthusiastic" and "hardworking." Both sets of adjectives are positive, but they encode very different assumptions about the source of success: one implies competence and agency, the other implies effort applied to compensate for something.

This adjective selection pattern is harder to measure than pronoun assignment because it requires semantic analysis rather than simple counting. Researchers typically address it through embedding-based analysis of the adjective distributions across groups, or through crowdsourced human rating of whether adjective sets connote equivalent or different underlying attributes. The pattern is reliable: it appears across multiple models and multiple demographic dimensions, and it aligns with the implicit association patterns documented in social psychology experiments using the Implicit Association Test (IAT).