Representation Harms › What Makes Representation Harmful

Before diving into specific categories, it's worth being precise about what makes a representation pattern harmful rather than merely statistically uneven.

Representation harms arise when model outputs systematically shape how people perceive or are perceived, and when those patterns disadvantage certain groups. The mechanism isn't necessarily intentional. A model trained on text from the internet inherits the representational patterns present in that text, including centuries of accumulated cultural bias. But the absence of intent doesn't reduce the impact.

Three features characterize representation harms:

• Systematicity: The pattern appears consistently across many inputs, not just in isolated outputs
• Group targeting: The pattern affects people along demographic dimensions such as gender, race, nationality, religion, disability status, or sexual orientation
• Harm potential: The pattern either reflects real-world disadvantage or risks reinforcing it by shaping how readers, users, or downstream systems perceive the affected group

This last criterion is important for distinguishing representation harms from mere statistical associations. A model that consistently associates "chef" with masculine pronouns isn't just reporting a statistical correlation from its training data. It's participating in the reproduction of a cultural pattern that has historically excluded women from professional kitchens and made the path harder for those who did enter. Language models are now embedded in writing assistants, educational tools, search systems, and content generators that millions of people use daily. The representational patterns they encode have real consequences at scale.

One feature of representation harms that distinguishes them from many other AI failure modes is their capacity to compound over time. When language models are used at scale to generate text, their outputs enter the information environment. That information environment becomes training data for the next generation of models, through web crawls, through instruction tuning on human preferences, and through the implicit reinforcement of what users accept and engage with. A model that systematically underrepresents nonwhite female scientists in generated content produces text that makes such representation seem unusual. If that text is crawled and included in future training sets, it teaches the next model that the underrepresentation is normal. The bias reproduces itself.

This feedback dynamic is sometimes called a data flywheel for harm. Unlike a single biased decision, which is bounded, representational biases in generative models can amplify across successive model generations. The implication for measurement and mitigation is that representation harms need to be addressed proactively in training data curation, not just corrected in output filtering, because the problem restates itself at a higher level with each training cycle.

A second conceptual distinction that clarifies what counts as a representation harm involves the difference between describing the world as it is and shaping how the world is perceived. These sound like the same thing but they aren't. A model that describes the current gender distribution of nursing accurately is reporting a fact. A model that generates the same distribution as if it were natural, inevitable, or definitional of the role is encoding a normative claim as an empirical one.

The formal test is counterfactual sensitivity. If a model's portrayal of an occupational group would change substantially given equivalent real-world statistics from a different historical period or a different country, that's evidence the model is encoding contingent cultural patterns as universal ones. If the model applies the same male-dominated framing to 2023 nursing that it would have applied to 1950 nursing, despite nursing having become heavily female-dominated in the interim, the model is not reflecting current reality; it's reflecting the accumulated text weight of a particular cultural perspective.