Bias in Language Models › Bias in Training Data: A Closer Look

To understand how bias enters a language model, it helps to trace the path from raw text to learned representation. The journey from a raw internet crawl to a model prediction involves multiple transformation steps, and bias can enter or be amplified at each one.

Language models learn from statistical patterns: which words appear near which other words. When a word like "aggressive" co-occurs more frequently with words associated with one demographic group than another, the model learns a higher-probability association between that group and that trait. This is not a flaw in the learning algorithm; it is exactly what the learning algorithm is supposed to do. The flaw lies in treating historically produced statistical patterns as objective truth about the groups being described.

Consider a large corpus of English-language newspaper articles. Such a corpus would associate:

• "Emotional" more often with women than men in professional contexts
• Criminal behavior more often with Black individuals than white individuals when reporting names and descriptions
• Technical expertise more often with men in articles about software engineering

None of these associations are accurate representations of reality, but they accurately reflect patterns in how certain newspapers have historically covered these topics. A language model trained on such data will reproduce these associations in its outputs. The model is not making a judgment; it is generalizing from its training distribution. But when that generalization is applied to new contexts, it carries the prejudices embedded in the historical distribution forward into present decisions.

The mechanism is worth understanding at a somewhat deeper level. Word embeddings and contextualized representations encode words as vectors in a high-dimensional space, where semantic proximity in the training data translates to proximity in embedding space. If "nurse" co-occurs frequently with "she," "care," and "gentle," and rarely with "he," "authority," and "leadership," then the embedding for "nurse" will be geometrically closer to the embeddings for "she," "care," and "gentle." When the model uses these embeddings to make predictions, that geometric proximity shapes the output distribution. The model is not reasoning about gender; it is traversing embedding space, and the space it is traversing was shaped by the statistical patterns of a biased corpus.

Beyond harmful associations, systematic underrepresentation creates a different kind of bias. When certain groups, languages, or perspectives appear rarely in training data, the model has weak, poorly calibrated representations of them. This causes the model to:

• Perform worse on tasks requiring knowledge about underrepresented groups
• Default to stereotyped or generic descriptions when asked about underrepresented subjects
• Conflate distinct subgroups, treating all members of a broad category as interchangeable
• Fail entirely on text from certain languages or dialects

Underrepresentation harms are often less visible than harmful associations, because the model's failure mode is absence rather than presence: the model does not say something offensive; it simply does not say much of anything accurate. A model asked to describe the history of Yoruba literature may generate something that sounds plausible but is factually impoverished, drawing on superficial patterns rather than deep knowledge. A user without independent expertise in Yoruba literature may have no way to know that the response is inadequate.

This invisibility makes underrepresentation particularly insidious. A model that outputs a slur is obviously broken. A model that responds to questions about Swahili literature with generic, inaccurate platitudes may appear to be working correctly to a user who lacks the background to evaluate its claims. The harm falls entirely on the user, who receives subtly misleading information without any signal that the model is operating outside its area of competence.

Underrepresentation also interacts with uncertainty calibration in important ways. A well-calibrated model should express lower confidence when it has less training evidence to draw from. In practice, many language models express similar surface-level confidence regardless of how much relevant training data existed. This means users cannot easily distinguish between the model's high-quality responses (where it has deep training coverage) and its low-quality responses (where it is essentially confabulating from indirect evidence). The burden of verifying accuracy falls on users, who are disproportionately harmed when they belong to underrepresented groups and may lack alternative sources of accurate information to compare against.

The path from the open internet to a training corpus involves many filtering decisions, each of which can introduce or amplify bias. Common Crawl data is typically filtered using quality signals such as perplexity relative to a reference language model or presence in curated lists of high-quality domains. Both of these signals systematically favor content that resembles text already produced by dominant groups. A perplexity filter trained on a reference model that embodies Western English writing conventions will filter out text that diverges from those conventions, including perfectly valid text from non-Western or non-standard English sources.

The implicit assumption in perplexity-based filtering is that low perplexity (relative to a reference model trained on "good" text) indicates quality. But "good" in this context means "resembles the reference model's training distribution," which is itself a biased sample. Text written in a non-native English style, text following non-Western rhetorical conventions, or text from communities that communicate differently from the reference model's training data will all receive higher perplexity scores and may be filtered out. The filter amplifies the demographic skew already present in the reference model's training data.

Deduplication, another common preprocessing step, can also introduce bias. When duplicate or near-duplicate text is removed, the relative frequencies of different content types change. If certain topics or perspectives appear in more duplicate forms (as in widely forwarded social media posts or syndicated news articles), deduplication will disproportionately reduce their representation. If community discussions about minority group experiences are more likely to spread virally in certain formats, deduplication may systematically trim those voices while leaving intact the original-content domains that dominant groups tend to control.

Domain and source selection adds another layer. Many curated datasets deliberately include text from Wikipedia, books, academic papers, and specific high-traffic websites. These sources are not demographically neutral. Wikipedia's gender gap in editorship is well-documented. Academic papers reflect the demographics and research priorities of academic institutions, which have their own historical biases. Books reflect publishing industry decisions about whose stories get told and whose receive the marketing resources to become widely read. Each source selection decision compounds the demographic and cultural skew already present in the internet crawl base.