Bias in Language Models › Sources of Bias

Bias in language models does not originate in a single place. It flows from multiple upstream sources, each of which contributes its own distortions to the final model. Understanding the provenance of each form of bias matters because different sources require different interventions. Filtering training data addresses one kind of problem; improving annotation processes addresses another; rethinking the objective function addresses a third.

The most fundamental source of bias is the training corpus. Modern language models train on vast collections of text scraped from the internet, digitized books, academic papers, news archives, and social media platforms. These sources reflect the demographics, values, and blind spots of the people who produced them.

Web crawl data, such as Common Crawl, is dominated by English text from North America and Western Europe. A 2022 analysis of the data powering many large language models found that over 90% of content originated from a small number of Western-centric domains. Languages spoken by hundreds of millions of people, such as Swahili, Bengali, or Javanese, receive tiny fractions of representation. Even within English text, certain communities, age groups, educational backgrounds, and geographic regions are systematically overrepresented. The result is a model whose "knowledge" about the world is shaped by a narrow slice of human experience.

The underrepresentation problem extends beyond language. Even in English, the voices that populate the internet skew sharply toward specific demographics. Wikipedia editors are predominantly male; Reddit users skew young and technically educated; news commentary overrepresents professional classes. Each of these sources shapes which topics get covered, how they are framed, what perspectives are treated as defaults, and which experiences are described with nuance versus which are glossed over with stereotypes. A model trained on text produced primarily by one demographic will have richer, more detailed representations of that demographic's experiences and shallower, less accurate representations of everyone else.

The temporal dimension of training data introduces a different kind of bias. Text collected from the internet reflects the social norms, power structures, and language conventions of the period when it was written. Historical texts may use terminology now considered offensive, encode now-discredited scientific beliefs, or simply reflect a world in which certain professions were exclusively male or certain races legally excluded from institutions. A model trained on such text will reproduce these historical distortions unless explicit care is taken to address them. When training data spans multiple decades, the model effectively learns a time-averaged view of society that blends past injustices with present-day conditions, without any mechanism to distinguish between them.

The mechanism by which corpus bias enters the model is statistical. The model does not "see" sentences as meaningful human communication. It sees sequences of tokens, and it learns which token sequences are likely. When certain demographic groups are consistently described in certain terms, associated with certain roles, or framed in certain emotional registers across millions of documents, those associations become part of the model's learned probability distribution. They are not stored as explicit beliefs but as implicit weightings embedded across billions of parameters. This distributed encoding makes corpus bias both pervasive and difficult to isolate or remove.

Many language models are fine-tuned or evaluated using human-labeled data, and human annotators carry their own biases. When workers label sentiment, assess toxicity, or evaluate the quality of model-generated text, their judgments are shaped by their cultural background, personal experiences, and social position.

Several systematic effects have been documented. Annotators from majority groups tend to rate text expressing minority group perspectives as more extreme or more offensive. Workers hired through crowdsourcing platforms like Amazon Mechanical Turk skew heavily toward specific demographics, particularly in the United States. Annotation guidelines written with one cultural context in mind may produce labels that are meaningful within that context but misleading when applied to text from other communities.

The practical consequences of annotation bias run deeper than simple labeling errors. When a toxicity detection model is trained on annotations made primarily by workers from one cultural background, it will tend to flag as toxic the speech patterns characteristic of other cultural backgrounds, even when those patterns are not intended or received as harmful by their communities. African American Vernacular English (AAVE) is a well-studied case: multiple research papers have documented that automated toxicity classifiers trained on crowd-annotated data flag AAVE at higher rates than equivalent content in Standard American English. The annotators were not being malicious. They were applying their learned cultural intuitions about what sounds aggressive, informal, or threatening, and those intuitions encoded cultural distance as danger.

As discussed in the Inter-Annotator Agreement chapter, disagreement between annotators is common. When inter-annotator agreement is low, the choice of how to resolve disagreements, whether by majority vote, expert adjudication, or averaging, encodes a particular social perspective into the gold-standard labels. A dataset where annotations are resolved by majority vote will systematically reflect the views of the majority annotator population, potentially disadvantaging minority perspectives. If the task involves judging the appropriateness, quality, or safety of content, majority-vote resolution means that content aligned with the majority annotator population will be systematically rated more favorably than equally valid content from minority communities.

Even accurate statistics can encode injustice. If a historical corpus accurately reflects that surgeons were predominantly male in 1980, a model trained on that corpus will learn a real statistical association between surgery and maleness. But deploying that association in 2024 as though it reflects current reality, or as though it should shape future expectations, perpetuates a historical pattern that society has actively worked to change.

This is the distinction between descriptive bias (the world as it was or is) and normative bias (the world as it should be). Language models typically encode descriptive statistics from their training data but deploy their outputs in contexts where users expect normative guidance or neutral description. When a student asks a career counseling chatbot about becoming an engineer and receives subtly discouraging language because the model associates engineering with maleness, the model has treated a historical correlation as a current norm. It is not describing reality accurately; it is projecting a historical snapshot into a present where that snapshot is no longer valid and should not be reinforced.

Historical bias is particularly insidious because it can resist filtering. You cannot remove it by filtering out inaccurate text, because the text is accurate about what the world looked like when it was written. You cannot remove it by including only high-quality, reputable sources, because many historically biased documents come from high-quality, reputable sources. The only interventions that reliably address historical bias operate at the training objective or fine-tuning stage, either by weighting recent data more heavily, by explicitly correcting for historical under-representation, or by using alignment techniques that teach the model to reason about what the world should be rather than just what it was.

Societal bias, as distinct from purely historical bias, refers to associations reflecting current prejudices and inequalities that persist in society today. These can be even harder to address, because they cannot simply be dismissed as outdated. They represent real patterns in real text that a model is supposed to learn from. The challenge is that learning these patterns accurately means reproducing them in model outputs, which then reinforces them in the culture. The model becomes a participant in maintaining the social structures its training data reflects.

The training process itself, beyond the data, introduces additional bias. Objective functions shape what the model optimizes for. A language model trained purely to predict the next token will optimize for statistical frequency, not for fairness or representativeness. This means patterns that appear frequently in data will be learned more robustly than patterns that appear rarely, even when the less frequent patterns represent an equally valid or important part of reality.

The standard cross-entropy training objective makes no distinction between harmful and harmless statistical patterns. A pattern associating a minority group with criminal behavior will be learned with exactly the same mechanism as a pattern associating warm weather with outdoor activities. The only difference is frequency and the strength of correlating signals. If the harmful association appears more frequently or is correlated with more other features in the training data, the model will learn it more strongly. This is not a design flaw in the specific model but a property of maximum likelihood estimation applied to language.

Fine-tuning with reinforcement learning from human feedback (RLHF) introduces another layer. Human raters provide feedback that shapes the model's behavior, but rater populations are not demographically representative. If the raters who provide feedback skew toward certain cultural backgrounds, the fine-tuned model will learn to produce outputs those raters prefer, which may not align with the preferences or needs of users from different backgrounds. This creates a systematic gap between the model's optimized behavior and the needs of underrepresented user populations.

Hyperparameter choices and architecture decisions can also amplify certain kinds of bias. Models with larger context windows can capture longer-range co-occurrence patterns, potentially learning subtler associative biases. Models with smaller vocabularies may fail to represent names and concepts from underrepresented cultures accurately, defaulting to "unknown token" behaviors that degrade performance for certain communities. The tokenization process itself can introduce disparities: if a tokenizer trained on English-dominant data splits non-English words into many subword fragments, models using that tokenizer will have worse performance on non-English languages not because of anything in the model weights but because of how the input is preprocessed.