Bias in Language Models › Bias Types

Bias in language models manifests in several distinct ways, each with different causes and consequences. The distinctions between these types matter taxonomically and practically: different types of bias require different measurement approaches and respond to different mitigation strategies.

Demographic bias encompasses skewed representations of, and associations with, human demographic groups. The most studied forms involve gender, race, and age, but demographic bias extends to disability, sexuality, religion, socioeconomic class, and nationality.

Gender bias is among the most well-documented. Word embedding studies beginning with the landmark Bolukbasi et al. (2016) paper demonstrated that embeddings trained on large corpora associate gendered terms with different occupations, personality traits, and social roles. The word "doctor" was closer in vector space to "he" than to "she," while "nurse" showed the reverse. These associations appear in static embeddings, masked language model predictions, and the completions of generative models. Critically, the magnitude of these associations has been shown to correlate with real-world gender representation statistics, suggesting that models are learning something real, but deploying it in contexts where the implication is that these associations should persist or be treated as natural.

Racial bias takes several forms. Language models have been shown to assign more negative sentiment to names and text associated with Black individuals compared to names and text associated with white individuals, holding all other content constant. This was demonstrated using the "name substitution" paradigm: take a sentence, swap in names strongly associated with Black versus white communities, and measure whether the model's sentiment assessment or continuation probabilities shift. They do, consistently and substantially. Models trained to detect toxicity in text systematically flag African American Vernacular English (AAVE) as more toxic than equivalent content in Standard American English, creating a disparate impact on communities that use AAVE.

Age bias involves both directions. Older adults are sometimes associated with decreased competence, technological inability, or diminished relevance. Young people may be associated with impulsiveness or irresponsibility. These associations shape model completions in ways that can disadvantage users across the age spectrum. An automated hiring tool built on a biased language model may subtly disadvantage older applicants through the language it generates in job descriptions or through the tone of AI-assisted screening conversations.

Demographic bias is particularly consequential because it maps directly onto legally protected characteristics in many jurisdictions. Discrimination on the basis of gender, race, age, disability, and religion is prohibited in employment, housing, education, and credit decisions in most developed legal systems. When language models used in decision-support systems exhibit demographic bias, the organizations deploying those models may face legal liability in addition to ethical criticism. This is one of the reasons that demographic bias has received more systematic research attention and regulatory focus than some other forms of model bias.

Cultural bias is distinct from demographic bias in that it involves assumptions about worldviews, values, and practices rather than stereotypes about specific groups. A model trained predominantly on English text from Western cultural contexts will encode Western frameworks as defaults and treat other cultural perspectives as deviations or exceptions.

This appears in many ways. Models may perform better on tasks that require knowledge of Western history, geography, and institutions than on equivalent tasks requiring knowledge of Asian, African, or South American contexts. They may apply Western concepts of individualism, linear time, or nuclear family structure in ways that are inappropriate for users from cultures organized around different values. Even the structure of language itself encodes cultural assumptions: models trained predominantly on English may struggle with grammatical categories, narrative conventions, or forms of politeness that are central to other languages.

Cultural bias is particularly challenging to address because it is often invisible to the researchers and engineers working on these systems. The default assumptions are so deeply embedded in the training data and in the backgrounds of the people building the systems that they do not register as assumptions at all. They register as facts, as common sense, as the obvious way that things are. A team of researchers from similar cultural backgrounds will often fail to notice that their benchmark is testing familiarity with Western cultural artifacts, that their example sentences embed Western grammatical conventions, or that their model evaluation metrics were designed with Western standards of quality in mind.

Consider an example from the domain of moral reasoning. Many Western ethical frameworks prioritize individual autonomy: the individual has rights, makes choices, and is the primary unit of moral concern. Many East Asian, African, and Indigenous ethical traditions prioritize relational or communal obligations, where one's identity and responsibilities are defined by one's place in a network of relationships, not by individual preferences. When a language model trained on Western-dominant text is asked to reason about ethical dilemmas, it will systematically apply individualistic framing. A question about whether to tell a white lie to protect a friend's feelings may be evaluated by the model in terms of individual honesty principles, ignoring the relational context in which protecting group harmony might be the culturally appropriate priority.

The same dynamic affects practical, non-philosophical tasks. Models trained on Western legal and political text may frame questions about governance in terms of rights and procedures, rather than the values of consensus, seniority, or communal deliberation that are central to political decision-making in many societies. Models trained on Western medical literature may default to biomedical explanatory models for illness and treatment, failing to engage directly with traditional medical knowledge systems that have clinical relevance in many communities. In each case, the Western framework is not presented as "one view among many" but as the default and obvious perspective, with alternatives appearing as edge cases or curiosities.

This problem scales as language models are deployed globally. A model built by a team in San Francisco, trained primarily on English-language internet text, and optimized for the preferences of demographically homogenous raters is now being used by hundreds of millions of people with vastly different cultural backgrounds, linguistic traditions, and social contexts. For users whose cultural frameworks align with the model's defaults, the model aligns with their expectations. For users whose frameworks differ, the model may give subtly misleading information, apply inappropriate norms, or fail to engage directly with cultural context that is central to the question being asked.

Occupational stereotyping involves associating certain professions, roles, or activities with particular demographic characteristics. The nurse/doctor gender association is the most famous example, but the pattern extends across many domains. Models may associate science, technology, engineering, and math (STEM) careers with men, humanities and care roles with women, and executive or leadership roles with white or Asian individuals while associating manual labor with specific ethnic groups.

These associations emerge from statistical patterns in text but become performative when deployed. A model that generates a job posting using male pronouns, or that depicts a scientist as a man when asked to write a story, actively reproduces historical patterns rather than simply reflecting them. The job posting will be read by real candidates, some of whom will feel implicitly excluded or subtly discouraged. The story will shape the mental imagery of readers who are developing their own sense of what is possible for people like them.

The performative aspect is important because it operates below the threshold of conscious intent. The person using the language model to draft a job posting did not ask for gendered language; the person who receives a resume-screened rejection did not know that their name was associated with a group the model disfavors. The bias acts invisibly, embedded in outputs that appear neutral on their surface. This invisibility makes occupational stereotyping in language models arguably more dangerous than overt discrimination: it is harder to challenge, harder to detect, and easier to rationalize away as "just what the data shows."

There is also a self-fulfilling dimension to this problem. When models consistently depict engineers as male, doctors as white, and service workers as belonging to particular ethnic groups, these depictions influence how users conceptualize these roles. Over time, this can normalize the very associations that the model reflects, reinforcing in culture what was already reflected from culture. The model becomes a participant in the social construction of occupational identity, not a passive mirror. A teenager receiving career advice from a language model, or browsing an AI-generated list of "typical" day-in-the-life descriptions for different professions, may internalize occupational stereotypes that limit their own sense of possibility.

Functional stereotyping extends beyond occupations into capabilities and traits. Models may associate emotional expressiveness with women and rational detachment with men, creativity with certain groups and analytical precision with others. These trait associations, embedded in model outputs, shape how models respond differently to equivalent requests from users who have different perceived demographic identities, and how models generate descriptions of people when demographic cues are present in the context.

Beyond who is described and in what roles, language models can assign systematically different emotional valence to content involving different groups. A model may generate more positive completions for prompts involving one group and more negative completions for prompts involving another. Sentiment classifiers, which are themselves language models or language model derivatives, may rate text about certain groups as more negative even when the semantic content is comparable.

This pattern has been documented across multiple experimental paradigms. In one common approach, researchers present a model with nearly identical sentences that differ only in the demographic group mentioned: "People in [Group A] are generally..." versus "People in [Group B] are generally...". When the model's completion probabilities are compared, systematic differences in positive versus negative continuation emerge. Studies using the BOLD (Bias in Open-Ended Language Generation) dataset have found that completions about certain religious groups, ethnic groups, or genders are consistently more negative on average, even when the prompts are semantically equivalent.

A particularly well-studied case involves co-reference resolution and sentiment. When a story describes both a positive and a negative actor and a model is asked to resolve which pronoun refers to whom, models consistently make errors that align with demographic stereotypes: associating professional success with male referents and domestic roles or emotional distress with female referents. This is not a surface-level bias. It reflects learned associations embedded in the model's representations of identity and action at a deep, distributional level.

Sentiment bias also compounds across model components. If a sentiment classifier is trained on annotated text where certain dialects or writing styles were labeled as more negative, and a downstream generation model uses that classifier as a reward signal (as in RLHF), the generation model will learn to avoid producing text that sounds like those dialects or styles. This creates a hidden pathway through which annotation bias becomes generation bias, without any single step in the pipeline being explicitly discriminatory.

This form of bias is particularly consequential in applications like content moderation, customer service, and mental health support, where sentiment assessment plays a direct role in how users are treated. A mental health chatbot that responds with less empathy to certain phrasing styles, or a content moderation system that removes more speech from certain communities, is not a neutral tool. It is a tool that has absorbed and operationalized social inequities, delivering differential treatment to users based on how their communication style maps onto patterns in the training data.