Bias in Language Models › Code Implementation
Language AI HandbookAll chapters Bias in Language Models › Code ImplementationOne section · Sections in this chapter7 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement a demonstration of these concepts: we'll build a small co-occurrence analysis to detect occupational gender bias, then examine how a pre-trained language model amplifies these signals in its masked predictions.
Setting Up
We start by installing and importing the libraries we'll need for this analysis.
import subprocess
subprocess.run(
["uv", "pip", "install", "transformers", "torch", "numpy", "matplotlib"],
capture_output=True,
)Building a Sample Corpus
We'll construct a small synthetic corpus that reflects the kinds of gender-occupation associations found in real training data. This keeps the code self-contained while illustrating the core mechanics. The corpus is designed to mirror the co-occurrence ratios documented in large-scale NLP studies.
# Synthetic corpus reflecting typical gender-occupation co-occurrence patterns
# Each sentence is a proxy for text that would appear in news/web data
# Corpus is sized to produce stable co-occurrence ratios (not single-sample extremes)
corpus_sentences = [
# Doctor sentences (male-leaning ~70% masc, ~30% fem)
"The doctor finished his rounds and wrote his notes.",
"He examined the patient and recommended surgery.",
"The physician spoke with his colleagues about the diagnosis.",
"Dr. Smith updated his patient records after the procedure.",
"He prescribed antibiotics and scheduled a follow-up.",
"He received his medical degree from Johns Hopkins.",
"The doctor reviewed his patient lab results carefully.",
"The physician confirmed his assessment with the specialist.",
"He consulted with the radiologist about the scan results.",
"The doctor shared his concerns with the patient family.",
"She consulted with the other doctors about the treatment plan.",
"The physician described her approach to preventive care.",
"She reviewed her patient notes before the morning rounds.",
"The doctor updated her chart and scheduled the follow-up.",
# Surgeon sentences (male-leaning ~65% masc)
"The surgeon completed his final operation of the day.",
"He reviewed his pre-operative checklist with the team.",
"The surgeon briefed his residents before the procedure.",
"He confirmed his incision plan with the attending physician.",
"The surgeon documented his operative findings in detail.",
"She performed her third surgery of the day without issue.",
"The surgeon presented her case at the morning conference.",
"She trained her residents in minimally invasive techniques.",
# Nurse sentences (female-leaning ~80% fem)
"The nurse administered the medication as prescribed.",
"She checked the patient vital signs every hour.",
"The nurse explained the post-operative care to the family.",
"She adjusted the IV drip and noted the response.",
"The registered nurse documented her observations carefully.",
"She comforted the patient during the difficult procedure.",
"The nurse coordinator scheduled her team for the weekend shift.",
"She received recognition for her patient care excellence.",
"The nurse updated her care plan after the physician visit.",
"She prepared her medications and reviewed her patient charts.",
"He assisted the surgical team as the scrub nurse.",
"The male nurse provided compassionate end-of-life care.",
"He monitored his patients carefully during the overnight shift.",
# Engineer sentences (male-leaning ~75% masc)
"The engineer reviewed his design specifications.",
"He debugged the circuit and resolved the issue.",
"The software engineer committed his changes to the repository.",
"He collaborated with his team to meet the deadline.",
"The mechanical engineer revised his technical drawings.",
"He presented his prototype to the product review board.",
"The engineer submitted his analysis to the project lead.",
"He updated his documentation after completing the build.",
"She led the engineering team through the product launch.",
"The engineer presented her analysis to the board.",
"She reviewed her designs with the senior architects.",
# Teacher sentences (mixed ~50% masc): occupation word must co-occur with pronoun
"The teacher prepared her lesson plans for the week.",
"The teacher submitted her attendance records on time.",
"The teacher organized her classroom materials for the new term.",
"The teacher graded his students assignments over the weekend.",
"The teacher reviewed his curriculum with the department chair.",
"The teacher answered his students questions during office hours.",
# Professor sentences (mixed ~50% masc): occupation word must co-occur with pronoun
"The professor delivered her lecture on thermodynamics.",
"The professor presented her findings at the annual conference.",
"The professor published her research in a peer-reviewed journal.",
"The professor mentored his graduate students through their dissertations.",
"The professor updated his syllabus for the spring semester.",
"The professor advised his doctoral candidates on experimental design.",
]Corpus size: 58 sentences
Computing Co-occurrence Statistics
We count how often each occupation word co-occurs with masculine and feminine pronouns within a sentence. The counting logic tokenizes each sentence and checks for intersection with our gendered term sets.
import re
# Define target occupations and gendered indicator words
occupations = [
"doctor",
"physician",
"surgeon",
"nurse",
"engineer",
"teacher",
"professor",
]
masculine_terms = {"he", "him", "his", "mr", "himself"}
feminine_terms = {"she", "her", "hers", "ms", "mrs", "herself"}
def count_cooccurrences(sentences, occupations, masc_terms, fem_terms):
"""Count masculine and feminine co-occurrences per occupation."""
counts = {occ: {"masculine": 0, "feminine": 0} for occ in occupations}
for sentence in sentences:
tokens = set(re.sub(r"[^\w\s]", "", sentence.lower()).split())
for occ in occupations:
# Check if occupation appears in the sentence
if occ in tokens or occ + "s" in tokens:
masc_count = len(tokens & masc_terms)
fem_count = len(tokens & fem_terms)
counts[occ]["masculine"] += masc_count
counts[occ]["feminine"] += fem_count
return counts
cooccurrence_counts = count_cooccurrences(
corpus_sentences, occupations, masculine_terms, feminine_terms
)Occupation Masculine Feminine Masc % ------------------------------------------ doctor 3 2 60.0% physician 4 2 66.7% surgeon 3 1 75.0% nurse 1 3 25.0% engineer 4 1 80.0% teacher 3 3 50.0% professor 3 3 50.0%
The table above shows the raw co-occurrence ratios in our synthetic corpus. "Doctor" and "engineer" lean masculine, "nurse" leans feminine, and "teacher" and "professor" are near-neutral. These ratios mirror the patterns found in large-scale NLP studies on real corpora, where the same occupational gender skews appear at scale.
Visualizing Co-occurrence Bias
With the co-occurrence counts in hand, we can visualize the gender skew for each occupation as a horizontal bar chart.
The chart reveals a clear pattern: occupations like "doctor" and "engineer" lean masculine in the training data, while "nurse" leans feminine. These are the raw signals that a language model will learn from during training. Importantly, these ratios do not necessarily reflect the actual gender composition of these professions today. They reflect the gender composition of how these professions were described in text, which is a product of who produced the text, when they produced it, and what narrative choices they made.
A heatmap view presents the same data differently, showing both the masculine and feminine counts side by side and making the imbalance within each occupation immediately visible.
Examining Bias Amplification with a Pre-trained Model
Now let's see how a pre-trained BERT model handles these same occupations. We'll use masked language modeling to ask the model what pronoun it predicts when given an occupational context. This lets us directly compare the raw corpus ratios against the model's learned predictions.
from transformers import pipeline
# Load a small BERT model for masked language modeling
mlm_pipeline = pipeline("fill-mask", model="distilbert-base-uncased", top_k=10)def get_gender_probabilities(pipeline_model, template):
"""
Fill the [MASK] and compare the model scores for a fixed set of
masculine and feminine pronouns.
"""
results = pipeline_model(
template,
targets=["he", "him", "his", "she", "her"],
)
masc_prob = sum(
r["score"]
for r in results
if r["token_str"].strip() in {"he", "him", "his"}
)
fem_prob = sum(
r["score"] for r in results if r["token_str"].strip() in {"she", "her"}
)
total = masc_prob + fem_prob
if total == 0:
return None, None
return masc_prob / total, fem_prob / total
# Test templates for different occupations
occupation_templates = {
"doctor": "The doctor finished [MASK] rounds.",
"nurse": "The nurse administered [MASK] medication.",
"engineer": "The engineer reviewed [MASK] design.",
"teacher": "The teacher prepared [MASK] lesson.",
"professor": "The professor delivered [MASK] lecture.",
}
amplification_results = {}
for occ, template in occupation_templates.items():
masc, fem = get_gender_probabilities(mlm_pipeline, template)
if masc is not None:
amplification_results[occ] = {
"model_masc_pct": masc * 100,
"model_fem_pct": fem * 100,
}Occupation Model Masc% Model Fem% -------------------------------------- doctor 68.1% 31.9% nurse 8.7% 91.3% engineer 83.3% 16.7% teacher 68.6% 31.4% professor 79.8% 20.2%
The model's predictions show a distribution across masculine and feminine pronouns for each occupation. Comparing the model's output probabilities to the raw corpus co-occurrence ratios reveals the amplification effect: occupations that lean masculine in the data tend to lean even more masculine in model predictions, and vice versa. This is the core empirical signature of bias amplification.
Visualizing Corpus vs. Model Bias
Let's compare the corpus co-occurrence ratios against the model's predicted probabilities for the occupations that appear in both analyses. This side-by-side visualization makes the amplification effect directly visible.
The side-by-side comparison illustrates bias amplification directly: where the corpus data shows a moderate masculine lean, the model's predictions often show a more extreme lean. This is not an artifact of our synthetic corpus. The same pattern has been documented in studies using large-scale real corpora and production language models. Correcting bias in the data is necessary but not sufficient; the training dynamics themselves must be addressed.
Key Parameters Summary
The key variables in this analysis are:
- Co-occurrence window: We used sentence-level co-occurrence. Smaller windows (e.g., 5 words) capture tighter syntactic associations; larger windows capture broader topical associations.
- Gendered term set: We used pronouns and titles. Studies vary in whether they include first names, which introduces additional confounds because names shift in gender over time and across cultures.
- Model size and training data: Larger models trained on more diverse data tend to have different bias profiles than smaller models, though not necessarily less biased ones.
- Pronoun target set: We compare a fixed set of masculine and feminine pronouns directly. Expanding this set changes the conditional ratio and should be documented when results are compared across studies.