Representation Harms › Code: Measuring Representation Harms
Language AI HandbookAll chapters Representation Harms › Code: Measuring Representation HarmsOne section · Sections in this chapter7 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's build practical tools for measuring each type of representation harm. We'll start with template-based stereotyping measurement, then implement embedding-based association analysis, and finally measure erasure through corpus-level frequency analysis.
Setup
We'll use numpy, scipy, and a simple word embedding approach for the association tests. For the text generation probes, we'll use the transformers library to interact with a masked language model.
import warnings
warnings.filterwarnings("ignore")Simulating MLM Occupational Stereotyping
We'll build a measurement framework that mimics what researchers use with masked language models, but using simulated probability distributions to illustrate the method without requiring GPU access.
# Simulated masked language model fill probabilities
# In real research, these would come from model.predict_masked_token()
# Values represent P(pronoun | context with occupation)
# Based on approximate patterns documented in WinoBias and similar benchmarks
OCCUPATION_PRONOUN_PROBS = {
"nurse": {"she": 0.91, "he": 0.05, "they": 0.04},
"receptionist": {"she": 0.93, "he": 0.02, "they": 0.05},
"teacher": {"she": 0.83, "he": 0.12, "they": 0.05},
"librarian": {"she": 0.87, "he": 0.08, "they": 0.05},
"social_worker": {"she": 0.88, "he": 0.07, "they": 0.05},
"engineer": {"she": 0.11, "he": 0.84, "they": 0.05},
"programmer": {"she": 0.14, "he": 0.81, "they": 0.05},
"CEO": {"she": 0.18, "he": 0.77, "they": 0.05},
"pilot": {"she": 0.04, "he": 0.91, "they": 0.05},
"surgeon": {"she": 0.25, "he": 0.70, "they": 0.05},
"teacher_m": {
"she": 0.24,
"he": 0.71,
"they": 0.05,
}, # male-dominated context
"doctor": {"she": 0.32, "he": 0.63, "they": 0.05},
}
# US Bureau of Labor Statistics 2023 data for reference
BLS_FEMALE_REPRESENTATION = {
"nurse": 0.87,
"receptionist": 0.90,
"teacher": 0.76,
"librarian": 0.79,
"social_worker": 0.82,
"engineer": 0.16,
"programmer": 0.25,
"CEO": 0.31,
"pilot": 0.07,
"surgeon": 0.39,
"teacher_m": 0.76,
"doctor": 0.41,
}
def stereotyping_gap(probs: dict, bls_rate: float) -> float:
"""Compute the gap between model feminine pronoun probability and true representation rate."""
model_she_prob = probs["she"]
return model_she_prob - bls_rateOccupation Stereotyping Analysis ------------------------------------------------------------ Occupation Model P(she) BLS Rate Gap ------------------------------------------------------------ teacher_m 0.24 0.76 -0.52 under surgeon 0.25 0.39 -0.14 under CEO 0.18 0.31 -0.13 under programmer 0.14 0.25 -0.11 under doctor 0.32 0.41 -0.09 under engineer 0.11 0.16 -0.05 near pilot 0.04 0.07 -0.03 near receptionist 0.93 0.90 +0.03 near nurse 0.91 0.87 +0.04 near social_worker 0.88 0.82 +0.06 over teacher 0.83 0.76 +0.07 over librarian 0.87 0.79 +0.08 over ------------------------------------------------------------ Mean absolute stereotyping gap: 0.113 Max overrepresentation: +0.080 Max underrepresentation: -0.520
The gaps reveal a systematic pattern. Occupations where women are underrepresented in reality (engineer, pilot, CEO) show further underrepresentation in model outputs: the model's probability of assigning feminine pronouns falls below even the already-low labor market rates. Occupations where women are overrepresented (nurse, receptionist) show feminine pronoun rates that exceed the real-world rates, reinforcing the clustering of women in care and support roles. Critically, the model doesn't merely reflect occupational distributions. It amplifies them, pushing associations further from parity in both directions.
Visualizing Stereotyping Bias
Let's visualize the relationship between model pronoun assignment and actual labor market representation. Points above the diagonal represent occupations where the model over-generates feminine pronouns relative to reality; points below represent occupations where the model under-generates feminine pronouns.
WEAT-Style Embedding Association Test
Now let's implement the core logic of the WEAT association test to measure how embeddings encode group-attribute associations. We'll use simulated embedding vectors that capture documented patterns from word2vec and GloVe studies.
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
"""Compute cosine similarity between two vectors."""
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
def mean_cosine_similarity(
target: np.ndarray, attribute_vectors: list
) -> float:
"""Compute mean cosine similarity between a target vector and a set of attribute vectors."""
similarities = [cosine_similarity(target, a) for a in attribute_vectors]
return np.mean(similarities)
def association_score(
target: np.ndarray, attribute_a_vectors: list, attribute_b_vectors: list
) -> float:
"""Differential association of target word with two attribute sets."""
return mean_cosine_similarity(
target, attribute_a_vectors
) - mean_cosine_similarity(target, attribute_b_vectors)
def weat_effect_size(
X_vectors: list, Y_vectors: list, A_vectors: list, B_vectors: list
) -> float:
"""
Compute WEAT effect size d.
X: target group 1 vectors
Y: target group 2 vectors
A: attribute 1 vectors (pleasant/positive)
B: attribute 2 vectors (unpleasant/negative)
"""
scores_X = [association_score(x, A_vectors, B_vectors) for x in X_vectors]
scores_Y = [association_score(y, A_vectors, B_vectors) for y in Y_vectors]
all_scores = scores_X + scores_Y
pooled_std = np.std(all_scores, ddof=1)
effect_size = (np.mean(scores_X) - np.mean(scores_Y)) / pooled_std
return effect_size
def permutation_test(
X_vectors, Y_vectors, A_vectors, B_vectors, n_permutations=5000
):
"""
Assess statistical significance via permutation test.
Returns p-value under null hypothesis that effect size is zero.
"""
observed_stat = sum(
association_score(x, A_vectors, B_vectors) for x in X_vectors
) - sum(association_score(y, A_vectors, B_vectors) for y in Y_vectors)
all_targets = X_vectors + Y_vectors
n_X = len(X_vectors)
n_extreme = 0
for _ in range(n_permutations):
perm = np.random.permutation(len(all_targets))
perm_X = [all_targets[i] for i in perm[:n_X]]
perm_Y = [all_targets[i] for i in perm[n_X:]]
perm_stat = sum(
association_score(x, A_vectors, B_vectors) for x in perm_X
) - sum(association_score(y, A_vectors, B_vectors) for y in perm_Y)
if perm_stat > observed_stat:
n_extreme += 1
return n_extreme / n_permutationsnp.random.seed(42)
# Generate simulated word embeddings in 50 dimensions
# that capture documented association patterns from WEAT studies
# In practice, you'd load actual GloVe or word2vec vectors
dim = 50
# Create a "pleasantness" direction and a "competence" direction
pleasant_dir = np.random.randn(dim)
pleasant_dir /= np.linalg.norm(pleasant_dir)
competence_dir = np.random.randn(dim)
competence_dir /= np.linalg.norm(competence_dir)
def make_word_vector(
pleasant_score: float, competence_score: float, noise: float = 0.3
) -> np.ndarray:
"""Create a word vector with given pleasantness and competence scores plus noise."""
v = pleasant_score * pleasant_dir + competence_score * competence_dir
v += noise * np.random.randn(dim)
return v / np.linalg.norm(v)
# European-American names (WEAT test 3): higher pleasant/competence scores in documented studies
european_names = [
make_word_vector(pleasant_score=0.7, competence_score=0.6) for _ in range(8)
]
# African-American names: lower scores in documented word embedding studies
african_american_names = [
make_word_vector(pleasant_score=0.3, competence_score=0.3) for _ in range(8)
]
# Pleasant attributes
pleasant_words = [
make_word_vector(pleasant_score=0.9, competence_score=0.5) for _ in range(8)
]
# Unpleasant attributes
unpleasant_words = [
make_word_vector(pleasant_score=-0.9, competence_score=-0.5)
for _ in range(8)
]WEAT Test: Names and Pleasantness (simulating WEAT Test 3) ============================================================ Effect size (d): 1.070 P-value: 0.0137 Interpretation: large differential association European-American names show stronger association with pleasant words than African-American names. Effect size guidelines (Cohen's d analog): d > 0.2: small effect d > 0.5: medium effect d > 0.8: large effect d > 1.0: very large effect Note: Real studies (Caliskan et al. 2017) found d = 1.41 for this test in GloVe embeddings trained on Common Crawl.
The WEAT effect size provides a standardized measure of association strength comparable across different embedding spaces and different group-attribute pairings. The original Caliskan et al. (2017) paper found large effect sizes () for name-pleasantness associations in GloVe embeddings, with p-values below 0.001 across thousands of permutations. The critical insight from these results is that the bias isn't a discrete error the model makes occasionally. It's a property of the geometric structure of the representation space, present everywhere the relevant words appear.
Measuring Erasure Through Generation Frequency
For erasure measurement, we'll analyze how often different demographic groups appear in generated text across different scenarios. The analysis computes an erasure index: the difference between a group's appearance rate in generated content and its population proportion. Negative values indicate the group is being systematically omitted.
# Simulated generation data: each entry is a generated text and which demographic
# groups appeared as main subjects/agents (protagonist roles)
# Patterns drawn from documented findings in studies of GPT-2 and GPT-3 generation
GENERATION_SCENARIOS = [
"Write about a scientist making a discovery",
"Write about a CEO making a strategic decision",
"Write about a software engineer solving a bug",
"Write about a nurse caring for a patient",
"Write about a teacher in a classroom",
"Write about a politician giving a speech",
]
# Simulated proportions from 1000 generations per scenario
# In real research these would be extracted from actual model outputs
# using demographic classifier or pronoun counting
GENERATION_DATA = {
"scientist": {
"white_male": 0.58,
"white_female": 0.19,
"nonwhite_male": 0.16,
"nonwhite_female": 0.07,
},
"CEO": {
"white_male": 0.63,
"white_female": 0.18,
"nonwhite_male": 0.13,
"nonwhite_female": 0.06,
},
"software_engineer": {
"white_male": 0.61,
"white_female": 0.14,
"nonwhite_male": 0.19,
"nonwhite_female": 0.06,
},
"nurse": {
"white_male": 0.08,
"white_female": 0.67,
"nonwhite_male": 0.07,
"nonwhite_female": 0.18,
},
"teacher": {
"white_male": 0.22,
"white_female": 0.58,
"nonwhite_male": 0.10,
"nonwhite_female": 0.10,
},
"politician": {
"white_male": 0.65,
"white_female": 0.17,
"nonwhite_male": 0.13,
"nonwhite_female": 0.05,
},
}
# US population reference proportions (approximate)
POPULATION_REFERENCE = {
"white_male": 0.30,
"white_female": 0.31,
"nonwhite_male": 0.19,
"nonwhite_female": 0.20,
}
def erasure_index(generated_probs: dict, reference_probs: dict) -> dict:
"""
Compute erasure index for each demographic group.
Negative values indicate underrepresentation (erasure).
Positive values indicate overrepresentation.
"""
return {
group: generated_probs[group] - reference_probs[group]
for group in reference_probs
}Erasure Index by Occupation and Demographic Group (Negative = underrepresented relative to population) ======================================================================== Occupation White M White F Non-w M Non-w F ------------------------------------------------------------------------ scientist +0.28 -0.12 -0.03 -0.13 CEO +0.33 -0.13 -0.06 -0.14 software_engineer +0.31 -0.17 +0.00 -0.14 nurse -0.22 +0.36 -0.12 -0.02 teacher -0.08 +0.27 -0.09 -0.10 politician +0.35 -0.14 -0.06 -0.15 ------------------------------------------------------------------------ MEAN +0.16 +0.01 -0.06 -0.11
The erasure index immediately reveals the pattern: white male subjects are strongly overrepresented in the scientist, CEO, software engineer, and politician scenarios, while nonwhite female subjects are underrepresented in every scenario. The nurse and teacher scenarios reverse the gender pattern, but both still erase nonwhite groups relative to population proportions. Notice that the sum of erasure indices across all groups equals zero for each occupation, since the generated proportions must sum to 1. This means every unit of overrepresentation for one group is a unit of underrepresentation for another. Erasure is fundamentally a distributional problem.
Visualizing Erasure Across Scenarios
Automated Toxicity Asymmetry Measurement
For demeaning associations, we can simulate a bias auditing approach that measures whether toxicity classifiers themselves show asymmetric false positive rates across demographic groups. This is not measuring whether model outputs are toxic toward certain groups. It's measuring whether the measurement tools themselves are biased, which is a meta-level representation harm: the community whose speech is most often pathologized by safety classifiers is also the community whose expression is most often suppressed.
# Simulate sentence pairs: same semantic content expressed in different dialects/styles
# Each pair (standard, aave) represents equivalent meaning with different register
DIALECT_PAIRS = [
{
"standard": "I am not going to do that.",
"aave": "I ain't gonna do that.",
"is_toxic": False,
},
{
"standard": "That was a foolish decision.",
"aave": "That was a real dumb move, fam.",
"is_toxic": False,
},
{
"standard": "You are always late.",
"aave": "You stay coming late.",
"is_toxic": False,
},
{
"standard": "That is incorrect.",
"aave": "That ain't right, no cap.",
"is_toxic": False,
},
{
"standard": "This is very unfair.",
"aave": "This is lowkey mad unfair.",
"is_toxic": False,
},
{
"standard": "I am exhausted from work.",
"aave": "I'm deadass tired from work.",
"is_toxic": False,
},
]
# Simulated toxicity scores from a classifier (0=clean, 1=toxic)
# Captures documented false positive rate asymmetry (Sap et al. 2019, Davidson et al. 2019)
# Real classifiers assign higher toxicity scores to AAVE even for non-toxic content
SIMULATED_TOXICITY_SCORES = {
"standard": [0.04, 0.07, 0.05, 0.03, 0.06, 0.04],
"aave": [0.23, 0.31, 0.19, 0.28, 0.22, 0.35],
}
def false_positive_rate(scores: list, threshold: float = 0.15) -> float:
"""Compute false positive rate at a given threshold for non-toxic sentences."""
return sum(1 for s in scores if s >= threshold) / len(scores)Toxicity Classifier Asymmetry Analysis ================================================== Threshold: 0.15 Standard American English: Mean toxicity score: 0.048 False positive rate: 0.0% African American Vernacular English (AAVE): Mean toxicity score: 0.263 False positive rate: 100.0% Relative false positive rate: 1000000000.0x higher for AAVE Note: Sap et al. (2019) found 3-4x higher annotation rates for toxicity in AAVE vs. Standard American English in real classifiers.
This asymmetry is a form of demeaning representation harm encoded in a measurement tool. When the tools researchers use to detect representation harms are themselves biased against non-dominant linguistic varieties, the harms are systematically undercounted for the most affected communities. The practical consequence is severe: content moderation systems that use these classifiers over-suppress Black social media posts, Black news content, and Black creative writing relative to equivalent content in Standard American English.
Summary Visualization: Harm Type Comparison
The following pair of plots summarizes the two quantitative analyses side by side. The left panel shows which occupations are most stereotyped (measured as the gap between model outputs and labor market reality), and the right panel shows which demographic groups are most consistently erased across all professional scenarios.
Key Parameters
The key parameters in the measurement framework are:
n_permutations: Number of random shuffles in the WEAT permutation test. Higher values give more precise p-values; 5,000 is standard in the literature.threshold(toxicity classifier): The decision boundary above which a score is treated as a positive prediction. The choice of threshold directly controls false positive and false negative rates and should be calibrated per use case.dim(embedding dimension): The dimensionality of the word embedding vectors. In practice, GloVe and word2vec use 50-300 dimensions; higher dimensions capture more nuance but require more data to estimate reliably.noise(simulation parameter): The amount of random variation added to simulated vectors. Controls how strongly the constructed embeddings deviate from the planted association direction.