Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Embedding Evaluation › Embedding Bias Detection

Michael BrenndoerferUpdated March 21, 20267 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 27Embedding Evaluation › Embedding Bias Detection

Reading section 7 of 13 · Embedding Bias Detection · 7 minSection 7 of 13 · 7 min

Section 7 of 13 · 7 min read

Embedding Evaluation › Embedding Bias DetectionOne section · Sections in this chapter7 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Word embeddings encode meaning and the societal biases present in their training data. This is a serious problem when embeddings are used in consequential applications like resume screening, loan approval, or content moderation.

The mechanism is straightforward: if training text contains patterns like "he is a programmer" and "she is a nurse" more often than their reverse, the model learns vector spaces where these associations are encoded geometrically. The cosine similarity between "programmer" and "he" ends up larger than between "programmer" and "she." These are not just abstract statistical artifacts; they reflect real stereotypes baked into the corpus, and they can cause real harm when deployed.

The bias problem is particularly insidious because it's invisible from intrinsic and extrinsic evaluations that don't test for it. A model can score 0.80 on SimLex-999 and 85% on sentiment classification while encoding large gender or racial biases. This means bias evaluation needs to be a deliberate, separate step in any responsible embedding assessment.

WEAT: Word Embedding Association TestLink Copied

The Word Embedding Association Test (WEAT) was introduced by Caliskan et al. in 2017 and provides a formal statistical test for bias in word embeddings. It's modeled after the Implicit Association Test (IAT) from psychology, which measures how quickly people associate concepts with attributes. WEAT adapts this to vector spaces by measuring cosine similarity rather than reaction time.

WEAT compares how strongly two sets of target words (for example, male names versus female names) are associated with two sets of attribute words (for example, career words versus family words).

The per-word association score $s(w, A, B)$ measures how much a word $w$ is more associated with attribute set $A$ than attribute set $B$ :

s(w, A, B) = \text{mean}_{a \in A} \cos(\mathbf{w}, \mathbf{a}) - \text{mean}_{b \in B} \cos(\mathbf{w}, \mathbf{b})

where:

$\mathbf{w}$ , $\mathbf{a}$ , $\mathbf{b}$ are the embeddings of words $w$ , $a$ , $b$
$\cos(\mathbf{u}, \mathbf{v})$ is cosine similarity
$A$ and $B$ are the two attribute word sets

The WEAT effect size $d$ then measures the differential association between two target sets:

d = \frac{\text{mean}_{x \in X} \, s(x, A, B) - \text{mean}_{y \in Y} \, s(y, A, B)}{\text{std}_{w \in X \cup Y} \, s(w, A, B)}

where:

$X$ and $Y$ are the two target word sets
The numerator is the difference in mean association scores between the two target sets
The denominator is the pooled standard deviation of all association scores across both target sets, giving a Cohen's $d$ -style standardization

An effect size near 0 indicates no differential association: both target sets are equally associated with both attribute sets. Positive values indicate that target set $X$ is more strongly associated with attribute $A$ than target set $Y$ is. The sign and magnitude mirror Cohen's $d$ conventions: values above 0.8 are considered large. Published WEAT studies on Word2Vec and GloVe consistently find effect sizes of 0.5-1.5 for gender-career bias, which is substantial by any standard.

In[20]:

Code

def weat_effect_size(embeddings, target_X, target_Y, attr_A, attr_B):
    """
    Compute WEAT effect size for bias between two target sets
    relative to two attribute sets.
    """

    def mean_cosine(word, attr_words):
        vecs = [embeddings[a] for a in attr_words if a in embeddings]
        if not vecs or word not in embeddings:
            return 0.0
        sims = [cosine_similarity(embeddings[word], v) for v in vecs]
        return np.mean(sims)

    def association(word, A, B):
        return mean_cosine(word, A) - mean_cosine(word, B)

    # Filter to words in vocabulary
    X = [w for w in target_X if w in embeddings]
    Y = [w for w in target_Y if w in embeddings]

    if not X or not Y:
        return None

    scores_X = [association(x, attr_A, attr_B) for x in X]
    scores_Y = [association(y, attr_A, attr_B) for y in Y]

    all_scores = scores_X + scores_Y
    pooled_std = np.std(all_scores, ddof=1)

    if pooled_std == 0:
        return 0.0

    effect_size = (np.mean(scores_X) - np.mean(scores_Y)) / pooled_std
    return effect_size


# Simulate bias test using our structured embeddings
# We'll manufacture a scenario with technology/animal words as targets
# and "work" vs "play" associations

# Extend embeddings with attribute words
np.random.seed(99)
tech_ref = np.mean(
    [embeddings[w] for w in ["computer", "laptop", "software"]], axis=0
)
animal_ref = np.mean([embeddings[w] for w in ["cat", "dog", "horse"]], axis=0)

embeddings["work"] = tech_ref * 0.7 + 0.3 * np.random.randn(50)
embeddings["career"] = tech_ref * 0.6 + 0.4 * np.random.randn(50)
embeddings["office"] = tech_ref * 0.5 + 0.5 * np.random.randn(50)
embeddings["play"] = animal_ref * 0.6 + 0.4 * np.random.randn(50)
embeddings["home"] = animal_ref * 0.4 + 0.6 * np.random.randn(50)
embeddings["leisure"] = animal_ref * 0.5 + 0.5 * np.random.randn(50)

# Normalize new embeddings
for word in ["work", "career", "office", "play", "home", "leisure"]:
    embeddings[word] /= np.linalg.norm(embeddings[word])

In[21]:

Code

# Test: are "technology" words more associated with "work" than "animals"?
target_tech = ["computer", "laptop", "software", "phone", "program"]
target_animal = ["cat", "dog", "horse", "kitten", "puppy"]
attr_work = ["work", "career", "office"]
attr_play = ["play", "home", "leisure"]

effect = weat_effect_size(
    embeddings, target_tech, target_animal, attr_work, attr_play
)

Out[22]:

Console

WEAT Bias Detection Example
=======================================================
Target X (Technology): ['computer', 'laptop', 'software', 'phone', 'program']
Target Y (Animals):    ['cat', 'dog', 'horse', 'kitten', 'puppy']
Attribute A (Work):    ['work', 'career', 'office']
Attribute B (Play):    ['play', 'home', 'leisure']

WEAT Effect Size: 1.734

Interpretation: Large association difference

Typical WEAT findings in real embeddings:
  - Career/family words correlate with male/female names
  - Science/art correlations align with gender stereotypes
  - Effect sizes of 0.5-1.5 are common in pretrained embeddings

Real WEAT studies on Word2Vec and GloVe find that European American names are more associated with pleasant words than African American names, and that male names are more associated with career words than female names. These findings sparked significant research into bias mitigation and fairness in NLP, which remains an active and important area.

What WEAT Captures and MissesLink Copied

WEAT provides a clean, statistically grounded measure of specific pre-defined biases. Its limitation is precisely this: you must choose the target and attribute word sets in advance, which means you can only find biases you already know to look for. A model might pass a gender-career WEAT test while still encoding age-based bias, nationality bias, or more subtle intersectional biases.

The WEAT framework also assumes that the bias is simple and linear in the embedding space. In high dimensions, biases can be distributed across many dimensions in complex, non-linear ways that WEAT doesn't capture. Models that have been explicitly debiased with linear projection methods may pass WEAT while retaining downstream discrimination, a finding documented in several follow-up studies.

Debiasing ApproachesLink Copied

Detecting bias is a prerequisite to addressing it, but debiasing is harder. The main approaches are:

Hard debiasing (Bolukbasi et al., 2016): Identify a "gender direction" in embedding space, defined as the first principal component of the differences between paired gender-specific words (he/she, man/woman, king/queen). Project gender-neutral words to be orthogonal to this direction, removing the component of their embedding that correlates with gender. This is fast and interpretable but has the limitation of treating gender as a single linear direction in a 300-dimensional space, which is surely an oversimplification.
Data augmentation: Train or fine-tune on data where gendered pronouns and names have been swapped, so the model sees equal exposure to both genders in all contexts. This is more principled but requires either labeled data or careful heuristics for pronoun swapping, and it doesn't address biases not mediated by explicit gender markers.
Counterfactual data augmentation: A more targeted version where you create counterfactual training examples by swapping demographic attributes and add them to the training set, forcing the model to produce invariant representations.
Post-hoc neutralization: Equalize distances for semantically equivalent pairs across the protected attribute. If "nurse" is closer to "she" than to "he", adjust both to be equidistant. This is clean but can produce representations that are technically equidistant while still encoding bias through other words.

Each approach has limitations. Hard debiasing can remove useful gender distinctions that are legitimately informative (actor vs actress as distinct roles) while leaving indirect biases intact through other words. There's no silver bullet, and any deployment of word embeddings in consequential settings requires ongoing monitoring and evaluation for bias.