Bias Measurement › Embedding Association Tests

Word embeddings and sentence encoders compress language into dense vectors. These vectors encode semantic relationships learned from text, which means they also encode whatever associations existed in that text. If the word "nurse" consistently appeared near words associated with women and "engineer" appeared near words associated with men, the model's vectors will reflect that. Embedding association tests are a family of methods for detecting and quantifying these encoded associations.

The motivation for this approach goes back to early observations about word2vec and GloVe embeddings. Researchers noticed that simple arithmetic in embedding space produced semantically meaningful results: "king" minus "man" plus "woman" pointed toward "queen." This was exciting because it demonstrated that embeddings captured valid semantic relationships. But the same mechanism that encodes "king:queen::man:woman" also encodes stereotypical relationships: "doctor:nurse::man:woman" turns out to approximate the same directional offset in some embedding spaces. If the geometric structure of an embedding space encodes these associations, it becomes possible to measure them quantitatively rather than anecdotally. Embedding association tests formalize exactly this measurement.

The Word Embedding Association Test (WEAT), introduced by Caliskan et al. in 2017, was the first systematic method for measuring bias in word embeddings. The core idea is elegant: bias can be detected by measuring whether two sets of target words (e.g., male names vs. female names) are more strongly associated with one set of attribute words (e.g., career-related words) than the other (e.g., family-related words).

The design of WEAT was directly inspired by the Implicit Association Test (IAT) from social psychology. The IAT measures the strength of associations in human cognition by measuring response times: people respond faster to pairings that match their implicit associations than to pairings that conflict with them. WEAT adapts this logic to embedding spaces: instead of response times, it uses cosine similarities. If male names are geometrically closer to career words than female names are, the model's embedding space encodes the same association that the IAT might detect in human cognition.

Formally, let $X$ and $Y$ be two equal-sized sets of target words, and let $A$ and $B$ be two sets of attribute words. The association of a target word $w$ with attribute sets $A$ and $B$ is defined as:

$$s(w, A, B) = \text{mean}_{a \in A}\, \cos(\vec{w}, \vec{a}) - \text{mean}_{b \in B}\, \cos(\vec{w}, \vec{b})$$

where:

• $\vec{w}$ is the embedding vector of word $w$
• $\cos(\vec{u}, \vec{v})$ is the cosine similarity between two vectors
• $\text{mean}_{a \in A}$ averages over all attribute words in set $A$

Intuitively, $s(w, A, B)$ measures how much more similar $w$ is to attribute set $A$ than to attribute set $B$. A positive value means $w$ leans toward $A$. A negative value means $w$ leans toward $B$. A value near zero means no preference. The function $s$ is computed for every target word in both groups.

The test statistic measures the differential association across the two target groups:

$$s(X, Y, A, B) = \sum_{x \in X} s(x, A, B) - \sum_{y \in Y} s(y, A, B)$$

where:

• $s(x, A, B)$ is the single-target association for one word in $X$, as computed above
• The sum is taken over all words in each target group, not just one

If this value is positive, the targets in $X$ associate more strongly with attributes in $A$ than the targets in $Y$ do. The raw statistic is a sum, so its magnitude grows with the size of the word sets, making it hard to compare across different test configurations. WEAT also defines an effect size $d$ analogous to Cohen's $d$ that normalizes for this:

$$d = \frac{\text{mean}_{x \in X}\, s(x, A, B) - \text{mean}_{y \in Y}\, s(y, A, B)}{\text{std}_{w \in X \cup Y}\, s(w, A, B)}$$

where:

• $\text{mean}_{x \in X}\, s(x, A, B)$ is the average single-target association for targets in $X$
• $\text{mean}_{y \in Y}\, s(y, A, B)$ is the average single-target association for targets in $Y$
• $\text{std}_{w \in X \cup Y}\, s(w, A, B)$ is the standard deviation of all individual target associations, pooled across both sets

The effect size $d$ ranges roughly from $-2$ to $+2$. A value near $0$ indicates no differential association. A value of $+1$ indicates that targets in $X$ associate with attributes in $A$ approximately one standard deviation more strongly than targets in $Y$. Values above $1.0$ are considered large effects.

The statistical significance of a WEAT result is assessed through a permutation test. The null hypothesis is that the two groups $X$ and $Y$ are indistinguishable in terms of their association with $A$ and $B$. Under this null, any partition of $X \cup Y$ into two equal groups should produce a test statistic similar to the observed one. The permutation test computes the fraction of random partitions that produce a statistic larger than or equal to the observed value. This fraction is the one-sided $p$-value.

WEAT was validated by showing that its results on word2vec and GloVe embeddings mirrored documented human implicit associations measured by the IAT. Career/family bias along gender lines, pleasantness associations with racial names, and mental health stigma all showed up as significant WEAT effects in embeddings trained on large corpora. The correspondence between WEAT and IAT results was striking: the same stereotypical associations that psychologists had documented in human cognition were legible in the geometric structure of word vectors.

Caliskan et al. defined 10 standard test configurations, each chosen to detect a specific type of bias. Each test specifies four word sets: two target groups and two attribute dimensions. The tests include:

• WEAT 1 and 2: Flower vs. insect names associated with pleasant vs. unpleasant words (a calibration test, confirming the method works)
• WEAT 3 and 4: European-American vs. African-American names associated with pleasant vs. unpleasant words (racial sentiment bias)
• WEAT 5: European-American vs. African-American names with positive/negative sentiment (a more recent name list)
• WEAT 6: Male vs. female names with career vs. family words (gender-career bias)
• WEAT 7: Math vs. arts words with male vs. female attribute words
• WEAT 8: Science vs. arts words with male vs. female attribute words
• WEAT 9: Mental illness vs. physical illness with young/pleasant vs. old/unpleasant attributes
• WEAT 10: Old vs. young age associations with pleasant vs. unpleasant attributes

Tests 1 and 2 serve as sanity checks: virtually everyone finds flowers more pleasant than insects, so a valid embedding should show this. If the method fails on these calibration tests, something is wrong with the embeddings or implementation. The remaining tests probe socially relevant biases that have been documented in humans.

The inclusion of calibration tests was an important methodological choice. Without tests that should obviously succeed, a large effect size on a gender or race test could mean the embedding space encodes that stereotypical association, or it could mean the cosine-similarity-based measurement approach is producing artifacts. The flower/insect test provides a baseline: if the method can reliably detect the obvious case, then large effects on the socially relevant tests are more credibly interpreted as systematic bias rather than measurement noise.

WEAT operates on individual word embeddings. But modern language models rarely produce context-free word vectors. BERT, GPT, and their descendants produce contextualized embeddings where the representation of "bank" depends on the surrounding sentence. For these models, WEAT's single-word approach does not apply.

The Sentence Encoder Association Test (SEAT), introduced by May et al. in 2019, adapts WEAT for sentence encoders. Instead of measuring the cosine similarity of isolated word vectors, SEAT embeds target words in neutral sentence templates and computes cosine similarities between those sentence embeddings.

A typical template looks like "This is [target]." or "[Target] is a person." The target word fills the slot and the full sentence is encoded. This gives the model context to work with while keeping the only varying element the target word itself.

The choice of template is not neutral. "This is John." and "John is a person." produce different sentence embeddings, and the relative associations of those embeddings with career vs. family words may vary depending on the template. May et al. found that effect sizes could differ substantially across templates, raising questions about which template to use and how to report variability. Best practice is to test multiple templates and report both the mean effect size and the range across templates rather than cherry-picking a template that maximizes or minimizes the apparent bias.

For example, to test gender-career bias in a sentence encoder:

• Target X: Sentences like "This is John." / "This is David." (male names)
• Target Y: Sentences like "This is Mary." / "This is Sarah." (female names)
• Attribute A: Career-related words embedded in template sentences
• Attribute B: Family-related words embedded in template sentences

The test statistic and effect size calculations are then identical to WEAT.

SEAT revealed that several popular sentence encoders, including ELMo and BERT variants, encode gender-career and gender-science biases with large effect sizes, despite their contextualized representations being more sophisticated than static word vectors. This finding was important because it showed that moving to contextualized embeddings does not automatically reduce stereotypical associations. The representations became more powerful, but the biases they encoded from the training data remained.

Let's implement WEAT from scratch and then extend it to work with a sentence encoder. We'll use sentence-transformers for the embeddings and scipy for statistical support.

First, install the required packages:

The two heatmaps show the cosine similarity matrix for male names (left) and female names (right). The white vertical line separates career words from family words. Comparing the left half (career) to the right half (family) across each heatmap shows the differential association pattern that WEAT quantifies.

Key ParametersLink Copied

The key parameters for a WEAT/SEAT evaluation are:

• n_permutations: Number of random permutations for the permutation test. More permutations yield more accurate p-values. Values of 5,000 to 10,000 are typical for publication-quality results.
• normalize_embeddings: Whether to L2-normalize sentence embeddings before cosine similarity. This should always be True for cosine-based tests, as it ensures similarity values fall in $[-1, 1]$.
• Sentence template: The template used to wrap target words for SEAT (e.g., "This is {}."). The choice of template can influence results; it is best to test multiple templates and report variability.
• Effect size threshold: By convention, $|d| > 0.8$ is considered a large effect, $|d| > 0.5$ medium, and $|d| < 0.5$ small (following Cohen's conventions).

A few cautions apply when interpreting association test results.

Effect size vs. statistical significance: A test can be statistically significant with a tiny effect size if the word sets are large. Conversely, a large effect size may not reach significance with small word sets. Always report both, and weight the effect size more heavily for practical interpretation.

The word list problem: WEAT results depend heavily on the chosen word lists. Different lists of "male names" or "career words" can yield different effect sizes. There is no canonical word list, so published results across papers are often not directly comparable. When running your own tests, document your word lists explicitly.

Effect size does not equal harm: A large WEAT effect size means the model's embedding space encodes an association. It does not tell you whether that association causes harm in a downstream task. A model could show strong gender-career WEAT effects but still assign credit scores fairly. Bias measurement is diagnostic, not automatically prescriptive.

Cosine similarity limitations: Cosine similarity measures the angle between vectors. In high-dimensional spaces, all vectors tend to have similar cosine similarities (a phenomenon called the concentration of measure), which can attenuate WEAT effect sizes. Newer embedding evaluation methods sometimes use other distance measures or operate in projected subspaces.

Representational vs. allocational harm: WEAT and SEAT measure representational harm, which refers to how a model represents groups in its internal geometry. This is distinct from allocational harm, which refers to concrete decisions that disadvantage certain groups. A model can show large WEAT effects without causing allocational harm if the biased associations are never activated in a way that affects downstream decisions. Conversely, allocational harm can occur in a classifier whose embedding associations look balanced but whose decision boundary happens to be poorly calibrated for one demographic.