Bias in Language Models › Worked Example: Gendered Occupational Associations

To make these concepts concrete, let's trace how a gender-occupation bias manifests in data and in model outputs, working through the mathematics explicitly.

Suppose we have a corpus of news articles, and we count how often each occupation co-occurs with gendered terms (pronouns, title words, names) within a window of 10 words.

We define the masculine co-occurrence ratio for an occupation as:

$$r_{\text{masc}}(w) = \frac{C_{\text{masc}}(w)}{C_{\text{masc}}(w) + C_{\text{fem}}(w)}$$

where:

• $w$: the occupation word being measured (e.g., "doctor," "nurse")
• $C_{\text{masc}}(w)$: the count of masculine-gendered terms co-occurring with $w$ in the corpus (e.g., "he," "him," "his," "Mr.")
• $C_{\text{fem}}(w)$: the count of feminine-gendered terms co-occurring with $w$ (e.g., "she," "her," "Ms.")
• $r_{\text{masc}}(w) \in [0, 1]$: values above 0.5 indicate masculine-leaning; values below indicate feminine-leaning

A ratio of exactly 0.5 represents equal masculine and feminine co-occurrence, which would be expected in an unbiased corpus if the actual population split for that occupation were 50/50.

For the occupation "doctor," suppose we find:

• Co-occurrences with masculine terms ("he," "him," "his," "Mr."): 4,200
• Co-occurrences with feminine terms ("she," "her," "hers," "Ms.," "Mrs."): 1,800

Applying the formula:

$$r_{\text{masc}}(\text{doctor}) = \frac{4{,}200}{4{,}200 + 1{,}800} = \frac{4{,}200}{6{,}000} = 0.70$$

For the occupation "nurse," suppose we find:

• Masculine co-occurrences: 800
• Feminine co-occurrences: 3,600

$$r_{\text{masc}}(\text{nurse}) = \frac{800}{800 + 3{,}600} = \frac{800}{4{,}400} \approx 0.18$$

The complement, $1 - r_{\text{masc}}(\text{nurse}) \approx 0.82$, is the feminine co-occurrence ratio for "nurse."

These co-occurrence ratios appear directly in the training signal. But due to the amplification effects described above, a language model trained on this data may assign probabilities that exceed these ratios. When prompted with "The doctor examined the patient and then", the model might assign:

• $P(\text{"she" or "her"}) = 0.12$
• $P(\text{"he" or "his"}) = 0.38$

We can quantify the amplification by comparing the corpus odds ratio to the model's output odds ratio. In the corpus, the masculine-to-feminine odds for "doctor" are:

$$\text{odds}_{\text{corpus}} = \frac{r_{\text{masc}}}{1 - r_{\text{masc}}} = \frac{0.70}{0.30} \approx 2.33$$

In the model's predictions, normalizing the gendered probabilities to sum to 1 (since $0.38 + 0.12 = 0.50$):

$$\text{odds}_{\text{model}} = \frac{0.38}{0.12} \approx 3.17$$

The amplification factor is:

$$\text{amplification} = \frac{\text{odds}_{\text{model}}}{\text{odds}_{\text{corpus}}} = \frac{3.17}{2.33} \approx 1.36$$

This represents a 36% increase in the relative advantage assigned to masculine pronouns, above what the raw training data contained. Measuring bias at the data level alone is therefore insufficient. The model is a nonlinear function of its training data, and the biases it outputs cannot be fully predicted by counting co-occurrences in the corpus.