Bias Measurement › Generation Bias Metrics

Embedding tests operate on internal representations. But what matters to users is what the model says, not what lives in its hidden states. Generation bias metrics measure bias in the text that models produce in response to prompts. They answer the question: does this model generate systematically different content for different demographic groups?

The shift from representation-level to generation-level measurement is significant. A model might encode stereotypical associations in its geometry while still generating balanced text, perhaps because other factors in the decoding process counteract the biased representations. Conversely, a model might produce biased outputs even with relatively balanced representations if the fine-tuning data or decoding strategy amplifies small asymmetries. Measuring generation bias directly captures the end-user experience in a way that embedding tests cannot.

The regard score, introduced by Sheng et al. (2019), measures the sentiment and social regard expressed when a model generates text about members of different demographic groups. The insight is that if you prompt a model with "The [man/woman] worked as a..." or "The [Black/White] person was known for...", the completions should be equally positive or neutral across groups if the model is unbiased.

The regard pipeline works in three steps:

1. Create minimal pairs: Start with sentence templates where only the demographic descriptor varies. For example:

   • "The woman walked into the room and..."
   • "The man walked into the room and..."

2. Generate completions: Use the model under evaluation to complete each sentence.

3. Classify regard: Pass the completions through a regard classifier (a fine-tuned text classifier) that labels the social portrayal as positive, neutral, negative, or other.

Bias is measured as the difference in the distribution of regard labels across demographic groups. A model that generates more negative completions for women than for men shows gender regard bias.

What makes the regard score useful is that it captures social connotation rather than raw sentiment. A completion like "She stayed home and raised her children" might score neutral or positive on a sentiment classifier, but the regard classifier is trained to recognize that this frames women in a restricted social role. The regard classifier was fine-tuned on annotations that specifically capture whether text portrays the subject in a positive, neutral, or negative social light, making it sensitive to stereotyping that pure sentiment analysis would miss.

Sheng et al. used the regard score to compare several GPT-2 variants and found consistent patterns: prompts about women generated completions with lower positive regard than prompts about men, and prompts mentioning certain racial identities generated completions with more negative regard than prompts with other identities. These disparities persisted even after standard fine-tuning, suggesting that the stereotypical associations in the training data were difficult to erase through conventional training procedures.

The Stereotype Score (SS) evaluates how strongly a model reinforces stereotypes in generation. It is typically measured using paired sentences like:

• "The programmer said that [PRONOUN] loves working with computers."
• "The nurse said that [PRONOUN] loves caring for patients."

The model assigns probabilities to "he" and "she" in the pronoun slot. If the model consistently assigns higher probability to "he" for programmer and "she" for nurse, it is reinforcing occupational gender stereotypes. The stereotype score quantifies this tendency.

More formally, for a set of stereotype-relevant sentence pairs $(s_+, s_-)$ where $s_+$ is the stereotypical version and $s_-$ is the anti-stereotypical version:

$$\text{SS} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\left[ P_{\theta}(s_{+,i}) > P_{\theta}(s_{-,i}) \right]$$

where:

• $N$ is the number of sentence pairs
• $P_{\theta}(s)$ is the probability the model assigns to sentence $s$
• $\mathbf{1}[\cdot]$ is the indicator function, equal to 1 if the condition is true and 0 otherwise

A stereotype score of $0.5$ indicates no preference between stereotypical and anti-stereotypical sentences. A score above $0.5$ indicates the model prefers the stereotypical framing. The ideal unbiased model scores exactly $0.5$.

An important subtlety in computing $P_{\theta}(s)$ for autoregressive language models is that the probability of a sentence depends on its length and the probability of each token in context. Longer sentences with more tokens will generally have lower absolute log-probabilities than shorter ones, purely because of chain rule decomposition. For this reason, length-normalized log-probabilities are often preferred when comparing sentences of different lengths. When both sentences in a pair have the same length (which is often arranged deliberately in benchmark design), raw log-probabilities are comparable. When lengths differ, normalization by the number of tokens is standard.

Toxicity measurement evaluates whether a model generates harmful content, including slurs, threats, explicit violence, or demeaning language, and whether this tendency varies across demographic groups.

The standard approach uses a pre-trained toxicity classifier to score model completions. Google's Perspective API, which was built on a classifier trained on the Jigsaw Civil Comments dataset, is one of the most widely used tools for this purpose. It returns scores across multiple toxicity categories, including:

• Toxicity: General harmful content
• Severe toxicity: Explicit threats or abuse
• Identity attack: Content targeting a demographic group
• Insult: Degrading or disrespectful language
• Threat: Content expressing intent to harm
• Profanity: Explicit language

To measure disparate toxicity, you generate completions from the model under evaluation by prompting it with minimally varied templates:

• "The [group] person said..."
• "[Group] people are known for..."
• Completions involving [group]-specific terminology

After scoring each completion with the toxicity classifier, you compare the distributions across demographic groups. If completions about Group A average 15% toxicity and completions about Group B average 5% toxicity, the model has learned to generate disproportionately harmful content in one demographic context, even when the prompt itself was neutral.

A key finding from studies of large language models is that prompts mentioning certain religious groups, racial groups, or political identities elicit consistently higher toxicity in model completions. This occurs even when the prompts are framed neutrally, suggesting that the model associates these identity terms with more contentious discourse from the training corpus. RealToxicityPrompts (Gehman et al., 2020) demonstrated that models trained on Common Crawl data generate toxic continuations at non-trivial rates and show systematic disparities across different demographic mentions.

One important limitation of toxicity-based bias measurement is that toxicity classifiers themselves may encode biases. A classifier trained predominantly on English text from Western cultural contexts may incorrectly flag vernacular language used by certain communities as toxic, leading to measurement errors that compound the underlying bias problem. When using toxicity metrics for bias evaluation, it is worth testing whether the classifier itself shows differential false positive rates across the demographic groups you are measuring.

Another complication is what researchers call the toxicity-helpfulness tradeoff. Models that refuse to generate any content about sensitive demographic groups will score perfectly on toxicity metrics, but they fail users who need to discuss those topics legitimately. A toxicity score of zero achieved by refusing to engage is not the same as a toxicity score of zero achieved by generating balanced, non-toxic content across all demographic groups. Measurement frameworks that track only toxicity rates without also tracking refusal rates can miss this distinction.

Let's build a simple demonstration of generation bias measurement using a language model and a probability-based stereotype score. We'll use the transformers library to examine how BERT's fill-mask probabilities reflect occupational gender stereotypes: