Bias Measurement › Classification Bias Metrics
Language AI HandbookAll chapters Bias Measurement › Classification Bias MetricsOne section · Sections in this chapter4 / 8 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Embedding tests probe representations. Generation tests probe outputs. Classification bias metrics probe decision quality. When a language model is used as a classifier (for sentiment analysis, toxicity detection, hiring screening, credit scoring, or anything else that assigns a label), fairness requires that performance should not differ systematically across demographic groups.
Classification bias metrics borrow from the algorithmic fairness literature. They define fairness mathematically, then measure whether a model satisfies those definitions. This formalization is powerful because it makes claims about fairness falsifiable and measurable. It also reveals a fundamental tension: different mathematical definitions of fairness are often mutually incompatible, which means the choice of metric is a normative decision, not a purely technical one.
Group Fairness Definitions
There are several competing mathematical definitions of fairness, and they are often mutually incompatible. Understanding the main ones is essential because the metric you choose reflects which type of fairness you are prioritizing.
Demographic Parity requires that the model's positive prediction rate be equal across groups:
where:
- is the predicted label
- is a protected attribute (e.g., race, gender)
The demographic parity difference (DPD) quantifies the violation:
A DPD of 0 means perfect demographic parity. Values closer to 1 indicate larger disparities.
Demographic parity is intuitive: if a hiring model selects 40% of applicants from group A and only 20% from group B, it is treating them unequally regardless of their qualifications. But demographic parity has a well-known limitation: it can be achieved by making the model worse for the advantaged group rather than better for the disadvantaged group. If you reduce the selection rate for group A to match the lower rate for group B, you have achieved parity without improving anything for group B. Critics argue that this "leveling down" shows why demographic parity alone is an insufficient fairness criterion.
Equalized Odds requires that both the true positive rate (TPR) and the false positive rate (FPR) are equal across groups:
where is the ground truth label. The equalized odds difference measures the maximum gap across both rates:
Equalized odds is often considered a stronger fairness criterion than demographic parity because it conditions on the actual outcomes, preventing a model from achieving parity by simply being wrong equally often for both groups. A model satisfying equalized odds is equally accurate across groups: it correctly identifies qualified candidates from both groups at the same rate (equal TPR) and incorrectly rejects unqualified candidates at the same rate (equal FPR).
Equal opportunity is a relaxed version of equalized odds that requires only TPR equality, accepting that FPR may differ. The argument for this relaxation is that false positives (incorrectly classifying an unqualified candidate as qualified) are less harmful than false negatives (rejecting a qualified candidate), so the primary concern is ensuring that qualified candidates from all groups are identified at equal rates.
Calibration (also called predictive parity) requires that predicted probabilities have the same meaning across groups. Formally, if a model assigns a probability to a positive outcome, then among all instances assigned probability , the actual positive rate should be , regardless of group membership:
A model is well-calibrated across groups if its confidence scores are equally trustworthy for all demographic groups. If the model assigns a 70% confidence to positive outcomes but is only correct 50% of the time for group B while being correct 70% of the time for group A, the model is miscalibrated for group B.
Disparate Impact Ratio uses a multiplicative rather than additive comparison:
The legal standard in US employment law is that a ratio below 0.8 (or above 1.25) indicates disparate impact, meaning one group receives positive predictions at less than 80% of the rate of the more-favored group.
It is mathematically impossible to simultaneously satisfy demographic parity, equalized odds, and calibration (equal prediction accuracy) when base rates differ across groups. Choosing which fairness criterion to optimize is a normative decision, not a purely technical one.
The impossibility theorem, proven independently by Chouldechova (2017) and Kleinberg et al. (2016), is one of the most important results in algorithmic fairness. It shows that when base rates (the actual prevalence of positive outcomes) differ across groups, you cannot simultaneously achieve demographic parity, equalized odds, and calibration. You have to choose. This means that any fairness measurement study implicitly makes a normative choice about which definition of fairness matters most, and different stakeholders with different values will legitimately prioritize different definitions.
In practice, the choice among fairness criteria often depends on the deployment context. Calibration is most important in medical risk scoring, where a miscalibrated score for a particular demographic could lead to under-treatment. Equalized odds is most important in criminal justice risk assessment, where both false positives (unjust detention) and false negatives (public safety) have serious consequences. Demographic parity is most important in hiring and lending, where the goal is proportional representation across groups.
Implementing Classification Bias Metrics
Let's compute fairness metrics on a synthetic dataset to illustrate how these calculations work:
def compute_fairness_metrics(y_true, y_pred, group):
"""
Compute demographic parity difference, equalized odds difference,
and disparate impact ratio for binary classification.
y_true: ground truth labels (0 or 1)
y_pred: predicted labels (0 or 1)
group: protected attribute labels (0 = group A, 1 = group B)
"""
mask_0 = group == 0
mask_1 = group == 1
ppr_0 = y_pred[mask_0].mean()
ppr_1 = y_pred[mask_1].mean()
tpr_0 = (
y_pred[mask_0 & (y_true == 1)].mean()
if (y_true[mask_0] == 1).any()
else 0.0
)
tpr_1 = (
y_pred[mask_1 & (y_true == 1)].mean()
if (y_true[mask_1] == 1).any()
else 0.0
)
fpr_0 = (
y_pred[mask_0 & (y_true == 0)].mean()
if (y_true[mask_0] == 0).any()
else 0.0
)
fpr_1 = (
y_pred[mask_1 & (y_true == 0)].mean()
if (y_true[mask_1] == 0).any()
else 0.0
)
acc_0 = (y_pred[mask_0] == y_true[mask_0]).mean()
acc_1 = (y_pred[mask_1] == y_true[mask_1]).mean()
return {
"ppr_0": ppr_0,
"ppr_1": ppr_1,
"demographic_parity_diff": abs(ppr_0 - ppr_1),
"tpr_0": tpr_0,
"tpr_1": tpr_1,
"fpr_0": fpr_0,
"fpr_1": fpr_1,
"equalized_odds_diff": max(abs(tpr_0 - tpr_1), abs(fpr_0 - fpr_1)),
"disparate_impact_ratio": ppr_0 / ppr_1 if ppr_1 > 0 else float("inf"),
"accuracy_0": acc_0,
"accuracy_1": acc_1,
}np.random.seed(42)
n = 1000
# Simulate a biased sentiment classifier.
# Group 0 is correctly classified at 85%, but Group 1 has a higher
# false negative rate (misses 45% of positive cases), reflecting a model
# that performs well on one demographic but poorly on another.
group = np.random.randint(0, 2, n)
y_true = np.random.randint(0, 2, n)
y_pred = np.zeros(n, dtype=int)
for i in range(n):
if group[i] == 0:
if y_true[i] == 1:
y_pred[i] = 1 if np.random.random() < 0.85 else 0
else:
y_pred[i] = 1 if np.random.random() < 0.15 else 0
else:
if y_true[i] == 1:
# Higher false negative rate for Group 1
y_pred[i] = 1 if np.random.random() < 0.55 else 0
else:
y_pred[i] = 1 if np.random.random() < 0.15 else 0
metrics = compute_fairness_metrics(y_true, y_pred, group)Fairness Metrics for Biased Classifier Metric Group 0 Group 1 Difference -------------------------------------------------------------------- Positive Prediction Rate 0.498 0.300 0.198 True Positive Rate 0.853 0.477 0.375 False Positive Rate 0.181 0.139 0.043 Accuracy 0.835 0.678 0.156 Demographic Parity Difference: 0.1980 Equalized Odds Difference: 0.3754 Disparate Impact Ratio: 1.6599 OK: DIR 1.660 >= 0.8 (passes the 80% rule).
The TPR gap is the most telling metric here. Group 1 is correctly identified as positive far less often than Group 0, meaning the model systematically misses positive cases for that group. This is the equalized odds violation. If this were a real classifier screening job applications, Group 1 members would be disproportionately rejected even when they qualified.
Notice that overall accuracy hides this disparity. If you reported only overall accuracy, the model might appear to be performing well. This is why fairness auditing cannot stop at aggregate performance metrics but must decompose performance by demographic group.
Key Parameters
The key parameters for classification bias analysis are:
demographic_parity_diff: The absolute difference in positive prediction rates between groups. Values above 0.05 to 0.10 are commonly flagged as problematic in practice.equalized_odds_diff: The maximum gap in either TPR or FPR across groups. This is generally considered a more stringent fairness criterion than demographic parity.disparate_impact_ratio: The ratio of positive prediction rates. A value below 0.8 may violate the legal 80% rule in US employment contexts. A value above 1.25 indicates reverse disparity.- Protected attribute: The demographic variable used to define groups. Each choice of protected attribute produces a different fairness analysis, and intersectional analysis requires examining all combinations.
Intersectionality in Classification Bias
One limitation of standard fairness metrics is that they examine one protected attribute at a time. In practice, bias often operates across combinations of multiple identities. A classifier might treat white women and Black men fairly when each group is examined alone, but still show severe disparity for Black women.
Intersectional analysis extends the standard fairness metrics by computing them for all pairwise (or higher-order) combinations of protected attributes. This requires substantially more data per subgroup, which is one reason it is rarely done in practice despite being more accurate.
Kimberlé Crenshaw's intersectionality framework from legal scholarship has directly influenced how researchers think about this problem. A model that appears fair under demographic parity for gender AND demographic parity for race may still concentrate harm on multiply-marginalized groups in ways that single-attribute analysis cannot detect.
The data requirements for intersectional analysis grow combinatorially. If you have three binary protected attributes (gender, race, age group), examining all intersections requires data for eight subgroups instead of six. Subgroups that are small in the population, for example, elderly disabled women of color, may have too few examples in any realistic evaluation set to estimate fairness metrics reliably. This creates a tension: the intersections where harm is most concentrated are often the same intersections where measurement is hardest. Researchers sometimes address this with conformal prediction intervals or by reporting which intersections have too few examples for reliable estimates rather than simply omitting them.