Bias Mitigation: Debiasing, CDA, and Fair Fine-tuning › Fine-tuning for Fairness

When you have access to model weights and some labeled fairness data, fine-tuning offers the most powerful mitigation pathway. Rather than adjusting the input distribution or post-processing embeddings, you directly modify what the model learns.

The core motivation is that data-level and embedding-level interventions operate on shallow representations. A model fine-tuned on CDA-augmented data learns to produce statistically balanced outputs at the word-frequency level, but its intermediate representations may still encode demographic information in subtler ways that surface under distribution shift. Fine-tuning methods that modify the model's representational objectives provide stronger guarantees.

Adversarial debiasing treats bias mitigation as a minimax game. The setup involves two networks: a predictor that performs the main task, and an adversary that tries to predict the sensitive attribute from the predictor's representations.

The training objective is:

$$\min_{\theta_p} \max_{\theta_a} \left[\mathcal{L}_{\text{task}}(\theta_p) - \lambda \mathcal{L}_{\text{adv}}(\theta_p, \theta_a)\right]$$

where:

• $\theta_p$ are the predictor's parameters
• $\theta_a$ are the adversary's parameters
• $\mathcal{L}_{\text{task}}$ is the main task loss (e.g., cross-entropy for classification)
• $\mathcal{L}_{\text{adv}}$ is the adversary's loss for predicting the sensitive attribute
• $\lambda$ controls the tradeoff between task performance and fairness

The predictor tries to minimize task loss while maximizing the adversary's loss. The adversary tries to predict the sensitive attribute from the predictor's representation. At equilibrium, the representation contains no information about the sensitive attribute, yet it still supports the task.

The gradient flow is the key implementation detail. During each training step:

1. Forward pass through predictor, compute task loss
2. Forward pass through adversary, compute adversary loss
3. Update adversary parameters to decrease adversary loss (adversary gets better at predicting)
4. Update predictor parameters to decrease task loss and increase adversary loss (predictor gets better at task while hiding demographic signal)

Step 4 uses gradient reversal: multiply the gradient from the adversary loss by $-\lambda$ before it reaches the predictor. This single operation turns the adversary's signal from helpful (for the adversary) to harmful (for the adversary) when it flows to the predictor.

The key insight behind gradient reversal is that the same gradient computation can serve two opposing purposes depending on where it is applied. The adversary network sees the predictor's hidden representation and learns to extract the demographic signal from it. If you allow that gradient to propagate back to the predictor unchanged, the predictor would learn to make its representation easier for the adversary to read, which is the opposite of what you want. By negating the gradient before it crosses the boundary, you force the predictor to actively hide the demographic information: it is penalized for any representation that the adversary can exploit.

In practice, adversarial debiasing is sensitive to the balance between the adversary's and predictor's learning rates. If the adversary learns too quickly, it can always track the predictor's representation and the minimax game never reaches equilibrium. If it learns too slowly, the predictor never receives a meaningful signal to suppress demographic information. Most implementations use a smaller learning rate for the adversary, on the order of 0.1 times the predictor's learning rate, and schedule the $\lambda$ weight to increase gradually from zero over the first few epochs. This curriculum gives the predictor time to learn the task before the fairness constraint becomes strict.

A deeper issue with adversarial training is that the demographic information encoded in representations may be distributed across many dimensions, none of which individually explains much variance. A single linear adversary will struggle to extract distributed information, which means a predictor that defeats the adversary may still have strong demographic information encoded in a way that a more powerful non-linear adversary could retrieve. More powerful adversaries provide stronger fairness guarantees but are harder to train stably. In practice, a two-layer MLP with dropout is a reasonable adversary architecture for most NLP tasks.

If your main concern is that the model's predictions change when you swap demographic attributes in the input, counterfactual fairness fine-tuning directly penalizes that instability.

For a classification model with output probabilities $P(y | x)$, the counterfactual fairness penalty for a demographic swap $\phi$ is:

$$\mathcal{L}_{\text{cf}} = \sum_{i} \text{KL}\!\left(P(y \mid x_i) \,\|\, P(y \mid \phi(x_i))\right)$$

where:

• $P(y \mid x_i)$: the model's predicted probability distribution over labels for the original input $x_i$
• $\phi(x_i)$: the counterfactually modified input with demographic attributes swapped (e.g., male pronouns replaced with female pronouns)
• $P(y \mid \phi(x_i))$: the model's predicted distribution for the swapped input
• $\text{KL}(P \| Q)$: the Kullback-Leibler divergence, measuring how different distribution $P$ is from $Q$; equals zero when the two distributions are identical

The KL divergence quantifies prediction instability across the demographic swap. If changing "he" to "she" causes the model's output to shift significantly, $\text{KL}$ will be large and the penalty is high. If the model produces identical predictions, $\text{KL} = 0$ and no penalty is applied.

A perfectly counterfactually fair model has $\mathcal{L}_{\text{cf}} = 0$ across all examples: the output distribution is invariant to demographic swaps.

Fine-tuning on the combined objective:

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{task}} + \lambda \mathcal{L}_{\text{cf}}$$

where $\lambda$ is the fairness weight. A larger $\lambda$ enforces stricter counterfactual consistency at the cost of some task accuracy. This combined loss pushes the model toward distributional consistency across demographic variants of the same input.

A practical note: the KL divergence is asymmetric. $\text{KL}(P \| Q) \neq \text{KL}(Q \| P)$ in general, and the asymmetry has a semantic consequence here. When $P$ is the original input's distribution, $\text{KL}(P \| Q)$ penalizes cases where the swapped prediction assigns low probability to outcomes the original prediction favored. This is the correct direction for most bias auditing scenarios: you want the model's behavior on the original input to be reproducible under demographic perturbation, not the other way around. For symmetric enforcement, you can replace the KL term with the Jensen-Shannon divergence, which equals $\frac{1}{2}\text{KL}(P \| M) + \frac{1}{2}\text{KL}(Q \| M)$ where $M = \frac{P + Q}{2}$, and has the advantage of being bounded in $[0, \log 2]$ regardless of how different the distributions are.

Reinforcement Learning from Human Feedback, which we covered in earlier chapters on alignment, can incorporate fairness directly into the reward signal. The insight is straightforward: if your reward model is trained to rate outputs partly based on their demographic neutrality, the policy model will learn to generate neutral outputs.

In practice, this means augmenting your annotation guidelines to explicitly flag and penalize biased outputs during reward model training. Annotators are shown pairs of model outputs and asked to prefer the one that is more demographically neutral, in addition to the standard quality dimensions. The resulting reward model then propagates that preference through RLHF training.

This approach has an important advantage: it shapes generation behavior across the output distribution rather than targeting specific demographic words or sentences. A model trained with fairness rewards learns to reason differently, not just to avoid specific keywords. The disadvantage is that it requires high-quality annotator guidelines and substantial disagreement resolution, since annotators themselves bring diverse intuitions about what constitutes fair language.

A subtle risk with RLHF fairness rewards is reward hacking: the model may learn to satisfy the fairness reward criterion superficially by adding explicit disclaimers ("people of all genders...") without changing the core content of its answers. Catching reward hacking requires red-teaming specifically designed to probe whether the model's demographic neutrality is surface-level or deep. A useful test is to strip any explicit demographic framing from the generated text and then evaluate the remaining content with a separate classifier trained to detect demographic associations. If the content itself is still biased even though the framing is neutral, the model has hacked the reward.