Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

CRF Training › Regularization

Michael BrenndoerferUpdated May 4, 20255 min read

Language AI HandbookAll chapters

Part 6 of 67Sequence Labeling

Chapter 43CRF Training › Regularization

Reading section 7 of 11 · Regularization · 5 minSection 7 of 11 · 5 min

Section 7 of 11 · 5 min read

CRF Training › RegularizationOne section · Sections in this chapter7 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Without regularization, CRF training can overfit to the training data. If a feature appears only once in training and perfectly predicts a label, the optimal weight is infinity. Regularization prevents extreme weights and improves generalization.

Regularization

Regularization adds a penalty term to the objective function that discourages large parameter values. This prevents overfitting by trading off training set fit against model complexity.

To understand why CRFs need regularization, consider what happens with a feature like "word=Schenectady" that fires exactly once in the training data, where the word was labeled B-LOC. If we maximize log-likelihood without any penalty, the optimal weight for this feature approaches infinity: making that weight arbitrarily large makes the CRF assign arbitrarily high probability to labeling "Schenectady" as B-LOC, which perfectly explains the one training example. But at test time, if the model sees "Schenectady" in a context where it is labeled differently (perhaps in a business name), the overly large weight will cause errors.

Regularization resolves this by adding a cost for large weights. The optimizer must balance two competing objectives: maximize the fit to training data, and keep weights small. For features that appear only once or twice, the training data provides weak evidence, so the optimizer accepts some loss in training fit to keep the weight reasonable. For features that appear hundreds of times with consistent labels, the training signal is strong enough to overcome the regularization cost, and the weight grows to reflect the recurring pattern.

L1 and L2 RegularizationLink Copied

The regularized log-likelihood objective is:

\mathcal{L}_{\text{reg}}(\boldsymbol{\lambda}) = \mathcal{L}(\boldsymbol{\lambda}) - \frac{1}{C_1} \sum_k |\lambda_k| - \frac{1}{2 C_2} \sum_k \lambda_k^2

where:

$\mathcal{L}(\boldsymbol{\lambda})$ : the original log-likelihood
$C_1$ : L1 regularization coefficient (larger = less regularization)
$C_2$ : L2 regularization coefficient (larger = less regularization)
$|\lambda_k|$ : absolute value of weight $k$ (L1 penalty)
$\lambda_k^2$ : squared weight (L2 penalty)

L1 and L2 regularization have different effects:

Comparison of L1 and L2 regularization for CRF training.
Regularization	Penalty	Effect	Use Case
L1	$\\|\lambda\\|$	Drives weights to exactly zero	Feature selection, sparse models
L2	$\lambda^2$	Shrinks all weights toward zero	General regularization
L1 + L2	Both	Combines sparsity with shrinkage	Often works best in practice

In[24]:

Code

# Compare different regularization strengths
regularization_strengths = [0.01, 0.1, 1.0, 10.0]
results = []

for c in regularization_strengths:
    crf_reg = sklearn_crfsuite.CRF(
        algorithm="lbfgs",
        c1=c,
        c2=c,
        max_iterations=100,
        all_possible_transitions=True,
    )
    crf_reg.fit(X_train, y_train)

    # Count non-zero features
    nonzero_features = sum(
        1 for _, weight in crf_reg.state_features_.items() if abs(weight) > 1e-6
    )

    # Get max weight magnitude
    max_weight = max(
        abs(weight) for _, weight in crf_reg.state_features_.items()
    )

    results.append(
        {"c": c, "nonzero": nonzero_features, "max_weight": max_weight}
    )

Out[25]:

Console

Effect of Regularization Strength
==================================================

C (reg coef)    Non-zero features    Max |weight|   
--------------------------------------------------
0.01            141                  3.3624         
0.10            122                  2.4189         
1.00            25                   1.4400         
10.00           3                    0.2209

Out[26]:

Visualization

Semi-log plot showing non-zero feature count increasing with regularization coefficient C. — Stronger regularization (smaller C) produces fewer non-zero features, creating sparser models that use only the most informative features.

Semi-log plot showing maximum weight magnitude increasing with regularization coefficient C. — Maximum weight magnitude decreases with stronger regularization, preventing extreme parameter values and improving generalization.

The regularization plots show clear trends. With strong regularization (small C), many features get zeroed out, creating a sparse model that uses only the most informative features. Maximum weight magnitudes also decrease, preventing the model from over-relying on any single feature.

Choosing the right regularization strength is a hyperparameter tuning problem. The common practice is to try a geometric grid of values, such as $C \in \{0.01, 0.03, 0.1, 0.3, 1.0\}$ , and evaluate each on a held-out development set using the F1 score or other task-specific metric. For NER, the metric of interest is usually entity-level F1, which requires predicting both the boundary and the type of each entity correctly. A regularization coefficient that maximizes token-level accuracy might produce many boundary errors; measuring F1 at the entity level is more informative for the deployment use case.

In practice, L2 regularization ( $c_2$ ) is the safer default choice because it does not produce discontinuous gradients. L1 regularization produces a gradient discontinuity at zero: the subgradient at $\lambda_k = 0$ is not unique, which requires special handling in gradient-based optimizers. L-BFGS in scipy handles L1 via the OWL-QN extension, which operates on the L-BFGS-B variant. The key practical difference is that L1 often produces better-compressed models (many exact zeros) while L2 tends to produce models where all features have small nonzero weights. For deployment scenarios where model size or inference speed matters, L1 or combined L1+L2 may be preferable despite the added optimization complexity.