CRF Training › Summary

Training Conditional Random Fields requires balancing computational efficiency with model expressiveness. The chapter covered each of the key components that make CRF training work in practice.

The log-likelihood objective is the foundation: CRF training maximizes the probability of correct label sequences. The objective is concave, guaranteeing a unique global optimum without the pathological local minima that complicate neural network training. This concavity follows from the log-sum-exp structure of the partition function and makes CRFs one of the most theoretically clean models in NLP.

The forward-backward algorithm is the computational engine. Dynamic programming computes the partition function and all marginal probabilities in $O(T \cdot |L|^2)$ time, making training tractable despite the exponential number of possible label sequences. The algorithm runs in log space to avoid numerical underflow, using the log-sum-exp trick at each step. Both forward and backward passes are required: the forward pass alone gives the partition function, but both together give the pairwise marginals needed for gradient computation.

Gradient computation follows directly from the forward-backward pass. The gradient equals observed feature counts minus expected feature counts. This moment-matching interpretation makes the convergence criterion transparent: training is complete when the model's distribution over label sequences reproduces the empirical feature statistics of the training data, subject to the regularization constraint.

L-BFGS optimization closes the loop between gradient computation and weight updates. Quasi-Newton methods converge much faster than gradient descent by approximating curvature information from gradient history. Most CRF implementations use L-BFGS by default because the convex CRF objective is well-suited to second-order methods and does not require the careful learning rate scheduling that gradient descent demands.

Feature templates encode domain knowledge into the model. Hand-designed features capture patterns like word shapes, context windows, morphological properties, and neighboring POS tags. Good feature design is often more important than optimizer choice or regularization strength, because the CRF can only learn patterns that the features can represent.

Regularization prevents overfitting by penalizing large weights. L1 regularization (c1 in sklearn-crfsuite) produces sparse models by driving many weights to exactly zero. L2 regularization (c2) shrinks all weights toward zero without enforcing sparsity. In practice, a small amount of both is often most effective. Regularization strength should be chosen by cross-validation on a development set.

CRFs established the foundation for structured prediction in NLP. While neural approaches now dominate many leaderboards, the principles of discriminative training, dynamic programming, and structured output remain central to modern sequence labeling systems. Neural CRF models directly inherit this framework, placing the CRF training objective on top of learned feature representations. Understanding how CRFs are trained makes these modern hybrid models much easier to reason about and debug.