Conditional Random Fields › Summary

Conditional Random Fields represent a fundamental advance in sequence labeling, shifting from generative to discriminative modeling to support richer feature representations.

Key concepts from this chapter:

Generative vs. Discriminative: HMMs model $P(X, Y)$ and derive predictions through Bayes' rule. CRFs directly model $P(Y|X)$, avoiding independence assumptions about observations. This allows arbitrary overlapping features of the input sequence.

Log-Linear Formulation: CRFs define probabilities through weighted feature functions:

$$P(\mathbf{y} | \mathbf{x}) = \frac{1}{Z(\mathbf{x})} \exp\left( \sum_{t=1}^{n} \sum_k \lambda_k f_k(y_{t-1}, y_t, \mathbf{x}, t) \right)$$

where $\mathbf{y}$ is the label sequence, $\mathbf{x}$ is the observation sequence, $\lambda_k$ are learned weights, $f_k$ are feature functions, and $Z(\mathbf{x})$ normalizes the distribution. Feature functions capture both label-observation relationships (like HMM emissions) and label-label relationships (like HMM transitions), plus arbitrary combinations.

Partition Function: The normalization constant $Z(\mathbf{x})$ sums over all possible label sequences. The forward algorithm computes it efficiently in $O(nK^2)$ time using dynamic programming.

Inference: The Viterbi algorithm finds the most likely label sequence in $O(nK^2)$ time. Beam search provides an approximate alternative for very large label sets.

Feature Functions: CRF power comes from rich features:

• State features link observations to labels (word suffixes, capitalization, POS tags)
• Transition features capture label-label patterns (valid BIO transitions)
• Context features use surrounding words and their properties

Neural CRFs: Modern architectures combine neural networks (BiLSTMs, transformers) for feature learning with CRF layers for structured prediction. The neural network learns representations; the CRF enforces output constraints.

Implementation: Libraries like sklearn-crfsuite provide efficient CRF training and inference. Feature engineering remains important for traditional CRFs, while neural CRFs learn features automatically but require more training infrastructure.

CRFs transformed sequence labeling by providing a principled framework for combining rich, overlapping features with structured output constraints. They remain the foundation of production NER, POS tagging, and chunking systems, either as traditional feature-based models or as structured output layers in neural architectures. The next chapter explores CRF training in detail: the forward-backward algorithm, gradient computation, and optimization techniques that make learning possible.