CRF Training › Limitations and Impact

CRF training has both strengths and limitations that shape when and how to use these models effectively.

The forward-backward algorithm makes CRF training tractable, but it still requires $O(T \cdot |L|^2)$ time per sequence. For tasks with many labels like fine-grained NER or semantic role labeling, this quadratic scaling becomes expensive. A sequence with 50 labels and 100 tokens requires computing 250,000 transition scores per training example. This cost compounds across thousands of training sentences and hundreds of optimization iterations. For real-time applications or very large training corpora, even this polynomial cost can become a bottleneck, and practitioners sometimes turn to approximate inference methods or batch the training examples to exploit parallelism.

Feature engineering remains both a strength and weakness. Hand-crafted features like word shapes, prefixes, and context windows capture linguistic knowledge effectively. But designing features requires domain expertise and extensive experimentation. Features that work well for English NER may perform poorly on German or Japanese: German capitalizes all nouns, so capitalization is a weaker signal for named entities; Japanese has no spaces between words, making the notion of a "word" itself a pre-processing challenge. Adapting a CRF to a new language or domain often requires redesigning the feature set from scratch, which is time-consuming and requires linguistic knowledge of the target language.

CRFs also struggle with long-range dependencies. The Markov assumption means each label depends only on the immediately preceding label. If a token's correct label depends on context five words away, the CRF must propagate that information through intervening transitions. In practice, this limits how much global context the model can exploit. You can partially compensate by including context features that look several words in either direction, but these are fixed features rather than learned representations, so they cannot capture the full richness of long-range dependencies.

Another fundamental limitation is the inability to handle overlapping entities. Standard CRFs produce exactly one label per token, so they cannot represent the case where "New York City" is simultaneously a location and part of an organization name. More complex models like semi-CRFs or factorial CRFs address some of these limitations but at greater computational cost.

Despite these limitations, CRFs made sequence labeling practical before deep learning. Their contribution extends beyond the models themselves to the training paradigms they established. The idea of discriminatively training structured models, using dynamic programming for efficient inference, and designing features to capture domain knowledge influenced subsequent developments in neural NLP. Modern neural CRF layers, which place a CRF on top of BiLSTM or transformer encoders, combine learned representations with the structured prediction framework that CRFs pioneered. When you use a BERT-based NER model with a CRF output layer today, you are using a direct descendant of the approach described in this chapter.

For practitioners today, CRFs remain valuable in several scenarios. When training data is limited (hundreds rather than thousands of examples), CRFs often outperform neural models because their features encode prior knowledge that the neural model cannot learn from a small sample. When interpretability matters, examining feature weights reveals what the model has learned in terms a domain expert can evaluate and critique. When computational resources are constrained, CRFs train on CPUs in minutes rather than requiring GPUs for hours. And as components in hybrid systems, CRF output layers on top of neural encoders often improve boundary accuracy compared to using softmax alone, because they enforce global consistency across the label sequence.

The historical importance of CRFs also lies in what they taught the NLP community about structured prediction. Before CRFs, many NLP systems used pipeline architectures: a POS tagger trained independently, then an NER model trained independently using POS tags as features. CRFs enabled joint training, where the output structure itself is modeled as a random variable and the model is optimized end-to-end over the full output. This shift from pipeline to joint modeling was one of the conceptual breakthroughs that set the stage for the end-to-end neural models that came later.