Hidden Markov Models › Key Parameters

When implementing HMM-based taggers, several parameters affect model performance. Understanding what each parameter controls and how it interacts with the others will help you tune your model effectively on new tasks and datasets.

• Smoothing constant ($\alpha$): Controls how much probability mass is reserved for unseen word-tag combinations. Higher values (e.g., 1.0 for Laplace smoothing) give more weight to unseen events but dilute the learned emission probabilities for known words. Lower values (e.g., 0.01) preserve learned distributions better but may leave unknown words dangerously undersmoothed. Start with $\alpha = 1.0$ and tune based on unknown word accuracy on a development set. If your test domain has very different vocabulary from training (e.g., medical text trained on news), you will likely need to increase $\alpha$.

• Tag set granularity: Finer tag sets (like Penn Treebank's 36 tags) capture more distinctions but require more training data and increase computational cost. Coarser sets (like Universal Dependencies' 17 tags) are easier to learn but lose information. Choose based on your downstream task requirements. For most information extraction applications, coarse tags are sufficient; for linguistic research or parsing support, fine-grained tags matter.

• Case normalization: Converting words to lowercase reduces vocabulary size and improves emission estimates for rare words, but loses capitalization signals that help identify proper nouns and sentence boundaries. The implementation above uses lowercase normalization, which helps with data sparsity but hurts on named entity recognition tasks where capitalization is a strong signal. A practical compromise is to lowercase everything but preserve a "starts with capital" feature in an augmented emission model.

• Unknown word handling: Beyond smoothing, you can improve unknown word accuracy by using morphological features (suffixes, prefixes, capitalization patterns) or by backing off to character-level models. The simple <UNK> token approach shown here provides a baseline, but production systems typically cluster unknown words by their suffix (last 3-4 characters) and learn separate emission distributions for each suffix class. This allows the model to recognize that "implementation" is very likely a noun based on its "-tion" suffix, even if "implementation" itself never appeared in training.

• Higher-order transitions: Switching from first-order (bigram) to second-order (trigram) transitions improves accuracy but requires more data and memory. For a 40-tag Penn Treebank model, the transition table grows from 1,600 to 64,000 entries. If your corpus has fewer than 100,000 sentences, trigram transitions may be too sparse to estimate reliably without additional smoothing (such as linear interpolation between trigram, bigram, and unigram estimates).