Hidden Markov Models › Hidden Markov Models
Language AI HandbookAll chapters Hidden Markov Models › Hidden Markov ModelsOne section · Sections in this chapter1 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Sequence labeling tasks like POS tagging require more than just looking up each word in a dictionary. The word "bank" could be a noun (a financial institution) or a verb (to bank on something). The word "flies" could be a noun (the insects) or a verb (the action of flying). Context determines the correct label, and that context often spans multiple words in a sequence.
Hidden Markov Models (HMMs) provide a principled probabilistic framework for sequence labeling. An HMM models two interacting processes: a hidden sequence of states (the tags you want to predict) and an observable sequence of outputs (the words you can see). The model learns statistical patterns about how states transition to each other and how each state generates observations. Given a new sentence, the HMM uses these learned patterns to infer the most likely hidden sequence.
Think of an HMM as a kind of hidden storyteller. Imagine a narrator who moves secretly through a sequence of rooms: the tags: and calls out a word from each room. You can hear the words, but you cannot see which room the narrator is in. Your job is to figure out the narrator's path through the rooms based solely on the words you hear. The HMM formalizes exactly this: it provides probabilities that describe which rooms connect to which, and which words get called out from each room. Given a sequence of heard words, the model can then work backwards to estimate the most probable path.
The main insight behind HMMs is that language is not a bag of independent tokens. When you see the word "the," you know with high confidence that a noun phrase is coming. When you see a verb, you know it probably follows a noun phrase. These regularities exist at two levels simultaneously: the syntactic level (how tags relate to one another) and the lexical level (which words are associated with which tags). HMMs capture both levels in a single probabilistic model, and the interaction between them is what makes disambiguation possible.
This chapter covers HMM components and their probabilistic foundations. You'll learn how to estimate HMM parameters from annotated data, understand the key assumptions that make HMMs tractable, and implement a working POS tagger. We'll also examine where HMMs succeed and where they fall short, setting up the motivation for more advanced models like Conditional Random Fields. By the end, you will have a working intuition for why this 1960s signal-processing technique became the backbone of practical NLP systems for nearly two decades.
Hidden Markov Models were originally developed for speech recognition in the late 1960s and early 1970s by Leonard Baum and colleagues at the Institute for Defense Analyses. The classic Baum-Welch algorithm for training HMMs dates to 1972. Throughout the 1980s, IBM researchers adapted HMMs to machine translation and speech, and by the early 1990s the technique had migrated into text-based NLP. Eric Brill's influential 1992 dissertation on transformation-based learning and the subsequent adoption of HMMs in systems like the Xerox tagger established HMM-based POS tagging as the standard approach for most of the decade. The Penn Treebank, released in 1993, provided the labeled data needed to train high-quality HMM taggers, and the combination of clean annotated corpora and HMM inference algorithms pushed tagging accuracy above 95% for the first time. HMMs were eventually displaced by Maximum Entropy Markov Models and Conditional Random Fields in the early 2000s, which themselves gave way to neural approaches, but the conceptual framework they established, that sequence labeling is a structured prediction problem over hidden state sequences, remains central to NLP today.