Hidden Markov Models › Worked Example: Tagging "Time flies"
Language AI HandbookAll chapters Hidden Markov Models › Worked Example: Tagging "Time flies"One section · Sections in this chapter9 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
To understand how an HMM resolves ambiguity, let's trace through the complete probability computation for the classic ambiguous sentence "Time flies." This two-word sentence has two plausible interpretations, and both words can function as either nouns or verbs:
- "Time flies" could mean "Time passes quickly" (NOUN VERB)
- "Time flies" could mean "Measure how fast flies move" (VERB NOUN)
The second interpretation is grammatically valid but pragmatically unusual. Most readers parse "Time flies" as the first interpretation without hesitation, which is exactly what we want the HMM to do. The question is: does the statistical evidence from the corpus lead the model to the same conclusion?
We will compute the joint probability for each interpretation and see which one the HMM prefers. Since we are multiplying many small probabilities, we will work in log space to avoid numerical underflow. Each log probability is a negative number (since all probabilities are between 0 and 1), so the interpretation with the higher (less negative) total log probability is the preferred one.
# Use parameters estimated from training data
def compute_log_joint(words, tags, pi, A, B):
"""Compute log P(words, tags) to avoid underflow."""
import math
log_prob = 0
# Initial probability
init_p = pi.get(tags[0], 1e-10)
log_prob += math.log(init_p)
# First emission
word = words[0].lower()
emit_p = B.get(tags[0], {}).get(
word, B.get(tags[0], {}).get("<UNK>", 1e-10)
)
log_prob += math.log(emit_p)
# Subsequent transitions and emissions
for i in range(1, len(words)):
trans_p = A.get(tags[i - 1], {}).get(tags[i], 1e-10)
word = words[i].lower()
emit_p = B.get(tags[i], {}).get(
word, B.get(tags[i], {}).get("<UNK>", 1e-10)
)
log_prob += math.log(trans_p) + math.log(emit_p)
return log_prob
words = ["Time", "flies"]
interpretations = [
(["NOUN", "VERB"], "Time (subject) flies (action)"),
(["VERB", "NOUN"], "Time (action) flies (object)"),
]
log_probs = []
for tags, description in interpretations:
log_p = compute_log_joint(words, tags, pi_train, A_train, B_train_smoothed)
log_probs.append((tags, log_p, description))Analyzing 'Time flies' ================================================== Interpretation: NOUN VERB Meaning: Time (subject) flies (action) Log probability: -19.7839 Interpretation: VERB NOUN Meaning: Time (action) flies (object) Log probability: -27.3867 ================================================== Most likely interpretation: NOUN VERB (Time (subject) flies (action)) NOUN VERB is 2003.79x more likely than VERB NOUN
To understand why the HMM prefers NOUN VERB, let's decompose the log probability into its components: initial probability, emissions, and transitions.
The HMM prefers "Time flies" as NOUN VERB. This makes sense: in the training data, sentences starting with nouns are more common than sentences starting with verbs, and noun-verb transitions are frequent. The emission probability of "time" from NOUN is also higher than from VERB, since "time" appears as a noun more often in general text.
This example illustrates how HMMs combine multiple sources of evidence: initial probabilities favor starting with nouns, transitions favor noun-verb patterns, and emissions favor "time" as a noun and "flies" as a verb. The bar chart above lets you see exactly how each component contributes to the total log probability. When the NOUN VERB interpretation wins, it is not because of a single dominant factor; it is because several moderately sized factors all point in the same direction.
The key insight from this example is that HMMs do not disambiguate words in isolation. "Flies" has substantial probability from both NOUN and VERB, so the emission alone would not clearly favor either interpretation. What tips the balance is the combination of initial probability (nouns are more common sentence starters), transition probability (noun-to-verb transitions are common), and emission probability (time-as-noun and flies-as-verb are both plausible). No single source of evidence is conclusive; together they provide enough signal to make the correct call. This is the power of the joint probability framework.