Hidden Markov Models (1970s) › The Architecture of Hidden Markov Models

HMMs represent systems with two distinct layers of information. The first layer consists of hidden states, the underlying system states that you cannot directly observe. In speech recognition, these might be the phonemes or words being spoken. When someone says the word "cat," the true linguistic units they're producing exist in their vocal tract and brain, but all you receive as an observer is a stream of acoustic energy. The second layer consists of observable outputs, the signals you can measure. In speech recognition, these are the acoustic features extracted from the audio signal, things like frequency components, energy levels, and spectral characteristics at each moment in time.

The relationship between these two layers is fundamentally probabilistic. A given hidden state doesn't deterministically produce a single observation. Instead, it generates observations according to a probability distribution. This captures the reality that the same phoneme can be pronounced slightly differently depending on context, speaker, or random variation. Similarly, transitions between hidden states follow probability distributions rather than fixed rules. Language unfolds over time in patterns that are regular enough to model but variable enough to require a probabilistic treatment.

The "Markov" part of the name refers to a specific mathematical property that makes these models computationally tractable. A Markov process has a memory of exactly one step, meaning the current state depends only on the immediately previous state, not on the entire history of how the system arrived there. This might seem like an oversimplification, and in some ways it is, but it's a simplification that buys us enormous computational advantages. Without the Markov assumption, calculating probabilities for sequences would require exponentially growing amounts of computation. With it, we can use dynamic programming techniques to efficiently compute exactly the quantities we need. The Markov assumption captures the essential temporal dependencies in many systems while keeping the mathematics tractable enough for practical use.