Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Conditional Random Fields › From Generative to Discriminative Models

Michael BrenndoerferUpdated May 6, 20257 min read

Language AI HandbookAll chapters

Part 6 of 67Sequence Labeling

Chapter 42Conditional Random Fields › From Generative to Discriminative Models

Reading section 2 of 13 · From Generative to Discriminative Models · 7 minSection 2 of 13 · 7 min

Section 2 of 13 · 7 min read

Conditional Random Fields › From Generative to Discriminative ModelsOne section · Sections in this chapter2 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Before diving into CRFs, let's understand the fundamental distinction between generative and discriminative approaches to classification. This distinction explains why CRFs exist and what problems they solve.

Generative vs. Discriminative Models

Generative models learn the joint probability $P(X, Y)$ of inputs and outputs, then use Bayes' rule to compute $P(Y|X)$ for prediction. Discriminative models directly learn the conditional probability $P(Y|X)$ without modeling the input distribution.

The HMM ApproachLink Copied

Hidden Markov Models are generative. They model how sequences are produced: first, a hidden state is generated according to transition probabilities, then an observation is emitted according to emission probabilities. For a sequence of observations $\mathbf{x} = (x_1, x_2, \ldots, x_n)$ and labels $\mathbf{y} = (y_1, y_2, \ldots, y_n)$ , an HMM models the joint probability of observations and labels:

P(\mathbf{x}, \mathbf{y}) = P(y_1) \prod_{t=2}^{n} P(y_t | y_{t-1}) \prod_{t=1}^{n} P(x_t | y_t)

where:

$\mathbf{x} = (x_1, x_2, \ldots, x_n)$ : the sequence of $n$ observations (e.g., words in a sentence)
$\mathbf{y} = (y_1, y_2, \ldots, y_n)$ : the sequence of $n$ labels (e.g., POS tags or entity types)
$P(y_1)$ : the initial state probability (probability of starting with label $y_1$ )
$P(y_t | y_{t-1})$ : the transition probability from label $y_{t-1}$ to label $y_t$
$P(x_t | y_t)$ : the emission probability of observing $x_t$ given label $y_t$

The first product captures transition probabilities (how states follow each other), and the second captures emission probabilities (how each state generates its observation). To predict labels for new observations, we apply Bayes' rule:

P(\mathbf{y} | \mathbf{x}) = \frac{P(\mathbf{x}, \mathbf{y})}{P(\mathbf{x})}

where:

$P(\mathbf{y} | \mathbf{x})$ : the posterior probability of label sequence $\mathbf{y}$ given observations $\mathbf{x}$
$P(\mathbf{x}, \mathbf{y})$ : the joint probability computed by the HMM formula above
$P(\mathbf{x})$ : the marginal probability of the observations, obtained by summing $P(\mathbf{x}, \mathbf{y})$ over all possible label sequences

This generative formulation imposes a critical constraint: the observation $x_t$ can only depend on its own label $y_t$ , not on other observations or labels. This independence assumption enables tractable inference but limits expressiveness.

The Independence ProblemLink Copied

Consider tagging the word "bank" in context. A good tagger should consider:

The word itself: "bank"
The previous word: "river" suggests location, "investment" suggests organization
The next word: "account" suggests financial meaning
Capitalization: "Bank" at sentence start vs. "bank" mid-sentence
Word suffixes: "-ing," "-tion," "-ly" carry grammatical information
Surrounding punctuation: quotation marks might indicate names

These features overlap and interact. The suffix "-ing" combined with a preceding auxiliary verb strongly suggests a verb tag. Capitalization plus a preceding determiner suggests a proper noun. HMMs cannot naturally express such feature combinations because each observation must be generated independently from its label alone.

In[3]:

Code

# Features that help disambiguate "bank"
def extract_context_features(words, position):
    """Extract overlapping contextual features for a word."""
    word = words[position]
    features = {
        "word": word.lower(),
        "is_capitalized": word[0].isupper(),
        "suffix_3": word[-3:] if len(word) >= 3 else word,
        "prefix_3": word[:3] if len(word) >= 3 else word,
    }

    if position > 0:
        features["prev_word"] = words[position - 1].lower()
        features["prev_capitalized"] = words[position - 1][0].isupper()
    else:
        features["prev_word"] = "<START>"
        features["prev_capitalized"] = False

    if position < len(words) - 1:
        features["next_word"] = words[position + 1].lower()
        features["next_capitalized"] = words[position + 1][0].isupper()
    else:
        features["next_word"] = "<END>"
        features["next_capitalized"] = False

    return features


# Two sentences with "bank"
sentence1 = ["The", "river", "bank", "was", "muddy"]
sentence2 = ["The", "investment", "bank", "collapsed"]

features1 = extract_context_features(sentence1, 2)
features2 = extract_context_features(sentence2, 2)

Out[4]:

Console

Context Features for 'bank' in Different Contexts:
=======================================================

Sentence 1: 'The river bank was muddy'
-------------------------------------------------------
  word                : bank
  is_capitalized      : False
  suffix_3            : ank
  prefix_3            : ban
  prev_word           : river
  prev_capitalized    : False
  next_word           : was
  next_capitalized    : False

Sentence 2: 'The investment bank collapsed'
-------------------------------------------------------
  word                : bank
  is_capitalized      : False
  suffix_3            : ank
  prefix_3            : ban
  prev_word           : investment
  prev_capitalized    : False
  next_word           : collapsed
  next_capitalized    : False

→ The previous word changes dramatically, but HMMs cannot
  directly use 'prev_word' as a feature for predicting the tag.

In an HMM, we could only condition on the previous tag, not the previous word. The rich contextual features visible in the data cannot be exploited directly.

The Discriminative SolutionLink Copied

CRFs take a fundamentally different approach. Instead of modeling how observations are generated, they directly model the probability of a label sequence given the observations:

P(\mathbf{y} | \mathbf{x})

This conditional formulation has one advantage: since we condition on $\mathbf{x}$ , we don't need to model the distribution of observations. Features of $\mathbf{x}$ can overlap arbitrarily, interact in complex ways, and depend on any part of the input sequence. The model only needs to learn which label sequences are likely given the observed features.

Out[5]:

Visualization

Diagram showing HMM generating observations from hidden states. — HMM (Generative): Models the joint probability P(X,Y) with arrows from hidden states to observations, matching the generative process.

Diagram showing CRF conditioning labels on observations. — CRF (Discriminative): Directly models P(Y|X) with arrows from observations to labels, allowing arbitrary features across positions.

The figure illustrates the fundamental difference. In HMMs, arrows point from hidden states to observations. This reflects the generative process. In CRFs, observations condition the labels, and features can connect any observation to any label. This reversal of direction is not merely notational; it fundamentally changes what the model can represent.