Conditional Random Fields (2001) › The Conditional Probability Framework

The mathematical foundation of CRFs rests on a principle called conditional modeling. Rather than trying to model the joint probability of both inputs and outputs together, which would require learning about the distribution of input sentences themselves, CRFs focus directly on what matters for prediction. They model the probability of a particular label sequence given an observed input sequence. This conditional approach makes the learning problem more tractable and focuses the model's capacity on the prediction task itself.

At the heart of a CRF lies a deceptively elegant formula. For an input sequence represented as $x$ and a candidate label sequence $y$, the model assigns a probability according to this expression:

This equation encodes several key ideas. The summation in the numerator represents a score for how well a particular label sequence fits the input. This score is computed by evaluating a collection of feature functions, each denoted $f_k$, that examine different aspects of the input-output pair. Each feature function has an associated weight $\lambda_k$ that the model learns during training, determining how much that particular feature should influence the final prediction.

The exponential function transforms these scores into unnormalized probabilities, ensuring that higher-scoring sequences receive higher probability. The denominator, denoted $Z(x)$ and called the partition function, serves as a normalizing constant. It sums over all possible label sequences to ensure that the probabilities across all potential labelings add up to one, as any valid probability distribution must.

What makes this framework powerful is its flexibility. The feature functions can capture any aspect of the relationship between inputs and outputs that a researcher can imagine and encode. This generality allows CRFs to incorporate domain knowledge and linguistic intuitions directly into the model structure.