Conditional Random Fields › Neural CRF Layers
Language AI HandbookAll chapters Conditional Random Fields › Neural CRF LayersOne section · Sections in this chapter8 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Modern sequence labeling combines neural networks with CRF layers. The neural network learns feature representations, while the CRF layer models label dependencies and ensures valid output sequences.
The BiLSTM-CRF architecture uses a bidirectional LSTM to encode contextual representations of each token, then feeds these representations to a CRF layer that models transition constraints between labels. The LSTM learns features; the CRF learns structure.
In this architecture:
- Word embeddings represent each token as a dense vector
- A BiLSTM processes the sequence, producing hidden states that capture bidirectional context
- A linear layer projects hidden states to label scores (emission scores)
- The CRF layer combines emission scores with learned transition scores
The CRF layer adds a transition matrix where is the score for transitioning from label to label . During training, we maximize the log-likelihood of correct label sequences. During inference, Viterbi decoding finds the best sequence.
import torch
import torch.nn as nn
class CRFLayer(nn.Module):
"""
CRF layer for sequence labeling.
Implements the forward algorithm for training (computing log-likelihood)
and Viterbi algorithm for inference (finding best path).
"""
def __init__(self, num_tags, start_tag_idx, end_tag_idx):
super().__init__()
self.num_tags = num_tags
self.start_tag_idx = start_tag_idx
self.end_tag_idx = end_tag_idx
# Transition scores: transitions[i, j] = score of j -> i
self.transitions = nn.Parameter(torch.randn(num_tags, num_tags))
# No transitions to START, no transitions from END
self.transitions.data[start_tag_idx, :] = -10000
self.transitions.data[:, end_tag_idx] = -10000
def forward_algorithm(self, emissions):
"""
Compute log partition function using forward algorithm.
emissions: (seq_len, num_tags) emission scores
Returns: log Z(x)
"""
seq_len = emissions.size(0)
# Initialize: score of starting in each tag
# Start from START tag, so first real tag scores from START transition
init_alphas = torch.full((self.num_tags,), -10000.0)
init_alphas[self.start_tag_idx] = 0.0
forward_var = init_alphas
for emission in emissions:
alphas_t = []
for next_tag in range(self.num_tags):
emit_score = emission[next_tag].view(1).expand(self.num_tags)
trans_score = self.transitions[next_tag]
next_tag_var = forward_var + trans_score + emit_score
alphas_t.append(torch.logsumexp(next_tag_var, dim=0).view(1))
forward_var = torch.cat(alphas_t)
# Transition to END tag
terminal_var = forward_var + self.transitions[self.end_tag_idx]
log_Z = torch.logsumexp(terminal_var, dim=0)
return log_Z
def score_sequence(self, emissions, tags):
"""
Compute score of a specific tag sequence.
emissions: (seq_len, num_tags)
tags: (seq_len,) ground truth tags
Returns: score of this sequence
"""
score = torch.zeros(1)
tags = torch.cat([torch.tensor([self.start_tag_idx]), tags])
for i, emission in enumerate(emissions):
score += self.transitions[tags[i + 1], tags[i]]
score += emission[tags[i + 1]]
score += self.transitions[self.end_tag_idx, tags[-1]]
return score
def neg_log_likelihood(self, emissions, tags):
"""
Compute negative log-likelihood for training.
NLL = log Z(x) - score(y)
"""
log_Z = self.forward_algorithm(emissions)
gold_score = self.score_sequence(emissions, tags)
return log_Z - gold_score
def viterbi_decode(self, emissions):
"""
Find best tag sequence using Viterbi algorithm.
"""
seq_len = emissions.size(0)
init_vvars = torch.full((self.num_tags,), -10000.0)
init_vvars[self.start_tag_idx] = 0.0
forward_var = init_vvars
backpointers = []
for emission in emissions:
bptrs_t = []
vvars_t = []
for next_tag in range(self.num_tags):
next_tag_var = forward_var + self.transitions[next_tag]
best_tag_id = torch.argmax(next_tag_var)
bptrs_t.append(best_tag_id.item())
vvars_t.append(next_tag_var[best_tag_id].view(1))
forward_var = torch.cat(vvars_t) + emission
backpointers.append(bptrs_t)
# Transition to END
terminal_var = forward_var + self.transitions[self.end_tag_idx]
best_tag_id = torch.argmax(terminal_var).item()
best_score = terminal_var[best_tag_id]
# Backtrack
best_path = [best_tag_id]
for bptrs_t in reversed(backpointers):
best_tag_id = bptrs_t[best_tag_id]
best_path.append(best_tag_id)
# Remove START tag
best_path.pop()
best_path.reverse()
return best_score, best_path
# Demonstrate CRF layer
num_tags = 5 # O, B-PER, I-PER, B-LOC, I-LOC
crf_layer = CRFLayer(num_tags, start_tag_idx=0, end_tag_idx=0)
# Fake emissions from a neural network
emissions = torch.randn(6, num_tags) # 6 tokens, 5 tags
fake_tags = torch.tensor([1, 2, 0, 3, 4, 0]) # Example tag sequenceCRF Layer Demonstration: ================================================== Log partition function: -19987.2617 Score of tag sequence: -49996.3750 Negative log-likelihood: 30009.1133 Viterbi decoding: Best path: [3, 3, 4, 3, 3, 3] Best score: -19989.8594
The CRF layer computes three key quantities. The log partition function sums over all possible tag sequences (exponentially many), computed efficiently via the forward algorithm. The sequence score measures how well a specific tag sequence fits the emission and transition scores. The negative log-likelihood (the training loss) is the difference between these: minimizing it encourages the model to assign high scores to correct sequences relative to all alternatives. Viterbi decoding finds the highest-scoring path for inference.
The neural CRF combines the representation learning power of neural networks with the structured prediction capabilities of CRFs. The neural network extracts features automatically from the input, while the CRF layer ensures that the output sequence satisfies structural constraints like valid BIO transitions.
