Encoder Architecture › Encoder for Understanding Tasks
Language AI HandbookAll chapters Encoder Architecture › Encoder for Understanding TasksOne section · Sections in this chapter4 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Bidirectional attention makes encoders powerful tools for tasks that require comprehending input text. Let's examine the major categories of tasks where encoder-only models excel.
Text Classification
Classification maps an entire input sequence to a single label: sentiment (positive/negative), topic (sports/politics/technology), or intent (question/command/statement). The encoder processes the full sequence, then a classification head converts the representation to class probabilities.
BERT and similar models prepend a special [CLS] token to every input. After passing through all encoder layers, this token's representation aggregates information from the entire sequence and is the sequence-level representation for classification.
Why use [CLS] rather than averaging all token representations? The [CLS] token starts with no inherent meaning. Through self-attention, it learns to collect task-relevant information from the actual content tokens. This learned aggregation often outperforms simple averaging, especially for tasks that depend on the full sequence.
def simulate_classification_flow(hidden_states, num_classes=3):
"""
Simulate how encoder output flows to classification.
Args:
hidden_states: Encoder output of shape (seq_len, d_model)
num_classes: Number of classification categories
Returns:
Class probabilities
"""
# Use [CLS] token representation (first position)
cls_representation = hidden_states[0] # Shape: (d_model,)
# Classification head: linear projection + softmax
d_model = cls_representation.shape[0]
W_classifier = np.random.randn(d_model, num_classes) * 0.02
b_classifier = np.zeros(num_classes)
logits = cls_representation @ W_classifier + b_classifier
probabilities = softmax(logits)
return probabilities, cls_representation
# Example: classify sentiment of a review
review_tokens = ["[CLS]", "This", "movie", "was", "fantastic", "!", "[SEP]"]
d_model = 256
hidden_states = np.random.randn(len(review_tokens), d_model)
# Get classification probabilities
probs, cls_repr = simulate_classification_flow(hidden_states, num_classes=3)Sentiment Classification Example ---------------------------------------- Input: This movie was fantastic ! Class Probabilities: Negative : 0.2675 ████████ Neutral : 0.3108 █████████ Positive : 0.4217 ████████████ [CLS] representation shape: (256,) [CLS] representation norm: 16.5427
Token Classification (Sequence Labeling)
Named entity recognition, part-of-speech tagging, and similar tasks require a label for each token, not just one label for the whole sequence. Here, every token's encoder representation passes through a classification head:
def token_classification(hidden_states, num_labels=5):
"""
Apply token-level classification to encoder outputs.
Args:
hidden_states: Encoder output of shape (seq_len, d_model)
num_labels: Number of possible token labels
Returns:
Per-token predictions
"""
seq_len, d_model = hidden_states.shape
# Classification head applied to each token
W = np.random.randn(d_model, num_labels) * 0.02
b = np.zeros(num_labels)
logits = hidden_states @ W + b # (seq_len, num_labels)
predictions = np.argmax(logits, axis=1)
return predictions, softmax(logits, axis=1)
# Example: Named Entity Recognition
ner_tokens = ["John", "works", "at", "Google", "in", "California", "."]
hidden_states_ner = np.random.randn(len(ner_tokens), d_model)
predictions, probs = token_classification(hidden_states_ner, num_labels=5)Named Entity Recognition Example -------------------------------------------------- Token | Predicted Label | Confidence -------------------------------------------------- John | B-PER | 0.2584 ← works | O | 0.3012 at | O | 0.3333 Google | B-ORG | 0.2534 ← in | O | 0.2496 California | B-LOC | 0.2801 ← . | O | 0.2998
Each token gets its own contextualized representation that incorporates information from the entire sentence. When classifying "Google," the encoder representation already knows that "John works at" precedes it and "in California" follows it, helping distinguish company names from other uses of the word.
Extractive Question Answering
Given a question and a passage, extractive QA identifies the span in the passage that answers the question. The encoder processes the concatenated question and passage, then two classification heads predict the start and end positions of the answer span:
def extractive_qa(hidden_states, question_len):
"""
Predict answer span from encoder representations.
Args:
hidden_states: Encoder output of shape (total_len, d_model)
question_len: Number of tokens in the question (to exclude from answer)
Returns:
Start and end position predictions
"""
seq_len, d_model = hidden_states.shape
# Separate heads for start and end positions
W_start = np.random.randn(d_model, 1) * 0.02
W_end = np.random.randn(d_model, 1) * 0.02
# Compute logits for each position
start_logits = (hidden_states @ W_start).squeeze()
end_logits = (hidden_states @ W_end).squeeze()
# Mask out question tokens (answer must be in passage)
start_logits[:question_len] = float("-inf")
end_logits[:question_len] = float("-inf")
# Get predictions
start_pos = np.argmax(start_logits)
end_pos = np.argmax(end_logits)
return start_pos, end_pos, softmax(start_logits), softmax(end_logits)
# Example
question = ["[CLS]", "Where", "does", "John", "work", "?", "[SEP]"]
passage = [
"John",
"Smith",
"works",
"at",
"Google",
"headquarters",
".",
"[SEP]",
]
full_sequence = question + passage
question_len = len(question)
hidden_states_qa = np.random.randn(len(full_sequence), d_model)
start, end, start_probs, end_probs = extractive_qa(
hidden_states_qa, question_len
)
Bidirectional attention enables this comparison. When evaluating whether "Google" is the answer, the model simultaneously considers the question "Where does John work?" and the surrounding context "works at ... headquarters." Access to both sides of the candidate answer is only possible with bidirectional attention.