Encoder Architecture › Encoder Output Usage
Language AI HandbookAll chapters Encoder Architecture › Encoder Output UsageOne section · Sections in this chapter7 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The encoder produces one contextualized representation per input token. How you use these representations depends on your downstream task.
Sequence-Level Tasks
For classification, sentiment analysis, or any task requiring a single output for the whole sequence, use the [CLS] token representation:
def get_sequence_representation(encoder_output, pooling="cls"):
"""
Extract sequence-level representation from encoder output.
Args:
encoder_output: Shape (seq_len, d_model)
pooling: "cls" (use first token), "mean" (average all),
or "max" (max pooling)
Returns:
Single vector of shape (d_model,)
"""
if pooling == "cls":
return encoder_output[0]
elif pooling == "mean":
return np.mean(encoder_output, axis=0)
elif pooling == "max":
return np.max(encoder_output, axis=0)
else:
raise ValueError(f"Unknown pooling: {pooling}")
# Compare pooling strategies
cls_repr = get_sequence_representation(output, "cls")
mean_repr = get_sequence_representation(output, "mean")
max_repr = get_sequence_representation(output, "max")
Mean pooling often works well when all tokens contribute equally (like sentence similarity). CLS pooling is preferred when the model was pre-trained to aggregate into the first position. Max pooling can capture salient features but may be sensitive to outliers.
Token-Level Tasks
For NER, POS tagging, or other sequence labeling tasks, use every token's representation:
def get_token_representations(encoder_output, special_tokens_mask=None):
"""
Extract per-token representations for sequence labeling.
Args:
encoder_output: Shape (seq_len, d_model)
special_tokens_mask: Boolean mask (True for special tokens to exclude)
Returns:
Token representations (possibly with special tokens removed)
"""
if special_tokens_mask is not None:
# Keep only non-special tokens
return encoder_output[~special_tokens_mask]
return encoder_output
# Example: exclude [CLS] and [SEP]
seq_len = output.shape[0]
special_mask = np.zeros(seq_len, dtype=bool)
special_mask[0] = True # [CLS]
special_mask[-1] = True # [SEP]
content_representations = get_token_representations(output, special_mask)Token-Level Representations ---------------------------------------- Total tokens: 8 Special tokens: 2 Content tokens: 6 Full encoder output shape: (8, 256) Content-only shape: (6, 256)
Span-Level Tasks
For question answering, relation extraction, or any task involving text spans, you might need to construct span representations from multiple tokens:
def get_span_representation(encoder_output, start, end, method="endpoints"):
"""
Extract representation for a span of tokens.
Args:
encoder_output: Shape (seq_len, d_model)
start: Start position (inclusive)
end: End position (inclusive)
method: "endpoints" (concat start and end), "mean" (average span),
"attention" (weighted by position)
Returns:
Span representation
"""
span_tokens = encoder_output[start : end + 1]
if method == "endpoints":
return np.concatenate([span_tokens[0], span_tokens[-1]])
elif method == "mean":
return np.mean(span_tokens, axis=0)
elif method == "maxpool":
return np.max(span_tokens, axis=0)
else:
raise ValueError(f"Unknown method: {method}")
# Example: represent the span "at Google" (positions 3-4 in "John works at Google")
span_repr_endpoints = get_span_representation(output, 3, 4, "endpoints")
span_repr_mean = get_span_representation(output, 3, 4, "mean")Span Representation Methods -------------------------------------------------- Span positions: 3 to 4 (2 tokens) Endpoints (concat): shape (512,) Mean pooling: shape (256,) Endpoint method doubles dimension but preserves boundary information. Mean method preserves dimension but loses position within span.