T5 Architecture › Working with T5
Language AI HandbookAll chapters T5 Architecture › Working with T5One section · Sections in this chapter8 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's use T5 for various tasks to see the text-to-text framework in action. We'll use T5-small for these examples to keep computational requirements modest.
Translation
import torch
# Translation example
input_text = "translate English to German: The weather is beautiful today."
# Tokenize
inputs = tokenizer(input_text, return_tensors="pt", padding=True)
# Generate
with torch.no_grad():
outputs = model.generate(
inputs.input_ids, max_length=50, num_beams=4, early_stopping=True
)
# Decode
translation = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f"Input: {input_text}")
print(f"Output: {translation}")Input: translate English to German: The weather is beautiful today. Output: Das Wetter ist heute schön.
Summarization
# Summarization example
article = """
The Amazon rainforest produces about 20% of the world's oxygen.
It spans across nine countries in South America and contains
10% of all species on Earth. Deforestation threatens this vital
ecosystem, with an area the size of a football field being cleared
every minute. Conservation efforts are necessary to preserve
biodiversity and combat climate change.
"""
input_text = f"summarize: {article}"
inputs = tokenizer(
input_text, return_tensors="pt", padding=True, truncation=True
)
with torch.no_grad():
outputs = model.generate(
inputs.input_ids, max_length=50, num_beams=4, early_stopping=True
)
summary = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f"Summary: {summary}")Summary: the amazon rainforest produces 20% of the world's oxygen. it spans across nine countries in south America and contains 10% of all species on earth.
Examining Internal Representations
Let's look at how T5 processes input through its encoder:
# Get encoder hidden states
text = "The model learns to understand language through pretraining."
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
encoder_outputs = model.encoder(
input_ids=inputs.input_ids, output_hidden_states=True
)
# Shape of hidden states from each layer
print("Encoder hidden state shapes:")
for i, hidden in enumerate(encoder_outputs.hidden_states):
print(f" Layer {i}: {hidden.shape}")Encoder hidden state shapes: Layer 0: torch.Size([1, 12, 512]) Layer 1: torch.Size([1, 12, 512]) Layer 2: torch.Size([1, 12, 512]) Layer 3: torch.Size([1, 12, 512]) Layer 4: torch.Size([1, 12, 512]) Layer 5: torch.Size([1, 12, 512]) Layer 6: torch.Size([1, 12, 512])
Each layer produces a tensor of shape (batch_size, sequence_length, hidden_dim). The sequence has 11 tokens, and each token gets a 512-dimensional representation. Layer 0 is the embedding layer, and layers 1-6 are the transformer blocks.
The progression across layers shows how the model builds meaning. Early layers capture surface-level features: token identity, simple co-occurrence patterns, and basic syntax. Middle layers develop richer syntactic structure, resolving phrase boundaries and grammatical relationships. Late layers encode semantic content most strongly, bringing together context from across the sequence into representations that encode meaning rather than form. This layer-by-layer refinement is not unique to T5 but is a general property of deep transformer stacks; it makes representations from different layers useful for different downstream tasks.
Attention Pattern Visualization
Let's visualize how attention patterns differ between encoder and decoder:
The encoder attention shows bidirectional patterns where tokens can attend freely to any position. The decoder attention shows the characteristic lower-triangular pattern from causal masking. Each token can only attend to itself and previous tokens, preventing information leakage from future positions during generation.
