Encoder-Decoder Architecture › Implementing Encoder-Decoder Generation
Language AI HandbookAll chapters Encoder-Decoder Architecture › Implementing Encoder-Decoder GenerationOne section · Sections in this chapter9 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
At inference time, the encoder processes the source sequence once. The decoder then generates the target sequence autoregressively, querying the cached encoder output at each step.
def generate_with_encoder_decoder(
model, src_tokens, max_len=50, sos_token=0, eos_token=1, temperature=1.0
):
"""
Generate target sequence given source sequence.
Args:
model: Encoder-decoder transformer
src_tokens: Source token indices
max_len: Maximum generation length
sos_token: Start-of-sequence token index
eos_token: End-of-sequence token index
temperature: Sampling temperature
Returns:
Generated token indices
"""
# Encode source (done once)
encoder_output = model.encode(src_tokens)
# Initialize target with SOS token
generated = [sos_token]
for _ in range(max_len):
# Decode current sequence
tgt_tokens = np.array(generated)
logits, _ = model.decode(tgt_tokens, encoder_output)
# Get logits for next token (last position)
next_logits = logits[-1] / temperature
probs = softmax(next_logits)
# Sample next token
next_token = np.random.choice(len(probs), p=probs)
generated.append(next_token)
# Stop if EOS generated
if next_token == eos_token:
break
return np.array(generated)
# Generate from our model
generated = generate_with_encoder_decoder(
model, src_tokens, max_len=10, temperature=0.8
)Encoder-Decoder Generation --------------------------------------------- Source tokens: [42, 100, 256, 789, 1000, 500] Generated tokens: [0, 9912, 2333, 2296, 7848, 7904, 6043, 4244, 5583, 1012, 11712] Source length: 6 Generated length: 11 Key observation: The encoder runs once, then the decoder generates autoregressively while attending to the cached encoder output at each step.
The generation demonstrates the encoder-decoder inference pattern: the encoder runs once to produce cached representations, then the decoder generates tokens autoregressively while querying that cache through cross-attention. The generated sequence (10 tokens) differs in length from the source (6 tokens), showing how the architecture naturally handles variable-length outputs. This "encode once, decode many times" pattern is particularly efficient for tasks like summarization, where a long input document is encoded once and then referenced repeatedly during output generation.
The generation process highlights the efficiency advantage of encoder-decoder models for tasks with long inputs. The encoder processes a 1000-word document once. The decoder then generates a 50-word summary, accessing the encoded document through cross-attention at each of the 50 generation steps. Compare this to a decoder-only approach that would need to include the full document in context at every step.