Encoder-Decoder Architecture › Building the Complete Encoder-Decoder Transformer
Language AI HandbookAll chapters Encoder-Decoder Architecture › Building the Complete Encoder-Decoder TransformerOne section · Sections in this chapter5 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
With the individual components understood (cross-attention for source access, masked self-attention for autoregressive generation, and feed-forward networks for transformation), we can now assemble the complete architecture.
The full encoder-decoder transformer stacks multiple blocks on each side. The encoder builds increasingly abstract representations of the source through successive bidirectional self-attention layers. The decoder mirrors this depth, with each layer first refining its generation context (self-attention), then consulting the encoder's final output (cross-attention), and finally applying nonlinear transformation (FFN).
This design saves computation: the encoder processes the source sequence exactly once. Its output is then reused by every decoder layer at every generation step. When generating a 50-word summary, the decoder queries the same encoder representations 50 times, once per generated token, without recomputing them.
class EncoderBlock:
"""Standard encoder block with bidirectional self-attention."""
def __init__(self, d_model, n_heads, d_ff):
self.d_model = d_model
self.self_attn = MultiHeadAttention(d_model, n_heads)
self.ffn = FeedForward(d_model, d_ff)
self.norm1_gamma = np.ones(d_model)
self.norm2_gamma = np.ones(d_model)
def __call__(self, x, mask=None):
# Self-attention (bidirectional, no causal mask)
normed = rms_norm(x, self.norm1_gamma)
attn_out, _ = self.self_attn(normed, mask=mask)
x = x + attn_out
# Feed-forward
normed = rms_norm(x, self.norm2_gamma)
ffn_out = self.ffn(normed)
x = x + ffn_out
return x
class TransformerEncoderDecoder:
"""
Complete encoder-decoder transformer.
Follows the original architecture from "Attention Is All You Need."
"""
def __init__(
self,
src_vocab_size,
tgt_vocab_size,
d_model,
n_heads,
d_ff,
n_encoder_layers,
n_decoder_layers,
max_seq_len,
):
self.d_model = d_model
self.n_encoder_layers = n_encoder_layers
self.n_decoder_layers = n_decoder_layers
# Embeddings
self.src_embedding = np.random.randn(src_vocab_size, d_model) * 0.02
self.tgt_embedding = np.random.randn(tgt_vocab_size, d_model) * 0.02
self.position_embedding = np.random.randn(max_seq_len, d_model) * 0.02
# Encoder stack
self.encoder_layers = [
EncoderBlock(d_model, n_heads, d_ff)
for _ in range(n_encoder_layers)
]
self.encoder_norm_gamma = np.ones(d_model)
# Decoder stack
self.decoder_layers = [
EncoderDecoderBlock(d_model, n_heads, d_ff)
for _ in range(n_decoder_layers)
]
self.decoder_norm_gamma = np.ones(d_model)
# Output projection (to target vocabulary)
self.output_projection = np.random.randn(d_model, tgt_vocab_size) * 0.02
def encode(self, src_tokens):
"""
Encode source sequence.
Args:
src_tokens: Source token indices of shape (src_len,)
Returns:
Encoder output of shape (src_len, d_model)
"""
src_len = len(src_tokens)
# Embed tokens and add positions
x = self.src_embedding[src_tokens]
x = x + self.position_embedding[:src_len]
# Pass through encoder layers
for layer in self.encoder_layers:
x = layer(x)
# Final normalization
x = rms_norm(x, self.encoder_norm_gamma)
return x
def decode(self, tgt_tokens, encoder_output):
"""
Decode target sequence given encoder output.
Args:
tgt_tokens: Target token indices of shape (tgt_len,)
encoder_output: Encoder output of shape (src_len, d_model)
Returns:
Logits of shape (tgt_len, tgt_vocab_size)
"""
tgt_len = len(tgt_tokens)
# Embed tokens and add positions
x = self.tgt_embedding[tgt_tokens]
x = x + self.position_embedding[:tgt_len]
# Causal mask for self-attention
causal_mask = create_causal_mask(tgt_len)
# Pass through decoder layers
cross_attention_weights = []
for layer in self.decoder_layers:
x, cross_weights = layer(x, encoder_output, causal_mask)
cross_attention_weights.append(cross_weights)
# Final normalization
x = rms_norm(x, self.decoder_norm_gamma)
# Project to vocabulary
logits = x @ self.output_projection
return logits, cross_attention_weights
def forward(self, src_tokens, tgt_tokens):
"""Complete forward pass."""
encoder_output = self.encode(src_tokens)
logits, cross_weights = self.decode(tgt_tokens, encoder_output)
return logits, cross_weightsEncoder-Decoder Transformer Test ================================================== Source sequence length: 6 Target sequence length: 4 Encoder layers: 4 Decoder layers: 4 Model dimension: 256 Logits shape: (4, 12000) (tgt_len × tgt_vocab_size) Cross-attention weights per layer: (8, 4, 6) (n_heads × tgt_len × src_len)
The complete encoder-decoder transformer successfully processes source and target sequences. The encoder compresses the 6-token source into 256-dimensional representations, which the decoder queries through cross-attention while generating 4 target tokens. Each decoder layer produces cross-attention weights of shape (8, 4, 6), representing how 8 heads × 4 target positions attend to 6 source positions. The final logits of shape (4, 12000) give probability distributions over the target vocabulary for each generated position.