Decoder Architecture › GPT-Style Decoder Architecture
Language AI HandbookAll chapters Decoder Architecture › GPT-Style Decoder ArchitectureOne section · Sections in this chapter6 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
GPT (Generative Pre-trained Transformer) established the dominant decoder-only paradigm. The full GPT architecture consists of:
- Token Embedding: Maps input token IDs to dense vectors
- Position Embedding: Adds positional information (learned or sinusoidal)
- Decoder Stack: Multiple decoder blocks in sequence
- Final Layer Norm: Normalizes the output of the last block
- Language Model Head: Projects to vocabulary size for next-token prediction
Let's build a complete GPT-style decoder.
class GPTDecoder:
"""
Complete GPT-style decoder-only transformer.
Takes token indices as input and produces logits over the vocabulary.
"""
def __init__(
self, vocab_size, max_seq_len, d_model, n_heads, d_ff, n_layers
):
self.vocab_size = vocab_size
self.max_seq_len = max_seq_len
self.d_model = d_model
self.n_layers = n_layers
# Token and position embeddings
self.token_embedding = np.random.randn(vocab_size, d_model) * 0.02
self.position_embedding = np.random.randn(max_seq_len, d_model) * 0.02
# Stack of decoder blocks
self.blocks = [
DecoderBlock(d_model, n_heads, d_ff) for _ in range(n_layers)
]
# Final layer norm
self.final_norm_gamma = np.ones(d_model)
# Language model head (projects to vocabulary)
self.lm_head = np.random.randn(d_model, vocab_size) * 0.02
def __call__(self, token_ids):
"""
Forward pass through the decoder.
Args:
token_ids: Array of token indices, shape (seq_len,)
Returns:
logits: Unnormalized scores over vocabulary, shape (seq_len, vocab_size)
"""
seq_len = len(token_ids)
# Get embeddings
tok_emb = self.token_embedding[token_ids] # (seq_len, d_model)
pos_emb = self.position_embedding[:seq_len] # (seq_len, d_model)
# Combine embeddings
x = tok_emb + pos_emb
# Pass through decoder blocks
for block in self.blocks:
x = block(x)
# Final normalization
x = rms_norm(x, self.final_norm_gamma)
# Project to vocabulary
logits = x @ self.lm_head # (seq_len, vocab_size)
return logits
def num_parameters(self):
"""Count total parameters."""
embed_params = self.vocab_size * self.d_model # Token embedding
embed_params += self.max_seq_len * self.d_model # Position embedding
block_params = sum(b.num_parameters() for b in self.blocks)
norm_params = self.d_model # Final norm
head_params = self.d_model * self.vocab_size # LM head
return embed_params + block_params + norm_params + head_paramsGPT Decoder Configuration ============================================= Vocabulary size: 10,000 Max sequence length: 512 Model dimension: 256 Number of heads: 8 FFN dimension: 1024 Number of layers: 6 Total parameters: 9,980,672
The parameter count breaks down into three main components: embeddings (token + position), the decoder stack, and the language model head. With a 10K vocabulary and 256-dimensional embeddings, the token embedding alone accounts for 2.56M parameters. The 6 decoder blocks contribute about 4.7M parameters (roughly 790K each), and the LM head adds another 2.56M for projecting back to vocabulary size.
