Learned Position Embeddings › GPT-Style Position Embeddings
Language AI HandbookAll chapters Learned Position Embeddings › GPT-Style Position EmbeddingsOne section · Sections in this chapter10 / 19 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
GPT-2 and GPT-3 use learned position embeddings in a straightforward way. The architecture adds token and position embeddings at the input layer, then processes the combined representation through transformer blocks. There are no segment embeddings, no special handling of sentence boundaries, and no modifications to the basic additive scheme.
The simplicity is part of the design philosophy of GPT-style models. By using a single contiguous sequence as input and learned position embeddings for position, the model places the full burden of learning useful representations on the training process. The architecture provides no inductive biases beyond "tokens at different positions have different representations." Everything else, including how position interacts with token identity, how attention uses position information, and what positional patterns are linguistically meaningful, emerges from training on large text corpora.
class GPTStyleEmbedding:
"""
GPT-style embedding layer with learned token and position embeddings.
This combines:
- Token embeddings: lookup table for vocabulary
- Position embeddings: lookup table for positions
"""
def __init__(self, vocab_size, max_seq_len, embed_dim, seed=None):
if seed is not None:
np.random.seed(seed)
# Token embedding table
self.token_embeddings = np.random.randn(vocab_size, embed_dim) * 0.02
# Position embedding table
self.position_embeddings = (
np.random.randn(max_seq_len, embed_dim) * 0.02
)
self.vocab_size = vocab_size
self.max_seq_len = max_seq_len
self.embed_dim = embed_dim
def forward(self, token_ids):
"""
Compute embeddings for a sequence of token IDs.
Args:
token_ids: Array of token indices, shape (seq_len,)
Returns:
Embeddings of shape (seq_len, embed_dim)
"""
seq_len = len(token_ids)
if seq_len > self.max_seq_len:
raise ValueError(
f"Sequence length {seq_len} exceeds maximum {self.max_seq_len}"
)
# Look up token embeddings
token_embeds = self.token_embeddings[token_ids]
# Look up position embeddings (positions 0 through seq_len-1)
positions = np.arange(seq_len)
pos_embeds = self.position_embeddings[positions]
# Add them together
return token_embeds + pos_embeds# Demonstrate GPT-style embedding
vocab_size = 50257 # GPT-2 vocabulary size
max_seq_len = 1024 # GPT-2 context length
embed_dim = 768 # GPT-2 embedding dimension
gpt_embed = GPTStyleEmbedding(vocab_size, max_seq_len, embed_dim, seed=42)
# Simulate processing a sequence
# (in practice, token_ids come from a tokenizer)
sample_token_ids = np.array(
[464, 3290, 318, 845, 1310]
) # "The cat is very small"
embeddings = gpt_embed.forward(sample_token_ids)GPT-Style Embedding Layer ================================================== Vocabulary size: 50,257 Max sequence length: 1,024 Embedding dimension: 768 Parameter counts: Token embeddings: 38,597,376 (38.6M) Position embeddings: 786,432 (0.79M) Total: 39,383,808 (39.4M) Input sequence: [ 464 3290 318 845 1310] Output shape: (5, 768)
The parameter count reveals an important insight: position embeddings are relatively cheap. In GPT-2 with its 50K vocabulary and 1024 positions, the position embedding table has only 0.79M parameters versus 38.6M for token embeddings. The position table represents about 2% of the embedding layer. This means increasing is computationally inexpensive from a parameter standpoint, though it increases the quadratic attention cost.
This asymmetry matters for understanding why modern models keep pushing context length. Adding another 1024 positions doubles the position table from 0.79M to 1.58M parameters, a trivial increase relative to the billions of parameters in the rest of the model. The real cost of longer contexts is the self-attention computation, which requires time and memory. This is why extending context length is an engineering challenge, not a parameter count challenge.