Sinusoidal Position Encoding › Complete Implementation
Language AI HandbookAll chapters Sinusoidal Position Encoding › Complete ImplementationOne section · Sections in this chapter9 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Here's a complete, production-ready implementation of sinusoidal position encoding that handles batched inputs:
class SinusoidalPositionEncoding:
"""
Sinusoidal position encoding as introduced in 'Attention Is All You Need'.
Generates deterministic position encodings using sine and cosine functions
at geometrically increasing wavelengths.
"""
def __init__(self, d_model, max_len=5000):
"""
Initialize the position encoding.
Args:
d_model: Dimension of the model (embedding size)
max_len: Maximum sequence length to pre-compute
"""
self.d_model = d_model
self.max_len = max_len
# Pre-compute position encodings
self.encoding = self._create_encoding(max_len, d_model)
def _create_encoding(self, max_len, d_model):
"""Generate the position encoding matrix."""
# Position indices: (max_len, 1)
position = np.arange(max_len)[:, np.newaxis]
# Dimension indices for pairs: (d_model/2,)
div_term = 10000 ** (np.arange(0, d_model, 2) / d_model)
# Compute encodings
encoding = np.zeros((max_len, d_model))
encoding[:, 0::2] = np.sin(position / div_term)
encoding[:, 1::2] = np.cos(position / div_term)
return encoding
def __call__(self, seq_len):
"""
Get position encodings for a sequence.
Args:
seq_len: Length of the sequence
Returns:
Position encodings of shape (seq_len, d_model)
"""
if seq_len > self.max_len:
# Extend encoding if needed
self.encoding = self._create_encoding(seq_len, self.d_model)
self.max_len = seq_len
return self.encoding[:seq_len]
def add_to_embeddings(self, embeddings):
"""
Add position encodings to token embeddings.
Args:
embeddings: Token embeddings of shape (seq_len, d_model)
or (batch_size, seq_len, d_model)
Returns:
Position-enhanced embeddings of the same shape
"""
if embeddings.ndim == 2:
seq_len = embeddings.shape[0]
return embeddings + self(seq_len)
elif embeddings.ndim == 3:
seq_len = embeddings.shape[1]
return embeddings + self(seq_len)[np.newaxis, :, :]
else:
raise ValueError(f"Expected 2D or 3D input, got {embeddings.ndim}D")Let's test the implementation:
# Create position encoder
pos_encoder = SinusoidalPositionEncoding(d_model=64, max_len=1000)
# Simulate token embeddings (batch of 2 sequences, length 10)
np.random.seed(42)
batch_embeddings = np.random.randn(2, 10, 64) * 0.1
# Add position information
positioned_embeddings = pos_encoder.add_to_embeddings(batch_embeddings)Position Encoding Integration Test ================================================== Input embeddings shape: (2, 10, 64) Output embeddings shape: (2, 10, 64) Position encoding magnitude (L2 norm): Position 0: 5.6569 Position 5: 5.6569 Position 9: 5.6569 Token embedding magnitude (sample): Before: 0.7284 After: 5.5914
The position encodings have similar magnitude to typical token embeddings (around 5-6 for 64 dimensions), which ensures that position information is meaningful but doesn't overwhelm the semantic content.
In practice, production transformer implementations often scale the token embeddings by before adding the position encoding. The intuition is that the learned token embeddings tend to have small norms when initialized, and scaling brings them to a magnitude roughly comparable to the position encoding. Without this scaling, the position signal might dominate in the early stages of training before the embeddings have grown to their natural size. The original paper applies this scaling explicitly: the embedding layer output is multiplied by , and then the position encoding is added.