Sinusoidal Position Encoding › In Practice: Integrating Sinusoidal Encoding
Language AI HandbookAll chapters Sinusoidal Position Encoding › In Practice: Integrating Sinusoidal EncodingOne section · Sections in this chapter11 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
When you use sinusoidal encoding in a real transformer, several practical considerations come up that are not obvious from the formula alone.
The first consideration is normalization. The position encoding values lie in for each dimension, and with dimensions, the L2 norm of a position encoding vector is approximately . If your token embeddings have a different typical norm, the relative strength of the position signal will change. Most implementations scale token embeddings by before adding the position encoding, as discussed earlier. Pay attention to this scaling factor in any implementation you borrow or build on.
The second consideration is where to apply the encoding. The standard approach adds the position encoding to the token embedding before the first transformer layer. This means the position information is available to the first self-attention layer and propagates through all subsequent layers. Some architectures experiment with adding fresh position encodings at each layer, but this is non-standard and introduces additional complexity.
The third consideration is dropout. Many transformer implementations apply dropout to the sum of the token embedding and position encoding before passing it to the attention layers. This is separate from dropout applied within the attention mechanism itself, and its purpose is to regularize the input representation. The original paper uses dropout with probability 0.1 in the base model, applied after the embedding plus encoding sum.
The fourth consideration is the interaction with layer normalization. If you apply layer normalization before attention (pre-norm, as in many modern architectures), the position information is normalized along with the semantic information. Post-norm architectures, like the original transformer, normalize after the residual connection, which means the position information persists more directly in the residual stream. Both approaches work, but the effect of position encoding relative to semantic content can differ between them.
def build_transformer_input(
tokens_embeddings, d_model, dropout_rate=0.1, seed=0
):
"""
Build the full input to a transformer layer.
Combines token embeddings with sinusoidal position encoding,
applies scaling and dropout as in the original transformer.
Args:
tokens_embeddings: Array of shape (seq_len, d_model)
d_model: Model dimension
dropout_rate: Dropout probability (applied in training)
seed: Random seed for dropout
Returns:
transformer_input: Array of shape (seq_len, d_model)
"""
seq_len = tokens_embeddings.shape[0]
# Step 1: Scale token embeddings by sqrt(d_model)
scaled_embeddings = tokens_embeddings * np.sqrt(d_model)
# Step 2: Generate position encodings
pos_enc = SinusoidalPositionEncoding(d_model=d_model, max_len=seq_len + 1)
position_encodings = pos_enc(seq_len)
# Step 3: Add position information
combined = scaled_embeddings + position_encodings
# Step 4: Apply dropout (simulated: zero out some values)
rng = np.random.default_rng(seed)
mask = rng.random(combined.shape) > dropout_rate
combined_dropped = combined * mask / (1.0 - dropout_rate)
return combined_dropped
# Example usage
np.random.seed(99)
seq_len_demo = 8
d_model_demo = 16
token_embs = np.random.randn(seq_len_demo, d_model_demo) * 0.1Transformer input construction: Token embeddings shape: (8, 16) Transformer input shape: (8, 16) Token emb norm (pos 0): 0.3980 Scaled emb norm (pos 0): 1.5920 After adding PE (pos 0): 3.4235
The scaling step is important: without it, the position encoding signal would be much stronger than the token embedding signal in the early stages of training, potentially slowing convergence. By scaling token embeddings up by , you ensure both signals start at comparable magnitudes.