Sinusoidal Position Encoding

The transformer's self-attention mechanism is permutation invariant: it produces the same output regardless of token order. Feed a transformer the sentence "The dog bit the man" and "The man bit the dog" and, without any additional signal, it will treat them identically. This is a fundamental problem, because word order is one of the primary carriers of meaning in language. A mechanism that cannot distinguish "not guilty" from "guilty" based on word order is useless for any serious language task.

To fix this, the original "Attention Is All You Need" paper introduced sinusoidal position encoding, an elegant solution that encodes each position as a unique pattern of sine and cosine waves at different frequencies. Rather than learning position representations from data, the encoding is computed from a fixed mathematical formula, requiring zero extra parameters. Every position in a sequence receives a distinct vector that the model can use to reason about order and distance.

This approach is one of the most instructive designs in the entire transformer architecture, because it solves a concrete engineering problem, uniquely identifying positions in a way that supports both absolute and relative reasoning, using pure mathematics rather than learned tables. Understanding sinusoidal encoding deeply will also give you the conceptual foundation for understanding modern alternatives like RoPE and ALiBi, which build directly on the same ideas.

Before we examine the formula, consider why the problem is harder than it looks. You might wonder: why append the position index to the token embedding? Why multiply by the position or add a one-hot vector? Each idea fails at least one property we need. Appending the position number means that position 1000 carries a value 1000 times larger than position 1, which would dominate the semantic embedding entirely. A one-hot vector of length equal to the maximum sequence length would require enormous memory and could not generalize to lengths not seen during training. The sinusoidal approach avoids these failure modes, and the rest of this chapter explains how.

Before diving into the mathematics, let's consider what properties we want from a position encoding. We need a function that takes a position index and produces a vector, and this function must satisfy several constraints that, taken together, are quite demanding.

The first constraint is uniqueness. Each position must map to a distinct vector. If positions 5 and 17 produce the same encoding, the model cannot distinguish them, and all the information about order is lost at those positions. This sounds simple but rules out many naive approaches: a constant function is not unique, and a linear function is unique but fails the next constraint.

The second constraint is bounded values. The encoding should not grow unboundedly with position. If position 1000 produces values 1000 times larger than position 1, the position signal would overwhelm the semantic content of the embeddings. The model would spend most of its capacity reasoning about raw position magnitude rather than meaning. We need the encoding values to stay within a fixed range regardless of how long the sequence is.

The third constraint is smooth progression. Nearby positions should have similar encodings. Position 50 should be more similar to position 51 than to position 500, giving the model useful gradient information during training and letting it generalize position-aware patterns across neighboring positions. An encoding that jumps chaotically between positions would be hard to learn from.

The fourth constraint is determinism. The same position should always produce the same encoding, without requiring any learned parameters. This is not strictly necessary for correctness (learned embeddings also satisfy the first three constraints), but it means the encoding requires zero parameters and can represent positions never seen during training.

Sinusoidal functions satisfy all these requirements elegantly. Sine and cosine oscillate smoothly between -1 and 1. This ensures bounded values. Different frequencies distinguish positions at different scales. And the encoding is purely deterministic, computed from a fixed formula. The use of multiple frequencies simultaneously is what makes the encoding both unique and smooth at the same time.

The core idea is to assign each position a unique "fingerprint" using waves of different frequencies. Think of how you might describe your location in a building: you could give the floor number (coarse scale), the room number (medium scale), and your position within the room (fine scale). Together, these scales uniquely identify any location. No single scale alone would suffice: the floor number alone could apply to hundreds of rooms, and the position-within-room alone could apply to rooms on every floor.

For position encoding, we use sine and cosine waves at different frequencies to achieve the same multi-scale identification. Each position $pos$ in the sequence receives a $d$-dimensional vector, where consecutive pairs of dimensions use sine and cosine at the same frequency:

where:

• $pos$: the position index in the sequence (0, 1, 2, ...)
• $i$: the dimension index pair (0, 1, 2, ..., $d/2 - 1$)
• $d$: the total embedding dimension
• $PE_{(pos, 2i)}$: the encoding value at position $pos$, even dimension $2i$
• $PE_{(pos, 2i+1)}$: the encoding value at position $pos$, odd dimension $2i+1$
• $10000$: a base constant that controls the frequency range

Notice that for a fixed position $pos$, as we scan across the $d$ dimensions, we compute sine and cosine of decreasing frequencies. The first two dimensions use the fastest oscillation, and the last two dimensions use the slowest. The encoding contains a full spectrum of waves, from very high frequency to very low frequency, stacked into a single vector.

The key to the formula is the denominator $10000^{2i/d}$. This term controls how fast each dimension pair oscillates as position increases. Let's unpack what happens at different dimension indices.

When $i = 0$ (the first dimension pair), the denominator is $10000^0 = 1$, so we compute $\sin(pos)$ and $\cos(pos)$. This oscillates rapidly: moving from position 0 to position 6 covers roughly one full cycle.

When $i = d/2 - 1$ (the last dimension pair), the denominator is approximately $10000^1 = 10000$, so we compute $\sin(pos/10000)$ and $\cos(pos/10000)$. This oscillates extremely slowly: you need 62,832 positions to complete one full cycle.

The exponent $2i/d$ creates a geometric progression of frequencies. As $i$ increases from 0 to $d/2 - 1$, the exponent increases from 0 to approximately 1, and the denominator grows from 1 to 10000. This exponential scaling ensures that each dimension pair captures position information at a different resolution. The spacing between frequencies is not arbitrary: it is chosen so that the wavelengths span a wide enough range to cover practical sequence lengths while keeping the total number of dimensions manageable.

The choice of 10000 as the base constant is also deliberate. A smaller base would cluster the wavelengths too close together. This provides redundant information in nearby dimensions. A larger base would spread them out but might leave some scale ranges uncovered. With 10000 and typical embedding dimensions of 256 or 512, the wavelengths range from about 6 positions up to 63,000 positions, which comfortably covers sequences of any practical length at the time the paper was written.

Each dimension pair uses both sine and cosine at the same frequency. This pairing is not arbitrary; it serves two distinct purposes that are both essential.

The first purpose is unique identification within a cycle. Sine alone cannot distinguish positions that differ by multiples of $2\pi$. At the first dimension's frequency, positions 0 and 6 have very similar sine values, and positions 0 and approximately 6.28 have identical sine values. But the (sine, cosine) pair at any frequency uniquely identifies a phase angle. Geometrically, as position increases, the (sin, cos) pair traces a circle in 2D space, and every point on that circle corresponds to a unique position within one cycle. The cosine value is 90 degrees out of phase with the sine value, so there is never a point where both are ambiguous simultaneously.

The second purpose is enabling relative position computation. The sine/cosine pairing allows relative positions to be computed through rotation matrices, a property we'll explore in detail later in this chapter. This mathematical structure means the model can potentially learn to attend to relative positions using simple linear operations, which is a powerful inductive bias for sequence modeling. This property is so useful that it directly inspired modern architectures: RoPE (Rotary Position Embedding), used in LLaMA and many other models, is essentially a refined version of this rotation insight, applied directly to the query and key vectors inside attention.

Now that we have the formula, let's build deeper intuition for why this multi-frequency approach works so well. The key insight is that different dimensions encode position at different scales, much like how we represent time using multiple units.

Consider a clock with both a second hand and an hour hand. The second hand rotates rapidly, completing one cycle per minute. If you only had the second hand, you could tell the difference between 3:00:15 and 3:00:45, but you couldn't distinguish 3:00:15 from 4:00:15, since both would show the second hand at the same position. The hour hand solves this problem: it moves slowly, completing one cycle every 12 hours, so it can distinguish times that the second hand cannot. A clock with both hands gives you two scales of information that together resolve ambiguities that neither could resolve alone.

But a two-handed clock has a limit: it cannot represent sub-second precision or multi-day periods. Real timekeeping systems solve this by adding finer and coarser time scales. The sinusoidal encoding generalizes this idea to $d/2$ "hands," each running at a different speed, chosen so that every positional difference up to the maximum sequence length is distinguishable by at least one hand.

Sinusoidal position encoding applies this clock principle to sequences. The first dimension pairs oscillate rapidly (like the second hand), distinguishing nearby positions with high precision. The later dimension pairs oscillate slowly (like the hour hand), distinguishing distant positions that the fast oscillators cannot. Together, they create a multi-resolution representation where any two positions, no matter how close or far apart, can be distinguished by at least one dimension pair.

The wavelength formula makes this precise. For dimension pair $i$, the wavelength (the number of positions needed for one complete oscillation cycle) is:

where:

• $\lambda_i$: the wavelength for dimension pair $i$ (measured in positions per cycle)
• $10000^{2i/d}$: the frequency denominator that grows geometrically with $i$
• $2\pi$: the angular measure of one complete cycle (in radians)

This formula reveals the geometric progression at the heart of sinusoidal encoding:

• First dimension pair ($i = 0$): Wavelength is $2\pi \approx 6.28$ positions. Positions 0 through 6 span roughly one full cycle. This fast oscillation distinguishes positions that differ by just 1 or 2.

• Middle dimension pairs: Wavelengths grow exponentially. By the time we reach the middle dimensions, wavelengths might be in the hundreds, suitable for distinguishing positions that differ by tens or hundreds.

• Last dimension pair ($i = d/2 - 1$): Wavelength is approximately $2\pi \cdot 10000 \approx 62,832$ positions. You'd need over 10,000 positions to complete one cycle. This slow oscillation can distinguish positions separated by thousands.

The geometric progression is deliberate and essential. If wavelengths grew linearly, nearby dimension pairs would be redundant, both distinguishing roughly the same positional differences. The exponential growth ensures each dimension pair contributes unique positional information at its own characteristic scale, creating a compact representation that efficiently covers all possible position differences.

The geometric spacing is deliberate. If wavelengths grew linearly, nearby dimension pairs would be redundant, encoding position at nearly the same resolution. The exponential growth ensures each dimension pair contributes unique positional information at a scale not covered by any other pair.

Think of it this way: with 256 dimension pairs and wavelengths growing exponentially from 6 to 63,000, each pair is responsible for roughly a factor-of-two increase in scale. This is like a biological sensory system that covers a wide range of intensities using logarithmic scaling: the eye can handle both starlight and sunlight because it responds logarithmically, not linearly. The sinusoidal encoding's geometric wavelength progression achieves the same kind of dynamic range in a compact representation.

With the formula and wavelength intuition in place, let's see what sinusoidal position encodings look like. We'll implement the encoding from scratch and visualize the resulting patterns to check our intuition.

The circular trajectories reveal why sine/cosine pairing works so well. In the first dimension pair (left), the fast oscillation means positions loop around the circle multiple times. Even if two positions land at similar angles on the circle, they'll be distinguished by other dimension pairs with different frequencies. In the middle dimension pair (right), positions are spread across a smaller arc. This provides coarse-grained discrimination.

Notice that in both cases, the trajectory lies exactly on the unit circle. This is a direct consequence of the identity $\sin^2(\theta) + \cos^2(\theta) = 1$. Every position encoding, in every dimension pair, has the property that its sine and cosine values are exactly one unit apart when viewed as a 2D point. This geometric regularity is part of what makes the relative position property, which we will derive in a moment, work out so cleanly.

Why does this encoding give each position a unique vector? Consider two positions $pos_1$ and $pos_2$. For them to have identical encodings, they would need to be indistinguishable across all dimension pairs. But with the geometric progression of wavelengths, this is virtually impossible for any pair of positions within a reasonable sequence length.

The key insight is that if two positions are the same distance apart in one dimension pair's "angular space," they will almost certainly be at different angles in every other dimension pair, because the frequencies are irrational multiples of each other. The wavelengths $2\pi \cdot 10000^{2i/d}$ for different $i$ values are not commensurate: you cannot find an integer number of cycles of one that equals an integer number of cycles of another. This incommensurability is what prevents aliasing.

If two positions differ by 1, the first dimension pair (wavelength approximately 6) will clearly distinguish them. If they differ by 100, middle dimension pairs will distinguish them. If they differ by 10,000, the later dimension pairs will distinguish them. The multi-scale representation captures position differences at any granularity.

Let's verify this by computing distances between position encodings:

The plot reveals an important property: distance grows quickly for small separations (positions 1-10 are clearly distinguishable from position 0) but then oscillates around a plateau for larger separations. The oscillation comes from the sinusoidal structure: at certain separations, the high-frequency dimensions happen to cycle back to similar values, temporarily reducing the distance. However, the low-frequency dimensions ensure that even these "aliased" positions remain distinguishable from one another. The plateau behavior also means that very distant positions are no more "different" than moderately distant ones, which is a reasonable property for a language model: the word at position 400 in a long document is not necessarily more distant in meaning from position 0 than the word at position 100.

Let's make the abstract concrete with a small, worked example. Consider a sentence with five tokens: "The cat sat on mat." We'll encode the positions 0 through 4 using a tiny encoding with $d = 8$ dimensions, small enough to trace through by hand for the first few values.

With $d = 8$, we have $d/2 = 4$ dimension pairs, numbered $i = 0, 1, 2, 3$. The frequency denominators for each pair are:

The wavelengths for each pair are:

• Pair 0: $\lambda = 2\pi \cdot 1 \approx 6.28$ positions
• Pair 1: $\lambda = 2\pi \cdot 17.78 \approx 111.7$ positions
• Pair 2: $\lambda = 2\pi \cdot 100 \approx 628$ positions
• Pair 3: $\lambda = 2\pi \cdot 562.3 \approx 3532$ positions

For position 0 (the token "The"), every sine term evaluates to $\sin(0) = 0$ and every cosine term evaluates to $\cos(0) = 1$, giving:

This is a special property of position 0 that makes it immediately recognizable: it is the only position where the odd dimensions (cosines) are all 1.

For position 1 (the token "cat"), we compute:

So $PE_1 \approx [0.841, 0.540, 0.056, 0.998, 0.010, 1.000, 0.002, 1.000]$.

The pattern is clear: in the first dimension pair ($i = 0$), there is already a large change from position 0 to position 1 (from $[0, 1]$ to $[0.841, 0.540]$), because the wavelength is only about 6. In the later dimension pairs ($i = 2$ and $i = 3$), the change is tiny, because a single step is a tiny fraction of their 628- or 3532-position wavelengths.

Now consider position 4 (the token "mat") in the first dimension pair: $\sin(4) \approx -0.757$ and $\cos(4) \approx -0.654$. This is very different from both position 0 and position 1, confirming that the high-frequency pair rapidly distinguishes even nearby positions. Meanwhile, the slow-frequency pairs at $i = 2$ and $i = 3$ have barely moved from their position-0 values, since 4 is much less than the wavelengths of 628 and 3532.

This worked example makes the abstract formula tangible. You can trace exactly how each token's position influences its encoding, and you can see that the information is spread across all dimensions simultaneously rather than concentrated in any single one. In practice, these position encodings are added element-wise to the token embeddings, so the final input to the transformer is the sum of what the token means (its semantic embedding) and where it sits in the sequence (its positional encoding).

One of the most elegant properties of sinusoidal encoding is that relative positions can be expressed as linear transformations. For any fixed offset $k$, there exists a matrix $M_k$ such that:

where:

• $PE_{pos}$: the position encoding vector at position $pos$ (a $d$-dimensional row vector)
• $PE_{pos+k}$: the position encoding vector at position $pos + k$
• $M_k$: a $d \times d$ transformation matrix that depends only on the offset $k$, not on the absolute position

This is a remarkable property. It means that the relationship between any two encodings that are $k$ steps apart is always the same, regardless of where they sit in the sequence. The transformation from position 5 to position 10 is the same linear map as the transformation from position 500 to position 505. In principle, a transformer that learns this linear map can attend to any token "k positions ahead" without needing to know the absolute positions, simply by applying a fixed linear operation to the encoding.

To understand why this works, we need to derive the relationship step by step using trigonometric identities.

Step 1: Recall the angle addition formulas. For any angles $a$ and $b$, trigonometry gives us:

where $a$ and $b$ are any angles (in radians). These identities let us express the sine or cosine of a sum in terms of the sines and cosines of the individual angles. They provide the algebra behind the relative position construction.

Step 2: Define the angular frequency. For dimension pair $i$, we define the angular frequency as:

where:

• $\omega_i$: the angular frequency for dimension pair $i$ (determines how fast this dimension oscillates)
• $i$: the dimension pair index (0, 1, 2, ..., $d/2 - 1$)
• $d$: the total embedding dimension
• $10000^{2i/d}$: the denominator that grows geometrically with $i$

This means the encoding at position $pos$ in dimension pair $i$ uses the argument $\omega_i \cdot pos$.

Step 3: Apply the addition formulas. To find the encoding at position $pos + k$, we substitute $a = \omega_i \cdot pos$ and $b = \omega_i \cdot k$ into the angle addition formulas:

where:

• $\omega_i = 1/10000^{2i/d}$: the angular frequency for dimension pair $i$
• $pos$: the current position in the sequence
• $k$: the position offset we want to shift by
• $\sin(\omega_i \cdot pos)$ and $\cos(\omega_i \cdot pos)$: the original encoding values at position $pos$ (these are $PE_{(pos, 2i)}$ and $PE_{(pos, 2i+1)}$)
• $\sin(\omega_i k)$ and $\cos(\omega_i k)$: constants that depend only on the offset $k$, not on the absolute position

The last two terms are fixed constants once you fix $k$ and $i$: the values $\sin(\omega_i k)$ and $\cos(\omega_i k)$ do not change as you vary $pos$. This is what makes the transformation linear in the original encoding values.

Step 4: Recognize the matrix structure. The key insight is that the right-hand sides of both equations are linear combinations of $\sin(\omega_i \cdot pos)$ and $\cos(\omega_i \cdot pos)$. This is exactly what matrix multiplication does. We can write:

This is a rotation in 2D. For each dimension pair, the encoding at $pos + k$ is the encoding at $pos$ rotated by angle $\omega_i k$. The rotation matrix for offset $k$ in dimension pair $i$ is:

where:

• $R_k^{(i)}$: the 2x2 rotation matrix for dimension pair $i$ with offset $k$
• $\omega_i = 1/10000^{2i/d}$: the angular frequency for dimension pair $i$
• $k$: the position offset (how many positions to shift)
• $\omega_i k$: the rotation angle, which depends on both the offset and the dimension's frequency

Step 5: Construct the full transformation matrix. The full transformation matrix $M_k$ is block-diagonal, with each 2x2 block being the rotation matrix for that dimension pair:

where:

• $M_k$: the full $d \times d$ transformation matrix for offset $k$
• $R_k^{(i)}$: the 2x2 rotation matrix for dimension pair $i$ (defined above)
• $0$: 2x2 zero matrices (indicating no interaction between dimension pairs)
• The matrix has $d/2$ blocks along the diagonal

This block-diagonal structure means relative position shifts act independently on each dimension pair, rotating the (sine, cosine) pair by an amount proportional to the offset. Each dimension pair encodes position at its own frequency, so shifting by $k$ positions rotates it by its characteristic angle $\omega_i k$. The faster the dimension's frequency (smaller $i$), the larger the rotation for a given offset $k$.

This rotation interpretation is more than a mathematical curiosity. It means that the inner product between two position encodings, $PE_{pos_1} \cdot PE_{pos_2}$, depends only on their relative distance $|pos_1 - pos_2|$, not on their absolute values. The attention mechanism computes inner products between queries and keys, both of which have position information added. If the model learns to use that position information, it can compute relative-position-sensitive attention scores. This is the theoretical basis for the claim that sinusoidal encoding supports relative position reasoning.

Let's verify this property numerically:

The visualization makes the rotation property tangible. Each colored arrow shows the transformation from position $pos$ (circle) to position $pos + 5$ (square). Notice that all arrows rotate by the same angle, confirming that the transformation depends only on the offset $k$, not on the starting position. This is the geometric essence of how sinusoidal encodings enable learning of relative positions.

A significant advantage of sinusoidal encodings is their ability to represent positions never seen during training. Unlike learned position embeddings that require a fixed vocabulary of positions, sinusoidal encodings are computed from a deterministic formula that works for any position value.

This matters in practice. During training, you choose a maximum sequence length and process all examples within that budget. But at inference time, users sometimes send inputs longer than anything in the training set. With learned embeddings, position 513 in a model trained on sequences of length 512 simply does not exist; the embedding table has no entry for it, and you must either truncate the input or handle a missing value. With sinusoidal encoding, position 513 is just another application of the formula, and the output is a perfectly valid encoding vector.

Let's examine how encodings behave beyond typical training lengths:

Here's a complete, production-ready implementation of sinusoidal position encoding that handles batched inputs:

The position encodings have similar magnitude to typical token embeddings (around 5-6 for 64 dimensions), which ensures that position information is meaningful but doesn't overwhelm the semantic content.

In practice, production transformer implementations often scale the token embeddings by $\sqrt{d\_model}$ before adding the position encoding. The intuition is that the learned token embeddings tend to have small norms when initialized, and $\sqrt{d\_model}$ scaling brings them to a magnitude roughly comparable to the position encoding. Without this scaling, the position signal might dominate in the early stages of training before the embeddings have grown to their natural size. The original paper applies this scaling explicitly: the embedding layer output is multiplied by $\sqrt{d\_model}$, and then the position encoding is added.

The choice between sinusoidal and learned position embeddings involves several trade-offs that have been explored extensively in the literature. Neither approach is universally superior, and the best choice depends on your specific task, training data, and inference requirements.

Sinusoidal encoding offers several important advantages. It requires no parameters to learn, which means faster training, no risk of overfitting position patterns, and a smaller model. The deterministic formula works for any position, enabling mathematically valid extrapolation to longer sequences. The mathematical structure, specifically the relative positions as rotations property, provides an inductive bias that may help the model learn position-aware patterns more efficiently. Finally, because the encoding is fixed, you can reason analytically about it: you know exactly what encoding any position will receive, which makes debugging and analysis easier.

The disadvantages of sinusoidal encoding are also real. The fixed formula may not capture task-specific positional patterns that deviate from a sinusoidal structure. Some tasks might benefit from non-linear position relationships that the formula cannot express. And while extrapolation is mathematically valid, it is practically unreliable: a model trained on 512-length sequences may fail on 1024-length sequences even with sinusoidal encoding, because the attention patterns were learned on shorter sequences.

Learned embeddings have the opposite set of trade-offs. They offer full flexibility to represent arbitrary position patterns, since the embeddings are optimized end-to-end with the rest of the model. They can learn task-specific positional biases directly from data, which may be important for tasks where certain positions have special semantic roles (e.g., the first token in a BERT-style model is often a special classification token). The implementation is also simpler: just another embedding lookup table.

The disadvantages of learned embeddings are equally important. They add parameters proportional to maximum sequence length times embedding dimension. They cannot generalize to positions beyond their training length, since those positions have no entry in the table. And they may overfit to position patterns in the training data, especially for small datasets.

The parameter savings are significant for long sequences. At 4096 positions with 768 dimensions, learned embeddings require over 3 million parameters just for position. Sinusoidal encoding requires none.

Interestingly, the original transformer paper found that both approaches performed similarly on machine translation. This result was somewhat surprising and suggested that the specific form of the position encoding matters less than the fact that some position signal is present. Modern practice varies: BERT uses learned embeddings, GPT-2 uses learned embeddings, but many newer architectures explore alternatives like relative position encodings (covered in later chapters) that build on the insights from sinusoidal design. In very large models, the position encoding parameters are a tiny fraction of total parameters, so the theoretical advantage of sinusoidal saving dissolves, and the practical flexibility of learned embeddings often wins.

When you use sinusoidal encoding in a real transformer, several practical considerations come up that are not obvious from the formula alone.

The first consideration is normalization. The position encoding values lie in $[-1, 1]$ for each dimension, and with $d$ dimensions, the L2 norm of a position encoding vector is approximately $\sqrt{d/2}$. If your token embeddings have a different typical norm, the relative strength of the position signal will change. Most implementations scale token embeddings by $\sqrt{d\_model}$ before adding the position encoding, as discussed earlier. Pay attention to this scaling factor in any implementation you borrow or build on.

The second consideration is where to apply the encoding. The standard approach adds the position encoding to the token embedding before the first transformer layer. This means the position information is available to the first self-attention layer and propagates through all subsequent layers. Some architectures experiment with adding fresh position encodings at each layer, but this is non-standard and introduces additional complexity.

The third consideration is dropout. Many transformer implementations apply dropout to the sum of the token embedding and position encoding before passing it to the attention layers. This is separate from dropout applied within the attention mechanism itself, and its purpose is to regularize the input representation. The original paper uses dropout with probability 0.1 in the base model, applied after the embedding plus encoding sum.

The fourth consideration is the interaction with layer normalization. If you apply layer normalization before attention (pre-norm, as in many modern architectures), the position information is normalized along with the semantic information. Post-norm architectures, like the original transformer, normalize after the residual connection, which means the position information persists more directly in the residual stream. Both approaches work, but the effect of position encoding relative to semantic content can differ between them.

The scaling step is important: without it, the position encoding signal would be much stronger than the token embedding signal in the early stages of training, potentially slowing convergence. By scaling token embeddings up by $\sqrt{d\_model}$, you ensure both signals start at comparable magnitudes.

Sinusoidal position encoding introduced key concepts that continue to influence modern position encoding research, but it also carries real limitations that motivated many follow-on papers.

The primary limitation is the disconnect between the encoding's mathematical properties and the model's learned behavior. While sinusoidal encodings can represent arbitrary positions, the transformer must still learn to use this information. If training data only contains sequences up to length 512, the model's attention patterns are calibrated for that range. Extrapolating to length 2048 provides valid encodings but potentially invalid learned behavior, because the attention heads may have learned specific patterns (e.g., "attend to high-frequency dimension pairs with values in this range") that break down at longer positions. Empirically, vanilla transformers with sinusoidal encoding tend to degrade significantly when asked to generalize to contexts longer than their training length, despite the theoretical extrapolation capability.

A second limitation is the absolute nature of the encoding. Each position has a fixed representation regardless of context. The word at position 50 has the same positional encoding whether it's in a 100-token sequence or a 1000-token sequence. This can make it harder for the model to learn purely relative patterns, like "attend to the previous word," because the relative displacement between two absolute positions encodes the same information whether those positions are in the middle or near the end of a sequence. Relative position encoding schemes, which we will cover in later chapters, address this directly by encoding relative distances rather than absolute positions.

A third limitation, less often discussed, is the potential for the position encoding to interfere with certain semantic structures. For example, if a task requires the model to recognize that two sentences contain the same content regardless of their absolute position in a longer document, the sinusoidal encoding provides a strong absolute position signal that the model must learn to ignore. Learned embeddings might also suffer from this, but at least they are optimized end-to-end and can in principle learn to minimize this interference. The sinusoidal encoding is fixed regardless of what the task requires.

Despite these limitations, sinusoidal encoding established foundational principles that remain central to position encoding research. The use of multiple frequencies to capture position at different scales appears in many modern schemes. The sine/cosine pairing for unique identification within each frequency is retained in RoPE. The geometric wavelength progression appears in various forms in ALiBi and other recent encodings. And the key insight, that relative positions should ideally be computable by simple linear operations on absolute position encodings, has driven a decade of work on relative and rotary position encodings.

The impact extends beyond position encoding itself. Sinusoidal functions are now used throughout deep learning as a tool for injecting continuous signals into discrete neural network processing: in neural radiance fields (NeRF) for 3D scene representation, in diffusion models for encoding timesteps, and in various forms of Fourier feature networks. The original transformer paper introduced the idea to a broad audience and demonstrated that periodic functions can serve as rich, well-behaved representations for continuous inputs.

When implementing sinusoidal position encoding, the following parameters control the encoding behavior:

• d_model: The dimension of the position encoding vectors, which must match the token embedding dimension. Larger values provide finer-grained positional discrimination because more dimension pairs are available to cover the full wavelength range. Common values range from 256 to 1024. If you use a model with d_model = 512, your position encodings have 256 dimension pairs, each contributing unique positional information at its own scale.

• max_len: The maximum sequence length to pre-compute encodings for. Setting this higher than your longest expected sequence avoids runtime recomputation but increases memory usage. Typical values range from 512 to 8192 depending on the task. For inference on unexpectedly long inputs, the SinusoidalPositionEncoding class above will recompute encodings on demand.

• Base constant (10000): The frequency scaling constant in the denominator. This value controls the range of wavelengths, from $2\pi$ at the fastest end to $2\pi \times 10000$ at the slowest. The original transformer paper uses 10000, but some implementations experiment with different values. A smaller base compresses the wavelength range and may suit shorter sequences; a larger base spreads it out and may help with very long sequences. The value 10000 was chosen empirically to work well for typical NLP sequence lengths at the time.

Sinusoidal position encoding provides a parameter-free method for injecting positional information into transformer models. By encoding each position as a unique pattern of sine and cosine values at geometrically spaced frequencies, it creates distinguishable representations for any sequence position, supports mathematical extrapolation beyond training length, and provides a geometric structure that enables relative position reasoning.

Key takeaways from this chapter:

• Multi-scale representation: Different dimension pairs capture position at different resolutions. High-frequency pairs distinguish nearby positions with precision; low-frequency pairs distinguish distant positions. Together they cover all positional scales from adjacent tokens to sequences of tens of thousands.

• Mathematical structure: The sine/cosine pairing enables relative positions to be computed as rotations. For any fixed offset $k$, the encoding at position $pos + k$ is a linear transformation (specifically a block-diagonal rotation) of the encoding at position $pos$. This property depends only on $k$, not on the absolute position, making it a powerful inductive bias for relative position reasoning.

• No learned parameters: The encoding is computed from a deterministic formula, eliminating position-related parameters and enabling representation of any position without a fixed vocabulary size.

• Bounded values: All encoding values lie in $[-1, 1]$. This ensures numerical stability regardless of position. The L2 norm of each position vector is approximately $\sqrt{d/2}$. This provides a consistent signal magnitude across all positions.

• Extrapolation caveat: While encodings are valid for any position, the model's learned attention patterns may not generalize to positions unseen during training. Mathematical validity and practical performance are separate concerns, and long-context generalization requires more than just a valid encoding formula.

• Trade-offs with learned embeddings: Sinusoidal encoding saves parameters and enables extrapolation but lacks the flexibility to learn task-specific position patterns. Modern practice often favors learned embeddings or more sophisticated alternatives like RoPE, but the design principles of sinusoidal encoding remain relevant and influential.

In the next chapter, we'll explore learned position embeddings in detail: how they're implemented, when they outperform sinusoidal encodings, and the design considerations that affect their performance.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about sinusoidal position encoding.