Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Sinusoidal Position Encoding › Uniqueness of Position Encodings

Michael BrenndoerferUpdated June 6, 20254 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 97Sinusoidal Position Encoding › Uniqueness of Position Encodings

Reading section 5 of 15 · Uniqueness of Position Encodings · 4 minSection 5 of 15 · 4 min

Section 5 of 15 · 4 min read

Sinusoidal Position Encoding › Uniqueness of Position EncodingsOne section · Sections in this chapter5 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Why does this encoding give each position a unique vector? Consider two positions $pos_1$ and $pos_2$ . For them to have identical encodings, they would need to be indistinguishable across all dimension pairs. But with the geometric progression of wavelengths, this is virtually impossible for any pair of positions within a reasonable sequence length.

The key insight is that if two positions are the same distance apart in one dimension pair's "angular space," they will almost certainly be at different angles in every other dimension pair, because the frequencies are irrational multiples of each other. The wavelengths $2\pi \cdot 10000^{2i/d}$ for different $i$ values are not commensurate: you cannot find an integer number of cycles of one that equals an integer number of cycles of another. This incommensurability is what prevents aliasing.

If two positions differ by 1, the first dimension pair (wavelength approximately 6) will clearly distinguish them. If they differ by 100, middle dimension pairs will distinguish them. If they differ by 10,000, the later dimension pairs will distinguish them. The multi-scale representation captures position differences at any granularity.

Let's verify this by computing distances between position encodings:

In[10]:

Code

def encoding_distance(PE, pos1, pos2):
    """Compute Euclidean distance between two position encodings."""
    return np.linalg.norm(PE[pos1] - PE[pos2])


# Compute pairwise distances for first 50 positions
max_pos = 50
distances = np.zeros((max_pos, max_pos))
for i in range(max_pos):
    for j in range(max_pos):
        distances[i, j] = encoding_distance(PE, i, j)

Out[11]:

Visualization

Heatmap showing pairwise distances between position encodings, with zero on the diagonal and increasing values off-diagonal. — Pairwise Euclidean distances between the first 50 position encodings. The diagonal is zero (each position is identical to itself). Nearby positions have smaller distances (warm colors near the diagonal), while distant positions have larger distances (cooler colors far from the diagonal). The symmetric banded structure confirms that distance grows smoothly with positional separation.

The distance matrix confirms that no two positions have identical encodings (no zeros off the diagonal). The banded structure shows that nearby positions have smaller distances, while distant positions have larger distances. This smooth distance gradient helps the model learn position-dependent patterns: when training on examples where semantically related tokens appear at similar relative positions, the model can exploit the continuity of the encoding to generalize.

Let's examine how distance varies with positional separation more precisely:

Out[12]:

Visualization

Line plot showing encoding distance on y-axis versus positional separation on x-axis, with smooth growth at small separations and oscillatory plateau at larger separations. — Encoding distance from position 0 to all other positions in a 100-position sequence. Distance rises steeply for small separations, then oscillates around a plateau for larger separations. The oscillation arises because the high-frequency dimensions occasionally return to similar values while the low-frequency dimensions still distinguish the positions. The dashed red line shows the mean distance for separations greater than 20.

The plot reveals an important property: distance grows quickly for small separations (positions 1-10 are clearly distinguishable from position 0) but then oscillates around a plateau for larger separations. The oscillation comes from the sinusoidal structure: at certain separations, the high-frequency dimensions happen to cycle back to similar values, temporarily reducing the distance. However, the low-frequency dimensions ensure that even these "aliased" positions remain distinguishable from one another. The plateau behavior also means that very distant positions are no more "different" than moderately distant ones, which is a reasonable property for a language model: the word at position 400 in a long document is not necessarily more distant in meaning from position 0 than the word at position 100.