Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Sinusoidal Position Encoding › Wavelength Intuition

Michael BrenndoerferUpdated June 6, 20255 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 97Sinusoidal Position Encoding › Wavelength Intuition

Reading section 3 of 15 · Wavelength Intuition · 5 minSection 3 of 15 · 5 min

Section 3 of 15 · 5 min read

Sinusoidal Position Encoding › Wavelength IntuitionOne section · Sections in this chapter3 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Now that we have the formula, let's build deeper intuition for why this multi-frequency approach works so well. The key insight is that different dimensions encode position at different scales, much like how we represent time using multiple units.

Consider a clock with both a second hand and an hour hand. The second hand rotates rapidly, completing one cycle per minute. If you only had the second hand, you could tell the difference between 3:00:15 and 3:00:45, but you couldn't distinguish 3:00:15 from 4:00:15, since both would show the second hand at the same position. The hour hand solves this problem: it moves slowly, completing one cycle every 12 hours, so it can distinguish times that the second hand cannot. A clock with both hands gives you two scales of information that together resolve ambiguities that neither could resolve alone.

But a two-handed clock has a limit: it cannot represent sub-second precision or multi-day periods. Real timekeeping systems solve this by adding finer and coarser time scales. The sinusoidal encoding generalizes this idea to $d/2$ "hands," each running at a different speed, chosen so that every positional difference up to the maximum sequence length is distinguishable by at least one hand.

Sinusoidal position encoding applies this clock principle to sequences. The first dimension pairs oscillate rapidly (like the second hand), distinguishing nearby positions with high precision. The later dimension pairs oscillate slowly (like the hour hand), distinguishing distant positions that the fast oscillators cannot. Together, they create a multi-resolution representation where any two positions, no matter how close or far apart, can be distinguished by at least one dimension pair.

The wavelength formula makes this precise. For dimension pair $i$ , the wavelength (the number of positions needed for one complete oscillation cycle) is:

\lambda_i = 2\pi \cdot 10000^{2i/d}

where:

$\lambda_i$ : the wavelength for dimension pair $i$ (measured in positions per cycle)
$10000^{2i/d}$ : the frequency denominator that grows geometrically with $i$
$2\pi$ : the angular measure of one complete cycle (in radians)

This formula reveals the geometric progression at the heart of sinusoidal encoding:

First dimension pair ( $i = 0$ ): Wavelength is $2\pi \approx 6.28$ positions. Positions 0 through 6 span roughly one full cycle. This fast oscillation distinguishes positions that differ by just 1 or 2.
Middle dimension pairs: Wavelengths grow exponentially. By the time we reach the middle dimensions, wavelengths might be in the hundreds, suitable for distinguishing positions that differ by tens or hundreds.
Last dimension pair ( $i = d/2 - 1$ ): Wavelength is approximately $2\pi \cdot 10000 \approx 62,832$ positions. You'd need over 10,000 positions to complete one cycle. This slow oscillation can distinguish positions separated by thousands.

The geometric progression is deliberate and essential. If wavelengths grew linearly, nearby dimension pairs would be redundant, both distinguishing roughly the same positional differences. The exponential growth ensures each dimension pair contributes unique positional information at its own characteristic scale, creating a compact representation that efficiently covers all possible position differences.

In[3]:

Code

import matplotlib.pyplot as plt  # noqa: F401
import numpy as np

# Calculate wavelengths for different dimensions
d = 512  # Embedding dimension
dimension_pairs = np.arange(d // 2)

# Wavelength formula: 2π × 10000^(2i/d)
wavelengths = 2 * np.pi * (10000 ** (2 * dimension_pairs / d))

Out[4]:

Visualization

Log-scale line plot showing wavelength increasing exponentially from about 6 to 63000 across 256 dimension pairs. — Wavelengths grow geometrically across dimension pairs in a 512-dimensional encoding. The first dimension pair has a wavelength of about 6.28 positions (one full cycle every six positions), while the final pair has a wavelength near 62,832 positions. This exponential spread ensures every scale of positional separation is covered by at least one dimension pair.

The geometric spacing is deliberate. If wavelengths grew linearly, nearby dimension pairs would be redundant, encoding position at nearly the same resolution. The exponential growth ensures each dimension pair contributes unique positional information at a scale not covered by any other pair.

Think of it this way: with 256 dimension pairs and wavelengths growing exponentially from 6 to 63,000, each pair is responsible for roughly a factor-of-two increase in scale. This is like a biological sensory system that covers a wide range of intensities using logarithmic scaling: the eye can handle both starlight and sunlight because it responds logarithmically, not linearly. The sinusoidal encoding's geometric wavelength progression achieves the same kind of dynamic range in a compact representation.