Sinusoidal Position Encoding › Key Parameters

When implementing sinusoidal position encoding, the following parameters control the encoding behavior:

• d_model: The dimension of the position encoding vectors, which must match the token embedding dimension. Larger values provide finer-grained positional discrimination because more dimension pairs are available to cover the full wavelength range. Common values range from 256 to 1024. If you use a model with d_model = 512, your position encodings have 256 dimension pairs, each contributing unique positional information at its own scale.

• max_len: The maximum sequence length to pre-compute encodings for. Setting this higher than your longest expected sequence avoids runtime recomputation but increases memory usage. Typical values range from 512 to 8192 depending on the task. For inference on unexpectedly long inputs, the SinusoidalPositionEncoding class above will recompute encodings on demand.

• Base constant (10000): The frequency scaling constant in the denominator. This value controls the range of wavelengths, from $2\pi$ at the fastest end to $2\pi \times 10000$ at the slowest. The original transformer paper uses 10000, but some implementations experiment with different values. A smaller base compresses the wavelength range and may suit shorter sequences; a larger base spreads it out and may help with very long sequences. The value 10000 was chosen empirically to work well for typical NLP sequence lengths at the time.