YaRN › Key Parameters

When implementing YaRN for a new model, several parameters control its behavior. Understanding what each controls makes it easier to configure YaRN correctly and tune it when needed.

• scale: The context extension factor (e.g., 4.0 for extending from 2048 to 8192 tokens). This is the most important parameter and must be set to the ratio of the desired context length to the original training context length. Larger values enable longer contexts but may require more fine-tuning to reach full quality.

• alpha: The lower wavelength threshold (default: 1.0). Dimension pairs with wavelength-to-context ratio below this value receive no interpolation. Setting $\alpha = 1$ means any pair whose wavelength fits within the training context is left untouched. Decreasing $\alpha$ below 1 would apply interpolation to some pairs that complete full rotations during training, which is generally unnecessary. Increasing $\alpha$ would apply less interpolation overall, potentially causing extrapolation for pairs with wavelengths slightly above the context length.

• beta: The upper wavelength threshold (default: 32.0). Dimension pairs with wavelength-to-context ratio above this value receive full interpolation. Increasing $\beta$ delays full interpolation to pairs with longer wavelengths, applying partial interpolation over a wider range. Decreasing $\beta$ would apply full interpolation earlier, which could be appropriate for models where even moderately slow-moving pairs have not generalized well to unseen rotation states.

• base: The RoPE frequency base (default: 10000). This should match the base used in the original model's RoPE implementation exactly. Using the wrong base shifts all wavelength computations and causes the ramp function to apply interpolation to the wrong dimension pairs. Many recent models (released in 2024 onward) use higher bases like 500,000, which significantly shifts the wavelength distribution and may require re-examining the $\alpha$ and $\beta$ thresholds.

• original_context: The context length the model was originally trained on (e.g., 2048 or 4096). This determines the wavelength ratios used in the ramp function. Getting this wrong would cause YaRN to apply the wrong amount of interpolation to each dimension pair.

For most applications, the default values of $\alpha = 1$ and $\beta = 32$ work well. The primary parameter to adjust is scale, which should be set to the desired extension factor. If the model uses a non-standard RoPE base, ensure base matches the original configuration. And if you observe poor long-context quality after fine-tuning, consider whether the $\alpha$ and $\beta$ thresholds are appropriate for your model's specific wavelength distribution.