Linear Attention › Key Parameters

When implementing or tuning linear attention, these parameters most directly affect performance and quality:

• feature_map: The function $\phi$ that transforms queries and keys. ELU+1 is fast and simple but produces poor softmax approximation. FAVOR+ with positive random features provides unbiased approximation but requires more computation. The choice depends on whether you prioritize speed or fidelity to softmax behavior.

• num_features (m): For random feature methods like Performer, this controls the trade-off between approximation quality and computational cost. Higher values reduce variance ($O(1/m)$) but increase the feature dimension. Typical values range from 64 to 512, with 256 offering a reasonable balance for most applications.

• eps: A small constant (typically 1e-6) added to the denominator during normalization to prevent division by zero. Critical for numerical stability, especially when feature maps can produce near-zero values.

• min_denom: An optional floor for the denominator (e.g., 1e-4) that clips extreme values. Useful when the sum of key features is poorly conditioned, which can occur with certain feature maps or input distributions.

• chunk_size: For memory-efficient implementations, this controls how many positions are processed together when accumulating the key-value aggregate. Smaller chunks reduce peak memory usage but may incur overhead from loop iterations. Values between 256 and 1024 work well in practice.

• D (feature dimension): The output dimension of the feature map. For simple maps like ELU, this equals the input dimension $d_k$. For random feature methods, it can be larger than $d_k$ to improve approximation quality, with typical ratios of 2x to 4x the original dimension.