Scaled Dot-Product Attention › Limitations and Considerations

Scaled dot-product attention is elegant and effective, but several limitations shape how it's used in practice. Understanding these limitations is not just academic: they directly motivate architectural choices in every major language model, and they explain why significant research effort has gone into attention variants over the past several years.

The most significant limitation is the $O(n^2)$ computational and memory cost. For a sequence of 1024 tokens processed with 12 attention heads, the attention matrices alone require storing millions of floating-point values per layer. Extending to 100,000 tokens (the range needed for document-level tasks such as analyzing entire research papers or legal contracts) makes standard attention impractical without modification. A sequence of 100,000 tokens would require $10^{10}$ entries per attention layer, far beyond the memory capacity of any current hardware. Research into efficient attention has produced many alternatives: sparse attention (attending to a fixed local window plus selected global tokens), low-rank approximations (approximating the $n \times n$ matrix as a product of smaller matrices), and kernel-based methods (reformulating attention to avoid materializing the full matrix). None has fully replaced standard attention for general-purpose models, but each has found niches where it offers meaningful advantages.

Softmax attention also has a tendency toward attention dilution on long sequences. Because softmax normalizes over all positions, each attention weight becomes smaller as sequence length grows: if the sequence has 1000 tokens, the average attention weight is 1/1000, even if the most relevant position truly deserves a weight of 0.5. This means the model must learn to produce very large raw scores to overcome the normalization and assign meaningful weight to relevant positions. In practice, this can cause trained models to show unexpectedly diffuse attention on long sequences, where they fail to focus on the most relevant context even when that context is clearly present. Researchers have explored alternatives like entmax (a sparse softmax that produces exactly zero weights for irrelevant positions) and attention temperature scaling at inference time, but softmax remains the dominant choice due to its well-understood behavior and hardware efficiency.

From a representational standpoint, standard scaled dot-product attention is permutation-equivariant: swapping two tokens in the input swaps the corresponding rows and columns of the attention matrix, and the output rows swap accordingly. The mechanism has no inherent notion of position or order. Feed the same sentence in different word orders, and the attention mechanism treats it as the same set of tokens in a different arrangement. This is why transformers require explicit positional encoding: without it, the model cannot distinguish "the cat sat on the mat" from "the mat sat on the cat." Positional encodings inject order information into the token representations before they enter the attention computation, effectively breaking the permutation equivariance. We'll cover positional encodings in Part XIV of the handbook.

There is also a more subtle limitation related to the form of the similarity function. Dot-product attention computes similarity as a bilinear form, which means the similarity score between query $i$ and key $j$ can only depend on the specific pair $(i, j)$ through their dot product. This is a restricted form of interaction: there is no way for the attention score to depend on the relationship between key $j$ and other keys, or on global properties of the sequence. More expressive attention variants have been proposed, but the dot product's simplicity and efficiency have kept it dominant.