Self-Attention Concept › Summary

Self-attention is the mechanism that allows a sequence to attend to itself, enabling each position to gather information from all other positions. This simple idea changes how we build language models.

Key takeaways from this chapter:

• Self-attention vs cross-attention: In cross-attention, one sequence attends to another. In self-attention, a sequence attends to itself, with each token serving in a QKV role simultaneously.
• All-pairs interaction: Self-attention computes direct connections between all $n^2$ pairs of positions (where $n$ is the sequence length), eliminating the need for information to flow through intermediate states.
• Contextual representations: By aggregating information from surrounding tokens, self-attention creates embeddings that capture word meaning in context, enabling disambiguation, coreference resolution, and syntactic awareness.
• Parallelization: Unlike recurrent models, self-attention computes all interactions simultaneously, enabling efficient training on modern hardware and allowing the field to scale to much larger models and datasets.
• Quadratic complexity: The all-pairs computation scales as $O(n^2)$, meaning computational cost grows with the square of sequence length $n$. This makes self-attention expensive for very long sequences and has motivated research into efficient attention variants.
• Position agnostic: Self-attention treats sequences as sets, requiring explicit positional encodings to capture word order. Without positional information, permuting the input tokens produces the same attention weights.
• Residual integration: In transformer blocks, self-attention output is added to the input via residual connections, meaning each layer enriches rather than replaces the existing representations.

The next chapter covers how self-attention computes these interactions through QKV projections. These learned linear transformations give self-attention its full expressiveness, allowing the model to compute asymmetric, task-specific attention patterns and to specialize different projections for different aspects of context. Understanding the QKV formulation is essential for implementing and reasoning about transformer models.