Self-Attention Concept › Why Self-Attention Works for Representation Learning
Language AI HandbookAll chapters Self-Attention Concept › Why Self-Attention Works for Representation LearningOne section · Sections in this chapter4 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Self-attention excels at building contextual representations. A word embedding from Word2Vec or GloVe assigns the same vector to "bank" regardless of context. Self-attention creates contextualized embeddings where each token's representation incorporates information from surrounding tokens.
The process works as follows. Each token starts with an initial embedding. Through self-attention, each token gathers information from all other tokens, weighted by relevance. The output is a new set of representations where each token "knows about" the full sequence context.
This contextual awareness enables several capabilities that static embeddings lack:
- Word sense disambiguation: The representation of "bank" differs based on surrounding words
- Coreference resolution: Pronouns can gather information from their antecedents
- Syntactic awareness: Verbs can connect to their subjects and objects regardless of distance
- Semantic composition: Phrases and sentences build meaning from their constituent words
To appreciate how powerful this is, it helps to compare it against what came before. Pre-transformer NLP relied on static word embeddings like Word2Vec and GloVe, which encode a word's meaning as a fixed vector that never changes regardless of context. These embeddings capture distributional statistics: "bank" ends up somewhere between "river" and "money" because it co-occurs with both kinds of words in training data. But this is a single point in embedding space, a compromise between two entirely different meanings, not a true representation of either. When the model processes "river bank," it uses the same "bank" vector as when it processes "bank account." Any downstream task must somehow disentangle these meanings using only context provided by neighboring tokens.
Self-attention solves this problem directly. When a transformer processes "river bank," the representation of "bank" is computed as a weighted sum of all tokens in the sentence, with "river" receiving very high weight. The resulting vector is shifted toward the "river"-related region of embedding space. When the same model processes "bank account," the high-weight neighbors are "account," "deposit," "money," and the representation shifts in the financial direction. The same input token produces different output representations depending on context. This is what makes transformer embeddings so much more useful for downstream tasks.
The mechanism also enables compositional understanding in a way that static embeddings cannot. Consider the phrase "not happy." In static embeddings, you might try to negate the "happy" vector, but there is no obvious way to do this without training a separate component. In self-attention, "not" directly attends to "happy" with high weight, and "happy" attends back to "not." The resulting representation of "happy" in this context incorporates information from "not," enabling the model to learn that negated words have different downstream behavior. This is the basis for handling composition, including cases such as negation and intensification.
Let's visualize how self-attention might weight different tokens when processing a simple sentence.
# Example sentence
tokens = ["The", "cat", "sat", "on", "the", "mat"]
n_tokens = len(tokens)
# Simulated attention weights for the word "sat" (position 2)
# In practice, these would be learned; here we illustrate the concept
attention_weights_for_sat = np.array([0.05, 0.45, 0.10, 0.15, 0.05, 0.20])
# The verb "sat" attends strongly to its subject "cat" and object "mat"
The visualization shows that when building a representation for "sat," the model attends most strongly to "cat" (the subject performing the action) and "mat" (the location). Function words like "the" and "on" receive less attention. This weighted combination creates a rich representation that captures the verb's relationship to other sentence elements.
This pattern, a verb attending to its syntactic arguments, is precisely the kind of structural awareness that transformers develop during training. The attention weights are not hand-coded. They emerge from optimization: the model learns that representing a verb well requires knowing who performed the action (subject) and what it was performed on or where it occurred (object or adjunct). In a real trained model, you would see systematic patterns: determiners attending to their head nouns, adjectives attending to the nouns they modify, verbs forming attention arcs to their subjects and objects. The attention mechanism provides a soft, differentiable way to implement the kinds of syntactic relationships that linguists have described for decades.