Self-Attention Concept › From Cross-Attention to Self-Attention

In encoder-decoder attention, we have two distinct sequences: the encoder outputs serve as keys and values, while the decoder state provides the query. The decoder asks, "Which parts of the input are relevant to what I'm generating right now?"

Self-attention simplifies this setup. Instead of attending across two different sequences, a single sequence attends to itself. Every token in the sequence serves simultaneously as a query, a key, and a value. The question becomes: "For each token in this sequence, which other tokens are most relevant to understanding its meaning in context?"

Consider the sentence "The animal didn't cross the street because it was too tired." To understand what "it" refers to, you need to connect it back to "animal" rather than "street." Self-attention enables this by allowing "it" to attend strongly to "animal," building a representation that captures this coreference relationship.

The key insight is that the meaning of a word depends heavily on its context. The word "bank" means something different in "river bank" versus "bank account." Self-attention lets each word gather information from surrounding words to disambiguate and enrich its representation.

Think of self-attention as a mechanism where every word in a sentence gets to vote on how much it contributes to every other word's meaning. The word "tired" votes heavily for "animal" when "it" is being interpreted, because "tired" is more naturally predicated of animals than streets. The word "cross" votes for "street" because streets are things you cross. All of this happens simultaneously, in a single computational pass over the sequence. The result is a set of enriched representations where each word's embedding combines its inherent meaning with its meaning in this specific sentence.

The transition from cross-attention to self-attention also has architectural implications. In an encoder-decoder model with cross-attention, the encoder and decoder are separate modules. The encoder reads the full source sequence, and the decoder generates the target sequence one token at a time, querying the encoder at each step. Self-attention enables a different design: a single stack of layers that can read a sequence, build increasingly rich representations, and either produce a classification or generate output. This is the architecture underlying BERT, GPT, and the entire modern transformer ecosystem.