Attention Intuition › Attention as Soft Lookup

Think of attention as a soft, differentiable dictionary lookup. In a traditional dictionary, you provide a key and get back exactly one value. Attention works similarly, but instead of retrieving a single value, it retrieves a weighted combination of all values based on how well each key matches your query.

Consider a translation task where you're generating the French word for "cat" from an English sentence. Rather than searching through the entire compressed representation, attention lets the decoder ask: "Which parts of the input are most relevant right now?" The answer comes as a probability distribution over all input positions.

The mechanics involve three components:

• The query represents what we are currently looking for. In encoder-decoder attention, this is the decoder's current hidden state.
• The keys represent what we're comparing against. These are the encoder's hidden states, one per input position.
• The values represent what we retrieve once we find a match. In basic attention, keys and values are the same encoder hidden states, though they can be separate learned projections.

The attention mechanism computes similarity scores between the query and all keys, normalizes these scores into a probability distribution using softmax, and returns a weighted sum of the values. This weighted sum is the context vector that the decoder uses alongside its own state to generate the next token.

The analogy to information retrieval is revealing. When you search a database, you have a query (what you are looking for) and you compare it against a set of keys (indexed fields). When you find a match, you retrieve the associated value (the full record). Attention makes this process soft and continuous: instead of a hard match, every key partially matches the query, and you retrieve a blend of all values weighted by match quality. This connection to information retrieval is not coincidental. Scaled dot product attention, the variant used in transformers, was explicitly designed to mirror this structure, as we'll see in upcoming chapters.

The query-key-value framing deserves close attention because it underpins every modern attention mechanism, from the original Bahdanau variant to the transformers that power today's language models. Understanding this framing deeply will make every subsequent chapter easier.

Imagine you are in a library with a very helpful librarian. You walk in and say "I'm looking for books about the French Revolution" (your query). The librarian looks through the catalog cards (the keys), which are brief descriptions of each book. They find three cards with high similarity to your query: one for a general history, one for a biography of Robespierre, and one on the economic causes of the revolution. They retrieve those books (the values) and bring them to you, with the most relevant book on top.

In this analogy, the catalog cards (keys) are compact representations designed for efficient comparison, while the books themselves (values) contain the content you need. This separation of roles is architecturally meaningful. You want the comparison (query against key) to be efficient and well-calibrated, while the retrieval (values) can be richer and more complex.

In basic seq2seq attention, keys and values are both just the encoder hidden states, so the distinction collapses. But in transformer self-attention, keys and values are separate learned projections of the same encoder states, allowing the model to optimize comparison separately from content retrieval. We will explore this separation fully in the Self-Attention chapter.

For now, the essential intuition is this: the query captures "what I need," the keys capture "what each position offers for comparison," and the values capture "what each position contributes." Attention computes the match between query and keys, then uses that match to blend the values.