Attention Intuition › Attention vs Pooling

Before attention became widespread, sequence models often used pooling operations to aggregate information. Mean pooling averages all hidden states, while max pooling takes the element-wise maximum. Understanding how attention differs from these simpler approaches clarifies exactly what attention adds.

Mean pooling treats all positions equally by computing a simple average:

$$c = \frac{1}{T} \sum_{j=1}^{T} h_j$$

where:

• $c$: the aggregated context vector, the same dimension as hidden states
• $T$: the number of positions in the sequence
• $h_j$: the hidden state at position $j$

Each position receives weight $\frac{1}{T}$, regardless of content. This works when all parts of the input contribute equally to the output, but fails when relevance varies. In sentiment analysis, the phrase "not good" carries more weight than "the movie was", yet mean pooling gives them equal importance.

Max pooling extracts the strongest signal at each dimension independently:

$$c_d = \max_j \, h_{jd}$$

where:

• $c_d$: the $d$-th dimension of the aggregated vector
• $h_{jd}$: the $d$-th dimension of the hidden state at position $j$
• $\max_j$: the maximum value across all positions $j = 1, \ldots, T$

This captures salient features by selecting the most activated value for each dimension. However, it loses information about which positions contributed and cannot combine graded contributions from multiple positions.

Attention provides learned, context-dependent weighting:

$$c_t = \sum_{j=1}^{T} \alpha_{tj} h_j$$

where the weights $\alpha_{tj}$ are computed dynamically based on what the model needs at each step $t$. Unlike mean pooling's uniform weights or max pooling's binary selection, attention learns which positions matter most for the current prediction.

Consider the sentence "The movie was absolutely terrible" for sentiment classification. The comparison below shows how mean pooling and a trained attention mechanism would weight each word:

The contrast is stark. Mean pooling treats "The" and "terrible" as equally important, diluting the sentiment signal. Attention learns to focus on "terrible" and to a lesser extent "absolutely", producing a context vector that emphasizes the words driving the classification.

One more consideration separates attention from both pooling methods: attention is query-dependent. The same input sequence produces different context vectors at different decoding steps, because the query (decoder state) changes. Mean pooling and max pooling produce the same aggregation regardless of what you are trying to do with it. Attention adapts. When generating "chat" (cat), the attention mechanism focuses on the source position for "cat." When generating "noir" (black), it shifts focus to "black." This step-specific adaptation is something that pooling simply cannot provide.