Bidirectional RNNs › Pooling Strategies for Sentence-Level Representations

So far we have used the BiRNN for sequence labeling, where each position in the output is used independently. But what if you want a single fixed-length vector representing the entire sentence? This is needed for sentence classification, textual entailment, or any task where the output is a single label for the whole input.

There are several common strategies for reducing a sequence of per-token representations to a single vector:

The most direct approach uses the final hidden states from each direction. For a BiLSTM processing a sequence of length $T$:

• The forward LSTM produces its most informative final state at position $T$, after seeing all tokens.
• The backward LSTM produces its most informative final state at position $1$, after having processed the sequence from right to left.

Concatenating these gives a vector of size $2d$ that contains the "summary" of the full sequence from both directions. This is the standard approach when using an LSTM encoder in sequence-to-sequence models: the encoder's final hidden states are concatenated and passed to the decoder as the initial context.

The limitation of this approach is that the final hidden states must compress the entire sequence into a vector of fixed size. For long sequences, important details from the middle of the sequence may be lost.

An alternative is to average all per-token hidden states:

$$\bar{h} = \frac{1}{T} \sum_{t=1}^{T} h_t$$

where $h_t = [\overrightarrow{h}_t ; \overleftarrow{h}_t]$ is the combined bidirectional representation at position $t$.

Mean pooling gives every position equal weight in the final representation. This is simple and often effective for tasks where the relevant information is distributed across the sequence (topic classification, sentiment analysis over long reviews). Its weakness is that it does not give more weight to the most informative tokens, treating every position equally regardless of importance.

Max pooling takes the element-wise maximum across all positions:

$$\bar{h}^{(j)} = \max_{t} h_t^{(j)}$$

for each dimension $j$. This selects the most strongly activated value in each dimension, effectively finding the tokens that most strongly express each feature. Max pooling is commonly used in sentence classification tasks because it automatically focuses on the most salient signals in the sequence rather than averaging them out.

The most expressive pooling strategy uses an attention mechanism to compute a weighted sum of per-token representations:

$$\bar{h} = \sum_{t=1}^{T} \alpha_t h_t$$

where the attention weights $\alpha_t$ are computed by a small network (often a single linear layer followed by softmax) that scores each token's relevance. This allows the model to learn which positions are most important for the task at hand, rather than treating all positions equally (mean pooling) or always selecting the maximum value (max pooling).

Attention-based pooling is conceptually related to the attention mechanisms in sequence-to-sequence models, which we cover in Part XII. The key difference is that pooling attention produces a fixed summary for the whole sequence, while sequence-to-sequence attention produces a different weighted summary for each output position.