RNN Architecture › Sequence Modeling Configurations

The RNN architecture is remarkably flexible. By choosing where to read inputs and where to produce outputs, you can adapt the same core equations to many different tasks. This flexibility is one of the reasons RNNs became the dominant approach to sequence modeling in the early deep learning era.

In many-to-one mode, the RNN reads an entire sequence and produces a single output from the final hidden state. The final hidden state $\mathbf{h}_T$ is a compressed representation of the whole sequence, which is then passed through a linear layer to produce the prediction:

$$\hat{y} = \mathbf{W}_{hy}\,\mathbf{h}_T + \mathbf{b}_y$$

This configuration is appropriate for tasks where you need a single label for an entire sequence:

• Sentiment analysis: classify a review as positive or negative
• Document classification: assign a topic label to a news article
• Spam detection: label an email as spam or legitimate
• Acceptability judgment: determine whether a sentence is grammatically well-formed
• Intent classification: identify the intent behind a user's utterance in a dialogue system

The intuition is that by the time the RNN has read the last word, $\mathbf{h}_T$ should contain enough information about the overall message to make a prediction. In practice, the final hidden state can have difficulty "remembering" information from the very beginning of a long sequence. This is the core limitation addressed in the Vanishing Gradients chapter, and it is part of why bidirectional RNNs (discussed below) are often preferred for classification tasks.

An important subtlety: using only the final hidden state discards the intermediate representations. An alternative is to use the average of all hidden states $\frac{1}{T}\sum_t \mathbf{h}_t$, or to take the max across positions (max pooling over time). These pooling strategies can sometimes outperform using only the final state, particularly when the relevant information is distributed throughout the sequence rather than concentrated near the end.

In one-to-many mode, the RNN receives a single input (or a conditioning vector) at the first step and then produces a sequence of outputs, with each output fed back as the input to the next step. This is the autoregressive generation pattern.

The typical setup uses an initial input $\mathbf{x}_0$ (perhaps a start-of-sequence token or a conditioning vector like an image embedding), then generates:

$$\mathbf{y}_t = \mathbf{W}_{hy}\,\mathbf{h}_t + \mathbf{b}_y, \quad \mathbf{x}_{t+1} = \text{embed}\!\left(\arg\max(\text{softmax}(\mathbf{y}_t))\right)$$

Applications of this configuration include:

• Image captioning: the image embedding is the initial input, and the RNN generates a caption word by word
• Music generation: a seed note or chord launches the generation of a melody
• Unconditional text generation: a start token launches free-form text
• Code generation from a specification: a short descriptor is encoded and the RNN expands it into a longer output

The generation process can use different decoding strategies. Greedy decoding always picks the token with the highest probability, which is fast but can produce suboptimal outputs. Beam search maintains a set of candidate sequences (beams) and selects the one with the highest cumulative log-probability. Temperature sampling adjusts how peaked or flat the probability distribution is. These strategies are covered in the Decoding Strategies chapter.

Many-to-many produces an output at every input step. There are two variants.

Synchronized (same-length input and output): Output $\mathbf{y}_t$ is produced from $\mathbf{h}_t$ at each step. This suits tasks where every input token needs a label, such as part-of-speech tagging or named entity recognition. The sequence labeling chapters in Part VI used CRF and HMM models for these tasks; RNNs provide an alternative that learns features directly from the text without hand-crafted feature templates.

Encoder-decoder (different-length input and output): The input sequence is first encoded into a context vector using a many-to-one RNN (the encoder), then a one-to-many RNN (the decoder) generates the output sequence from that context. This is the foundation of sequence-to-sequence models and is explored in detail in Part XII. The classic application is machine translation: the encoder reads a sentence in English and compresses it into a fixed-size context vector, and the decoder generates the corresponding French sentence one word at a time. The attention mechanism, introduced later, addresses the bottleneck of compressing an entire sentence into a single fixed-size vector.