Stacked RNNs › ELMo as a Stacked Bidirectional LSTM Example

One of the most influential applications of stacked RNNs is ELMo (Embeddings from Language Models), introduced by Peters et al. in 2018. ELMo uses a 2-layer stacked bidirectional LSTM trained as a language model, and the resulting representations became one of the first major successes of contextualized word embeddings before the transformer era.

ELMo combines several ideas discussed in earlier chapters:

1. Bidirectional RNN (see Chapter 8 of this part): Two separate LSTMs, one reading forward and one reading backward
2. Stacking: 2 layers of bidirectional LSTMs, where each direction is stacked independently
3. Layer combination: ELMo doesn't use just the top layer. It uses a learned weighted combination of all layers

The architecture processes each sentence through:

• A character-level CNN to produce token embeddings (avoiding the fixed vocabulary problem)
• A forward 2-layer stacked LSTM reading left to right
• A backward 2-layer stacked LSTM reading right to left
• Concatenation of forward and backward states at each layer

For a token $t$ in a sequence of length $T$, ELMo produces representations at three levels:

• $E_0(t)$: The character-CNN embedding (context-independent)
• $E_1(t)$: The concatenated forward/backward states from layer 1 (local context)
• $E_2(t)$: The concatenated forward/backward states from layer 2 (global context)

The final ELMo embedding for downstream tasks is:

$$\text{ELMo}(t) = \gamma \sum_{\ell=0}^{2} s_\ell \cdot E_\ell(t)$$

where:

• $s_\ell$ are task-specific softmax weights (learned scalars that sum to 1)
• $\gamma$ is a task-specific scalar that scales the overall magnitude
• The sum runs over all three levels: character embedding + 2 LSTM layers

The key insight is that different layers capture different types of linguistic information:

• Layer 1 captures syntax: part-of-speech, dependency structure
• Layer 2 captures semantics: word sense disambiguation, coreference

By learning to combine layers, different downstream tasks can emphasize the information that matters most. A POS tagger uses mostly layer 1. A coreference model uses mostly layer 2. The task itself determines which level of the hierarchy is most relevant.

Before ELMo, most NLP systems used static word embeddings like Word2Vec or GloVe, which give every token the same representation regardless of context. ELMo demonstrated that contextual representations from stacked recurrent networks could dramatically improve downstream tasks. Improvements of 4–14% absolute on tasks like question answering, textual entailment, and coreference resolution were reported at the time of publication.

ELMo also highlighted an important property of stacked RNNs that pure depth metrics miss: different layers specialize. The lower layers specialize in syntax, the upper layers in semantics. Using all layers together, rather than just the final layer, extracted the most value from the stack.