Bidirectional RNNs › Bidirectionality in Sentence Embeddings: ELMo

The ELMo (Embeddings from Language Models) system, published by Peters et al. in 2018, brought bidirectional LSTMs to contextual word representation learning at scale. It is worth understanding ELMo's design in detail because it represents the culmination of the BiLSTM era and directly motivates the transformer-based models that followed.

ELMo uses a deep two-layer bidirectional LSTM trained on a large corpus using a combination of forward and backward language modeling objectives. The forward LM predicts each word given all preceding words; the backward LM predicts each word given all following words. These two objectives are trained jointly on the same corpus.

The key architectural choice is that ELMo exposes all intermediate LSTM layer representations, rather than only the top layer. For a two-layer BiLSTM, this means three vectors are available for each token: the character-CNN embedding (which encodes morphology), the first BiLSTM layer's output, and the second BiLSTM layer's output. When ELMo is used downstream, these three representations are combined using learned scalar weights:

$$\text{ELMo}_t = \gamma \sum_{k=0}^{K} s_k h_{t,k}$$

where $s_k$ are softmax-normalized task-specific weights, $h_{t,k}$ is the $k$-th layer representation at position $t$, and $\gamma$ is a task-specific scalar. Different NLP tasks learn different weighting profiles.

The finding that different ELMo layers encode different linguistic properties was empirically validated through a series of probing experiments. When the scalar weights are examined after training on different tasks:

• Syntactic tasks (POS tagging, constituency parsing) assign high weight to lower BiLSTM layers, which capture local morphological and syntactic patterns.
• Semantic tasks (coreference, SRL, NER) assign higher weight to upper layers, which capture broader semantic relationships.

This layer-wise specialization emerged purely from training on language modeling: no explicit supervision about which layers should encode syntax versus semantics was provided. The hierarchy of linguistic abstraction emerged naturally from the architecture and training objective.

ELMo demonstrated that contextual representations dramatically outperform fixed word embeddings, establishing the paradigm of pretraining a large language model and fine-tuning it for downstream tasks. However, ELMo's BiLSTM architecture has inherent limitations that BERT's transformer architecture overcame.

The most fundamental limitation is that ELMo's bidirectionality is not truly joint. The forward and backward LSTMs are trained separately (each on its own language modeling objective) and then concatenated. The forward LSTM has never "seen" the backward representations during its own training; it is optimized to predict words from left context alone. Similarly, the backward LSTM is optimized to predict words from right context alone. The concatenation happens only at the representation level, not during learning.

BERT's self-attention mechanism, by contrast, is truly jointly bidirectional: every token attends to every other token simultaneously in a single operation. There is no separate forward and backward pass. The model is trained to predict masked tokens using context from both sides at once. This joint training produces qualitatively richer representations because the model learns to integrate left and right context in a unified way rather than concatenating separately trained unidirectional features.

This distinction explains why BERT substantially outperformed ELMo despite using broadly similar pretraining concepts. The joint bidirectionality of attention improved on the concatenated-LSTM approach.