BERT (2018) › The Problem

Before BERT, language representation learning faced fundamental limitations that constrained the effectiveness of pretrained models for downstream understanding tasks. Traditional language models, including early neural approaches, processed text unidirectionally. Left-to-right language models could predict the next word given previous context, while right-to-left models could predict the previous word given subsequent context. However, understanding tasks like question answering, natural language inference, and sentiment analysis required bidirectional context: to understand a word's meaning, you often need to consider both what came before and what comes after.

Consider a sentence like "The bank announced it would increase interest rates." Understanding "bank" requires knowing the context: both the preceding mention of "announced" and the following "interest rates" help disambiguate whether this refers to a financial institution rather than a riverbank. A unidirectional language model processing left-to-right would encode "bank" before seeing "interest rates," so its representation would omit that disambiguating phrase. A right-to-left model would face the opposite problem. Neither direction could combine the preceding and following context in a single representation.

The problem extended beyond word sense disambiguation to syntactic and semantic relationships. To determine if "The cat sat on the mat" implies "The mat is under the cat," a model needs to understand relationships that flow in multiple directions. Subject-verb-object relationships, modifier connections, and anaphoric references all require considering the entire sentence context, not just one direction. Unidirectional models struggled with these tasks because they encoded information progressively in one direction, losing the ability to revise understanding based on later information.

Existing approaches to pretraining also suffered from a mismatch between pretraining objectives and downstream tasks. Traditional language models were trained to predict the next word in a sequence, an objective that naturally aligned with text generation tasks. However, understanding tasks like classification, question answering, and entailment detection required different capabilities: identifying important spans of text, comparing sentence pairs, and making judgments about relationships. The gap between pretraining objectives and understanding tasks limited how effectively pretrained representations could transfer to downstream applications.

Feature-based approaches using pretrained representations, such as ELMo, attempted to address some limitations by combining representations from bidirectional LSTMs. However, these approaches were constrained by the sequential nature of LSTM processing and the depth limitations of stacked bidirectional networks. The representations were powerful but still didn't fully exploit bidirectional context in the same way that BERT would enable.

Additionally, existing pretraining approaches required substantial task-specific architecture modifications. Transferring representations to new tasks often meant designing custom architectures or significant engineering to integrate pretrained components. This limited the general applicability of pretrained models and made pretraining harder to apply across diverse tasks.