BERT (2018) › The Solution

BERT addressed these limitations through a combination of architectural choices and pretraining objectives specifically designed for bidirectional understanding. The model used the transformer encoder architecture, which enabled parallel processing of all tokens and full bidirectional attention. Unlike autoregressive models that masked future tokens to prevent information leakage, BERT's encoder could attend to all tokens simultaneously in both directions, enabling true bidirectional context understanding.

The key innovation was masked language modeling (MLM), a pretraining objective that enabled bidirectional learning. During pretraining, BERT randomly masked approximately 15% of input tokens and trained the model to predict these masked tokens from the surrounding context. Unlike next-token prediction in language models, masked language modeling allowed the model to use information from both left and right context when predicting each masked token. This objective trained BERT to combine both directions in each token representation, matching the needs of understanding tasks.

The transformer encoder made BERT's bidirectional training possible. The self-attention mechanism allowed each token to directly attend to all other tokens in the sequence, creating dense connections that could capture complex relationships. Unlike LSTMs that processed sequences sequentially, transformers processed all tokens in parallel, enabling efficient training on large corpora while maintaining full bidirectional context at every layer. Multi-head attention could capture relationships at the syntactic and semantic levels, including pragmatic context, in parallel.

BERT also introduced a next sentence prediction (NSP) task during pretraining, where the model learned to predict whether one sentence followed another in the original text. This objective helped the model understand relationships between sentences, which proved valuable for tasks like question answering and natural language inference that required reasoning across sentence boundaries. Together, NSP and masked language modeling exposed BERT to sentence-pair and token-level prediction during pretraining.

The architecture included special tokens that became standard in subsequent language models: [CLS] for classification tasks and [SEP] to separate sentence pairs. The [CLS] token's final hidden state was designed to represent the entire sequence, making it suitable for classification tasks. The [SEP] token allowed BERT to handle both single sentences and sentence pairs, enabling application to a wide range of tasks with minimal architectural changes.

Fine-tuning BERT for downstream tasks was remarkably simple. For most tasks, researchers could take the pretrained BERT model and add a small task-specific layer on top. For classification, the [CLS] token's representation was fed through a linear layer. For question answering, span prediction layers were added to identify answer boundaries. For sentence pair tasks, both sentences were concatenated with a [SEP] token between them. This minimal architecture modification, combined with fine-tuning the entire model on task-specific data, enabled BERT to achieve state-of-the-art results across diverse tasks with relatively little task-specific engineering.