XLNet, RoBERTa, ALBERT (2019) › The Problem

Despite BERT's impressive success, several limitations became apparent as researchers deployed it widely and analyzed its behavior more carefully. These limitations fell into three categories: fundamental issues with the masked language modeling objective, suboptimal training procedures, and inefficient parameter usage that constrained model scale.

BERT's masked language modeling approach, while effective, introduced artifacts that limited its ability to learn optimal representations. The model never saw actual words during pretraining, only masked tokens. During fine-tuning, however, the model encountered real words, creating a mismatch between pretraining and downstream tasks. This pretrain-finetune discrepancy meant that representations learned during pretraining might not transfer optimally to real-word contexts.

Masked language modeling also suffered from independence assumptions. When predicting multiple masked tokens in the same sequence, BERT predicted each token independently, even though these tokens appeared in the same context. In reality, masked tokens were conditionally dependent on each other. For example, in "The capital of France is [MASK]", predicting "Paris" for the mask should depend on understanding that we're discussing geography, but BERT's parallel prediction could not account for relationships between simultaneously masked tokens.

The bidirectional context captured by BERT came at the cost of not learning a generative model. BERT could not generate text autoregressively because it was trained with bidirectional attention. This limitation mattered for tasks requiring generation, such as text summarization or dialogue systems, where autoregressive capabilities were essential. The inability to generate text meant BERT was limited to understanding tasks rather than both understanding and generation.

BERT's original training procedure, while sufficient to demonstrate the model's potential, left room for improvement in several areas. The model was trained for relatively few steps on a dataset that, while large, could be expanded significantly. The training used static masking, where the same masking pattern was applied to each sequence during multiple epochs, potentially leading to overfitting to specific masking patterns.

The next sentence prediction (NSP) task, intended to capture sentence-level relationships, showed mixed results. Some analyses suggested that NSP was too easy and provided limited signal. The binary classification task of predicting whether two sentences were consecutive or not might not have been the most effective way to learn sentence relationships. Some experiments showed that removing NSP entirely could improve performance, suggesting the task was not as valuable as initially thought.

BERT's training hyperparameters had been chosen conservatively. The learning rate schedule, batch size, and other training details had been selected to ensure stability, but might not have been optimal for maximizing performance. As computational resources became more available and understanding of transformer training deepened, more aggressive training regimes became feasible.

BERT's architecture, while powerful, used parameters inefficiently in several ways. The model stored separate embeddings for token embeddings, segment embeddings, and position embeddings, with the vocabulary size and hidden dimension size determining embedding parameters. For large models, these embedding parameters consumed significant memory but might not have needed the full hidden dimension size.

The transformer layers themselves consumed most parameters, but researchers questioned whether all layers needed independent parameters. If lower layers learned similar transformations across positions, parameter sharing might enable learning more general patterns with fewer parameters. Additionally, the attention mechanism and feedforward networks scaled quadratically and linearly with sequence length respectively, making longer sequences computationally expensive.

The parameter inefficiency meant that training larger BERT models required proportionally more computational resources. Doubling the model size roughly doubled the computational cost, limiting the scale of models that could be practically trained. For a fixed computational budget, this inefficiency prevented exploring larger model architectures that might have captured more complex patterns.