BERT (2018) › Limitations

Despite its benchmark gains and widespread adoption, BERT had several limitations that shaped subsequent research directions. The most significant limitation was computational cost. BERT-base contained 110 million parameters, while BERT-large had 340 million parameters. Training these models required substantial computational resources, and even fine-tuning could be expensive for researchers and practitioners with limited access to GPUs or TPUs. Inference was also computationally intensive, making real-time applications challenging, especially for mobile or edge deployments.

The model's input length limitation was another constraint. BERT could process sequences up to 512 tokens, which was sufficient for many tasks but insufficient for longer documents or contexts. BERT could not process entire documents, multi-turn conversations, or long-form content without truncation or complex chunking strategies that lost global context. This limitation motivated research into models capable of handling longer sequences.

BERT's masked language modeling objective, while effective for understanding tasks, created a mismatch with generation tasks. Unlike autoregressive language models that naturally generated text token by token, BERT's bidirectional encoding made direct text generation difficult. The model excelled at understanding and classification but wasn't designed for generation, limiting its applicability to tasks requiring both understanding and generation capabilities.

The model also struggled with certain types of reasoning tasks. While BERT captured rich contextual information, it didn't explicitly model structured reasoning, symbolic manipulation, or multi-step logical inference. Tasks requiring explicit reasoning chains, mathematical problem-solving, or complex logical deduction often exceeded BERT's capabilities, even when fine-tuned on relevant data. This limitation highlighted the difference between statistical pattern matching and true reasoning.

The next sentence prediction task, intended to help the model understand sentence relationships, was later shown to be less useful than initially thought. Subsequent research found that NSP didn't contribute significantly to BERT's performance, and models like RoBERTa achieved better results by removing NSP and focusing solely on masked language modeling. This finding showed why researchers must test which pretraining objectives improve model capabilities.

BERT's representations, while powerful, weren't optimized for all downstream tasks equally. Some tasks benefited more from BERT's bidirectional understanding than others. Tasks requiring explicit sequential generation or tasks with very different structures from the pretraining corpus sometimes saw smaller improvements. This variability in transfer effectiveness highlighted that pretraining objectives need careful design to match intended applications.

The model's training data, while large, reflected biases present in web text and corpora. BERT inherited and sometimes amplified social, cultural, and linguistic biases present in its training data. These biases could manifest in downstream applications, affecting fairness and appropriateness in real-world deployments. Addressing bias in pretrained models became an important research direction following BERT's release.