BERT for IR (2019) › Limitations

Despite its significant advances, BERT for information retrieval faced several important limitations that prevented it from fully replacing more efficient approaches. The computational cost of BERT-based retrieval was substantially higher than dual encoder methods, making it challenging to deploy at the full scale required for web search. Processing each query-document pair through BERT required significant computation, and for search systems that needed to rank millions of documents, this cost could be prohibitive. While reranking architectures addressed this by limiting BERT to smaller candidate sets, the fundamental efficiency challenge remained.

The latency of BERT-based retrieval was another significant limitation. Even with reranking, running BERT on hundreds of documents could introduce noticeable delay, particularly when using large BERT models with many layers. For real-time search applications where sub-second response times were expected, this latency could impact user experience. Optimizations such as model distillation, quantization, and specialized hardware helped address this, but the fundamental trade-off between accuracy and speed remained.

The scalability challenge was particularly acute for first-stage retrieval, where BERT's computational requirements made it impractical to process millions of documents. While BERT excelled at reranking small candidate sets, the initial retrieval stage typically still relied on more efficient methods like BM25 or dual encoders. This meant that BERT's sophisticated understanding was only applied to documents that passed the initial filter, potentially missing highly relevant documents that the first stage failed to retrieve. This limitation motivated research into more efficient transformer architectures that could be used for first-stage retrieval.

The training data requirements for effective BERT-based retrieval were substantial. Fine-tuning BERT on retrieval tasks required large amounts of labeled query-document relevance data, which was expensive to collect and maintain. While some systems could use click logs or other implicit feedback, high-quality training data often required expert judgments or carefully curated relevance labels. Organizations without access to large-scale relevance data found it challenging to train effective BERT-based retrieval models, creating a barrier to adoption.

The interpretability of BERT-based retrieval systems was limited, making it difficult to understand why documents were ranked highly or to debug ranking problems. While attention weights could provide some insight into which parts of queries and documents the model focused on, the deep transformer architecture made it challenging to trace how specific features or interactions contributed to relevance scores. This opacity made it difficult to identify and fix ranking issues, understand model failures, or explain results to users.

The handling of very long documents remained challenging even with BERT. While BERT could process sequences up to 512 tokens, many documents were much longer, requiring document splitting or truncation strategies that might lose important information. The attention mechanism helped BERT focus on relevant passages, but if a document's most relevant content was split across multiple passages or fell outside the processed window, the system might miss it. This limitation was particularly problematic for long-form content like research papers, technical documentation, or legal documents.

The domain adaptation challenge meant that BERT models fine-tuned on general web search data might not perform well on specialized domains with different terminology, writing styles, or relevance criteria. Fine-tuning BERT for a specific domain required domain-specific training data, which might be limited for specialized areas. This made it challenging to apply BERT-based retrieval to domains like medicine, law, or specialized technical fields without investing in domain-specific training data collection and model adaptation.