XLM (2019) › The Problem

The traditional approach to multilingual natural language processing had relied on training separate models for each language or using translation-based approaches that required intermediate translation steps. This approach was resource-intensive and often resulted in inconsistent performance across languages, especially for low-resource languages with limited training data. Training separate models for each language meant that knowledge learned in one language couldn't be transferred to another, requiring substantial computational resources and data for each language pair.

The statistical translation-based approaches had additional limitations. They required intermediate translation steps, often involving English as a pivot language, which introduced errors and inefficiencies. A question in Spanish might need to be translated to English first, processed, then translated back to Spanish, with each translation step potentially introducing errors. These approaches struggled to capture cross-lingual semantic relationships and often required significant adaptation for each new language.

Monolingual language models like BERT had achieved remarkable success, but they were language-specific. A BERT model trained on English couldn't understand French or Spanish without retraining, and the learned representations didn't capture relationships between words in different languages. This limitation meant that models needed to be retrained from scratch for each language, wasting computational resources and preventing transfer of knowledge between languages.

For low-resource languages with limited training data, the problem was even more severe. High-resource languages like English had abundant training data, enabling sophisticated language models. Low-resource languages might have only a fraction of this data, making it difficult or impossible to train effective language models. The inability to transfer knowledge from high-resource to low-resource languages meant that the gap between language capabilities would continue to grow.

The field needed a solution that could enable language models to work across multiple languages simultaneously, sharing knowledge between languages while maintaining the performance benefits of large-scale pretraining. This required new training objectives that explicitly encouraged the model to learn cross-lingual representations and new architectures that could handle multilingual data effectively.