XLM (2019) › Limitations
History of Language AIAll chapters XLM (2019) › LimitationsOne section · Sections in this chapter5 / 7 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Despite its significant contributions, XLM faced several limitations that would shape subsequent research directions. The model's performance varied significantly across language pairs, with stronger performance for languages that were typologically similar or had abundant training data. Languages that were very different from those in the training data, or languages with limited representation in the training corpus, showed weaker cross-lingual transfer.
The model's reliance on parallel text data for translation language modeling was also a limitation. While parallel data enabled strong cross-lingual learning, such data was not available for all language pairs, and creating parallel corpora was expensive and time-consuming. Language pairs without parallel data couldn't benefit from the translation language modeling objective, limiting the model's applicability to language pairs with existing translation resources.
The shared vocabulary approach, while effective for related languages, sometimes struggled with languages that used different writing systems or had very different morphological structures. Languages with non-Latin scripts or complex morphology might not benefit as much from the shared subword vocabulary, limiting the effectiveness of cross-lingual transfer.
The model's performance on low-resource languages, while improved compared to monolingual approaches, still lagged behind high-resource languages. The cross-lingual transfer helped but didn't completely eliminate the gap between languages with abundant training data and those with limited resources. This limitation highlighted the continuing importance of having adequate training data for each language.
The computational requirements for training multilingual models were also substantial. Training on multiple languages required processing much more data than monolingual models, increasing training time and computational costs. The need for parallel text data added complexity to the data preparation process, requiring alignment and preprocessing of multilingual corpora.