XLM (2019) › The Solution

XLM addressed these limitations by using a single model architecture trained on multilingual data that included text from many different languages. The model used a shared vocabulary and embedding space across all languages, allowing it to learn representations that captured semantic similarities between words and phrases in different languages. The key innovation was the use of translation language modeling, which trained the model to predict words in one language given context in another language, encouraging the model to learn cross-lingual representations.

The model's architecture was based on the transformer, with shared parameters across all languages. Unlike monolingual models that had separate parameters for each language, XLM used the same transformer layers for all languages, forcing the model to learn representations that worked across linguistic boundaries. This shared architecture meant that the model had to find commonalities between languages, learning abstract representations that captured universal linguistic patterns.

The shared vocabulary included subword tokens that were common across languages, as well as language-specific tokens for words that were unique to particular languages. Byte Pair Encoding (BPE) was applied to the concatenated corpora from all languages, creating a unified vocabulary where frequent subword units were shared across languages. This shared vocabulary enabled the model to recognize that "cat" in English and "chat" in French referred to similar concepts, even though they were spelled differently.

The key innovation in XLM was translation language modeling (TLM), a training objective that explicitly encouraged cross-lingual learning. In addition to the standard masked language modeling used in BERT, XLM used parallel text data where the same content was available in multiple languages. The model was trained to predict words in one language given context from both languages, forcing it to learn cross-lingual correspondences.

For example, given a parallel sentence pair "The cat sat on the mat" (English) and "Le chat s'est assis sur le tapis" (French), the model might be asked to predict "chat" in the French sentence given context from both languages. This training objective explicitly taught the model that "cat" and "chat" were related, encouraging the learning of cross-lingual representations that captured semantic similarities.

The training process for XLM involved several key components. First, the model was trained on large amounts of monolingual text from many different languages, learning to predict the next word in each language using causal language modeling. Second, the model was trained on parallel text data using translation language modeling, learning to predict words in one language given context in another. This cross-lingual training encouraged the model to learn representations that captured semantic similarities across languages.

XLM's architecture enabled several mechanisms for cross-lingual transfer. The shared parameters meant that improvements learned from high-resource languages could benefit low-resource languages. When the model learned to recognize grammatical patterns from English, these patterns could transfer to other languages with similar structures. The shared embedding space allowed the model to map similar concepts across languages, enabling knowledge transfer at the semantic level.

The model also used language embeddings to indicate which language each token belonged to, allowing it to learn language-specific patterns while maintaining cross-lingual representations. These language embeddings enabled the model to adapt its behavior based on the language, while the shared transformer layers ensured that knowledge could be transferred across languages.