Language Identification › Language Filtering in LLM Data Curation

Large language model training corpora are assembled from web crawls that span hundreds of languages. The data curation pipeline must decide which languages to include, at what quantities, and with what quality filters. Language identification is the mechanism that makes language-specific decisions possible, but the policy decisions around how to use those identifications are equally important and have large downstream effects on model capabilities.

Models like GPT-2, GPT-3, and most early LLMs were predominantly English. Their data pipelines applied a simple filter: keep only documents identified as English with high confidence. The threshold is typically 0.5 to 0.9 depending on the tool and the desired strictness.

This filter dramatically reduces corpus size relative to the raw web crawl. CommonCrawl data, after language filtering for English, retains roughly 50-60% of the original document count (varying by time period and filtering aggressiveness), but with much higher quality for downstream English language modeling. The discarded documents are either non-English or are mixed-language pages where the model cannot confidently assign English as the dominant language.

For an English-focused model, this is the right tradeoff. The model does not need Mandarin Wikipedia, French news articles, or German academic papers, and including them would dilute the signal from English text at the same number of training tokens. However, this policy implicitly assumes that the pipeline wants a monolingual model. Many modern deployment contexts require multilingual capabilities, which demands a different approach.

Multilingual LLMs like mBERT, XLM-R, mT5, and BLOOM handle dozens to hundreds of languages. Their curation pipelines face additional challenges that single-language pipelines never encounter.

Language balance is a central question. Web content is deeply skewed toward high-resource languages: English accounts for a large fraction of indexed web content, with a long tail of languages that have minimal web presence. If you train on data proportional to web prevalence, your model will be predominantly English with poor capabilities in most other languages. Most multilingual model developers upsample low-resource languages, typically using a temperature-based sampling scheme where the probability of sampling from language $L$ is proportional to $p_L^{1/T}$ where $p_L$ is the language's fraction of available data and $T > 1$ is a temperature parameter that flattens the distribution.

Cross-language contamination is another concern. A document misclassified as one language and included in that language's training split introduces noise. For high-resource languages, occasional contamination is tolerable. For a low-resource language where the training split might have only millions of tokens, contamination from a closely related language could represent a significant fraction of the training signal.

Language-specific quality filters add another layer of complexity. Quality heuristics developed for English, such as perplexity filtering against a reference language model, minimum word count requirements, and duplicate removal, must be calibrated separately for each language. The average sentence length differs across languages. The distribution of common words differs. A quality filter tuned on English may over-filter or under-filter text in other languages if applied naively.

The BLOOM model (Scao et al., 2022) addressed these challenges through careful language-specific data sourcing. Rather than relying entirely on web crawls, the team curated dedicated datasets for low-resource languages from organizations like Masakhane (African languages) and AmericasNLP (Indigenous American languages). This required language identification as both a filter and a verification step to confirm that sourced data matched its claimed language labels. Even professionally curated datasets sometimes contain mislabeled content, and applying LangID as a verification step on supposedly monolingual sources catches these errors before they enter the training pipeline.

At the scale of a CommonCrawl-based corpus, even a 1% misclassification rate introduces tens of millions of documents in the wrong language bucket. For a model training on 100 billion English tokens, a 1% contamination means 1 billion tokens of non-English content mixed into the English training split. This has a measurable effect on English language modeling quality, particularly on tasks involving rare vocabulary, idioms, or cultural knowledge that depends on consistent English exposure.

The effect on low-resource languages is potentially much larger in relative terms. A small language with only 100 million tokens of training data in the corpus, where 20% consists of misclassified documents from related languages, receives 80 million tokens of target-language signal and 20 million tokens of noise. The noise is not random: it comes from the most closely related languages, which means the model may learn to confuse the target language with its relatives, producing outputs that mix the languages in unnatural ways.

This is why production pipelines often use multiple LangID systems in ensemble, or use LangID as an initial filter followed by secondary quality checks like perplexity filtering against a reference language model. If a document passes the LangID filter but scores poorly under a language-specific perplexity model, it is likely either misclassified or low-quality content within the target language. Both cases benefit from additional scrutiny.

A robust production pipeline does not rely on a single LangID tool. Using two or more independent systems and requiring agreement before accepting a document reduces the false positive rate at the cost of some additional computation and potentially lower recall. The systems to combine depend on the target languages: FastText for broad coverage and speed, CLD2 for multilingual byte-fraction estimation, and lingua for high-precision short-text identification.

Secondary validation with a language-specific perplexity model is particularly effective at catching misclassified documents. A character-level or subword language model trained on high-quality text in the target language assigns low perplexity to grammatically coherent text in that language and high perplexity to text in other languages or low-quality content. Running this filter after LangID catches the documents that tricked the classifier but cannot fool a language model trained on the target language itself.

The computational cost of secondary validation is significant at scale. A perplexity-based filter requires a forward pass through a language model for every document, which is orders of magnitude more expensive than a FastText classification. In practice, this filter is applied only to documents that pass the primary LangID filter and are destined for lower-resource language buckets where quality is more critical.