Language Identification › Summary

Language identification is the foundation of multilingual data curation. Before any language model can learn from a document, the pipeline must know what language that document is in. The concepts from this chapter:

• Character n-gram models exploit the distinctive statistical fingerprints of each language's character sequences. They assign a document to the language whose model assigns the highest average per-n-gram log-probability. Laplace smoothing handles unseen n-grams, and length normalization makes scores comparable across documents of different sizes.
• Rank-based methods (Cavnar and Trenkle, 1994) build language profiles from the most frequent n-grams and measure the out-of-place distance between a document's n-gram ranks and each language profile. This approach is robust to short text and avoids probability smoothing.
• FastText-based systems like LID.176 use discriminative training to sharpen language boundaries and cover 176 languages with extremely high throughput, making them the standard choice for large-scale web filtering.
• CLD2 and CLD3 explicitly handle multilingual documents by estimating byte fractions per language, which is valuable when you need to identify and extract content from pages that mix languages.
• Confidence thresholds control the precision-coverage tradeoff: higher thresholds give cleaner language buckets at the cost of discarding more documents. Practical deployments use thresholds between 0.7 and 0.9 depending on whether corpus purity or volume is the priority.
• Short text degrades accuracy substantially below 10 to 20 words. Data curation pipelines should impose minimum document length requirements or apply separate handling for short-form content.
• Code-switching produces text that belongs to multiple languages. Document-level LangID assigns a single dominant language label; sentence-level approaches provide finer granularity at higher computational cost. Code-switching is a natural linguistic phenomenon, not a noise artifact, and should be treated accordingly.
• Multilingual pipelines face additional challenges around language balance, per-language quality calibration, and the compounding effect of misclassification at scale. Language upsampling with temperature-based sampling is a standard technique for giving low-resource languages equitable representation.
• Real-world curation often combines multiple LangID tools in ensemble, applies minimum document length requirements, and uses secondary quality checks like perplexity filtering to catch misclassifications that pass the primary filter.