Language Identification › Handling Multilingual Documents

Real web pages frequently contain text in multiple languages. A single Wikipedia article may have inline quotations in a foreign language. A news article may quote speakers in their native language. An e-commerce product page might have a description in English and user reviews in a mix of languages. Technical documentation might mix English code identifiers with prose in another language. These multilingual documents require more sophisticated handling than simple binary language classification.

Most LangID systems operate at the document level: they return a single language label for an entire document. This works well when a document is predominantly monolingual, but fails for multilingual documents where neither the majority-language label nor any single label captures the full picture.

Sentence-level LangID applies language detection to each sentence independently. This is more computationally expensive but provides finer granularity. For a data curation pipeline, sentence-level detection allows you to extract only the sentences in your target language from otherwise mixed documents, build a multilingual corpus with per-sentence language labels, and detect and study code-switching patterns at the sentence boundary level.

The tradeoff is that sentence-level detection is more affected by the short-text problem: individual sentences are shorter than full documents, so confidence tends to be lower. A sentence like "Thank you for your support" embedded in a French article will correctly classify as English, but a five-word French sentence at the beginning of a mostly-English document may not classify confidently enough to trust the label.

There is also the question of sentence tokenization quality. Sentence boundaries are not always obvious, especially in noisy web text. A sentence splitter may combine two sentences from different languages into one "sentence," creating an artificial code-switching scenario. The pipeline quality of document-level operations cascades into sentence-level ones.

CLD2's approach to multilingual documents estimates the fraction of bytes in each detected language. If a 1,000-character document has 600 characters classified as English and 400 as French, the tool reports English at 60% and French at 40%. This allows downstream filtering based on dominance thresholds: accept documents where the target language exceeds 70% of content, and reject those with more evenly split language mixes.

The byte fraction approach is more informative than a single label for mixed documents. A data pipeline building an English corpus can use byte fraction to filter documents where at least 80% of the content is English, retaining pages like English technical documentation that contains occasional French quotations while rejecting pages where the content is truly split.

The limitation is that byte fraction is an approximation. CLD2 classifies fixed-length chunks of text and aggregates the results, which means that language boundaries within a paragraph may not align with chunk boundaries. For fine-grained analysis of multilingual documents, more sophisticated segmentation approaches are needed.

For some downstream tasks, you want to segment a multilingual document into monolingual spans. The procedure is as follows: first, run sentence tokenization to split the document into sentences; second, run LangID on each sentence; third, merge consecutive sentences with the same language label into spans; finally, apply a minimum span length filter to discard isolated sentences in foreign languages.

This pipeline is more robust than naive per-sentence filtering because it handles the common case where a foreign-language sentence is surrounded by target-language content. A single French quotation embedded in an English article will form a short, isolated French span. If the pipeline requires spans of at least three sentences in the same language, that isolated quotation will be attributed to the surrounding English context rather than creating a separate French span.

Language-aware segmentation is also the foundation for building code-switching corpora, where you specifically want to identify and preserve the points at which language boundaries occur. Research on multilingual models, cross-lingual transfer, and code-switching phenomena all benefit from this finer-grained view.