How language identification works

Language identification (LID) is the step that decides which language a piece of text is in. A 2022 paper from Meta AI's No Language Left Behind (NLLB) project defines it as predicting the primary language for a span of text. A survey by Jauhiainen and colleagues calls it a key part of many text processing pipelines, because later steps generally assume the language of the input is known. Two openly released models show how the step works and where it fails: fastText's 176-language models and the NLLB project's 218-label model.

The task and the label

LID is a classification problem: given a string, pick a label from a fixed list. The survey, posted in 2018, says automatic LID has been researched for over fifty years. It points to a design choice that matters in practice. A classifier with no "unknown" output is forced to pick a language for every document, and for a language it has never seen, the choice may be arbitrary and unpredictable. Thresholding on the classifier's score is the easiest way to add an "unknown" result.

The NLLB team compared two designs and kept a single multiclass classifier with one score per language, then tuned a threshold for each language. The paper gives two reasons: changing one language's threshold does not change the others, and the design generalized better to out-of-domain data, which in their case meant moving from Wikipedia text to web text.

fastText's language ID models

The fastText documentation offers two models that recognize 176 languages. They were trained on Wikipedia, Tatoeba, and SETimes data and expect UTF-8 input. The full model, lid.176.bin, is 126MB and slightly more accurate. The compressed lid.176.ftz is 917kB. Both are released under the Creative Commons Attribution-Share-Alike License 3.0.

An October 2, 2017 fastText blog post by Edouard Grave explains the method. Each word is represented by its character n-grams, runs of a few letters, so a misspelled or unseen word still gets a usable representation. In the post's tutorial, on a small Tatoeba sample, accuracy rose from around 96.5 percent to above 98.5 percent when character n-grams of length 2 to 4 were added. That is a tutorial result, not a test of the released models. The underlying classifier comes from Joulin and colleagues, who report training on more than one billion words in less than ten minutes on a standard multicore CPU.

Counts differ by source. The NLLB paper describes fasttext as covering 187 languages, citing a 2018 paper, while the documentation lists 176 for its distributed models.

The NLLB model and its reported results

The NLLB team trained its own fasttext model on character n-grams of length 2 to 5, with all settings tuned on the FLORES-200 development set. Training data was heavily imbalanced: English made up 10.1 percent and Minangkabau in Latin script 0.06 percent. The paper reports micro F1 of 95.85 across all of FLORES-200, with Arabic varieties and Akan/Twi merged after linguistic analysis. On the 95 labels it shares with the CLD3 tool, it scored 98.5 against 97.0.

Web text was harder. Annotators checked web sentences in 74 low-resource languages whose prediction scores from an earlier model fell between 50 and 90 percent. On that challenge set, micro F1 fell to 79.14, against 64.41 for CLD3. The authors note that the set is biased by the earlier model. The project README lists the LID model under a CC-BY-NC 4.0 license, and the repository was archived on March 20, 2026.

Short texts and close languages

Length is the first failure point. The survey cites a 2006 study of four methods that all lost substantial accuracy on 25-character texts compared with 125-character texts. The NLLB authors also found that predictions held up better on long sentences.

Similar languages are the second. The NLLB confusion analysis names Akan and Twi, Dyula and Bambara, Faroese and Icelandic, Western Persian and Dari, and Bosnian and Croatian as the hardest pairs. Two Twi varieties can share identical sentence translations, and the authors suggest accepting multiple labels in future work.

Held-out test scores can also hide failures in the field. Caswell and colleagues trained models on up to 1,629 languages with quality on held-out test sets comparable to published models. Human-judged accuracy for the web-crawl corpora built with them was only around 5 percent for many lower-resource languages. They trace the errors to domain mismatch, class imbalance, language similarity, and insufficiently expressive models. Their filters and semi-supervised models raised median dataset precision from 5.5 percent to 71.2 percent.

Code-switching and mixed documents

The survey says most LID methods assume a document contains text from a single language. In one corpus it cites, 4 percent of the documents contained more than one language. Code-switching, the use of two or more languages interchangeably, breaks that assumption even inside a single sentence.

The first shared task on code-switched data, held in 2014, labeled each word in Twitter data for four pairs: Modern Standard Arabic and dialectal Arabic, Mandarin and English, Nepali and English, and Spanish and English. Seven teams submitted 42 runs. Word-level identification scored lowest for the closely related Arabic pair and highest for Spanish-English and Nepali-English.

The NLLB data pipeline handles mixing by discarding. It labels each web paragraph, splits it into sentences, and labels each sentence again. A sentence whose label does not match its paragraph is dropped, as in the paper's example of "Best véto ever!", labeled French as a document and Czech as a sentence.

What to ask

• Which model and version is in use, how many labels does it support, and is the needed language on the list?
• Was accuracy reported on clean benchmark sentences such as FLORES-200, or on text like the program's own documents?
• How does the system perform on strings of about 25 characters, such as titles, captions, and chat lines?
• Can it return "unknown," or more than one label for a close pair such as Bosnian and Croatian?
• Does it label mixed-language text by document, sentence, or word, and what happens to a sentence that disagrees with its document?
• Which license covers the model: the Creative Commons Attribution-Share-Alike 3.0 license for fastText's models, or CC-BY-NC 4.0 for the NLLB model?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.

Sources

1. 1No Language Left Behind: Scaling Human-Centered Machine Translation · arXiv (NLLB Team, 2022; version 3)
2. 2Automatic Language Identification in Texts: A Survey · arXiv (Jauhiainen, Lui, Zampieri, Baldwin, Lindén; version 2, November 2018)
3. 3Language identification · fastText documentation (Facebook)
4. 4Fast and accurate language identification using fastText · fastText blog (Edouard Grave), October 2, 2017
5. 5Bag of Tricks for Efficient Text Classification · arXiv (Joulin, Grave, Bojanowski, Mikolov, 2016)
6. 6fairseq, NLLB branch: No Language Left Behind (README) · Meta AI Research (facebookresearch on GitHub)
7. 7Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus · arXiv (Caswell, Breiner, van Esch, Bapna), accepted to COLING 2020
8. 8Overview for the First Shared Task on Language Identification in Code-Switched Data · Association for Computational Linguistics (First Workshop on Computational Approaches to Code Switching, 2014)

Where this shows up in our work

• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.