What is FLORES-200?

FLORES-200 is a human-translated evaluation set for machine translation: the same 3,001 English sentences, professionally translated into 204 language-script combinations, so that any translation direction can be scored on identical content. Meta AI released it in 2022 with the NLLB-200 models, and the No Language Left Behind paper describes how it was built. Searchers write it as FLORES-200, Flores 200, or flores200, and the name also covers its predecessor FLORES-101 and its successor FLORES+. For anyone reading a vendor's translation scores, the benchmark is the hidden variable: a number means nothing until the sentence set, the split, the metric, and the tokenizer behind it are known.

From FLORES-101 to FLORES-200

The first FLORES sets, released in 2019, covered a handful of low-resource pairs with English. The FLORES-101 paper, posted to arXiv on June 6, 2021 and later published in the Transactions of the Association for Computational Linguistics (volume 10, pages 522-538), scaled the idea to 101 languages. The authors argued that existing benchmarks either covered few low-resource languages, were limited to narrow domains, or were built with semi-automatic methods and so were low quality. Their answer was a single set of English sentences translated by professionals into every language, which they called a many-to-many benchmark because the translations are aligned across all languages, not only to English.

FLORES-200 kept the English source sentences and roughly doubled the coverage. The NLLB paper's Section 4.1 describes it as a significant extension of FLORES-101 and says the 3,001 sentences were professionally translated into 200-plus languages. The FLORES-200 README adds a point that matters for anyone comparing old and new scores: some languages were translated from Spanish, French, Russian, or Modern Standard Arabic rather than from English, and five languages that were already in FLORES-101 (Quechua, Aymara, Cebuano, Kimbundu, and Umbundu) were improved, so their FLORES-101 and FLORES-200 results differ.

The FLORES repository changelog dates the two releases: FLORES-101 on June 4, 2021, and FLORES-200, together with NLLB-Seed, NLLB-MD, and Toxicity-200, on June 30, 2022.

3,001 sentences from 842 Wikimedia articles

The NLLB paper's "Flores-200 at a glance" section gives the headline numbers: translations from 842 distinct web articles, totaling 3,001 sentences, with sentences approximately 21 words long on average. Roughly one third of the sentences come from each of three English Wikimedia projects: Wikinews, a collection of news articles; Wikijunior, written for children; and Wikivoyage, a travel guide. The FLORES-101 paper's Table 1 breaks this down to 993 sentences from 309 Wikinews articles, 1,006 sentences from 284 Wikijunior articles, and 1,002 sentences from 249 Wikivoyage articles, and says the authors manually labeled every sentence with one of 10 sub-topics such as crime, disasters, entertainment, science, politics, and travel.

The sampling kept some document context. The FLORES-101 paper says one paragraph was selected per document, from the beginning, middle, or end of the article, and that about 3 sentences were selected from each document, which the authors say allows document-level evaluation. The articles are also rich in metadata: 40 percent contain hyperlinks to other pages and 66 percent contain images. The facebook/flores dataset card on Hugging Face lists the fields each row carries: an id, the sentence, the URL of the source English article, the domain, the topic, and two flags for whether the article has an image and whether the sentence has a hyperlink.

Dev, devtest, and the hidden test set

FLORES splits its 3,001 sentences three ways. The FLORES-101 paper's Table 1 gives the sizes: dev has 997 sentences from 281 articles, devtest has 1,012 sentences from 281 articles, and test has 992 sentences from 280 articles. Documents, not individual sentences, were assigned to splits, so sentences from one article stay together.

The three splits are not equally public. The NLLB paper says the team releases the full text of dev and devtest and keeps the test set hidden through an evaluation server. The facebook/flores card lists dev at 997 examples and devtest at 1,012 and describes test as hidden. Most published scores, including the NLLB paper's own Table 39, are on devtest. When a vendor quotes "FLORES-200" without naming a split, the question to ask is which one, and whether dev was used for tuning before devtest was scored.

The hidden split has not moved with the benchmark's maintenance. The FLORES+ dataset card says the separate blind test set was developed by Meta, is not managed by the Open Language Data Initiative, and is not part of the FLORES+ repository.

The 204 language codes and script tags

Every FLORES-200 language is named by a code made of a three-letter ISO 639-3 language code, an underscore, and a four-letter ISO 15924 script code. The README gives eng_Latn for English in Latin script, pbt_Arab for Southern Pashto in Arabic script, and ace_Arab and ace_Latn for Acehnese written in Arabic and in Latin script. The NLLB-200 model cards use the same codes to select source and target languages, which is why the codes appear in so many search queries.

The README's language table has 204 rows. Eight language codes appear twice because they are written in more than one script: Acehnese (ace_Arab, ace_Latn), Banjar (bjn_Arab, bjn_Latn), Central Kanuri (knc_Arab, knc_Latn), Kashmiri (kas_Arab, kas_Deva), Minangkabau (min_Arab, min_Latn), Tamasheq (taq_Latn, taq_Tfng), Chinese in Simplified and Traditional script (zho_Hans, zho_Hant), and Modern Standard Arabic, which appears in Arabic script (arb_Arab) and romanized (arb_Latn). That is how 204 entries cover roughly 200 languages, and why the NLLB paper describes the model as covering 200 languages while the benchmark covers 204. The README also carries a table mapping FLORES-101 codes to FLORES-200 codes, since FLORES-101 used shorter codes without script tags.

How translation quality was controlled

Both papers describe the workflow in detail, and the details explain why the set is trusted more than web-mined test data. The FLORES-101 paper says the authors ran a pilot in which 100 sentences in 8 language pairs were translated by several language service providers (LSPs) and evaluated by another, then chose the two LSPs that produced the highest-quality translations. Each LSP's translations went through editing, automated checks, and a comparison against commercial translation engines to catch copying; the paper defines a translation as a suspected copy when its sentence-level spBLEU against one engine exceeds the other by more than 20 points and exceeds 50.

An evaluation LSP then scored each language on a 0 to 100 Translation Quality Score. The score counted errors of grammar, punctuation, spelling, capitalization, addition or omission of information, mistranslation, unnatural translation, untranslated text, and register, each rated minor, major, or critical. The paper states "we set the acceptable translation quality score to 90%" and that languages below it went back for re-translation. The paper's Table 2 reports that 45 languages required re-translation, with an average of 1 round, a maximum of 3, and an average of 35 days to re-translate. About 50 percent of languages scored above 95, and some candidate languages never met the bar and were left out.

FLORES-200 added requirements for the people. The NLLB paper says translators had to be native speakers educated in the target language with high English fluency, with 2 to 3 years of translation experience if they held a degree in translation or linguistics and 3 to 5 years if not, and had to pass a translation test every 18 months. Reviewers had to be native speakers with, typically, a translation degree and at least 5 years of experience. A new alignment phase had translators and reviewers agree on script, standardization, spelling, borrowed terms, and neologisms before work began, and an initial batch of 200 sentences was translated and reviewed before the full set. The paper calls these stringent standards and says that for many languages finding a qualified reviewer was very challenging, so the rules were relaxed to accept reviewers with general language degrees or more than 10 years of commercial experience. Figure 5 of the paper plots the final quality score for each language, with 90 percent as the minimum.

spBLEU, chrF++, and how scores are reported

BLEU depends on how text is split into words, and the FLORES-101 paper spends its Section 5 on this problem. Moses tokenization, the default in sacreBLEU, works for European languages but not for Burmese or Khmer, which do not separate words with spaces, and Hindi, Tamil, and Japanese each have their own community tokenizers, which makes scores hard to compare. The authors' answer was SentencePiece BLEU, or spBLEU: a single SentencePiece model with 256,000 tokens trained on monolingual data from all the FLORES-101 languages, with temperature upsampling so low-resource languages are represented. Both the system output and the reference are tokenized with that model and BLEU is computed on the pieces. The paper's Table 6 checked spBLEU against WMT 2020 human rankings for English to Pashto, Russian, and Chinese and found it ranked the Pashto systems identically to the human evaluators.

The NLLB paper extended the tokenizer to SPM-200, trained on 100 million sampled sentences with a vocabulary of 256,000 and a sampling temperature of 5, and released it with FLORES-200. The same paper is explicit that spBLEU inherits BLEU's weakness: "A major downside of BLEU is that it is tokenization-dependent." A spBLEU score computed with SPM-100 and one computed with SPM-200 are not the same metric, and the paper's Table 30 uses the FLORES-101 tokenizer for comparability with earlier systems while footnote 28 notes minor differences between the two models on FLORES-101 languages. The sacreBLEU README exposes both as the flores101 and flores200 tokenizers, downloading the model file on first use.

The NLLB team's primary metric was not spBLEU at all. Section 7.1 of the paper says the team primarily evaluates with chrF++, a character n-gram F-score over n-grams of length 1 to 6 plus word unigrams and bigrams, computed with sacreBLEU's settings, and uses BLEU and spBLEU only where needed to compare against published work. A reader who sees "FLORES-200 score" should therefore ask three questions: which split, which metric, and for spBLEU which SentencePiece model.

License and where to download it

The FLORES repository's Licenses section lists FLORES-200, NLLB-Seed, and Toxicity-200 under CC-BY-SA 4.0 and NLLB-MD under CC-BY-NC 4.0. The facebook/flores dataset card repeats CC BY-SA 4.0 and asks users to cite the NLLB paper, the FLORES-101 TACL article, and the 2019 FLORES paper. The share-alike condition travels with derived data, which matters if an organization intends to build its own test set from FLORES sentences.

Access is gated rather than open. The facebook/flores card says users must accept conditions before accessing files, to protect the integrity of the evaluation, and the FLORES+ card asks users not to redistribute the files publicly without protecting them from automatic scraping and states that the data "should not be used as training data." The concern is contamination: a model trained on the benchmark sentences would score well on them for the wrong reason.

The original home is no longer maintained. GitHub's banner on the facebookresearch/flores repository says the owner archived it on November 21, 2023, and the README says the repository is no longer being updated and that newer versions of FLORES and NLLB-Seed are managed by the Open Language Data Initiative.

FLORES+ and the Open Language Data Initiative

The Open Language Data Initiative (OLDI) is a community effort to expand open datasets for machine translation and natural language processing. The site lists nine organizers from the University of Pretoria, George Mason University, the Common Crawl Foundation, Google, Meta FAIR, Apple, Johns Hopkins University, and McGill University, and describes FLORES+ as "an extended and improved version of the FLORES-200 dataset" covering over 200 languages. A companion dataset, OLDI-Seed, extends NLLB-Seed with 6,193 sentences from English Wikipedia translated into many languages.

The FLORES+ dataset card on Hugging Face gives the current shape: version 5.0, 230 language varieties, 997 dev and 1,012 devtest sentences per language, released under CC BY-SA 4.0. Each row carries the ISO 639-3 code, the ISO 15924 script code, a Glottolog code, a variant tag, the source article URL, domain, topic, the image and hyperlink flags, and a last_updated field naming the FLORES+ version in which the row last changed. That last field is how a reader can tell whether a language's sentences were revised after the Meta release. Contributions follow the Developer Certificate of Origin and OLDI's guidelines, and the site mentions a shared task at WMT26.

The GitHub home of FLORES+ is also closed. The openlanguagedata/flores repository was archived on November 12, 2024, and its README directs users to the Hugging Face repository for the most recent version. The practical result is that "FLORES-200" now names a frozen 2022 snapshot and "FLORES+" names the version that receives corrections.

Known criticisms

Four limitations appear in the public record.

Some languages were translated badly. A 2024 paper, Correcting FLORES Evaluation Dataset for Four African Languages by Abdulmumin and colleagues, had native speakers review the Hausa, Northern Sotho, Xitsonga, and isiZulu sets. The paper's Table 1 reports that 632 of the 997 Hausa dev sentences (63.4 percent) and 70 of the 1,012 devtest sentences (6.9 percent) were corrected, along with 190 (19.1 percent) and 226 (22.3 percent) for isiZulu, 67 and 62 for Northern Sotho, and 83 devtest sentences for Xitsonga. For Hausa, the authors write that a significant part of the translations were suspected to have been automatically generated. The same paper lists issues raised on the FLORES GitHub repositories, including near-identical translations across several Arabic dialects, Mandarin used for Cantonese, and unspecified orthography for Lombard and Sardinian. The NLLB paper's own quality section reports that mistranslation and unnatural translation were still the most common errors found, driven by borrowing from dominant regional languages and by low standardization that produced spelling inconsistencies.

The domain is Wikimedia. Every sentence comes from Wikinews, Wikijunior, or Wikivoyage. The NLLB paper's Section 8.4 evaluates on 238 directions from 8 other benchmarks precisely to test domains other than the Wikimedia text of FLORES-200, and notes elsewhere that the devtest set contains only 3 toxic items because of that domain. A FLORES score is evidence about encyclopedic and travel prose, not about maintenance manuals, chat messages, or scanned forms.

The unit is the sentence. Systems are scored one sentence at a time against one reference. The FLORES-101 paper's own comparison table claims document-level evaluation is possible because sentences come from the same articles, but the released splits, metrics, and leaderboards are sentence level.

The sentences are now widely known. Because dev and devtest are public and have been downloaded for years, the FLORES+ card's instruction not to use the data for training is a request, not a guarantee. The hidden test split exists for this reason, and the tokenizer fairness study by Petrov and colleagues shows a second use of the public text: as a parallel corpus for measuring how many tokens each language costs, a purpose the benchmark was never designed for.

What to check

• Which split was scored, dev or devtest, and was dev used for tuning before devtest was reported?
• Which metric is quoted, chrF++, BLEU, or spBLEU, and for spBLEU which SentencePiece model: SPM-100 from FLORES-101 or SPM-200 from FLORES-200?
• Which version of the sentences was used: the archived 2022 FLORES-200 release or a FLORES+ version, and for which languages did FLORES+ revise the text?
• Does the language appear in the record of corrected sets, such as Hausa or isiZulu, where a score on the original data measures the reference as much as the system?
• Does the program's content look like Wikinews, Wikijunior, or Wikivoyage prose, or does it need a test set in its own domain?
• Was the model trained on data that could include FLORES sentences, and is the vendor willing to score on the hidden test split or on a held-out set the program supplies?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.

Questions people ask

What is FLORES-200?

FLORES-200 is a human-translated evaluation benchmark for machine translation released by Meta AI with the NLLB-200 models in 2022. The No Language Left Behind paper describes it as 3,001 sentences sampled from English Wikinews, Wikijunior, and Wikivoyage, drawn from 842 articles, and professionally translated into 204 languages. Because every language holds the same sentences, any translation direction can be scored on the same content.

What is the difference between FLORES-101 and FLORES-200?

FLORES-101, described in a 2021 arXiv paper and a 2022 TACL article by Goyal and colleagues, translated 3,001 English Wikipedia sentences into 101 languages. FLORES-200, described in the NLLB paper, kept the same English sentences and roughly doubled the coverage to 204 language-script entries, added an alignment phase to the translation workflow, and allowed some languages to be translated from Spanish, French, Russian, or Modern Standard Arabic instead of English.

How many sentences are in FLORES-200?

The dataset holds 3,001 sentences in total, averaging about 21 words each, according to the NLLB paper. The FLORES-101 paper gives the split sizes: dev has 997 sentences from 281 articles, devtest has 1,012 sentences from 281 articles, and test has 992 sentences from 280 articles. The Hugging Face dataset card for facebook/flores lists the same 997 and 1,012 counts for dev and devtest.

What are dev, devtest, and test in FLORES?

The three splits serve different purposes. Dev is for tuning a system, devtest is the public set most papers report on, and test is held back. The NLLB paper says the team releases the full text of dev and devtest and keeps the test set hidden through an evaluation server. The FLORES+ dataset card says the blind test set was developed by Meta and is not managed by the Open Language Data Initiative.

What do FLORES-200 language codes like eng_Latn mean?

Each code pairs an ISO 639-3 language code with an ISO 15924 script code, separated by an underscore. The FLORES-200 README gives eng_Latn for English in Latin script and pbt_Arab for Southern Pashto in Arabic script. Eight language codes appear twice because they are written in two scripts, for example ace_Arab and ace_Latn for Acehnese and zho_Hans and zho_Hant for Chinese, which is why 204 entries cover roughly 200 languages.

What is spBLEU?

spBLEU is SentencePiece BLEU. The FLORES-101 paper proposes it as BLEU computed after tokenizing both system output and reference with a single SentencePiece model trained on monolingual data from all the FLORES languages, with 256,000 tokens. The point is to score every language with the same tokenizer instead of language-specific ones. The NLLB paper released SPM-200, a 200-language version, and sacreBLEU exposes both as the flores101 and flores200 tokenizers.

What license does FLORES-200 have?

The FLORES repository on GitHub lists FLORES-200 under CC-BY-SA 4.0, alongside NLLB-Seed and Toxicity-200, while NLLB-MD is under CC-BY-NC 4.0. The Hugging Face dataset cards for facebook/flores and for FLORES+ both state CC BY-SA 4.0. The FLORES+ card adds that the data is for evaluation and should not be used as training data, and the repository is gated so users must accept conditions before download.

What is FLORES+?

FLORES+ is the maintained successor to FLORES-200. The OLDI website describes it as an extended and improved version of the FLORES-200 dataset covering over 200 languages. The Hugging Face card for openlanguagedata/flores_plus gives 230 language varieties at version 5.0, with 997 dev and 1,012 devtest sentences per language. The README of the original FLORES-200 GitHub repository points users to the Open Language Data Initiative for newer versions.

Is FLORES-200 reliable for every language?

Not uniformly. A 2024 arXiv paper by Abdulmumin and colleagues corrected the Hausa, Northern Sotho, Xitsonga, and isiZulu sets and reported that 632 of the 997 Hausa dev sentences, 63.4 percent, needed correction, with a significant part suspected to be automatically generated. The same paper notes GitHub issues about near-identical translations across Arabic dialects. The NLLB paper itself reports that mistranslation and unnatural translation were the most common errors found during quality review.

Is FLORES-200 sentence level or document level?

The unit of translation is the sentence. The FLORES-101 paper says about 3 sentences were selected per article, which allows document-level evaluation in principle, and the dataset ships with article URLs, domains, and topics. In practice, systems are scored sentence by sentence against a reference, and the content comes only from Wikinews, Wikijunior, and Wikivoyage, so a score says nothing about manuals, forms, or messages.

Sources

1. 1The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation · arXiv (Goyal, Gao, Chaudhary, Chen, Wenzek, Ju, Krishnan, Ranzato, Guzmán, Fan), June 6, 2021
2. 2The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation · Transactions of the Association for Computational Linguistics, volume 10, pages 522-538 (2022)
3. 3No Language Left Behind: Scaling Human-Centered Machine Translation · arXiv (NLLB Team, Meta AI), v3 of August 25, 2022
4. 4FLORES (repository) · Meta AI Research (facebookresearch on GitHub)
5. 5FLORES-200 (README) · Meta AI Research (facebookresearch on GitHub)
6. 6facebook/flores (dataset card) · Meta AI Research, hosted on Hugging Face
7. 7openlanguagedata/flores_plus (dataset card) · Open Language Data Initiative, hosted on Hugging Face
8. 8Open Language Data Initiative · Open Language Data Initiative
9. 9openlanguagedata/flores (archived repository) · Open Language Data Initiative on GitHub
10. 10sacreBLEU project README · sacreBLEU project (Matt Post and contributors)
11. 11Correcting FLORES Evaluation Dataset for Four African Languages · arXiv (Abdulmumin, Mkhwanazi, Mbooi, Muhammad, Ahmad, Putini, Mathebula, Shingange, Gwadabe, Marivate), v2 of October 5, 2024
12. 12Language Model Tokenizers Introduce Unfairness Between Languages · arXiv (Petrov, La Malfa, Torr, Bibi), v2 of October 20, 2023; NeurIPS 2023

Where this shows up in our work

• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.