Named entity recognition across languages

Named entity recognition (NER) finds the names in a text and labels each one by type, such as person, organization, or location. The CoNLL-2003 shared task gives the standard example: "[ORG U.N.] official [PER Ekeus] heads for [LOC Baghdad]" (Tjong Kim Sang and De Meulder). The best system in that task scored 88.76 F1 on English news. Scores fall in other languages, and among the lowest are languages written without spaces between words or outside the Latin script. In the XTREME benchmark, one multilingual model trained on English alone scored 84.7 F1 in English and 1.3 in Thai (Hu and colleagues).

The task and how it is scored

The CoNLL-2002 task paper defines named entities as phrases that contain the names of persons, organizations, locations, times, and quantities (Tjong Kim Sang, 2002). Both CoNLL tasks used four types: persons (PER), locations (LOC), organizations (ORG), and miscellaneous names (MISC) that fit none of the others. The data has one word per line, each with a tag that marks whether the word begins or continues a name and of which type.

Scoring is strict. Precision is the share of names found by the system that are correct. Recall is the share of names in the data that the system finds. A name counts as correct only if it is an exact match of the annotated name. F1 is the balanced combination of the two. Under that rule, a system that tags only "Real" in "Real Madrid" gets no credit for that name.

The CoNLL-2002 and CoNLL-2003 shared tasks

CoNLL-2002 covered Spanish and Dutch. The Spanish data came from EFE News Agency wire stories of May 2000, and the Dutch from four 2000 editions of the Belgian newspaper De Morgen. Twelve systems took part. The best, a boosted decision tree system by Carreras and colleagues, scored 81.39 F1 on Spanish and 77.05 on Dutch.

CoNLL-2003 covered English, from Reuters stories of 1996, and German, from the Frankfurter Rundschau of 1992. Sixteen systems took part. The best, a combined system by Florian and colleagues, scored 88.76 on English and 72.41 on German. A simple baseline scored 59.61 and 30.30. The German result had precision of 83.87 percent but recall of 63.71 percent, so the system missed more than a third of German names. Eleven of the 16 teams used information beyond the training data, and all of those used gazetteers, which are lists of known names. A majority vote of five systems reached 90.30 on English and 74.17 on German.

Scores across many languages

Pan and colleagues built automatic "silver-standard" name annotations for 282 languages in Wikipedia, the data later benchmarks call WikiANN. The labels were transferred from English through cross-language links. On Wikipedia text, F1 was generally above 80 where a language had more than 10,000 name mentions and dropped significantly below 250. Three groups scored low: languages with fewer than 2,000 mentions; languages with no spaces between words, such as Chinese, Thai, and Japanese; and languages not written in Latin script, such as Tamil, Telugu, Kannada, and Malayalam. On human-annotated news and forum text from the DARPA LORELEI program, models trained on silver data scored 6.5 to 17.6 points below models trained on gold data. Bengali scored 61.6 with gold training and 44.0 with silver.

XTREME used WikiANN across 40 languages, testing models trained only on English data. For the XLM-R model:

Language	Script, as XTREME lists it	F1
English	Latin	84.7
German	Latin	78.8
Russian	Cyrillic	69.1
Urdu	Perso-Arabic	56.4
Arabic	Arabic	53.0
Mandarin	Chinese ideograms	33.1
Japanese	Ideograms	23.2
Thai	Brahmic	1.3

The authors found that the mBERT model did better on Latin-script data than on Chinese or Japanese ideograms. For Georgian, the model recognized entities written in Latin script well and entities in Georgian script less well.

African languages and MasakhaNER

MasakhaNER, published in 2021, annotated local news in ten African languages for PER, LOC, ORG, and dates, which replaced MISC. The annotators were native speakers from the same regions as the news sources. The authors note that Amharic's Fidel script has no capitalization. The mBERT model scored 0.0 F1 on Amharic because it had not seen the script during pretraining. XLM-R-large, whose pretraining included Amharic, scored 76.18 on Amharic, 90.54 on Hausa, and 67.90 on Wolof, with an average of 80.49 across the ten. Between 33.40 and 72.94 percent of test entities did not appear in the training data, and with a basic model the languages with fewer unseen entities, such as Hausa, Igbo, and Swahili, scored higher. Moving from Wikipedia-trained models to news without in-language training gave less than 40 F1 for all five languages tested.

Why names in non-Latin scripts are harder

Capital letters are a strong clue in English. Khaled Shaalan's 2014 survey of Arabic NER explains what happens without them. Capitalization does not distinguish names in Arabic script, and most Arabic proper nouns look the same as common nouns and adjectives. The name Ashraf can also be read as a verb meaning supervised or as a superlative meaning most honorable. Prefixes and suffixes, such as the conjunction for "and," attach directly to names. Some Arabic systems borrow capitalization from the English gloss of each word.

The other studies point the same way. Pan and colleagues list missing word spaces and non-Latin scripts among the causes of low scores. XTREME found weaker transfer across scripts, and MasakhaNER found a model blind to a script it had never seen.

What to ask

• Was the F1 measured on news like CoNLL, on Wikipedia silver data like WikiANN, or on human-annotated text like the target documents?
• Was the model trained on the target language, or only on English and transferred?
• Which entity types does the score cover: PER, ORG, LOC, MISC, geopolitical entities, or dates?
• Does the score require an exact match of each name's boundaries?
• Did the model's pretraining include the script, given mBERT's 0.0 on Amharic?
• For Arabic, how does the system separate names from ordinary words and attached prefixes?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.

Sources

1. 1Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition · CoNLL-2002, ACL Anthology
2. 2Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition · CoNLL-2003 at HLT-NAACL 2003, ACL Anthology
3. 3Cross-lingual Name Tagging and Linking for 282 Languages · ACL 2017, Vancouver (ACL Anthology)
4. 4XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization · arXiv (version 5, September 4, 2020); ICML 2020
5. 5MasakhaNER: Named Entity Recognition for African Languages · Transactions of the Association for Computational Linguistics, Volume 9 (2021)
6. 6A Survey of Arabic Named Entity Recognition and Classification · Computational Linguistics, Volume 40, Issue 2 (June 2014)

Where this shows up in our work

• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.