BLEU, chrF, and COMET: reading a translation score

A vendor that quotes a translation score is quoting a number that depends on a human reference translation, a test set, and a scoring configuration. The score measures how closely the system's output matches the reference, or how a trained model rates that output. It does not measure correctness directly, and it is not the share of sentences translated correctly. BLEU counts overlapping word sequences, chrF counts overlapping character sequences, and COMET uses a neural model trained on human quality judgments, so a program office comparing systems needs to know which one produced the number and under what settings.

Default tokenizer (13a)2.1 BLEU

Japanese tokenizer (ja-mecab)14.5 BLEU

Nothing about the translation changed. A score without its settings cannot be compared with another score.

Source: the sacreBLEU documentation's Japanese example, as cited in this post.

BLEU counts word sequences

BLEU was introduced in 2002 by Papineni, Roukos, Ward, and Zhu at the Association for Computational Linguistics (ACL 2002). The NLTK documentation describes the computation and cites that paper. It has three parts:

• Modified n-gram precision. An n-gram is a run of n consecutive words. Each reference word can be matched only once, so a system cannot raise its score by repeating a word the reference contains.
• A geometric mean. The default uses n-grams up to length four with equal weights.
• A brevity penalty. Output shorter than the closest reference is penalized, because short output can still score high precision.

Matt Post describes BLEU as a precision metric. The maximum n-gram length is almost always four, and because BLEU is computed over a whole corpus, zero n-gram counts are rare (Post, 2018).

chrF counts character sequences

chrF was introduced by Maja Popović at the 2015 Workshop on Statistical Machine Translation (WMT 2015). The sacreBLEU documentation lists the defaults in that tool: character n-grams up to order 6, no word n-grams, and a beta of 2, which sets the importance of recall relative to precision. Setting the word n-gram order to 2 gives chrF++, a variant that also counts word n-grams.

Because the unit is a character sequence, two words that differ only in an ending still share many of their character n-grams. A word-level metric counts them as different words.

COMET is a family of trained models

COMET, from Rei, Stewart, Farinha, and Lavie, is a framework for training evaluation models (2020 paper). It builds on a cross-lingual pretrained language model and takes three inputs: the source text, the system output, and a reference. The authors trained three models on three kinds of human judgment: Direct Assessments, Human-mediated Translation Edit Rate, and Multidimensional Quality Metric. In the paper's design, the estimator variants predict a quality score, and the ranking variant produces a similarity bounded between 0 and 1.

Agreement with human judgments was measured with a Kendall's Tau-like statistic on data from the 2019 WMT metrics task. The paper's argument for a learned metric is that neural translation often departs from word-for-word correspondence, so n-gram counting no longer gives reliable quality estimates.

The name COMET therefore covers a family of models. A score is meaningful together with the name of the model that produced it.

Why scores do not travel

A score belongs to a system, a test set, and a scoring configuration together. Change any of the three and the number moves.

• Number of references. Post scored one system on a WMT 2017 English-Finnish test set and got 22.04 BLEU with one reference and 25.25 with two.
• Reference processing. Differences in how references were tokenized and normalized changed BLEU by as much as 1.8 on one language pair, German-English, and averaged around 1.0.
• Tokenizer and language. The sacreBLEU documentation shows the same Japanese files scoring 2.1 under the default 13a tokenizer and 14.5 under a Japanese-specific one, and says the default tokenizer produces poor results for Japanese.
• Casing and smoothing. Post observes that many papers do not say whether they report cased or case-insensitive BLEU. The sacreBLEU documentation lists case-sensitive scoring and exponential smoothing as defaults, with a lowercase option and other smoothing methods available. Each choice is a different configuration.
• Metric and model. The three metrics measure different things, and the COMET models in the paper were each trained on a different kind of human judgment.

Post's paper calls BLEU a parameterized metric whose values can vary widely with its settings. His recommendation is to use the processing scheme that WMT uses and to be explicit about the settings. The tool he released prints a signature such as BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0, which records the number of references, casing, tokenizer, smoothing, and software version. The same README recommends sharing signatures in papers. The chrF signature has the same form, with fields for character and word order.

A score supports one comparison well: system A against system B on the same test set under the same settings.

What to ask

• Which metric produced the number, and for COMET, which model?
• Which test set was used, and how many references did it have?
• Can the vendor supply the full signature: references, casing, tokenizer, smoothing, and software version?
• Were the systems being compared scored on the same test set under the same settings?
• Was the tokenizer suited to the language, given that the same Japanese files scored 2.1 under one tokenizer and 14.5 under another?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from. The long read on offline machine translation goes further.

Sources

1. 1Bleu: a Method for Automatic Evaluation of Machine Translation · Association for Computational Linguistics (ACL 2002)
2. 2chrF: character n-gram F-score for automatic MT evaluation · Association for Computational Linguistics (WMT 2015)
3. 3COMET: A Neural Framework for MT Evaluation · Association for Computational Linguistics (EMNLP 2020)
4. 4A Call for Clarity in Reporting BLEU Scores · Association for Computational Linguistics (WMT 2018)
5. 5sacreBLEU project README · sacreBLEU project (Matt Post and contributors)
6. 6nltk.translate.bleu_score module documentation · NLTK Project