What is sacreBLEU?

sacreBLEU is a Python tool and library for computing machine translation scores that another team can reproduce. Matt Post introduced it in A Call for Clarity in Reporting BLEU Scores, a paper at the Third Conference on Machine Translation (WMT) in Brussels in 2018, after measuring how much a BLEU score for the same system output could move depending on hidden settings. The tool expects detokenized output, applies its own tokenization to output and reference, downloads standard test sets so the user never touches the references, and prints a signature string that names every parameter behind the score. The name plays on a French exclamation, and the motivation section of the project README opens with the problem the tool solves: "Comparing BLEU scores is harder than it should be."

Why sacreBLEU exists

BLEU, defined by Papineni, Roukos, Ward, and Zhu in a 2002 ACL paper, became the dominant automatic metric for machine translation because it is cheap, language independent in principle, and correlates reasonably with human judgment. Post's paper is not about whether BLEU is a good proxy for quality. The paper's narrower claim is that the field had a reporting problem: people say "the" BLEU score, but BLEU is a parameterized metric whose values vary with the number of references, the length penalty in multi-reference settings, the maximum n-gram length, smoothing for zero counts, and, most of all, how the reference translation is tokenized and normalized before scoring.

The paper quantified the damage with a single WMT 2017 system, online-B, scored across 12 language pairs under four processing schemes. "basic" used the Moses tokenizer supplied by the user; "split" added compound splitting; "unk" mapped rare words to an unknown token; and "metric" applied only the metric-internal tokenization of WMT's official mteval-v13a.pl script. Table 1 of the paper shows the range between schemes, which excludes the unk scheme, reaching 1.8 BLEU for German to English and averaging around 1.0, which the author notes is larger than many reported gains between systems. Case handling added more variance: uncased scores for the same output run from 0 to 1.3 points higher than cased scores in the table. A second example in the paper shows reference count alone moving English to Finnish from 22.04 with one reference to 25.25 with two.

The paper then tried to recover the configuration used by seven influential papers and found most were unclear about whether their tokenization was user supplied or metric internal, and whether scores were cased. The conclusion was that scores across papers could not be directly compared, which impedes replication, and that the fix had to be mechanical rather than a plea for better footnotes.

What the tool does

Post's paper describes three design choices, and the README restates them. First, sacreBLEU expects detokenized outputs and applies its own metric-internal preprocessing, producing the same values as WMT. The paper draws the pipeline as a figure in which the user controls preprocessing of the source and postprocessing of the output, and the metric alone touches the reference. The paper's proposal is that the community use only metric-supplied reference tokenization when sharing scores, and sacreBLEU is the enforcement mechanism.

Second, the tool downloads and stores the WMT 2008 to 2018 and IWSLT 2017 test sets, which the paper calls a "protective layer" between the references and the user. The README's examples show the workflow: sacrebleu -t wmt17 -l en-de --echo src prints the source side of a test set to translate, and sacrebleu -t wmt17 -l en-de -i output.detok.txt scores the detokenized output against the stored reference. The README says downloading is triggered when a test set is requested and that sacrebleu --list shows the available sets. The paper also flags a trap this solves: WMT 2014 English to German exists in two versions, one of 2,737 sentences used for the official evaluation and a restored version of 3,004 sentences, and many researchers did not know which they were using. sacreBLEU names them wmt14 and wmt14/full.

Third, the tool prints a version string recording the settings. The README lists further features that followed: the official WMT mteval-v13a.pl behavior, support for Japanese and Chinese tokenizers, chrF, chrF++, and TER, and paired significance tests through bootstrap resampling and approximate randomization.

The paper contrasts this with what existed. Moses' multi-bleu.perl requires user-supplied preprocessing, so the paper rules it out. MultEval explicitly advocates user tokenization. mteval-v13a.pl uses metric-internal processing and is what WMT uses, but it requires the data to be wrapped in XML. Nematus' multi-bleu-detok.perl removes the XML requirement but still makes the user handle the references. sacreBLEU's contribution was to combine metric-internal tokenization, reference management, and a recorded signature in one command.

How to read a sacreBLEU signature

The signature is the part of sacreBLEU that a reader of a proposal or paper will meet most often. The README's current example is:

nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0

Each field is a key and a value separated by a colon, and the fields are separated by vertical bars. In the README's examples and the BLEU source, the fields mean the following.

The effective-order field is the least obvious. The BLEU class source describes the effective_order parameter as stopping the inclusion of n-gram orders for which precision is 0, with a default of False, so a corpus-level BLEU signature normally shows eff:no. Effective order matters for sentence-level scoring, where a short sentence may have no matching 4-grams at all, and the README's chrF examples show eff:yes because chrF defaults to effective-order smoothing. The same source sets the maximum n-gram order to 4, lists floor, add-k, exp, and none as the smoothing methods with exp as the default, and says that when no tokenizer is given the tool picks 13a unless the target language is known to be Chinese, Japanese, or Korean, in which case it picks zh, ja-mecab, or ko-mecab. The raw README lists the smoothing defaults for the methods that take a value: 0.1 for floor and 1 for add-k.

The 2018 paper shows the original format, which readers will find in older papers. There the signature was a plus-delimited string of dotted key-value pairs, BLEU+c.mixed+l.en-de+#.1+s.exp+t.wmt14+tok.13a+v.1.2.10, read as mixed case, English to German, one reference, exponential smoothing, the WMT14 dataset, the 13a tokenizer, and sacreBLEU version 1.2.10. The -sh flag prints a short form with abbreviated keys. Either way, a signature is a complete statement of how a number was produced, and a BLEU score published without one leaves the reader to guess.

sacreBLEU tokenize options

Tokenization is the setting with the largest effect on a BLEU score, and the --tokenize flag controls it. The README describes the options.

• 13a is the default for BLEU and mimics the mteval-v13a script from Moses, the metric-internal tokenization used by WMT.
• intl applies international tokenization and mimics Moses' mteval-v14 script.
• zh separates out Chinese characters and tokenizes the non-Chinese parts with 13a.
• ja-mecab and ko-mecab tokenize Japanese and Korean with the MeCab morphological analyzer, and the README notes they require pip install "sacrebleu[ja]" or "sacrebleu[ko]".
• char performs language-agnostic character-level tokenization.
• none applies no tokenization at all, for users who have a reason to score pre-tokenized text and understand that the result is not comparable to 13a scores.
• flores101 and flores200 apply the SentencePiece models released with the FLORES-101 and FLORES-200 datasets, fetching the model file automatically if it is missing, and the README also lists spBLEU-1K, a multilingual SentencePiece tokenizer trained on more than 1,000 data sources.

The FLORES options implement spBLEU. The FLORES-101 paper proposed SentencePiece BLEU because Moses tokenization does not extend to languages such as Burmese or Khmer that do not separate words with spaces, and because languages like Hindi and Japanese had custom tokenizers whose use was scattered across publication footnotes. A single 256,000-token SentencePiece model trained on all 101 languages gave one tokenizer for every language. The NLLB paper released SPM-200 for FLORES-200 and reports minor differences between SPM-200 and the FLORES-101 model when measuring on the FLORES-101 languages, so a score computed with the flores101 model and one computed with flores200 are not interchangeable. A signature that reads tok:flores200 settles the question; a bare "spBLEU" does not.

chrF, chrF++, and TER support

sacreBLEU is no longer only a BLEU tool. The README's example sacrebleu -t wmt17 -l en-de -i output.detok.txt -m bleu chrf ter prints all three metrics, each with its own signature, and all three support multiple references.

chrF was proposed by Maja Popović in a 2015 WMT paper as an F-score over character n-grams, which the paper describes as language-independent and tokenization-independent. The raw README lists the defaults: --chrf-char-order 6, --chrf-word-order 0, and --chrf-beta 2, which sets how much recall counts relative to precision. Setting --chrf-word-order 2 adds word unigrams and bigrams, and the README says that with that setting "the metric is referred to as chrF++," the variant Popović described in a 2017 WMT paper. Further flags control lowercasing, whether whitespace is included in character n-grams, and epsilon smoothing in the style of the original chrF++.py script instead of effective-order smoothing.

TER, Translation Edit Rate, comes from Snover, Dorr, Schwartz, Micciulla, and Makhoul at AMTA 2006. It counts the edits a human would have to perform to change the system output into the reference, so a lower score is better, the reverse of BLEU and chrF. The README lists TER as case insensitive by default, with flags for case sensitivity, basic normalization and tokenization, special treatment of Asian characters, and punctuation removal.

The practical consequence is that one command, one input file, and one set of references can produce three metrics with three signatures, and a report that quotes all three gives a reader more than any single number. The post on reading BLEU, chrF, and COMET scores covers what the metrics measure and how they track human judgment.

Command-line usage from the README

The README's workflow, as it stands, runs as follows. Install with pip install sacrebleu, which the README says requires Python 3.9 or later; the PyPI page lists version 2.6.0, released January 12, 2026, under the Apache-2.0 license. Get the source side of a test set to translate:

sacrebleu -t wmt17 -l en-de --echo src > wmt17.en-de.en

Translate it with any system, detokenize the output, and score it:

sacrebleu -t wmt17 -l en-de -i output.detok.txt

Or, for text that is not a built-in test set, pass the reference file directly: sacrebleu ref.txt -i output.txt. Add -m bleu chrf ter to print several metrics, -b to print only the score, -sh for the short signature, and -f text for the old one-line format; the README says the default output format changed to JSON as of version 2.0.0. The text format output looks like BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 20.8, the signature and score on one line.

The README also documents paired significance tests. --paired-bs runs paired bootstrap resampling, with the number of resamples set by --paired-bs-n and a default of 1000, and --paired-ar runs paired approximate randomization with 10,000 trials by default. These answer whether a difference between two systems on the same test set is likely to be noise, which matters when the gap is smaller than the 1.8 points the 2018 paper attributed to tokenization alone.

For use from Python, the README shows the metric classes:

from sacrebleu.metrics import BLEU, CHRF, TER
bleu = BLEU()
bleu.corpus_score(sys, refs)
bleu.get_signature()

With the README's two-reference example the call prints BLEU = 48.53 82.4/50.0/45.5/37.5 (BP = 0.943 ratio = 0.944 hyp_len = 17 ref_len = 18), which lists the score, the four n-gram precisions, the brevity penalty, the length ratio, and the hypothesis and reference lengths in tokens, followed by a signature of nrefs:2|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0. The README notes that the older sacrebleu.corpus_bleu compatibility function may be removed in a future release.

Why scores without signatures cannot be compared

Post's argument rests on one fact about BLEU: it is a precision metric, so changing how the reference is processed changes the set of n-grams the system is scored against. Two scores are comparable only when the reference processing is the same, and the paper states that user-supplied preprocessing is error prone and inadequate for comparing across papers. The paper draws an analogy to parsing, where the PARSEVAL metric and the evalb program with a shared parameter file let F1 scores be compared across almost thirty years of papers on the Penn Treebank because one codebase answered every edge case.

A signature makes the same thing possible for translation. If two reports both carry tok:13a|case:mixed|smooth:exp on the same test set, the numbers can be set side by side. If one carries tok:intl and the other tok:13a, or one is lowercased and the other mixed, the comparison is invalid even though both are labeled BLEU. If one report quotes spBLEU with the flores101 model and another with flores200, the same applies. And if a report carries no signature at all, the reader cannot tell which case holds, which is why the README strongly recommends sharing signatures in papers.

The paper's own summary of the fix is that research groups should report BLEU computed only with metric-internal tokenization and preprocessing of the reference, and be explicit about the parameterization. The same logic applies to a vendor demonstration or a proposal appendix: a translation score is a claim about a specific test set under specific settings, and the signature is the part of the claim that lets someone else check it.

What to check

• Does every quoted BLEU, chrF, or TER score carry a sacreBLEU signature, or at least name the tokenizer, case handling, smoothing, reference count, and tool version?
• Was the system output detokenized before scoring, and was the reference left untouched by the vendor, as the 2018 paper recommends?
• Which tokenizer does the signature show, and is it the same across every score being compared, including flores101 versus flores200 for spBLEU?
• Which test set and version was used, for example wmt14 against wmt14/full, or FLORES-200 dev against devtest?
• Is the difference between two systems larger than what the paired bootstrap or approximate randomization tests call noise?
• When only BLEU is quoted, can the vendor also provide chrF++ and TER from the same run, since one command produces all three?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.

Questions people ask

What is sacreBLEU?

sacreBLEU is an open-source Python tool and library for computing machine translation scores that other people can reproduce. Matt Post introduced it in the 2018 WMT paper A Call for Clarity in Reporting BLEU Scores. The project README says it wraps the original reference implementation of BLEU, scores detokenized output with WMT standard tokenization, downloads and manages common test sets, and prints a signature string recording the parameters used. It also computes chrF, chrF++, and TER.

What is a sacreBLEU signature?

The signature is a short string that records the settings behind a score. The README's example is nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0, which reads as one reference, mixed case, effective order off, the 13a tokenizer, exponential smoothing, and sacreBLEU version 2.0.0. The 2018 paper's original format used plus signs and dotted keys, such as BLEU+c.mixed+l.en-de+#.1+s.exp+t.wmt14+tok.13a+v.1.2.10. The README says to share signatures in papers.

Why can BLEU scores from different papers not be compared?

Because BLEU is a parameterized metric, not a single formula. Post's 2018 paper lists the parameters: the number of references, the length penalty, the maximum n-gram length, smoothing, and above all how the reference is tokenized and normalized. Scoring one WMT 2017 system with different reference processing changed BLEU by as much as 1.8 points for German to English, and the paper found that well-cited papers often did not report which scheme they used, so their numbers cannot be lined up.

What does tokenize 13a mean in sacreBLEU?

13a is sacreBLEU's default tokenizer for BLEU. The README says it mimics the mteval-v13a script from Moses, which is the metric-internal tokenization the annual WMT evaluation uses. The intl option mimics the mteval-v14 international tokenization, zh separates Chinese characters and applies 13a to the rest, ja-mecab and ko-mecab use MeCab analyzers for Japanese and Korean, char splits every character, none applies no tokenization, and flores101 and flores200 use the FLORES SentencePiece models.

How do I compute chrF with sacreBLEU?

Pass the metric flag. The README's example is sacrebleu -t wmt17 -l en-de -i output.detok.txt -m bleu chrf ter, which prints all three scores. chrF uses a character n-gram order of 6 and a beta of 2 by default, which weights recall twice as much as precision. Setting --chrf-word-order 2 adds word unigrams and bigrams, and the README says the metric is then referred to as chrF++. Options also cover lowercasing, whitespace handling, and epsilon smoothing.

What is the difference between BLEU, chrF, and TER in sacreBLEU?

BLEU, from Papineni and colleagues in 2002, is a precision over word n-grams of length 1 to 4 with a brevity penalty. chrF, from Popović in 2015, is an F-score over character n-grams, and chrF++ from 2017 adds word n-grams. TER, from Snover and colleagues in 2006, counts the edits a human would need to turn the output into the reference, so lower is better. sacreBLEU implements all three, and each has its own options and its own signature.

How do I install and run sacreBLEU?

The README gives pip install sacrebleu, with Python 3.9 or later; the PyPI page lists version 2.6.0 released on January 12, 2026 under the Apache-2.0 license. To score against a WMT test set, the README shows sacrebleu -t wmt17 -l en-de -i output.detok.txt, where the test set is downloaded on first use. sacrebleu --list shows the available test sets, --echo src prints a test set's source side, and -b prints the score alone.

Should output be tokenized before running sacreBLEU?

No. The 2018 paper and the README both say sacreBLEU expects detokenized output and applies its own metric-internal tokenization to both the system output and the reference. The paper calls user-supplied reference preprocessing the main cause of incompatible scores and argues the reference should be kept away from the user entirely. The --force flag exists only to insist that tokenized-looking input is really detokenized.

What does eff mean in a sacreBLEU signature?

eff records whether effective order was used. The BLEU class in the sacreBLEU source describes effective_order as stopping the inclusion of n-gram orders for which precision is 0, and the default is False, so corpus-level BLEU signatures show eff:no. The field matters for sentence-level scores, where short sentences often have no matching 4-grams. The README's chrF examples show eff:yes, because chrF defaults to effective-order smoothing.

What is spBLEU and does sacreBLEU support it?

spBLEU is BLEU computed after tokenizing with a shared SentencePiece model, proposed in the FLORES-101 paper so that one tokenizer could score 101 languages. sacreBLEU supports it through the flores101 and flores200 tokenizer options, which fetch the SentencePiece model file automatically, and the README also lists a spBLEU-1K tokenizer trained on more than 1,000 data sources. The NLLB paper, which released SPM-200 for FLORES-200, reports minor differences between SPM-200 and the FLORES-101 model on the FLORES-101 languages.

Sources

1. 1A Call for Clarity in Reporting BLEU Scores · Proceedings of the Third Conference on Machine Translation: Research Papers (WMT 2018), pages 186-191, Association for Computational Linguistics
2. 2sacreBLEU project README · sacreBLEU project (Matt Post and contributors)
3. 3sacreBLEU README (raw) · sacreBLEU project on GitHub
4. 4sacreBLEU source: sacrebleu/metrics/bleu.py · sacreBLEU project on GitHub
5. 5sacrebleu (PyPI project page) · Python Package Index
6. 6Bleu: a Method for Automatic Evaluation of Machine Translation · Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (2002), pages 311-318
7. 7chrF: character n-gram F-score for automatic MT evaluation · Proceedings of the Tenth Workshop on Statistical Machine Translation (2015), pages 392-395
8. 8chrF++: words helping character n-grams · Proceedings of the Second Conference on Machine Translation (2017), pages 612-618
9. 9A Study of Translation Edit Rate with Targeted Human Annotation · Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (2006), pages 223-231
10. 10The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation · arXiv (Goyal et al.), June 6, 2021
11. 11No Language Left Behind: Scaling Human-Centered Machine Translation · arXiv (NLLB Team, Meta AI), v3 of August 25, 2022

Where this shows up in our work

• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.