Machine translation quality estimation, explained

Quality estimation (QE) is the task of scoring a machine translation from the source text and the output alone, without a human reference translation (CometKiwi, 2022). That separates it from metrics such as BLEU and COMET, which compare the output with a reference. A document translated for use has no reference, so QE is the automatic score available at run time. A 2009 paper named two purposes: filtering out poor translations so professional translators do not spend time on them, and telling end users how good a translation is (Specia and colleagues). In the 2024 shared task, the best system's scores reached a Spearman correlation of 0.553 with professional translators' judgments, averaged over language pairs (WMT 2024).

From confidence estimation to shared tasks

In a 2009 paper for the European Association for Machine Translation (EAMT), Lucia Specia and four colleagues at Xerox Research Centre Europe and the University of Bristol called the task confidence estimation. They defined it as predicting the quality of a system's output for a given input without any information about the expected output. Earlier work, they note, treated it as a binary choice between good and bad translations. Finding a clear boundary is hard, they argued, so they predicted a continuous score for each sentence instead. The line they drew between this task and machine translation evaluation was the reference: evaluation needs one.

The Conference on Machine Translation (WMT) now runs QE as a shared task. In the 2023 edition, a sentence-level system predicts a score, ranked by Spearman rank correlation with human scores. A word-level system tags each output word OK or BAD, scored mainly with the Matthews correlation coefficient (MCC). Professional translators supply the human labels as 0 to 100 direct assessments, post-edits, or Multidimensional Quality Metrics (MQM) annotations. For MQM, the 2023 task counted 1 point for each minor error, 5 for each major error, and 10 for each error at the top severity, subtracted the total from 100, and divided by sentence length in words.

How COMET-QE and CometKiwi score a sentence

COMET began as a reference-based metric. In its 2021 WMT paper, the Unbabel and Instituto Superior Técnico (IST) team described COMET-QE, which keeps COMET's dual-encoder design but replaces the reference translation with the source sentence. The team reported that reference-free models were becoming competitive with reference-based ones. The same paper shows word-level tags at work. For a Chinese source, an output read "the main purpose of this project is to design a car for blind driving." A QE model tagged every word OK except "blind driving," which it tagged BAD, and gave the sentence an overall score of 0.2.

CometKiwi, the team's 2022 entry, joins the COMET framework with the predictor-estimator architecture of OpenKiwi. The translation and the source are passed together into one pretrained encoder, which produces a sentence score and an OK or BAD label for each word. The authors report the best multilingual results in all three 2022 tracks, among them a Spearman correlation of 0.572 at the sentence level and an MCC of 0.341 at the word level.

What the 2023 to 2025 tasks found

In WMT 2023, seven teams entered the sentence-level task. The best multilingual Spearman correlation was 0.594, against 0.372 for the COMET-QE baseline. The best score per language pair ranged from 0.394 for English-Telugu to 0.775 for English-Tamil. At the word level, the best multilingual MCC was 0.329. The organizers also tested hallucinations, translations with content detached from the source. Some systems detected them perfectly or nearly so in some language pairs, yet the top systems still missed localized errors such as oscillations, including a German output that repeated Bett-Bett-Bett.

WMT 2024 used CometKiwi 2022 as its baseline, at a multilingual Spearman of 0.520 against 0.553 for the best entry. In error span detection, scored by F1, a balance of precision and recall, the baseline's 0.278 beat the only other entry, and the organizers saw ample room for improvement. Systems built on large language models were still outperformed at the sentence level by predictor-estimator systems.

The WMT 2025 general translation task used QE inside its own ranking. CometKiwi-XL was one of the reference-free metrics in AutoRank, the automatic score that chose systems for human evaluation. The organizers name a known QE pitfall, being fooled by fluent output in the wrong language. In one check, CometKiwi-XL scored an untranslated copy of the source at 0.23 and the human reference at 0.55.

Flagging sentences for a linguist

One use is triage, sending doubtful sentences to a person first. Turchi, Negri, and Federico tested this in a 2015 paper. Four professional translators post-edited an English information technology user manual into Italian in the MateCat tool. Half of the 847 test segments carried a green or red flag from a QE classifier trained for each translator, and the rest a neutral grey flag. Green meant post-editing should be faster than translating from scratch. The boundary was a human-targeted translation edit rate (HTER) of 0.4.

The flags helped less than expected. Average post-editing time was 8.086 seconds per word with colored flags and 9.592 with grey ones, not a significant difference (p = 0.33). Pair by pair, the flagged segment was faster 51.7 percent of the time, which was significant (p = 0.039). The time saving was significant only for sentences of 5 to 20 words whose HTER fell between 0.2 and 0.5, where it was 0.663 seconds per word. The classifiers averaged about 80 percent accuracy.

A word-level tag points within the sentence, like the BAD tag on "blind driving." Given the 2024 span results, such a tag marks where to look first, not a verdict.

What to ask

• Which QE model and version produced the score, and was it trained on direct assessments, post-edits, or MQM labels?
• What is its correlation with human judgments for this language pair, given that the best 2023 scores ranged from 0.394 to 0.775 across pairs?
• Does it flag word spans or only sentences, and how was span accuracy measured?
• Was it tested on hallucinations, repeated words, copied source text, and fluent output in the wrong language?
• What threshold sends a sentence to a linguist, and was it checked with timed post-editing, as in the 2015 test?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.

Sources

1. 1Estimating the Sentence-Level Quality of Machine Translation Systems · Specia, Cancedda, Dymetman, Turchi, and Cristianini, EAMT 2009 (ACL Anthology)
2. 2Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task · Rei and colleagues, WMT 2021 (ACL Anthology)
3. 3CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task · Rei and colleagues, WMT 2022 (ACL Anthology)
4. 4Findings of the WMT 2023 Shared Task on Quality Estimation · Blain and colleagues, WMT 2023 (ACL Anthology)
5. 5Findings of the Quality Estimation Shared Task at WMT 2024: Are LLMs Closing the Gap in QE? · Zerva and colleagues, WMT 2024 (ACL Anthology)
6. 6MT Quality Estimation for Computer-assisted Translation: Does it Really Help? · Turchi, Negri, and Federico, ACL 2015 (ACL Anthology)
7. 7Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets · Kocmi and colleagues, WMT 2025 (ACL Anthology)

Where this shows up in our work

• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.