Terminology and glossaries in machine translation

Terminology control means making a machine translation system use the approved target-language term for each source term, as listed in a glossary or term base. Without it, a system can choose a synonym, or translate one term two ways in the same document. The Conference on Machine Translation (WMT) ran shared tasks on the problem in 2021 and 2023 (WMT 2021; WMT 2023). Three methods recur: forcing terms into the output during decoding, training a model to copy terms placed in its input, and listing glossary entries in a prompt to a large language model. Each trades term accuracy against speed or fluency. The 2021 task found that a good overall score does not guarantee correct terms.

Why controlled terms matter in defense documents

DoD Instruction 5025.12, Standardization of Military and Associated Terminology, effective March 5, 2020, with Change 1 effective January 21, 2022, sets the policy. Standardized terminology is meant to improve communications and mutual understanding within the Department of Defense (DoD), with other federal departments and agencies, and between the United States and its international partners. DoD Components are to use the DoD Dictionary of Military and Associated Terms as the primary terminology source for correspondence, including policy, strategy, doctrine, and planning documents. Terms that are user-specific or subject-focused but not universally accepted go in a separate Terminology Repository.

The WMT 2023 organizers show the two failures a glossary is meant to prevent. In German medical abstracts, one system rendered the same word, Krankheit, as condition in one sentence and disease in another. Their opening example shows the fix. The German source "Der Bericht entspricht FOG" came out as "The report is in accordance with FOG"; a hint mapping FOG to ROA gave "in accordance with ROA," the abbreviation the reference used.

Forcing terms during decoding

Lexically constrained decoding changes beam search, the step that assembles the output, so that pre-specified words and phrases must appear. Post and Vilar, of Amazon Research, set out the costs in a 2018 paper. Grid beam search, from Hokamp and Liu in 2017, grows linearly with the number of constraints, and constrained beam search grows exponentially. Their dynamic beam allocation keeps the cost constant in the number of constraints.

Dinu and colleagues at Amazon AI tested English-German translation with terms from Wiktionary and from IATE, the multilingual terminology database of European Union institutions and bodies. A baseline model already used the required term 76.9 percent of the time on the Wiktionary test set. Constrained decoding raised that to 99.5 percent, but its 99th-percentile latency was 0.68 seconds, against 0.19 seconds for the baseline. On IATE terms it reached only 82.0 percent with a beam size of 5. Forcing can also break grammar: one constrained output put the noun Festnahme in front of the verb festzunehmen. Training the model to copy target terms written into the source sentence, in place of the source terms, reached 93.4 and 94.5 percent on the two sets at baseline speed.

Glossaries in prompts

A large language model can take a glossary as plain text. Moslem and colleagues of the ADAPT Centre in Ireland tested a commercial model this way, in a paper for the 2023 conference of the European Association for Machine Translation. They placed up to five glossary terms that matched the source sentence into the prompt. Professional linguists rated 100 sentences per setting for quality on a 1 to 4 scale and for term use on a 0 to 1 scale. For English to Arabic, term use rose from 0.67 without glossary terms to 0.94 with them, and quality from 2.80 to 3.19. Term use rose from 0.87 to 0.96 for Spanish and from 0.89 to 0.97 for French. The authors add that forcing glossary terms was sometimes detrimental to grammatical accuracy, and that contradictory terms can hurt quality.

Entries in WMT 2023 mixed the methods. One team placed term pairs in front of the sentence as context. Another used two-shot prompting with hints, then found and penalized mistranslated terms if the hints failed. A third forced its decoder to generate the target terms.

What the shared tasks found

The 2021 task covered the COVID-19 domain in five language pairs: English to French, Chinese, Russian, and Korean, and Czech to German. For the English-source pairs, linguists working with domain experts in the TICO-19 project supplied about 600 terms per language. The test references had not been written with those terms, so professional translators revised them, changing 9.25 percent of the French sentences and 26.34 percent of the Korean. Nine teams sent 43 submissions. Winners were systems that no other system beat on both term exact-match accuracy and COMET, a trained metric of overall quality. English-French reached 0.974 exact match; the single English-Korean entry, 0.569. In English-Russian, a system that ranked first on COMET ranked last on exact match, at 0.788. The organizers concluded that a narrow terminology with unambiguous translations works better than a broad one with several target options.

The 2023 task gave each sentence its own term dictionary, in German to English, English to Czech, and Chinese to English, and drew 21 submissions from 7 teams. Most systems translated 60 to 70 percent of terms correctly. Adding the dictionary raised chrF, a character n-gram overlap score, by 0.2 to 10.1 points for every system but one, which lost 10.2. The control was the surprise: random aligned phrases from the reference, not terms at all, improved scores about as much. The organizers say this challenges the idea that terminology carries the gain, or shows that the metrics are not terminology-centric. They did no statistical testing.

What to check

• Does the glossary give one approved target per source term, and is each entry traced to the DoD Dictionary or the Terminology Repository?
• How does the system enforce terms, by constrained decoding, inline training, or prompt, and what does that cost in speed?
• Is term accuracy reported separately from overall quality, given the English-Russian system that led on COMET and trailed on terms?
• Was a random-phrase control run, as in WMT 2023, to show that the gain comes from the terms?
• Does the system inflect a forced term to fit the sentence, or paste in the dictionary form?
• Does it render the same term the same way across a whole document?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.

Sources

1. 1DoD Instruction 5025.12, Standardization of Military and Associated Terminology (March 5, 2020, Change 1 January 21, 2022) · Office of the Director of Administration and Management, Department of Defense
2. 2Findings of the WMT Shared Task on Machine Translation Using Terminologies · Alam and colleagues, WMT 2021 (ACL Anthology)
3. 3Findings of the WMT 2023 Shared Task on Machine Translation with Terminologies · Semenov, Zouhar, Kocmi, Zhang, Zhou, and Jiang, WMT 2023 (ACL Anthology)
4. 4Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation · Post and Vilar, NAACL-HLT 2018 (ACL Anthology)
5. 5Training Neural Machine Translation to Apply Terminology Constraints · Dinu, Mathur, Federico, and Al-Onaizan, ACL 2019 (ACL Anthology)
6. 6About IATE (Interactive Terminology for Europe) · European Union, IATE terminology database
7. 7Adaptive Machine Translation with Large Language Models · Moslem, Haque, Kelleher, and Way, EAMT 2023 (ACL Anthology)

Where this shows up in our work

• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.