Tokenizers and language cost, explained

A tokenizer is the step that splits text into the units, called tokens, that a language model reads and writes. The same sentence can take very different numbers of tokens in different languages. In a paper at the 2023 Neural Information Processing Systems (NeurIPS) conference, Petrov and colleagues found that the tokenizer used by ChatGPT and GPT-4 needs about 1.6 times as many tokens for Italian as for English, 2.6 times for Bulgarian, 3 times for Arabic, and up to 15 times for Shan, a language of Myanmar. Price, context limits, and processing time are all counted in tokens, so the gap carries into each.

How a tokenizer splits text

Most current models use subword tokenizers. A subword tokenizer learns its vocabulary from how often strings occur in a training corpus. A common design, byte-level byte pair encoding, starts from the 256 possible byte values and adds frequent combinations, as Ahia and colleagues describe.

The corpus shapes the result. Petrov's paper shows an older vocabulary, from GPT-2, holding a single token for the string BuyableInstoreAndOnline, likely from an online store's code. The common Arabic word for "why" was broken into one token per letter. The Bulgarian word for "why" took 6 tokens for 4 letters. The newer tokenizer behind ChatGPT and GPT-4 still needs three tokens for more than 65 percent of Japanese kanji.

Encoding adds another layer. Chapter 2 of the Unicode Standard, version 17.0, notes that what a reader sees as one character can be several code points. UTF-8, which the standard calls the preferred encoding form for the Internet, stores the basic Latin characters of ASCII (American Standard Code for Information Interchange), enough for English, in one byte each. Compared with the UTF-16 form, the standard says, UTF-8 is significantly larger for scripts such as Hindi, Thai, and Chinese.

How much more, by language

Petrov measures a tokenization premium: the token length of a sentence in one language divided by the length of its translation in another. The test set is FLORES-200, the same 2,000 Wikipedia sentences translated by people into 200 languages.

For the ChatGPT and GPT-4 tokenizer, Portuguese came closest to English and still needed about 50 percent more tokens. Processing German or Italian cost about 50 percent more. Dzongkha, Odia, Santali, and Shan cost more than 12 times as much. One Shan pronoun is written as one consonant and three diacritics, four code points, and became 9 tokens. The English equivalent, a three-letter word, is a single token.

Tokenizers built for many languages narrowed the gap without closing it. All five multilingual models tested had languages with premiums above 2.5. Byte-level models also showed premiums of more than 4 times for some language pairs. Tokenizers built for other languages still favored English: one model trained on 16 Indian languages and English encoded English most efficiently.

What the cost study measured

Ahia and colleagues, at the Conference on Empirical Methods in Natural Language Processing (EMNLP) in December 2023, studied 22 typologically diverse languages through a commercial application programming interface (API). Languages with their own scripts, such as Telugu, needed up to 5 times as many tokens as Latin-script languages for the same content. Most mid-resourced Indic languages cost close to 5 times as much as English. On a news summarization benchmark, prompting and generation cost up to 4 times as much in Telugu and Amharic.

The cause is not only how much of each language the tokenizer saw. The authors trained tokenizers on parallel text, one language per script, at vocabulary sizes from 5,000 to 50,000. The rates still differed across scripts. They conclude that language and script properties matter as well. They also found a negative correlation, close to -0.5, between a language's token count and the Human Development Index of the country with the most speakers.

Context limits and processing time

A model accepts a fixed number of tokens. Petrov's paper gives GPT-4's context sizes at the time as 8,000 and 16,000 tokens and concludes that a user could fit less than a tenth as much Burmese or Dzongkha content as English. Ahia reports that the 4,096-token limit of the model studied left no room for even one worked example in Telugu for most of the summarization test set, so only zero-shot prompting, with no worked examples, was possible.

Time follows length. Petrov timed one model, RoBERTa, and found processing time grew linearly with token count. Shan took almost twice as long as English.

The tokenizer affects quality and training cost too. Rust and colleagues, at the 2021 meeting of the Association for Computational Linguistics (ACL), found that replacing a multilingual model's tokenizer with a dedicated monolingual one improved results for almost every task and language they tested. Ali and colleagues, in the Findings of the 2024 North American Chapter of the ACL (NAACL) conference, trained 24 models at 2.6 billion parameters. English-centric tokenizers in multilingual models caused severe performance loss and up to 68 percent more training cost. The authors add that parity and fertility, a tokens-per-word measure, do not always predict performance.

The studies state their own limits. Petrov notes that FLORES-200 contains many English-centric names and institutions, which might skew results toward English. Ahia notes that professionally translated text can differ from original writing.

What to check

• What is the token premium of the system's tokenizer for each language it must read, measured on parallel text rather than assumed?
• How many pages of a document in that language fit in the context window, compared with the same document in English?
• Was response time measured in each working language, or only in English?
• Was the tokenizer built for the languages in use, or inherited from an English-centric model?
• Does a claim of fairness across languages rest on one parallel corpus, and does that corpus favor English names and topics?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.

Sources

1. 1Language Model Tokenizers Introduce Unfairness Between Languages · Advances in Neural Information Processing Systems 36 (Petrov, La Malfa, Torr, and Bibi; NeurIPS 2023)
2. 2Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models · ACL Anthology (Ahia et al., EMNLP 2023)
3. 3How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models · ACL Anthology (Rust et al., ACL-IJCNLP 2021)
4. 4Tokenizer Choice For LLM Training: Negligible or Crucial? · ACL Anthology (Ali et al., Findings of NAACL 2024)
5. 5The Unicode Standard, Version 17.0, Chapter 2: General Structure · The Unicode Consortium

Where this shows up in our work

• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.