What is SentencePiece?

SentencePiece is a tokenizer: a program that splits text into pieces a neural network can number, and that puts the pieces back together afterward. Taku Kudo and John Richardson of Google described it in a 2018 EMNLP system demonstration paper as a language-independent subword tokenizer and detokenizer that trains directly from raw sentences, implementing two segmentation algorithms, byte-pair encoding (BPE) and the unigram language model. The project README calls it an unsupervised text tokenizer for neural text generation systems where the vocabulary size is fixed before training. Released translation models such as Meta's NLLB-200 carry a SentencePiece model file beside their weights, and the spBLEU metric used on FLORES is BLEU computed over SentencePiece pieces. The tokenizer is where a language's cost is set: a sentence that becomes 20 pieces in English may become 60 in a language the model saw little of.

Why translation models need subword tokenization

Neural translation models predict one token at a time from a fixed list, so every word the model can output has to be on that list, and a list of whole words is either enormous or incomplete. Neural Machine Translation of Rare Words with Subword Units, by Rico Sennrich, Barry Haddow, and Alexandra Birch at ACL 2016, framed translation as an open-vocabulary problem and proposed encoding rare and unknown words as sequences of subword units, so that names, compounds, cognates, and loanwords could be translated piece by piece. The arXiv version reports gains of up to 1.1 BLEU on WMT 15 English to German and 1.3 BLEU on English to Russian over a baseline that fell back to a dictionary for unknown words. The ACL Anthology page notes the paper received the ACL 2026 Test of Time Award and that BPE is now the default tokenization method for most large language models.

BPE, as adapted in that paper, starts from characters and repeatedly merges the most frequent adjacent pair into a new symbol, for a fixed number of merge operations. Frequent words end up as single symbols and rare words as several. The approach has one dependency the SentencePiece paper later targeted: the original subword-nmt tool expects input already split into words, so a pre-tokenizer has to run first, and for languages without spaces between words that pre-tokenizer is a separate, language-specific program.

The second algorithm came from Kudo's own Subword Regularization paper at ACL 2018. A sentence can usually be segmented into subwords several valid ways, and the paper asks whether that ambiguity can serve as training noise. It proposes training on multiple segmentations sampled probabilistically, and to make the sampling work it introduces a segmentation algorithm based on a unigram language model, in which each piece has a probability and the best segmentation of a sentence is the most probable one. The paper reports consistent gains, especially in low-resource and out-of-domain settings. SentencePiece implements both algorithms behind one interface.

The SentencePiece paper and its four components

The 2018 paper describes four components. The Normalizer converts semantically equivalent Unicode characters to canonical forms; by default it applies a subset of Unicode NFKC normalization, and users can supply custom rules as a tab-separated file. The Trainer learns the subword model from the normalized corpus, with the model type given as a parameter. The Encoder runs the Normalizer and then segments text into pieces, or directly into integer ids. The Decoder converts pieces or ids back into normalized text.

The paper's command-line example trains a model with spm_train --input=data/input.txt --model_prefix=spm --vocab_size=1000, encodes "Hello world." into the pieces _He ll o _world . or the ids 151 88 21 887 6, and decodes either form back to the original string. The same operations are available through C++, Python, and TensorFlow APIs, which the paper notes call the same native code, so there is no performance penalty for using the wrappers.

Two engineering choices matter for deployment. First, the model file is self-contained: the paper says it includes not only the vocabulary and segmentation parameters but also the pre-compiled normalization rules, stored as a Protocol Buffer, so the tokenizer's behavior is determined only by the model file and has no external dependencies. The paper ties this to reproducibility, citing Post's 2018 finding that subtle preprocessing differences change BLEU scores, and noting that even NFKC normalization may differ by Unicode version. Second, the paper reports training complexity: a naive BPE implementation costs O(N squared) per sentence, while SentencePiece uses a binary heap for an O(N log N) algorithm, and unigram training and segmentation are linear in the input size.

SentencePiece also manages ids. The paper says the vocabulary size is set with --vocab_size, and that ids are reserved for special symbols: unknown, beginning of sentence, end of sentence, and padding. The training options document gives the defaults as 0, 1, 2, and -1, where -1 disables padding, and allows user-defined symbols such as language tags, which the paper mentions as the way multilingual models mark a target language with tokens like <2ja>.

The two model types, BPE and unigram

The options document lists four values for --model_type: unigram, which is the default, bpe, char, and word. The document marks unigram as recommended and says the word type is only useful for languages that use spaces.

The paper explains one design difference between SentencePiece and the earlier BPE tool that often confuses newcomers. subword-nmt specifies the number of merge operations, because that is a BPE-specific parameter. SentencePiece specifies the final size of the vocabulary instead, because a merge count means nothing to the unigram algorithm. A user who asks for 32,000 pieces gets 32,000 pieces under either model type.

The two algorithms differ in how they segment. BPE is deterministic: given a trained merge table, a word always splits the same way. The unigram model assigns probabilities to pieces, picks the most probable segmentation by default, and can sample alternatives. The paper's Figure 6 shows the Python call SampleEncodeAsPieces returning varying segmentations of "New York" across five calls, including samples that split New or York into single letters, which is the subword regularization mechanism from the 2018 ACL paper. The README lists on-the-fly sampling for both types: subword regularization for unigram and BPE-dropout for BPE.

The paper's own experiment used the unigram model. On the Kyoto Free Translation Task, English-Japanese Wikipedia text with 440,000 training sentences, a word model with 80,000-word vocabularies on each side scored 28.24 BLEU for Japanese to English and 20.06 for English to Japanese. SentencePiece with an 8,000-piece shared vocabulary scored 29.55 and 21.62. Adding Japanese pre-tokenization before SentencePiece made little difference into English and lowered the score into Japanese to 20.86, which the authors read as evidence that pre-tokenization acts as a constraint on the vocabulary that unsupervised segmentation can do without.

Training from raw text, independent of language

The paper's argument for raw-text training starts from the pipeline that preceded it. Moses, the standard statistical machine translation toolkit, provided a pre- and postprocessor built on hand-crafted, language-dependent rules designed for European languages where words are separated by whitespace. Training a system for Chinese, Korean, or Japanese meant running a word segmenter first. The paper says this language-dependent processing also made multilingual models hard to train, because the configuration of pre- and postprocessors had to be managed per language while the network itself was language independent.

SentencePiece removes the step. The README describes the tokenizer as purely data driven and language independent, with no requirement for language-specific pre-tokenizers such as Moses, MeCab, or KyTea, and the paper describes the result as a purely end-to-end system. The input is treated as a raw sequence of Unicode characters, so the same training command works for Thai, Burmese, Pashto, or English. The options document adds the knob that governs rare characters: --character_coverage, default 0.9995, with a recommendation of 1.0 for languages with small alphabets such as English or German and the default for Chinese, Japanese, and Korean. Characters outside the covered set map to the unknown token unless --byte_fallback is on, in which case the options document says they decompose into UTF-8 byte tokens such as <0xE3>.

The FLORES-101 paper shows why this property mattered for evaluation as well as training. The authors needed one tokenizer to score 101 languages with BLEU, because the Moses tokenizer did not handle Burmese or Khmer and because Hindi, Tamil, and Japanese each had their own community tools. They trained a single SentencePiece model with 256,000 tokens on monolingual data from all 101 languages and computed BLEU over its pieces, calling the result spBLEU. The FLORES-101 paper describes SentencePiece as a system that learns subword units from training data and does not require tokenization, and whose logic does not depend on language because all sentences are sequences of Unicode.

Lossless tokenization and the whitespace symbol

The paper's central technical idea is lossless tokenization. Conventional tokenization is not reversible: once "Hello world." becomes [Hello] [world] [.], the information that no space preceded the period is gone, and restoring the sentence requires a language-dependent detokenizer that knows English puts spaces between words and Japanese does not.

SentencePiece treats whitespace as an ordinary symbol. The paper states that the tool "first escapes the whitespace with a meta symbol" at U+2581 and then segments the text into pieces, so "Hello world." becomes [Hello] [_wor] [ld] [.], with the position of every space preserved. The Decoder is defined as the inverse of the Encoder: Decode(Encode(Normalize(text))) equals Normalize(text). Detokenization is then a one-line string join that replaces the meta symbol with a space, and the README describes the result as reversible and lossless. The paper contrasts this with subword-nmt's @@ marker, which marks boundaries inside words rather than spaces and cannot encode consecutive whitespace.

For a program that must show a translation beside its source, or search a translated document, this property is practical rather than academic. A tokenizer that loses spacing or normalizes characters irreversibly cannot reconstruct the exact text it was given, and the paper's design makes reconstruction a guarantee of the model file rather than a property of a separate script.

How NLLB-200 and other translation models use it

The No Language Left Behind paper describes the tokenizer for its 200-language model in Section 8.1.1. The team trained a single SentencePiece model for all languages from a sample of 100 million sentences drawn from its primary bitext. Because 150 of the 200 languages are low resource, uniform sampling would have over-represented high-resource languages and fragmented low-resource text, so the team applied temperature sampling with a temperature of 5, which downsamples high-resource and upsamples low-resource languages. Even then, Simplified Chinese, Traditional Chinese, and Cantonese showed high unknown-token rates because of their very large character sets, and the team upsampled those three a further 5 times until the unknown rate for every language fell below 1 percent.

The vocabulary was set at 256,000. The paper gives two reasons: capacity to represent the wide spectrum of languages, and the tokenization rate, the average number of tokens per sentence per language. The team did not try 384,000 or more because a larger vocabulary would significantly increase the number of model parameters. The same model, SPM-200, was released with FLORES-200 and is what sacreBLEU applies under its flores200 tokenizer option, so the tokenizer that trains NLLB-200 is also the tokenizer that scores it.

Other released models carry SentencePiece files for the same reason. The repository README's benchmark section names T5-base with a 32,000-piece unigram model and Gemma 3 with a 256,000-piece BPE model as its test cases, and the paper describes multilingual models marking the target language with user-defined tokens. The common pattern is one shared vocabulary across all languages a model serves, with the balance between languages set by how the training sample was drawn.

Vocabulary size tradeoffs

The vocabulary size is the main decision a SentencePiece user makes, and the public sources show the range: 1,000 in the paper's toy example, 8,000 as the options document's default and in the paper's English-Japanese experiment, 32,000 for T5, 256,000 for NLLB-200 and Gemma 3.

A larger vocabulary means longer pieces and fewer tokens per sentence. The NLLB paper names the cost: a larger vocabulary would significantly increase the number of model parameters, and 256,000 was the point at which the team judged tokenization quality reasonable and stopped. A smaller vocabulary means more pieces per sentence, which the NLLB paper says makes modeling harder, especially long-range dependencies and the synthesis of words from near character-level tokens, and which lengthens every sequence the model must process.

The balance also depends on how many languages share the vocabulary. Eight thousand pieces served one language pair in the 2018 paper. The FLORES-101 authors used 256,000 for 101 languages, and the NLLB team kept 256,000 for 200, which means each language received, on average, fewer dedicated pieces than in the 101-language model. The paper's tokenization-rate discussion is about exactly this: a shared budget that frequency-based training spends mostly on the languages with the most text, unless sampling corrects for it.

The effect on low-resource languages and non-Latin scripts

The NLLB paper states the mechanism plainly. Because SentencePiece selects subword units by frequency, underrepresented languages tend to have a higher tokenization rate than high-resource ones, "leading to a near character-based model for those languages." Temperature sampling and the large vocabulary were the team's two mitigations.

Two 2023 papers measured the effect on public tokenizers using FLORES-200 as the parallel corpus. Language Model Tokenizers Introduce Unfairness Between Languages, by Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi at NeurIPS 2023, defines a tokenizer's premium for one language relative to another as the ratio of token counts for the same sentence, and parity as a ratio near 1. Measured against English, the ChatGPT and GPT-4 tokenizer needed about 1.6 times as many tokens for German, about 3 times for Arabic, and up to 15 times for Shan, a language of Myanmar. The paper's Table 1 gives Burmese at 16.89 for GPT-2 and 11.70 for ChatGPT, Standard Arabic at 4.40 and 3.04, and Portuguese, the closest to parity, still at 1.48 for ChatGPT. The appendix tables list Southern Pashto at 5.39 for GPT-2 and 3.83 for ChatGPT and GPT-4. Byte-level encodings are not neutral either: the paper reports UTF-8 representations over 4 times longer for Burmese or Tibetan than for Chinese.

Tokenizers trained for many languages narrow the gap without closing it. The paper's Table 4 gives NLLB's SPM-200 premiums of 1.11 for Simplified Chinese, 1.40 for Standard Arabic, 1.94 for Shan, and 2.49 for Santali, against 4.43 for Shan under XLM-R and 12.06 under BLOOM, and the appendix puts Southern Pashto at 1.45 under NLLB. The authors also observe that models aimed at one language treat other languages in the same script more fairly, noting that "Sharing writing systems seems to improve tokenization parity." The practical reading is that a Perso-Arabic script language pays a modest premium under a tokenizer trained to cover it and a large one under a tokenizer trained mostly on English.

Do All Languages Cost the Same?, by Orevaoghene Ahia and colleagues, connects the token count to money. The authors tokenized a FLORES-200 subset with the ChatGPT tokenizer and found Latin-script languages represented with substantially fewer tokens, Cyrillic and Japanese close behind, and languages with their own scripts such as Telugu and Georgian needing up to 5 times more tokens for the same content. Because the API charges per token, the paper estimates close to a 5 times cost increase for most mid-resourced Indic languages with non-Latin scripts relative to English, and, for the BLOOM tokenizer, finds Arabic an outlier with more tokens than some other mid-resourced languages. The authors identify two causes, the language's share of pretraining data and properties of the language and its script, and report that training their own byte-level BPE tokenizers with vocabularies from 5,000 to 50,000 on parallel text could not achieve a uniform fragmentation rate across scripts even when content and data size were held equal.

For a program evaluating a model on a non-Latin script language, the tokenizer is therefore the first thing to measure. An input limit stated in tokens holds several times less Pashto or Burmese text than English under an English-centric tokenizer, by the premiums above, and the same sentence costs more time and more money to process.

What to check

• Which tokenizer does the model ship with, and is the SentencePiece model file the same one used at training time, since the paper says behavior is determined entirely by that file?
• What is the vocabulary size and model type, and how many of the pieces are useful for the program's languages rather than for English?
• What is the tokenization rate, tokens per sentence, for the program's languages compared with English on a parallel sample such as FLORES-200?
• What is the character coverage, and do the program's scripts, including Pashto-specific letters or rare characters, fall outside it and become unknown or byte tokens?
• Does the vendor's input limit, stated in tokens, translate into enough text in the target language to hold a full paragraph or form field?
• Was the tokenizer's training sample balanced across languages, as NLLB's temperature-5 sampling was, or drawn in proportion to available text?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.

Questions people ask

What is SentencePiece?

SentencePiece is an open-source tokenizer and detokenizer for neural text processing, described by Taku Kudo and John Richardson in a 2018 EMNLP system demonstration paper. It splits text into subword pieces using either byte-pair encoding or a unigram language model, trains those models directly from raw sentences without a language-specific pre-tokenizer, and maps pieces to integer ids. The project README describes it as an unsupervised tokenizer released under the Apache 2.0 license.

What is the difference between SentencePiece and BPE?

BPE is an algorithm; SentencePiece is a tool that implements two algorithms, BPE and unigram, plus the surrounding machinery. The 2018 paper explains that the earlier subword-nmt BPE tool needs pre-tokenized words and marks subwords with @@, while SentencePiece treats the input as a sequence of Unicode characters, keeps whitespace as a symbol, and sets a final vocabulary size instead of a number of merge operations. The training options document lists unigram as the default model type.

What is a unigram tokenizer?

A unigram tokenizer, from Taku Kudo's 2018 ACL paper Subword Regularization, treats each sentence as having several possible segmentations and picks the most probable one under a unigram language model over pieces. Because the model assigns probabilities, it can also sample different segmentations of the same sentence during training, which the paper shows improves translation, especially in low-resource and out-of-domain settings. SentencePiece implements this as its default model type.

Why does SentencePiece train on raw text?

The 2018 paper argues that tools such as Moses depend on hand-written, language-specific rules built for European languages with spaces between words, and that Chinese, Korean, and Japanese need separate word segmenters before subword training can start. Training directly from raw sentences removes that step, makes the pipeline the same for every language, and allows lossless tokenization, where decoding the encoded text returns exactly the normalized input.

What is the underscore symbol in SentencePiece output?

SentencePiece escapes every whitespace with the meta symbol U+2581, a lower one eighth block that looks like an underscore, and includes it in the tokens. The paper gives the example of Hello world. becoming [Hello] [_wor] [ld] [.], so the position of each space is preserved and detokenization is a plain string join followed by replacing the symbol with a space. The README calls this reversible and lossless tokenization.

How does NLLB-200 use SentencePiece?

The No Language Left Behind paper says the team trained a single SentencePiece model for all 200 languages from 100 million sentences sampled from its bitext, with a vocabulary of 256,000. Sampling used a temperature of 5 to downsample high-resource and upsample low-resource languages, and Chinese and Cantonese were upsampled a further 5 times to push their unknown-token rate below 1 percent. The same model, called SPM-200, was released with FLORES-200 for computing spBLEU.

What vocabulary size should a SentencePiece model have?

There is no single answer, and the public sources show the tradeoff. The training options document gives a default vocab_size of 8,000. The 2018 paper's English-Japanese experiment used an 8,000-piece shared vocabulary against an 80,000-word baseline and improved BLEU. NLLB-200 chose 256,000 pieces for 200 languages and stopped there because a larger vocabulary would significantly increase model parameters. Larger vocabularies mean fewer tokens per sentence but more parameters in the embedding layers.

Why do low-resource languages get more tokens?

The NLLB paper explains that SentencePiece picks subword units by frequency, so languages with little training text end up with a higher tokenization rate, meaning more tokens per sentence, approaching a character-level model. The paper lists that as a reason for temperature sampling and a large vocabulary. Petrov and colleagues measured the effect on FLORES-200 in 2023: the ChatGPT tokenizer needed roughly 3 times the English token count for Arabic and up to 15 times for Shan.

Does tokenization cost more for some languages?

Yes, when a service charges by token. Ahia and colleagues' 2023 paper Do All Languages Cost the Same? tokenized a FLORES-200 subset with the ChatGPT tokenizer and found Latin-script languages took substantially fewer tokens, while languages with their own scripts such as Telugu and Georgian needed up to 5 times more, which the paper translates into close to a 5 times cost increase for most mid-resourced Indic languages relative to English.

What does character_coverage do in SentencePiece?

character_coverage sets the share of characters in the training data that the model must represent with their own pieces. The training options document gives a default of 0.9995 and recommends 1.0 for languages with small alphabets such as English or German, keeping the default for Chinese, Japanese, and Korean, whose character sets are very large. Characters outside the coverage become unknown tokens unless byte_fallback is enabled, which decomposes them into UTF-8 byte tokens.

Sources

1. 1SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Association for Computational Linguistics
2. 2SentencePiece (repository README) · google/sentencepiece on GitHub
3. 3SentencePiece training options (doc/options.md) · google/sentencepiece on GitHub
4. 4Neural Machine Translation of Rare Words with Subword Units · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (2016), pages 1715-1725
5. 5Neural Machine Translation of Rare Words with Subword Units (arXiv version) · arXiv (Sennrich, Haddow, Birch), v5 of June 10, 2016
6. 6Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (2018), pages 66-75
7. 7No Language Left Behind: Scaling Human-Centered Machine Translation · arXiv (NLLB Team, Meta AI), v3 of August 25, 2022
8. 8The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation · arXiv (Goyal et al.), June 6, 2021
9. 9Language Model Tokenizers Introduce Unfairness Between Languages · arXiv (Petrov, La Malfa, Torr, Bibi), v2 of October 20, 2023; NeurIPS 2023
10. 10Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models · arXiv (Ahia, Kumar, Gonen, Kasai, Mortensen, Smith, Tsvetkov), May 23, 2023

Where this shows up in our work

• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.