Chinese word segmentation for search and translation

Chinese word segmentation is the task of dividing a run of Chinese characters into words, because written Chinese puts no spaces between them. The organizer of the 2005 international bakeoff, Thomas Emerson, calls finding the boundaries an essential first step for information retrieval and text-to-speech systems (Emerson, 2005). The Unicode Consortium's text segmentation annex agrees: reliable word boundaries in Chinese require dictionary lookup or other mechanisms (UAX #29). The best system in that bakeoff scored an F of 0.972, and unknown words remained the main problem.

Why the boundaries are ambiguous

The lack of spaces is not the heart of the problem, Nianwen Xue argues in a 2003 paper. The problem is that most characters can sit in different positions in different words. The character 文 ends the word 日文 (Japanese) and begins the word 文章 (article). The string 日文章魚怎麼說, "How to say octopus in Japanese?", can be split as 日文 章魚 怎麼 說 or as 日 文章 魚 怎麼 說 (Japan, article, fish, how, say). Every one of those words is in the dictionary, so a dictionary alone cannot choose.

Unknown words are a second problem. Xue lists compounding, abbreviation (suoxie), new personal names, and transliterated foreign names as steady sources of words no dictionary lists. The paper treats segmentation as tagging each character with its position in a word, and reports precision of 95.01 percent and recall of 94.94 percent from a 237K-word training set.

Unicode's default word rules, in the annex's revision 49 dated September 1, 2026, end with "break everywhere (including around ideographs)." Under those defaults, each Chinese character stands as its own segment.

The 2005 bakeoff: corpora, scores, and unknown words

SIGHAN, the Association for Computational Linguistics (ACL) Special Interest Group for Chinese Language Processing, ran the first international bakeoff in 2003. In the second, held in summer 2005, 23 groups submitted 130 result sets. One group was the U.S. Department of Defense. The closed track allowed only the supplied training data, and the open track any outside data.

Corpus	Characters	Training words	Test words	OOV rate	Best F, closed	Best F, open
Academia Sinica (AS)	Traditional	5.45M	122K	0.043	0.952	0.956
City University of Hong Kong (CityU)	Traditional	1.46M	41K	0.074	0.943	0.962
Microsoft Research Beijing (MSR)	Simplified	2.37M	107K	0.026	0.964	0.972
Beijing University (PKU)	Simplified	1.1M	104K	0.058	0.95	0.969

F balances precision and recall. The out-of-vocabulary (OOV) rate is the share of test words absent from the training data. Across all runs the average F was 0.918 and the median 0.941. A baseline that matched only training-set words recalled almost no unknown words. The best recall on unknown words was 0.872 in the open track and 0.813 in the closed. Emerson concludes that "OOV handling is still the Achilles heel of segmentation systems," and cautions that high scores on one test set may not scale to other applications.

Why the corpora disagree on what a word is

Sproat and Emerson wrote in 2003 that no single segmentation standard is accepted, and that the standard best for retrieval likely differs from the one best for speech synthesis. The Chinese national standard GB 13715 specifies 毛/泽东 for Mao Zedong, with a break after the first character. An earlier evaluation within China also accepted 毛泽东. The 2003 bakeoff refused such leniency and scored strictly against each corpus's own standard.

The 2005 corpora used four standards: PKU one derived from GB 13715, AS a Taiwanese national standard, CityU the LIVAC corpus standard, and MSR Microsoft's internal standard. A 2019 ACL paper shows the effect on one sentence, "Yao Ming reaches the final": the Chinese Treebank standard gives 姚明 进入 总决赛, and the PKU standard gives 姚 明 进入 总 决赛 (Meng, Li, and colleagues). A system's F score is therefore a score against one standard.

Better segmentation did not always help

Peng and colleagues ran retrieval on the 164,768-document TREC-5 and TREC-6 Chinese collection with 12 segmenters ranging from 44 to 95 percent accuracy. Retrieval rose steadily up to 70 percent accuracy, levelled off to 77 percent, then fell slightly at 85, 90, and 95 percent. On TREC-5, average precision was 0.3721 at 70 percent and 0.3189 at 95 percent. The authors found that accurate segmenters kept compounds whole, while less accurate ones broke them up, to the benefit of retrieval. Both bakeoff reports add that short strings with little context, such as search queries, had not been evaluated.

Chang, Galley, and Manning tested a phrase-based Chinese-English translation system. On its test set, treating each character as a word scored 29.36 BLEU, a standard translation metric, against 30.73 for a simple dictionary segmenter. A statistical segmenter with a higher F, 0.877 against 0.828, scored lower, 30.35 BLEU. The authors found that consistency and word size can matter more than F, and that words slightly shorter than the Chinese Treebank standard worked best.

Character and subword models changed the comparison

In the 2008 test, characters lost. The 2019 study by Meng, Li, and colleagues compared word-based and character-based neural models on four tasks and found that the character-based models consistently did better. In Chinese-to-English translation trained on 1.25 million sentence pairs, characters averaged 0.83 BLEU higher in a standard attention model. In English-to-Chinese, the gain reached 3.13 BLEU, 45.17 against 42.04. A subword model on the Chinese side, built with byte-pair encoding, beat the word model but trailed characters by about 0.8 to 0.9 BLEU.

The authors blame sparse data. In the Chinese Treebank, segmented with the Jieba tool, 24,458 of 50,266 distinct words appear once: 48.7 percent of the vocabulary but 4.0 percent of the text. They state that they are not making a conclusive argument that segmentation is unnecessary.

What to check

• Which segmentation standard does the system follow, and does the test data follow the same one?
• Was a reported F score measured closed or open, and on which corpus?
• What is the recall on out-of-vocabulary words such as new names and transliterated foreign names?
• How does the system segment short search queries, which neither bakeoff evaluated?
• Does the translation or search system work on characters, subwords, or words, and was that choice tested on the task itself?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.

Sources

1. 1The Second International Chinese Word Segmentation Bakeoff · Fourth SIGHAN Workshop on Chinese Language Processing (2005), ACL Anthology
2. 2The First International Chinese Word Segmentation Bakeoff · Second SIGHAN Workshop, Sapporo (July 2003), ACL Anthology
3. 3Chinese Word Segmentation as Character Tagging · Computational Linguistics and Chinese Language Processing, Vol. 8, No. 1 (February 2003)
4. 4Unicode Standard Annex #29: Unicode Text Segmentation · The Unicode Consortium (Unicode 18.0.0, revision 49, 2026-09-01)
5. 5Investigating the Relationship between Word Segmentation Performance and Retrieval Performance in Chinese IR · COLING 2002, ACL Anthology
6. 6Optimizing Chinese Word Segmentation for Machine Translation Performance · Third Workshop on Statistical Machine Translation (June 2008), ACL Anthology
7. 7Is Word Segmentation Necessary for Deep Learning of Chinese Representations? · ACL 2019, ACL Anthology

Where this shows up in our work

• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.