Korean for machines: Hangul blocks, spacing, and particles

Korean is written in Hangul, which the Unicode Standard calls a featural syllabic script: consonant and vowel letters, called jamo, are grouped into square syllable blocks (Unicode 17.0, Chapter 18). Unicode encodes every modern block as a single character and derives its letters by arithmetic. Words are separated by spaces, but the national spelling rules attach particles, the grammatical markers for roles such as subject and object, to the word before them (Korean Orthography). A system that searches or translates Korean has to settle which characters it compares, where it splits words, and what unit it feeds a model. In one 2020 study, the unit alone moved Korean-to-English scores from 7.07 to 40.34 on BLEU, a measure of overlap with reference translations (Park and colleagues).

How Unicode builds a syllable block

Chapter 18 explains that a modern block holds a leading consonant and a vowel, with an optional trailing consonant. There are 19 leading consonants, 21 vowels, and 27 trailing consonants, which makes 399 two-jamo blocks and 10,773 three-jamo blocks, 11,172 in all. They sit in the Hangul Syllables block, U+AC00 to U+D7AF, and are taken from the Korean standard KS C 5601-1992.

Section 3.12 of Chapter 3 sets the base at U+AC00 and the counts at 19 leading consonants, 21 vowels, and 28 trailing values. The 28 is one more than the number of trailing consonants, so that index 0 can mean none. Each leading consonant therefore owns 588 consecutive code points, 21 times 28, and 19 times 588 gives 11,172. To decompose a syllable, subtract U+AC00, divide by 588 for the consonant, divide the remainder by 28 for the vowel, and keep the last remainder for the trailing consonant.

The standard's own example is U+D4DB, HANGUL SYLLABLE PWILH. The index 10459 yields 17, 16, and 15, which point to the jamo U+1111, U+1171, and U+11B6. Composition runs the same arithmetic backward. The character name is computed the same way.

One syllable, several encodings

Unicode also encodes conjoining jamo, starting at U+1100, that spell a syllable as a sequence of letters. Chapter 18 notes that thousands of archaic syllables can be written only that way. The arithmetic above is the normative canonical decomposition, so a precomposed syllable and its jamo sequence are canonically equivalent, and normalizing text to one form makes them the same string.

A third set looks alike and behaves differently. The Hangul Compatibility Jamo block, U+3130 to U+318F, holds spacing letters with no joining behavior, provided solely for compatibility with the KS X 1001:1998 standard. Unicode says the compatibility normalization forms, NFKD and NFKC, convert them to conjoining jamo. A pipeline that skips normalization can store the same syllable as two different strings, and exact string matching treats them as different words.

Spacing rules and particles

The Korean Orthography, issued as Ministry of Culture, Sports and Tourism Notice No. 2017-12 and effective March 28, 2017, states the base rule in Article 2: each word in a sentence is written with spaces. Article 41 makes particles the exception. A particle is written attached to the word before it, and the commentary says this holds when particles are stacked or follow a verb ending, as in 학교에서처럼 and 나에게만이라도. The reason given is that a particle cannot stand alone. One particle can also take two shapes: the commentary to Article 1 says 이 in 하늘이 and 가 in 바다가 are the same subject morpheme.

A space-separated chunk is therefore often a noun with its particles, not a dictionary word. The rules also allow choices. Article 47 writes auxiliary verbs apart in principle but allows them joined. Article 49 does much the same for proper nouns other than personal names, as in 대한 중학교 and 대한중학교. Article 50 says technical terms are spaced word by word in principle but may be joined, and gives 중거리 탄도 유도탄 and 중거리탄도유도탄 as an example. The Standard Korean Language Dictionary files that term under military vocabulary, as a guided missile with a range of 1,000 to 3,000 km, and lists seven similar words, among them 중거리 탄도 미사일 and 아이아르비엠, a Hangul spelling of IRBM.

Which unit to feed a model

Karl Stratos reported in a 2017 parsing study that 2,703 of 4,698 word types in the held-out part of a treebank, over 57 percent, were unseen in training. Replacing words with jamo shrank the input space by 98.4 percent and raised the parser's labeled attachment score from 82.77 to 91.46. Moon and Okazaki show why letters help: 하다, 한다, 할거야, and 합니다, forms of the verb "to do," begin with four different syllables but share the jamo ㅎ and ㅏ.

Park and colleagues trained Transformer translation models on 800,000 Korean-English news sentence pairs and compared units. On the test set, whole words scored 7.07 BLEU from Korean to English and 18.42 the other way, with an out-of-vocabulary rate of 26.20 percent. Syllables scored 38.75 and 38.45, and subwords alone 38.69 and 36.98. Splitting into morphemes first and then into subwords scored best, 40.34 and 40.41. In the 32K-vocabulary subword model, 30.26 percent of tokens crossed a morpheme boundary. The morphological analyzer drops the original spaces, so the authors inserted a marker to rebuild Korean output.

The same 2020 paper notes that Korean was then absent from the language list of WMT, the Conference on Machine Translation. The WMT 2025 general task added English to Korean, scored by professional translators with Multidimensional Quality Metrics (MQM). The human reference scored -1.9 and shared the top rank range, 1 to 3, with two machine systems.

What to check

• Does the system normalize Hangul to one form before it indexes, compares, or counts characters?
• Does it map compatibility jamo (U+3130 to U+318F) to conjoining jamo, or treat them as different letters?
• Does search find a technical term written both spaced and joined, as Article 50 allows, and its listed synonyms?
• Which token unit does the translation model use, and was the score measured on the same test set as a competitor's?
• Does the pipeline keep the original spaces, so Korean output can be rebuilt after morphological analysis?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.

Sources

1. 1The Unicode Standard, Version 17.0, Core Specification, Chapter 3: Conformance (Section 3.12, Conjoining Jamo Behavior) · The Unicode Consortium
2. 2The Unicode Standard, Version 17.0, Core Specification, Chapter 18: East Asia (Section 18.6, Hangul) · The Unicode Consortium
3. 3한글 맞춤법 (Korean Orthography), Ministry of Culture, Sports and Tourism Notice No. 2017-12, effective March 28, 2017 · National Institute of Korean Language (Korean language norms)
4. 4표준국어대사전 (Standard Korean Language Dictionary), entry 중거리 탄도 유도탄 · National Institute of Korean Language
5. 5An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks · Park, Lee, Jang, and Jung, AACL-IJCNLP 2020 (ACL Anthology)
6. 6Jamo Pair Encoding: Subcharacter Representation-based Extreme Korean Vocabulary Compression for Efficient Subword Tokenization · Moon and Okazaki, LREC 2020 (ACL Anthology)
7. 7A Sub-Character Architecture for Korean Language Processing · Karl Stratos, EMNLP 2017 (ACL Anthology)
8. 8Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets · Kocmi and colleagues, WMT 2025 (ACL Anthology)

Where this shows up in our work

• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.