Kurdish for machines: Kurmanji, Sorani, and two scripts

Kurdish is an Indo-European language whose main varieties use different scripts. Kurmanji, or Northern Kurdish, is mostly written in a Latin-based alphabet; Sorani, or Central Kurdish, along with Southern Kurdish and Laki, mostly in an Arabic-based one, according to a 2020 paper by Sina Ahmadi on KLPT, the Kurdish Language Processing Toolkit. The paper cites 20 to 30 million speakers. The ISO 639-3 code tables list the varieties as individual languages under one macrolanguage. Before a system can tokenize, search, or translate Kurdish, it has to know which variety and which script a document uses, and it has to repair letters that arrive as the wrong code points. Published Sorani results show what each step costs.

Codes and varieties

The ISO 639-3 macrolanguage table lists Kurdish (kur) with three active individual languages: Central Kurdish (ckb), Northern Kurdish (kmr), and Southern Kurdish (sdh). The KLPT paper adds Laki, code lki, which the table does not place under kur. Ahmadi notes that scholars disagree on whether these are dialects of one language or separate languages, and adopts the dialect view. The same paper places Zazaki and Gorani, often taken for Kurdish dialects, in a separate Zaza-Gorani language family.

The grammars differ in ways a translation system meets directly. KLPT notes that Sorani has no grammatical gender or case but a full set of article markers for definite, indefinite, and demonstrative nouns in singular and plural, while Kurmanji has fewer article markers, for feminine and masculine genders. Sorani has no future tense and uses adverbs instead. In the past tense of transitive verbs both varieties shift to an ergative pattern, but Kurmanji marks it with oblique case and Sorani only with pronoun markers. Kurdish also builds many verbs from a noun, adjective, or preposition plus one of around 300 single-word verbs, such as kirdin, to do: siław kirdin means to greet, and pîroz kirdin to congratulate.

Two alphabets

The Unicode Common Locale Data Repository (CLDR) lists the letters each locale uses. The chart for Kurdish (ku) gives 31 main Latin letters: the basic 26 plus ç, ê, î, ş, and û. The chart for Central Kurdish (ckb) gives 33 Arabic-script main letters. They include ڕ (U+0695), ڵ (U+06B5), ڤ (U+06A4), ە (U+06D5), ۆ (U+06C6), and ێ (U+06CE), along with keheh (U+06A9) and Farsi yeh (U+06CC). Arabic kaf (U+0643) and Arabic yeh (U+064A) appear only in the auxiliary list.

The Unicode Standard's Chapter 9 calls the Arabic-based alphabet Soraní. U+06C6 writes the vowel [o], and the long vowel [uː] is a digraph of two waw characters, U+0648 twice. Two letters are used only for Soraní: U+0695 for the trilled r and U+06B5 for the velarized l. Unicode names look-alike variants, U+0692, U+0694, U+06B6, U+06B7, and U+06CA, and says none is required or desired in a regular implementation.

The two alphabets do not map one to one. Ahmadi writes that waw (U+0648) and yeh (U+06CC) each stand for two Latin values, u or w and î or y, and that the Arabic-based script has no letter for the vowel i. The rule-based transliterator in KLPT detects that i with an accuracy of 39 percent.

One letter, several code points

KLPT describes the cleanup problem. Kurdish keyboard layouts assign different encodings to similar-looking letters, and many writers also type on Arabic, Turkish, or Persian keyboards. The letter for î or y can arrive as U+064A, U+0649, U+FEF2, U+FEF1, or U+06CC, and only U+06CC should be used. Numerals arrive in three digit sets. In Kurmanji, a suffix on a year can be written 2020'an, 2020-an, or 2020an. Word boundaries differ as well: the Arabic-based script tends to join clitics and words, so hîwaşyane, "(it) is also their hope," is one written word holding four tokens, while Latin-script Kurmanji writes the phrase as hêvîya wan jî ew e.

Ahmadi and Antonios Anastasopoulos measured the cost. They call this unconventional writing: Sorani typed with Arabic or Persian conventions, such as سويدي, Swedish, spelled with Arabic yeh (U+064A) instead of the Sorani سویدی with U+06CC. On the 1,012-sentence FLORES-200 devtest set, a public 600-million-parameter translation model scored 30.81 BLEU, a measure of overlap with reference translations, from clean Sorani into English. At the paper's highest level of Arabic-convention noise, the score fell to 4.72. Running the authors' normalization model first brought it back to 24.53, recovering almost 20 of the 26 points lost.

Data and translation results

Data is thin and leans toward one variety. KLPT's review of 53 publications on Kurdish language processing found Sorani in almost 90 percent of them, a smaller share for Kurmanji, and none for Southern Kurdish or Laki. Only 18 of the 53 released their resources or tools under an open-source license.

At a 2020 workshop on machine translation for low-resource languages, Ahmadi and Mariam Masoud trained neural Sorani-English systems. The largest parallel source was Tanzil, a collection of Quran translations with 92,354 parallel sentences; the TED talk subtitle collection had 2,358. On Tanzil test data the best Sorani-to-English score was 22.03 BLEU. English to Sorani reached 59.28, which the authors explain in part by one Kurdish translation being aligned with 11 English ones. On TED, every setup scored 1.00 BLEU or less from Sorani to English, and translation edit rate (TER) was one in almost all cases. The authors point to imbalanced data, sentence types, alignment quality, and domain-specific terms. A system trained mostly on one kind of text did poorly on another.

What to check

• Does the system label Kurdish at the level of ckb, kmr, and sdh, or only as kur?
• Does it read Sorani in Arabic script and Kurmanji in Latin script, and how does it restore the unwritten vowel i when it transliterates?
• Does it map Arabic yeh, alef maksura, presentation forms, and Arabic kaf to U+06CC and U+06A9 before search and translation?
• Does search treat the three digit sets as the same numbers?
• Was translation quality measured on documents like the ones at hand, or on religious text and subtitles?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.

Sources

1. 1ISO 639-3 Macrolanguage Mappings · SIL Global (ISO 639-3 code tables)
2. 2The Unicode Standard, Version 17.0, Core Specification, Chapter 9: Middle East-I · The Unicode Consortium
3. 3Locale Data Summary for Central Kurdish [ckb] · The Unicode Consortium (Common Locale Data Repository charts)
4. 4Locale Data Summary for Kurdish [ku] · The Unicode Consortium (Common Locale Data Repository charts)
5. 5KLPT - Kurdish Language Processing Toolkit · Sina Ahmadi, NLP-OSS 2020 (ACL Anthology)
6. 6Towards Machine Translation for the Kurdish Language · Ahmadi and Masoud, LoResMT 2020 (ACL Anthology)
7. 7Script Normalization for Unconventional Writing of Under-Resourced Languages in Bilingual Communities · Ahmadi and Anastasopoulos, ACL 2023 (ACL Anthology)

Where this shows up in our work

• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.