Farsi, Dari, and Pashto: what a language system has to tell apart

Before a system can translate or search a document, it has to know which language the document is in. Persian (also called Farsi), Dari, and Pashto make that hard, because all three are written in the Arabic script with added letters, so the script alone does not say which one a document is in. Dari is a close variant of Persian, and ISO 639-3, the standard that assigns three-letter language codes, treats Pashto as a language separate from Persian. Translation and search both depend on the label chosen at this step, and on which code records it.

Two macrolanguages, five individual languages

In its macrolanguage mappings, ISO 639-3 lists Persian (fas) as a macrolanguage that contains two individual languages: Iranian Persian (pes) and Dari (prs). Pashto, listed there as Pushto (pus), is a separate macrolanguage with three: Southern Pashto (pbt), Northern Pashto (pbu), and Central Pashto (pst). All five entries are marked active.

This produces a granularity problem. A label that says Persian can mean the macrolanguage or either individual language, depending on the system that wrote it. A pipeline that stores one code per document has to decide which level it records, and a rule written against one level will not match records labeled at another. A filter for prs will not return a document that another system labeled fas.

What the script does and does not reveal

The Unicode Standard says the Arabic script has been extended to represent languages including Persian, Urdu, Pashto, Sindhi, and Uyghur (Unicode 17.0, Chapter 9). The additions are extra letters. Peh, for example, is described there as the Persian and Urdu letter with three dots below.

The Unicode locale charts show how the additions differ. The Persian chart lists these among its main letters: پ, چ, ژ, ک, and گ. The Pashto chart lists پ, چ, ژ, and ک as well, then adds ټ, ځ, څ, ډ, ړ, ږ, ښ, ڼ, ې, and ۍ, none of which appears in the Persian list. A letter found only in the Pashto list is a useful clue. A text that uses only the shared letters gives no letter-level evidence either way.

Other characters are shared more widely. Farsi yeh (U+06CC) is used in Persian, Urdu, Pashto, Azerbaijani, Kurdish, and other languages written in the script, with two dots below in initial and medial forms and no dots in final and isolated forms. Digits come in two code point ranges. Unicode describes the Eastern Arabic-Indic digits (U+06F0 to U+06F9) as those used in Afghanistan, India, Iran, and Pakistan, and the Arabic-Indic digits (U+0660 to U+0669) as those used in other parts of the Arabic world. Unicode provides a single digit sequence for Persian, Sindhi, and Urdu. A digit code point identifies a digit set, not a language.

Invisible characters belong to the script as well. Unicode says the zero width non-joiner may be used to prevent joining, as in the Persian plural suffix, and both the Persian and Pashto charts list the zero width non-joiner (U+200C) among their auxiliary characters. A word typed with that character and the same word typed without it are different strings, and a system that compares text has to reconcile them.

Persian and Dari are close and still measurable

A 2015 study by Shervin Malmasi describes Persian and Dari as two close variants: Persian, also known as Farsi, and Dari, which is Eastern Persian spoken predominantly in Afghanistan. The author built a corpus of 28,000 sentences, 14,000 per language, and trained classifiers on character and word n-grams, which are runs of consecutive characters or words. Accuracy reached 96 percent on that corpus. On 79,000 sentences from the Uppsala Persian Corpus, a different collection, accuracy fell to 87 percent.

The study reports lexical, morphological, and orthographic differences between the two. It describes Dari as low-resourced and as too different from Persian for Persian resources to be applied to it directly. A system that folds Dari into Persian therefore gives up distinctions that a classifier can measure.

Why identification stays hard

The drop from 96 to 87 percent shows how much a result depends on the corpus the classifier meets. The authors of PALI, a benchmark for languages written in Perso-Arabic scripts, say identification is difficult in general, and especially so in bilingual communities that write in unconventional ways and in low-resource settings. PALI proposes a hierarchical model that groups the languages classifiers often confuse with one another.

What to check

• Does the system tell Persian, Dari, and Pashto apart, or fold Dari into Persian?
• At which level does it record the language: the macrolanguage (fas) or the individual language (pes, prs)?
• Was identification accuracy measured on documents like these, or on a different collection?
• What does it report for a text that uses only the letters Persian and Pashto share?
• Does it treat a word typed with the zero width non-joiner and the same word without it as one word?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from. The long read on offline machine translation goes further.

Sources

1. 1ISO 639-3 Macrolanguage Mappings · SIL Global (ISO 639-3 code tables)
2. 2The Unicode Standard, Version 17.0, Core Specification, Chapter 9: Middle East-I · The Unicode Consortium
3. 3Locale Data Summary for Pashto [ps] · The Unicode Consortium (Common Locale Data Repository charts)
4. 4Locale Data Summary for Persian [fa] · The Unicode Consortium (Common Locale Data Repository charts)
5. 5Discriminating similar languages: Persian and Dari · Tiny Transactions on Computer Science, Volume 3 (2015), record hosted by Macquarie University
6. 6PALI: A Language Identification Benchmark for Perso-Arabic Scripts · Association for Computational Linguistics (VarDial 2023)