Pashto machine translation: script, resources, and pitfalls

Pashto is an Eastern Iranian language of Afghanistan and Pakistan, written right to left in a Perso-Arabic script with a dozen letters that Arabic does not have. Meta's NLLB-200 covers it under the code pbt_Arab, which the FLORES-200 README spells out as Southern Pashto in Arabic script, and the No Language Left Behind paper classifies it as a low-resource language. Searchers ask for Pashto translation AI, Pashto NLLB, Pashto OCR, and how accurate Pashto to English translation is. The answers depend on five things that a benchmark score does not show: the script, the dialect, the available text, the condition of the document, and the spelling of names.

The codes for Pashto, from ps to pbt_Arab

The language has several names and several codes, and systems disagree about which to use. The ISO 639-3 registration authority lists pus, reference name Pushto, as a macrolanguage whose two-letter ISO 639-1 equivalent is ps, the code Wikipedia and OPUS use. Inside it sit three individual languages: pbt, Southern Pashto; pbu, Northern Pashto; and pst, Central Pashto. FLORES-200 and NLLB-200 chose pbt, the southern variety on which the written standard is based, and attached the ISO 15924 script code Arab. A system labeled ps and a system labeled pbt_Arab are both Pashto, but only the second says which Pashto.

The Linguistic Data Consortium's language specification for Pashto, written for a speech corpus collected in Pakistan, lists the alternative names Pushtu, Afghan, Pushto, Pashto, Pashtoe, Pashtu, and Pukhto, and explains that the variation in the name's own spelling comes from dialect pronunciation of its second consonant. The document estimates about 8 million first-language speakers in Afghanistan, around 28 percent of the population, and about 9 million in Pakistan, around 12 percent, and notes that Pashto was designated one of Afghanistan's two official languages in 1936, alongside Dari.

A Perso-Arabic script with extra letters

Pashto uses the Arabic script as extended for Persian and then extended again. The Unicode core specification's Chapter 9 says the Arabic script has been extended to represent languages including Persian, Urdu, Pashto, Sindhi, and Uyghur, and that the extended letters live in the Arabic block and its supplements. The U.S. National Geospatial-Intelligence Agency's Romanization of Pashto, the BGN/PCGN 1968 system in its 2017 revision, gives the count: the alphabet "contains twelve additional consonants not present in standard Arabic," plus three additional vowel characters and an additional vowel point. Four of the twelve are the Persian additions for p, ch, zh, and g; the other eight are Pashto's own. The LDC specification counts the same inventory another way: 27 standard Arabic letters plus 17 extended symbols, 14 of them characteristic of Pashto, namely 6 retroflex consonants, 3 affricates and fricatives, a ringed kaf pronounced g, and 3 additional y letters.

The Unicode Arabic code chart for version 18.0 annotates the following characters as Pashto.

The chart also marks U+0682, hah with two dots vertical above, as not used in modern Pashto, and notes that the kaf with ring may appear like an Arabic kaf with a ring below the base. Several of these letters are visually one dot or one ring away from a common Arabic letter, which is the root of both the OCR problem and the spelling problem below.

Right-to-left text and joining behavior

Two properties of the script affect every stage of processing. First, the Unicode chapter states that the Arabic script is written from right to left, so a Pashto document mixes right-to-left text with left-to-right numerals and any embedded English, and software has to apply the bidirectional algorithm to display it. Second, Arabic letters join. The chapter says a rendering system must select a glyph for each letter according to its joining context, and that dual-joining letters have four distinct forms, for isolated, final, medial, and initial positions, while right-joining letters have only two. The PsOCR paper counts 44 Pashto letters that can each take up to four contextual forms, with ligatures and diacritical marks expanding the range of glyph shapes further.

The LDC specification notes one consequence for text data: a zero-width non-joiner, U+200C, is placed between two characters that would otherwise connect, to force final and initial forms, and the document is careful to say that this character is not a word boundary. A pipeline that strips or miscounts such invisible characters will change what the text looks like without changing its letters.

The vowels are the other gap. The NGA document states that the points Arabic uses to mark short vowels are not written in Pashto, so a reference source may be needed to identify standard spellings and the correct vowels. The LDC document adds that spelling standardization is a significant issue, and that in Pakistan writers tend to substitute the Urdu forms of letters, which fail to distinguish some Pashto sounds, including the five y letters. A translation system is therefore reading unvowelled text in which the same letter string can be more than one word and in which the writer may have used a neighboring language's alphabet.

Kandahari, Peshawari, and the five consonants that split the dialects

Pashto dialects differ mainly in pronunciation and vocabulary, and the differences are large enough to show in the spelling of the language's name. The LDC specification, prepared by Appen for a Pashto speech corpus, defines four dialect regions by their phoneme distributions: South-West, around Kandahar, which the document calls the written standard; North-East, around Peshawar and Jalalabad and parts of Kabul; South-East, around Quetta; and Central/North-West, covering less common variants such as Central Ghilzai and Wardak. The document describes the South-West dialect as the most prestigious and says the Kandahar pronunciation is the basis of the standard written form, while noting that about 9.7 million people speak northern dialects against about 2.7 million for southern ones, with roughly 80 percent lexical similarity between the two.

The split rests on five consonants, which the document tabulates with their Unicode code points. The letter U+069A, the second consonant of the language's name, is a voiceless retroflex fricative in Kandahari, a velar fricative in the North-East, and a postalveolar sh sound around Quetta. The voiced partner U+0696 is a retroflex fricative in the south and a g sound in the north. The affricates U+0685 and U+0681, ts and dz in the south, become plain s and z in the North-East, and U+0698 shifts from zh to a j sound. The written form does not change, so a Kandahari and a Peshawari reader see the same word and say it differently; speech systems, and transliterated names, carry the difference into text.

Wilma Heston's Digital Pashto-English Dictionary at the University of Chicago uses the same consonant to sort dialects into hard and soft groups, classing Yusufzai, which the LDC specification places in the North-East group around Peshawar, as hard and Khattak as soft, which is why the language's name is romanized Pakhto for the hard dialects and Pashto for the soft ones.

Public corpora from FLORES-200, OPUS, and Wikipedia

Pashto has been a test language longer than most low-resource languages. The NLLB paper describes the original FLORES set of 2019 as pairing four low-resource languages, Khmer, Nepali, Pashto, and Sinhala, with English in the Wikimedia domain. The FLORES-101 paper used English to Pashto as one of three directions to validate its spBLEU metric against WMT 2020 human evaluations, and reports that spBLEU ranked the five Pashto systems identically to the human judges. The WMT20 news translation task page lists Pashto-English among its pairs and says the Pashto and Khmer parallel data came mostly from OPUS and points to the parallel corpus filtering task for details, with Pashto added to the Common Crawl monolingual data and a Wikipedia dump supplied.

The OPUS API lists what is available for English-Pashto today, and the list shows how concentrated the data is.

The total is about 11.9 million pairs, and 95 percent of it is the NLLB corpus, which is text mined automatically from the web by the NLLB project rather than translated by people. The human-translated sets are tiny: TED talks, the tico-19 COVID-19 set, and software interface strings from GNOME and translatewiki.

Monolingual text is thin too. The List of Wikipedias on Wikimedia Meta ranks Pashto Wikipedia 138th with 21,356 articles, 77,351 total pages, 369,473 edits, 40,355 registered users, 61 active users, and 3 administrators, and the Pashto Wikipedia's statistics page shows nearly the same live figures. Since FLORES sentences come from English Wikimedia projects and much monolingual training data comes from Wikipedia, a language with 21,000 articles has little encyclopedic text of its own to learn from.

What NLLB-200 reports for Pashto on FLORES-200

The NLLB repository README links a metrics file for each released checkpoint, each a table of FLORES-200 scores for every translation direction. The dense model files carry chrF++ only; the mixture-of-experts file adds spBLEU computed with both the SPM-200 and the SPM-100 tokenizers. The 3.3B dense file and the 54.5B MoE file give the Pashto rows below; the other dense models' files sit at the same host.

Three things stand out. The direction into English scores 17.6 to 20.6 chrF++ points higher than the direction into Pashto, depending on the model. Model size buys little: the gap between the 600M distilled model and the 54.5B model is 2.5 points into Pashto and 5.5 points into English. And the two spBLEU columns differ, 22.9 against 22.5 and 39.9 against 39.0, the tokenizer dependence the NLLB paper warns about in its metrics section.

The paper's Table 39 adds a comparison on FLORES-200 devtest between bilingual models trained on single low-resource directions and the multilingual models trained on the same data. For pbt_Arab the bilingual English-to-Pashto model scored 37.1 chrF++ against 35.3 for the multilingual dense model and 36.5 for the MoE model, a small loss, while Pashto to English rose from 47.2 to 56.6 and 56.9, gains of 9.4 and 9.7 points. The caption says most low-resource languages benefit from multilingual transfer and that gains are more consistent when translating into English. For Pashto the transfer helped reading, not writing.

Tokenization sets a second ceiling. Petrov and colleagues' tokenizer fairness study measured, on FLORES-200, how many more tokens each language needs than English. Their appendix tables put Southern Pashto at 5.39 times English under GPT-2, 3.83 under the ChatGPT and GPT-4 tokenizer, and 1.45 under NLLB's own SPM-200. A model that spends almost four tokens of context per English-token equivalent has less room for a Pashto paragraph, and a sentence-level system like NLLB-200, which the paper evaluates one sentence at a time, never sees the paragraph at all.

Pashto OCR, Naskh, and Nastaliq

Much Pashto reaches an analyst as an image, and recognition comes before translation. The PsOCR paper, by Ijazul Haq and colleagues, describes the obstacles: a cursive right-to-left script of 44 letters with up to four contextual forms each, ligatures, inconsistent use of diacritical marks, and no clear word boundaries. Before PsOCR, the paper says, the available Pashto datasets were small, naming the KPTI database of about 17,000 handwritten and printed text-line images and the HPCID set of about 15,000 handwritten samples. PsOCR itself is synthetic: one million images rendered in 1,000 font families with word, line, and document bounding boxes. On a 10,000-image benchmark subset, the paper's Table 4 reports Gemini at a character error rate of 0.10 and a word error rate of 0.31, GPT-4o at 0.30 and 0.60, and the best open model, Qwen-7B, at 0.34 and 0.73. A word error rate of 0.31 on clean synthetic text means roughly one word in three wrong before any translation begins.

The calligraphic style of the page matters as much as the font. The Unicode chapter notes that Urdu is often written in the ornate Nastaliq style. Naz and colleagues, writing on Urdu recognition in SpringerPlus in 2016, say Arabic uses the Naskh style while Urdu and Persian follow Nastaliq, and list what Nastaliq does to a recognizer: a diagonal writing direction, ascenders and descenders that defeat baseline detection, several morphologies of the same character rather than four, and gaps after non-joining letters that look like word spaces. The same paper cites earlier work on Pashto ligature recognition, and PsOCR's own top-ten font list includes a Naskh face. Which style a given Pashto document uses depends on its publisher, and an engine trained on one style meets the problems above on the other. The post on Urdu and the Nastaliq script covers the published recognition results for that style.

Name transliteration variance

Pashto names arrive in Latin letters through several systems, and the systems disagree. The NGA's BGN/PCGN Pashto system is the U.S. and U.K. standard for geographic names; it renders the Pashto-specific letters with diacritics, so U+067C becomes t with a line below, U+069A becomes s͟h with a double underline, and U+06BC becomes n with a line below, giving examples such as Pas͟htūn Kōṯ, Kandahār, Khōst, and Jalālābād. Because short vowels are not written, the document says a reference source may sometimes be required to settle standard spellings and remove dialectal and idiosyncratic variation, and it adds that the pronunciation of short vowels will vary. The LDC corpus used a modified Buckwalter scheme instead, an ASCII-only system built for computers. Heston's dictionary uses its own phonetic transcription, and its comparison table notes that the ALA-LC library scheme distinguishes letters such as seen and sad that sound the same in Pashto.

Dialect adds a layer. The same letter is kh in Peshawar and a retroflex sh in Kandahar, so a northern speaker's name written from sound and a southern one's can differ in the Latin alphabet while being identical in Pashto script. Combine unwritten vowels, three romanization systems, and two pronunciations, and one person can appear under several spellings across documents, watchlists, and translations. The post on transliterating names covers the standards and why matching across them is hard.

U.S. government interest in Pashto

Pashto has been a priority language for U.S. forces since the Afghanistan deployments. A February 24, 2010 Army STAND-TO! article on the Defense Language Institute Foreign Language Center (DLIFLC) described an Afghan Languages Portal gathering the institute's Dari and Pashto products for deploying troops: basic language guides, HeadStart programs of 80 to 100 hours of instruction, and material for linguists at higher proficiency levels. The article reported that the commander in Afghanistan required all troops to learn at least 200 words of Dari or Pashto and one soldier per platoon to study a language intensively. The National Security Education Program's list of critical languages and the current DLIFLC language catalog could not be confirmed from a public government page for this post, so no claim is made here about Pashto's current designation.

What to check

• Which Pashto does the system claim: the ISO 639-1 code ps, the macrolanguage pus, or pbt_Arab, Southern Pashto in Arabic script, as NLLB-200 uses?
• Which direction and which metric does a quoted score describe, since NLLB's own files put Pashto to English 17.6 to 20.6 chrF++ points above English to Pashto, and spBLEU differs by tokenizer?
• Does the system's tokenizer cover the twelve Pashto letters and the zwarakay vowel sign, or do they fall to unknown or byte tokens?
• Are the documents typed text or scans, and if scans, are they in Naskh or Nastaliq, and what character and word error rates does the OCR stage report on Pashto specifically?
• Which dialect region produced the documents, and does the vocabulary of the training data match it?
• Which romanization system does the program use for names, and does the matching step allow for unwritten vowels and dialect differences in the letters the systems render differently?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.

Questions people ask

Does NLLB-200 support Pashto?

Yes. The FLORES-200 README lists Southern Pashto under the code pbt_Arab, and the No Language Left Behind paper's language table classifies it as a low-resource language written in Arabic script. The metrics files linked from the NLLB repository give FLORES-200 chrF++ scores for every model: for the 3.3B dense model, 58.2 translating Pashto into English and 38.6 translating English into Pashto. The paper notes that Pashto was already one of four languages in the original 2019 FLORES set.

What is the NLLB language code for Pashto?

pbt_Arab. The first part is the ISO 639-3 code pbt, which the ISO 639-3 registration authority defines as Southern Pashto, one of three individual languages inside the macrolanguage pus, Pushto, alongside pbu Northern Pashto and pst Central Pashto. The second part is the ISO 15924 script code Arab. The two-letter ISO 639-1 code ps, used by Wikipedia and OPUS, maps to the macrolanguage, not to a specific dialect.

What script is Pashto written in?

A modified Perso-Arabic script written right to left. The NGA's Romanization of Pashto document says the alphabet contains twelve consonants not present in standard Arabic, three additional vowel characters, and an additional vowel point, and that the points Arabic uses for short vowels are not written in Pashto. The Unicode Arabic block chart annotates letters such as U+067C teh with ring, U+0685 hah with three dots above, U+069A seen with dot below and dot above, and U+06BC noon with ring as Pashto.

What are the main Pashto dialects?

The LDC's language specification for Pashto describes four dialect regions: South-West around Kandahar, which is the basis of the written standard; North-East around Peshawar and Jalalabad; South-East around Quetta; and Central/North-West. The split turns on five consonants. The letter for the sound in the language's own name is a retroflex fricative in Kandahari and a velar fricative in Peshawari, which is why the name appears as both Pashto and Pakhto. The document gives about 80 percent lexical similarity between northern and southern varieties.

How accurate is Pashto to English translation?

On FLORES-200, the NLLB metrics files give chrF++ scores into English of 54.5 for the 600M distilled model, 57.0 for the 1.3B dense model, 58.2 for the 3.3B dense model, and 60.0 for the 54.5B mixture-of-experts model, with the MoE model's spBLEU at 39.9. The NLLB paper's Table 39 shows the direction into English gaining about 9 chrF++ points from multilingual training over a bilingual baseline, while English into Pashto did not gain. These are sentence-level scores on Wikimedia text.

Where can I get Pashto parallel data?

The OPUS collection's API lists English-Pashto corpora with their sizes: NLLB with 11,348,028 aligned pairs, CCAligned with 299,634, GNOME with 95,312, XLEnt with 54,923, translatewiki with 37,301, ParaCrawl with 26,322, and smaller sets such as TED2020, tico-19, and WikiTitles, about 11.9 million pairs in total. The WMT20 news task page says its Khmer and Pashto parallel data came mostly from OPUS. FLORES-200 itself is an evaluation set and its card says it should not be used for training.

How big is Pashto Wikipedia?

Small. The List of Wikipedias on Wikimedia Meta ranks Pashto Wikipedia 138th with 21,356 articles, 77,351 total pages, 369,473 edits, 40,355 registered users, 61 active users, and 3 administrators, and the Pashto Wikipedia's own statistics page shows nearly the same live counts. For comparison, the OPUS wikimedia corpus for English-Pashto holds 9,116 aligned pairs. The thin Wikipedia matters because FLORES sentences and much monolingual training text come from Wikimedia projects.

Why is Pashto OCR difficult?

The PsOCR paper describes Pashto as written in a cursive right-to-left script of 44 letters, each taking up to four contextual forms, with ligatures, inconsistently used diacritics, and no clear word boundaries. The same paper notes that only small Pashto datasets existed before it, such as 17,000 KPTI text-line images and 15,000 HPCID handwritten samples, and its own benchmark of a one-million-image synthetic set found the best model at a character error rate of 0.10 and a word error rate of 0.31.

What is the difference between Naskh and Nastaliq for Pashto?

Naskh and Nastaliq are two calligraphic styles of the same script. Naz and colleagues' 2016 paper on Urdu OCR says Arabic uses Naskh while Urdu and Persian follow Nastaliq, and that Nastaliq's diagonal writing, shifting baseline, and many shapes per character create recognition problems Naskh does not. Pashto shares its letters with both traditions, so a Pashto document may arrive in either style, and an OCR engine trained on Naskh will struggle with Nastaliq pages.

Why do Pashto names have so many spellings?

Because short vowels are not written and several romanization systems coexist. The NGA's BGN/PCGN Pashto system notes that a reference source may be needed to recover standard spellings and eliminate dialectal variation, and it renders the Pashto-specific letters with diacritics, giving Pas͟htūn Kōṯ and Kandahār as examples. The LDC specification used a modified Buckwalter scheme instead, and Heston's Pashto dictionary uses its own phonetic transcription, so the same name can be spelled three ways by three careful sources.

Sources

1. 1No Language Left Behind: Scaling Human-Centered Machine Translation · arXiv (NLLB Team, Meta AI), v3 of August 25, 2022
2. 2FLORES-200 (README) · Meta AI Research (facebookresearch on GitHub)
3. 3fairseq, NLLB branch: No Language Left Behind (README) · Meta AI Research (facebookresearch on GitHub)
4. 4NLLB-200 3.3B dense model: FLORES-200 metrics (metrics.csv) · Meta AI Research
5. 5NLLB-200 54.5B MoE model: FLORES-200 metrics (metrics.csv) · Meta AI Research
6. 6The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation · arXiv (Goyal et al.), June 6, 2021
7. 7ISO 639-3 code pbt: Southern Pashto · SIL Global, ISO 639-3 Registration Authority
8. 8ISO 639-3 code pus: Pushto · SIL Global, ISO 639-3 Registration Authority
9. 9Arabic, Range: 0600-06FF (code chart, The Unicode Standard, Version 18.0) · Unicode, Inc.
10. 10The Unicode Standard, Core Specification, Chapter 9: Middle East-I · Unicode, Inc.
11. 11Romanization of Pashto, BGN/PCGN 1968 System, 2017 Revision · U.S. Board on Geographic Names and Permanent Committee on Geographical Names, hosted by the National Geospatial-Intelligence Agency
12. 12Language Specific Peculiarities Document for Pashto as Spoken in Pakistan and Afghanistan · Linguistic Data Consortium (documentation for LDC2016S09)
13. 13A Digital Pashto-English Dictionary with Audio (front matter) · Wilma L. Heston, Digital South Asia Library, University of Chicago, April 2005
14. 14OPUS API: corpora for English-Pashto (source=en, target=ps) · OPUS, the open parallel corpus (University of Helsinki / NLPL)
15. 15Translation Task, Fifth Conference on Machine Translation (WMT20) · Conference on Machine Translation (WMT20)
16. 16List of Wikipedias · Wikimedia Meta-Wiki
17. 17Pashto Wikipedia: Statistics (Special:Statistics) · Wikimedia Foundation
18. 18PsOCR: Benchmarking Large Multimodal Models for Optical Character Recognition in Low-resource Pashto Language · arXiv (Haq, Zhang, and co-author), v2 of January 9, 2026
19. 19Urdu Nasta'liq text recognition using implicit segmentation based on multi-dimensional long short term memory neural networks · SpringerPlus (2016), open-access copy on PubMed Central
20. 20Language Model Tokenizers Introduce Unfairness Between Languages · arXiv (Petrov, La Malfa, Torr, Bibi), v2 of October 20, 2023; NeurIPS 2023
21. 21The Defense Language Institute Foreign Language Center (STAND-TO!) · U.S. Army, February 24, 2010

Where this shows up in our work

• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.