Why Arabic is hard for machines

An analyst searching Arabic documents, or an office buying a system to read them, runs into three properties that compound: a script that leaves out most vowel information, words that carry prefixes and suffixes attached to a stem, and a gap between the formal written standard and the spoken dialects. Each one breaks an assumption that search indexes and translation systems make about what a word is. The practical result is a query that misses documents containing the very word it asked for.

Direction and shapes: strings that look the same

Arabic is written from right to left. The Unicode Standard stores characters in logical order, the order in which they are read, and requires a display process to convert that to the visual order on screen (Unicode 17.0, Chapter 9). A tokenizer, the step that splits text into units a model can process, therefore sees characters in reading order, and direction is handled only when text is drawn.

Letters are cursive. A letter takes one of up to four shapes depending on its neighbors: isolated, initial, medial, or final. Unicode gives each letter a single code point regardless of how many shapes it shows, and leaves shape selection to the renderer. The Arabic Presentation Forms-A block is encoded primarily for compatibility reasons, and Unicode gives those characters a preferred representation using ordinary Arabic letters. Two strings that look identical on screen can therefore differ in code points, and a system that compares strings has to reconcile them before it can match them.

Missing vowels leave the machine to guess

The short vowels are the larger problem. Vowel and other marks, called tashkil, are encoded as combining marks on base letters, and in normal writing they are omitted. A reader supplies the vowels from context. A machine has to do the same.

Fadel and colleagues give an example: one unvocalized three-letter verb form can be read as a verb meaning talked or as one meaning wounded. In their Arabic-English neural translation experiments on one million sentence pairs, adding diacritics as a separate input raised the BLEU score (a standard translation metric) from 33.01 to 34.34. The missing marks carry information the translation model can use. That is one result on one dataset, not a general figure for Arabic.

One written word, several tokens

Arabic morphology is concatenative, meaning affixes attach to stems. The AraBERT paper gives a plain case. The definite article Al is always prefixed, yet it is not an intrinsic part of the word it attaches to. Without segmentation, the forms kitAb and AlkitAb both enter the vocabulary as separate items, which the authors call unnecessary redundancy.

Their fix was to split words into stems, prefixes, and suffixes with the Farasa segmenter before training a subword vocabulary. Pre-segmentation improved sentiment analysis and question answering but reduced performance on named-entity recognition. Segmentation changes what a model sees, and the effect depends on the task.

A word-level search index has the same problem. It stores kitAb and AlkitAb as different terms, so a query for one can miss documents containing the other unless the system splits or normalizes the words. A vocalized word and its bare form are likewise different strings, because tashkil are separate combining code points, until a system decides to strip the marks. Compatibility characters add a third mismatch: a document can use them where the query uses ordinary letters.

Dialects and the standard

Modern Standard Arabic (MSA) is the standard variety, and the dialects differ from it and from each other. In the Inoue et al. study, the MSA models were pre-trained on sources such as news and Wikipedia text, while the dialectal models were trained mostly on transcripts and social media. The authors found that matching the variant of the pre-training data to the variant of the task data mattered more than the amount of pre-training data.

Translation adds a decision step. Shapiro and Duh note that a translator of a diglossic language like Arabic, one with a formal standard and spoken varieties, may receive text whose dialect is unknown. The traditional design identifies the dialect and then routes the text to a dialect-specific translation system. With neural translation, a shared multi-dialect system has become a natural alternative, and their paper studies when identifying the dialect first is worth doing.

What to check

• Does search match a word with and without vowel marks?
• Does a query for kitAb return documents containing AlkitAb, and the reverse?
• Does the system reconcile compatibility characters with ordinary Arabic letters before it compares strings?
• Was the system trained and tested on Modern Standard Arabic, or on the dialect of the documents it will read?
• Does it identify the dialect first and route the text, or use one shared system, and what evidence supports that choice?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from. The long read on offline machine translation goes further.

Sources

1. 1The Unicode Standard, Version 17.0, Core Specification, Chapter 9: Middle East-I · The Unicode Consortium
2. 2AraBERT: Transformer-based Model for Arabic Language Understanding · arXiv, from the Proceedings of the Twelfth International Conference on Language Resources and Evaluation (LREC 2020)
3. 3Neural Arabic Text Diacritization: State of the Art Results and a Novel Approach for Machine Translation · arXiv, published in the Proceedings of the 6th Workshop on Asian Translation (Association for Computational Linguistics, 2019)
4. 4Comparing Pipelined and Integrated Approaches to Dialectal Arabic Neural Machine Translation · Association for Computational Linguistics (VarDial 2019)
5. 5The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models · arXiv, accepted to the Workshop on Arabic Natural Language Processing (WANLP 2021)