Languages
Offline speech recognition: what the Whisper paper reported
What the 2022 Whisper paper reported: 680,000 hours of training audio, how word error rate is measured, how accuracy falls with less training data per language, and the failure modes the authors state.
Oct 21, 20264 min read6 sources
Why it matters
Whisper's word error rate was 3.0 percent in Spanish and 93.7 percent in Pashto on the same benchmark, so a single accuracy claim says little about a given language.
The short version
- 1Of 680,000 training hours, 65 percent was English transcription, 17 percent other-language transcription, and 18 percent translation into English.
- 2The authors found that word error rate halves for every 16-fold increase in a language's training data.
- 3The authors name repeat loops, dropped words at segment edges, and complete hallucination as remaining failure modes.
Automatic speech recognition (ASR) turns recorded speech into text, and a model whose weights are released can run on a local machine with no network connection. The Whisper paper, posted by OpenAI researchers on December 6, 2022, describes models trained on 680,000 hours of audio paired with transcripts from the internet (Radford and colleagues). The project README states that the code and model weights are released under the MIT License. The paper's main caution for multilingual use is that performance on a language is well predicted by how much training data that language had.
How word error rate is measured
Word error rate (WER) is the standard ASR metric. The Jurafsky and Martin textbook chapter on speech recognition, in its August 2026 draft, defines it as 100 times the sum of substitutions, insertions, and deletions, divided by the number of words in the correct transcript. Because insertions count, WER can exceed 100 percent. The chapter's example has six substitutions, three insertions, and one deletion against 13 reference words, for 76.9 percent.
The Whisper authors add that WER penalizes every difference from the reference, including harmless differences in transcript style. They therefore normalize text before scoring and report WER drops of up to 50 percent on some datasets from quirks such as split contractions. They caution that tuning the normalizer risks overfitting to Whisper's own style. For Chinese, Japanese, Thai, Lao, and Burmese, which do not separate words with spaces, they space every letter, which measures character error rate instead. A WER figure therefore depends on the normalizer behind it.
What went into the training data
The paper's Figure 11 splits the 680,000 hours three ways:
- 438,218 hours (65 percent) of English audio with English transcripts.
- 117,113 hours (17 percent) of other-language audio with same-language transcripts.
- 125,739 hours (18 percent) of other-language audio with English translations.
The text says the non-English transcription data covers 96 other languages and that speech recognition data exists for 75. The model card says the non-English data represents 98 languages. The authors write that the dataset is very English-heavy because collection drew mainly on English-centric parts of the internet, and that most languages have less than 1,000 hours.
Noisy labels leak in. The text reports about 9,000 hours of supposed Welsh translation data (Figure 11 lists 8,263), most of it English audio misidentified as Welsh. Welsh translation scored 13 BLEU. The paper's Table 1 lists five sizes, from 39 million to 1,550 million parameters, and the README gives approximate memory needs from about 1 GB to about 10 GB. Jurafsky and Martin note that the models and inference code are public but the training code and training data are not.
Accuracy by language
For a wide test, the authors used FLEURS, a benchmark of the same sentences read in 102 languages with about 12 hours of speech per language (Conneau and colleagues). They found a squared correlation of 0.83 between the log of WER and the log of training hours per language. Their fit implies that WER halves for every 16-fold increase in training data. Many of the largest outliers were languages with unique scripts and more distant from the Indo-European majority, such as Hebrew, Telugu, Chinese, and Korean.
The paper's figures for the large-v2 model show the spread:
| Language | Transcribed training hours | FLEURS WER (%) |
|---|---|---|
| Spanish | 11,100 | 3.0 |
| Russian | 9,761 | 5.6 |
| Arabic | 739 | 16.0 |
| Urdu | 104 | 22.6 |
| Persian | 24 | 32.9 |
| Swahili | 5.4 | 39.3 |
| Bengali | 1.3 | 104.1 |
| Pashto | none listed | 93.7 |
Pashto appears in Figure 11 only as 63 hours of translation data. The model card says the models show strong ASR results in about 10 languages.
Failure modes the authors state
Section 6 of the paper says larger models steadily reduced perception errors such as confusing similar-sounding words. Many remaining errors, especially in long recordings, are "decidedly non-human/perceptual." The authors list getting stuck in repeat loops, skipping the first or last few words of a segment, and complete hallucination, a transcript unrelated to the audio.
Their long-form decoding uses heuristics against these failures. Beam search with 5 beams reduces repetition looping. The decoder raises its sampling temperature in steps of 0.2, up to 1.0, when the average log probability falls below -1 or the text's gzip compression ratio exceeds 2.4.
The model card adds that the models may output text not spoken in the audio, that repetition is mitigated "but not perfectly," and that both are likely worse in lower-resource languages. The card recommends against use in high-risk decision-making contexts.
Outside testing found the same failure. Koenecke and colleagues ran 13,140 English audio segments through the hosted Whisper service in 2023. The abstract says roughly 1 percent of transcriptions contained entire hallucinated phrases or sentences, and the body reports 1.4 percent of segments. The authors found that 38 percent of hallucinations included explicit harms, and that hallucinations occurred more for speakers with longer pauses.
What to check
- Which model size and version produced the score, and was it tested on the target language and type of audio?
- Was WER computed with the same text normalizer as the comparison, or is it character error rate?
- How many hours of transcribed training audio does the language have in Figure 11?
- Does the transcript keep timestamps so a linguist can check each segment against the audio?
- Are long silences and pauses, where hallucinations cluster, flagged for review?
- Will the transcript feed a decision the model card advises against?
Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.
Sources
- 1Robust Speech Recognition via Large-Scale Weak Supervision · arXiv (Radford, Kim, Xu, Brockman, McLeavey, Sutskever; version 1, December 6, 2022)
- 2Model Card: Whisper · OpenAI (whisper repository on GitHub)
- 3Whisper (README) · OpenAI (whisper repository on GitHub)
- 4Speech and Language Processing, Chapter 16: Automatic Speech Recognition · Daniel Jurafsky and James H. Martin, draft of August 19, 2026
- 5FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech · arXiv (Conneau and colleagues, May 25, 2022)
- 6Careless Whisper: Speech-to-Text Hallucination Harms · arXiv (version 2, May 3, 2024); ACM FAccT 2024
Where this shows up in our work
- BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.