Cross-lingual retrieval: asking in English, finding it in Russian

A multilingual embedding model maps text from many languages into one vector space, so an English question and a Russian passage with the same meaning land near each other and the passage can be retrieved. For a reader who cannot read Russian, and whose documents cannot be uploaded to an outside service, that lets a search run in English over material that stays on the local machine. The match is imperfect: published studies describe difficulty with differing scripts, uneven language resources, a preference for documents in the query's language, and names that are spelled differently in each language.

How one vector space is built

An embedding model turns a piece of text into a list of numbers. Retrieval ranks passages by how close their vectors are to the vector of the query. For an English query to reach a Russian passage, the model has to place text with the same meaning close together whatever the language.

Two widely cited models show how. LaBSE, the Language-agnostic BERT Sentence Embedding, combines masked language modeling, translation language modeling, dual-encoder translation ranking, and additive margin softmax. The translation ranking step trains the model to rank a sentence's true translation above other candidates. The authors report that starting from a pre-trained multilingual model cuts the parallel training data needed by 80%, that their best model reaches 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, and that they released a model for more than 109 languages.

Multilingual E5 follows a different recipe. The technical report describes contrastive pre-training on 1 billion multilingual text pairs, followed by fine-tuning on labeled datasets, and offers small, base, and large sizes.

Testing English queries against Russian documents

Tatoeba measures matching a sentence with its translation. Finding a passage that answers a question is a different task and needs its own test.

The HC4 test collections were built for that task. Their topics are written in English and in the document languages, and the documents are Common Crawl News articles in Chinese, Persian, and Russian. Relevance was judged on a three-grade scale. The Russian collection has 54 topics and five million documents. The authors built new collections because older ones, assembled by pooling results from traditional systems, have blind spots when used to evaluate neural methods.

Where it falls short

Script and alignment. A study of cross-lingual ranking describes cross-lingual retrieval as challenging because of disparities in resources, differences in scripts, and weak alignment of meaning across languages in embedding models. Among its four representative language pairs, Arabic to Russian was the hardest for most encoders, which the authors attribute to cross-script retrieval and distant language families. English to Russian was not one of the four pairs.

Preference for the query's language. The same study measures query-language bias, the probability that a model retrieves a document written in the query's language. It reports that multilingual E5, despite comparatively strong cross-lingual retrieval, showed very strong query-language bias, almost on par with BM25, a keyword-matching method. With such a bias, an English query tends to pull English documents ahead of a better Russian match.

Low-resource languages. The authors note that high-resource languages still dominate overall performance, and that gains over keyword and translate-first baselines are most pronounced for low-resource and cross-script pairs. They also report that contrastive training can reduce language bias in encoders that start with weak alignment.

Domain data. A study on Arabic and English corporate documents found retrieval was the main bottleneck, with performance dropping noticeably when the query and the supporting document were in different languages. The authors say open-domain benchmarks built from Wikipedia can mask such problems through language imbalance, pretraining overlap, and memorized content. Two simple remedies improved results: retrieving equally from both languages, or translating the query.

Names. A multilingual RAG study across 13 languages found that evaluation metrics need adjustment for variation in how named entities are spelled. The example is a Russian name rendered as sofya kovalevskaya in one transliteration and sofia kovalevskaia in another. The authors scored generated answers by character 3-gram overlap for this reason. They also report that models tend to insert English named entities into answers written in the user's language, and that frequent code-switching remains a limitation in languages with non-Latin alphabets.

That study measured answers, not retrieval, but the same variation affects any step that matches names as exact strings.

A retrieved passage is still in the original language

A cross-lingual retriever ranks passages. It does not translate them, and the passage it returns is in the document's own language. A reader who cannot read that language needs a translation, and anyone checking the result needs the original passage and its location beside it.

What to ask

• Was the retriever tested on English queries against Russian documents, or only on another language pair or on sentence matching?
• Does an English query return English documents ahead of a better Russian match?
• Are names matched as exact strings, or across spellings such as sofya kovalevskaya and sofia kovalevskaia?
• Was it evaluated on documents like these, or only on open-domain benchmarks built from Wikipedia?
• Does each result show the original passage and its location beside any translation?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.

The long read on offline machine translation goes further.

Sources

1. 1Language-agnostic BERT Sentence Embedding · arXiv (Feng et al., ACL 2022)
2. 2Multilingual E5 Text Embeddings: A Technical Report · arXiv (Wang et al.)
3. 3HC4: A New Suite of Test Collections for Ad Hoc CLIR · arXiv (Lawrie et al., ECIR 2022)
4. 4What Drives Cross-lingual Ranking? Retrieval Approaches with Multilingual Language Models · arXiv (Goworek et al.)
5. 5Retrieval-augmented generation in multilingual settings · arXiv (Chirkova et al.)
6. 6The Cross-Lingual Cost: Retrieval Biases in RAG over Arabic-English Corpora · arXiv (Amiraz et al., ArabicNLP 2025)