BEIR, explained

BEIR, short for Benchmarking IR, is a benchmark that tests how well a retrieval system finds relevant documents in collections it was never trained on. The paper, BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models by Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych, was first posted on April 17, 2021 and accepted at the NeurIPS 2021 Datasets and Benchmarks track. It gathers 18 datasets from 9 kinds of retrieval task, puts them in one format, and scores every system with the same metric. Because the Massive Text Embedding Benchmark (MTEB) reuses it, a BEIR score is behind most published claims about how well an embedding model retrieves. This page covers what the benchmark contains, what the original paper found, where the benchmark's authors say it falls short, what has followed it, and how to run it on a machine with no network.

What zero-shot retrieval means

A retrieval model is usually trained on a collection with relevance judgments, pairs of queries and the documents a person marked as answering them. MS MARCO, a web-search collection, supplied most of the training data for the neural retrievers of 2020 and 2021. The BEIR paper's motivation is that real deployments rarely look like that. Building judgments for a new collection is costly, so systems are, in the paper's words, applied in a zero-shot setup, with no training data for the target. The open question in 2021 was whether a model trained on MS MARCO would still work on medical abstracts, legal arguments, or tweets.

BEIR answers that question by construction. Every system is trained elsewhere and then run, unchanged, against the 18 test collections. MS MARCO is reported as a nineteenth column but excluded from the zero-shot comparison, because most systems trained on it and their scores there are in-domain. The benchmark therefore measures transfer, not peak performance, and that is the right question for a program whose documents look like none of the public collections. The page on held-out test sets covers why training on the test collection invalidates a score.

The 18 BEIR datasets and 9 task types

The paper's Table 1 lists the datasets with their corpus sizes and test query counts. Grouped by task:

Two features of this table matter when reading a BEIR score. The query counts are uneven: Touché-2020 has 49 queries and TREC-COVID 50, so one hard query moves those columns noticeably, while CQADupStack has 13,145. And the corpora range from a few thousand documents to nearly 15 million, so a system's speed and memory figures depend heavily on which dataset they were measured on. The paper's appendix also records the licenses. FEVER, NQ, DBPedia, and Signal-1M are CC BY-SA 3.0; HotpotQA is CC BY-SA 4.0; ArguAna and Touché-2020 are CC BY 4.0; CQADupStack is Apache 2.0; SCIDOCS is GPL v3; SciFact is CC BY-NC 2.0; MS MARCO is MIT for non-commercial research; TREC-COVID has a dataset license agreement; and the TREC-NEWS, Robust04, and BioASQ collections are under copyright. The beir README marks those last three plus Signal-1M as not publicly downloadable, which leaves 14 zero-shot datasets, plus MS MARCO, in the public set that everyone can run.

Why nDCG@10

Every main result in the paper is normalized discounted cumulative gain at rank 10, with Recall@100 in a separate table. The authors give their reasoning: precision and recall at a cutoff are unaware of rank order, and mean reciprocal rank and mean average precision cannot handle graded relevance, where a document can be marked partly relevant rather than yes or no. nDCG rewards a relevant document more the higher it is ranked and accepts both binary and graded judgments, which the paper says provides a good balance for both kinds of task in the benchmark. The cutoff at 10 reflects a search result page, or the handful of passages a question-answering system reads. The page on retrieval metrics defines each measure in full; the point here is that a BEIR number is not recall, and a system with high nDCG@10 can still miss relevant documents ranked eleventh and below.

What BEIR found: BM25 as the baseline

The paper evaluated 10 systems in four families. Lexical: BM25, run through Anserini. Sparse: DeepCT, SPARTA, and docT5query, which expand or reweight the document's terms before indexing. Dense: DPR, ANCE, TAS-B, and GenQ, which encode queries and documents as vectors. And two more expensive designs: ColBERT, which keeps a vector per token and matches late, and BM25 followed by a cross-encoder (CE) reranker that rescores the top 100 BM25 hits.

The headline is in the abstract: "Our results show BM25 is a robust baseline." Averaged over the 18 datasets, each system's nDCG@10 relative to BM25 was: DeepCT minus 27.9 percent, SPARTA minus 20.3 percent, docT5query plus 1.6 percent, DPR minus 47.7 percent, ANCE minus 7.4 percent, TAS-B minus 2.8 percent, GenQ minus 3.6 percent, ColBERT plus 2.5 percent, and BM25 plus cross-encoder plus 11 percent. Three systems beat BM25 on average, and only one of them by a wide margin. The abstract adds that the reranking and late-interaction models achieved the best zero-shot performance at high computational cost.

The per-dataset figures for BM25 and the three systems that beat it:

Even the best system lost to BM25 somewhere. The reranker trailed on ArguAna and Touché-2020, the two argument-retrieval sets, which the paper attributes to their distance from the reranker's training data. ColBERT lost on 9 of the 18, including both news sets and both argument-retrieval sets. The dense models' pattern is the sharpest: DPR, the only system not trained on MS MARCO, beat BM25 on NQ, the set it was trained on, with 0.474 against 0.329, and lost almost everywhere else, as low as 0.077 on SCIDOCS and 0.127 on BioASQ. The dense vs sparse retrieval page sets these figures beside the dense models' in-domain results, and the BM25 and hybrid search page covers the ranking function itself.

The lexical bias problem

The paper does not take its own numbers at face value, and the caveat is one of the more useful parts of it. Many of the 18 datasets were built by having annotators judge candidate documents that a keyword system had retrieved. A relevant document with no word overlap with the query was never shown to an annotator, so it sits in the judgments as irrelevant. In the paper's phrasing, "such a lexical bias disfavours approaches that don't rely on lexical matching."

The authors measured it on TREC-COVID with Hole@10, the share of a system's top 10 results that no annotator had judged. For BM25 it was 6.4 percent; for TAS-B, a dense model, 31.8 percent. They then judged 980 of the unjudged query-document pairs themselves. ANCE's nDCG@10 rose from 0.654 to 0.735, moving from just below BM25 to well above it. BM25 moved only from 0.656 to 0.668 and docT5query from 0.713 to 0.714. The lesson generalizes: a benchmark whose judgments came from keyword pools gives keyword systems a structural advantage, and the size of the BM25 margin on BEIR is partly an artifact of how the datasets were built. The lesson also cuts the other way for a program building its own test set, because judgments made only on what one system retrieved will favor that system.

How BEIR is used on MTEB, and where it stops

The MTEB paper by Muennighoff, Tazi, Magne, and Reimers, posted October 13, 2022, built a benchmark of 8 task types across 58 datasets and 112 languages, and for retrieval it did not start over. Section 3.2 states that MTEB reuses datasets and evaluation from BEIR, and the appendix lists the 15 publicly available BEIR datasets it includes: ArguAna, ClimateFEVER, CQADupstack, DBPedia, FEVER, FiQA2018, HotpotQA, MSMARCO, NFCorpus, NQ, Quora, SCIDOCS, SciFact, Touche2020, and TRECCOVID. The metric is nDCG@10. The paper's own finding on that subset was that SGPT-5.8B-msmarco was the best embedding model, and that models tuned for semantic similarity, such as SimCSE and SGPT-nli, performed badly on retrieval.

This means the retrieval column of that original benchmark, now labeled MTEB(eng, v1), is a BEIR average over 15 English datasets. The MMTEB paper (Enevoldsen et al., first submitted February 19, 2025) later introduced MTEB(eng, v2) as a zero-shot benchmark that excludes MS MARCO and Natural Questions because they are frequently used in fine-tuning; its Table 16 lists 10 retrieval tasks, all drawn from BEIR datasets and several in hard-negative versions. The caveats above travel with both. The embedding model page covers how to read the rest of the leaderboard.

Limits: English only, domain mix, document length

The benchmark's authors state its boundaries: "all datasets covered in the BEIR benchmark are currently English," and multilingual and cross-lingual tasks are named as future work. The datasets are web questions, scientific abstracts, news, forum posts, tweets, and Wikipedia-derived fact checks; there are no technical manuals, regulations, or scanned documents, and the authors note that including every dataset is not feasible. Document length is short: most tasks have average document lengths of up to a few hundred words, and the neural systems were run on only the first 512 word pieces of each document, so a long document was scored on its opening. A program whose collection is 300-page manuals in several languages, split into chunks that carry tables and part numbers, is outside every one of these boundaries, and a BEIR average predicts little about it. The cross-lingual retrieval page covers the language gap.

Successors and multilingual counterparts

An official leaderboard. The beir wiki says the current leaderboard is hosted on Eval AI with automatic evaluation, open to anyone's models and runs, and that the earlier leaderboard kept in two Google Sheets was deprecated on January 10, 2023. The paper Resources for Brewing BEIR by Kamalloo, Thakur, Lassance, Ma, Yang, and Lin, posted June 13, 2023, explains the move. The original BM25 baseline had been run with Elasticsearch, later moved to Pyserini, and never packaged as a reproducible baseline, and the stale spreadsheet left researchers reporting on partially overlapping subsets of the 18 datasets. The paper supplies reference runs through Anserini and Pyserini, with dense models searched by flat Faiss indexes, and reports BM25 at 0.429 average nDCG@10 with multi-field indexing and 0.424 with a single field, alongside SPLADE at 0.474, Contriever at 0.448, uniCOIL at 0.428, and TAS-B at 0.424. Two years after the original paper, a learned sparse model led and BM25 still matched a dense model.

MIRACL. Making a MIRACL, by Zhang, Thakur, and colleagues, posted October 18, 2022, is a retrieval benchmark over Wikipedia in 18 languages: Arabic, Bengali, English, Spanish, Farsi, Finnish, French, Hindi, Indonesian, Japanese, Korean, Russian, Swahili, Telugu, Thai, and Chinese, plus two surprise languages for a WSDM 2023 Cup challenge. The abstract gives over 700,000 relevance judgments for around 77,000 queries, all made by native speakers hired by the team, and the paper's Table 1 counts more than 90 million passages across the known languages. Metrics are nDCG@10 and Recall@100. The baselines show the BEIR pattern does not hold uniformly across languages: averaged over the known languages, BM25 scored 0.393 nDCG@10 and the dense mDPR 0.415, with a hybrid of the two at 0.578, but BM25 led in Indonesian, 0.449 to 0.272, while mDPR led in Chinese, 0.512 to 0.180. The dataset is Apache 2.0.

mMARCO. mMARCO, by Bonifacio and colleagues, posted August 31, 2021, is the MS MARCO passage ranking dataset machine-translated into 13 languages. It is a training resource rather than a test benchmark, and its finding was that multilingual models fine-tuned on the translated data were more effective than models fine-tuned on English alone. Together, MIRACL supplies the multilingual test and mMARCO the multilingual training data that BEIR left as future work.

How to run BEIR offline

The benchmark was built to run locally, which suits a disconnected evaluation. The beir README gives the steps. The package installs with pip install beir, is tested on Python 3.9 and later, and is Apache 2.0 licensed. Each public dataset is a zip file at a fixed URL on the project's public server, fetched once with util.download_and_unzip and unpacked into a folder; the README's dataset table gives MS MARCO at 6,980 test queries over 8.84 million passages and SciFact at 300 queries over about 5,000 documents. The BeIR organization on Hugging Face mirrors them as 50 dataset repositories. Once on disk, GenericDataLoader(data_folder).load(split="test") returns the corpus, queries, and qrels with no further network access.

The format is three plain files, documented on the wiki's custom dataset page. corpus.jsonl has one JSON object per line with _id, an optional title, and text. queries.jsonl has _id and text. The qrels file is tab-separated with a header row and three columns, query-id, corpus-id, and score. A retriever is wrapped in EvaluateRetrieval, retrieve produces ranked results, and evaluate returns nDCG, MAP, recall, and precision at k of 1, 3, 5, 10, 100, and 1,000, with MRR available through evaluate_custom. The same loader reads a private collection written in those three files, which is how BEIR's machinery becomes a test harness for a program's own documents. The mteb package, also Apache 2.0, wraps the same datasets for embedding models with pip install mteb and a command-line mteb run, and its README points to a guide on cache management for repeated runs.

Running the public benchmark offline proves the harness works. Running the program's own queries and judgments through it, with BM25 as the baseline on the same table, is the measurement that matters, and the lexical-bias finding above says to judge the pooled results of every system under test, not just one.

What to check

• Which BEIR datasets does a quoted average cover, all 18, the 14 public ones, or MTEB's 15, and is the figure nDCG@10 or something else?
• Was BM25 run as a baseline on the program's own collection, and did the candidate retriever beat it there, not only on BEIR?
• Does the collection resemble any BEIR dataset in language, domain, and document length, given that BEIR is English with documents of a few hundred words?
• Were the program's relevance judgments made on pooled results from several systems, or only on what one system retrieved?
• Can the evaluation run on a disconnected machine from the three-file format, and are the dataset licenses compatible with the program's use?
• For a multilingual collection, were MIRACL or an equivalent per-language test run, and were results reported per language rather than averaged?

Azcuna's TRACE is one offline viewer for S1000D, MIL-STD-3001 XML, and PDF technical manuals. It keeps the source location and the markings of everything it shows and never modifies the source. It was proposed to NAVAIR under Navy SBIR Phase I topic DON26BZ05-NV078, and the proposal was submitted on September 23, 2026.

Questions people ask

What is the BEIR benchmark?

BEIR, Benchmarking IR, is a benchmark for zero-shot evaluation of information retrieval models, published by Thakur, Reimers, Rücklé, Srivastava, and Gurevych at the NeurIPS 2021 Datasets and Benchmarks track. It collects 18 publicly available retrieval datasets from 9 task types and domains, converts them to one corpus, queries, and relevance-judgments format, and scores a retrieval system on all of them with nDCG@10 without letting the system train on any of them. The paper evaluated 10 systems and found BM25 a robust baseline.

What does zero-shot mean in BEIR?

That the retrieval model is tested on a dataset it was never trained on. The BEIR paper describes real deployments as a zero-shot setup with no training data for the target collection, because building relevance judgments is costly, and asks how well neural models trained on one collection, usually MS MARCO, transfer to others. In BEIR every system is trained elsewhere and then run on the 18 test collections as they are. MS MARCO itself is reported but excluded from the zero-shot comparison because most systems trained on it.

What are the BEIR datasets?

The 18 datasets, grouped by task, are TREC-COVID, NFCorpus, and BioASQ for bio-medical retrieval; NQ, HotpotQA, and FiQA-2018 for question answering; Signal-1M for tweet retrieval; TREC-NEWS and Robust04 for news retrieval; ArguAna and Touche-2020 for argument retrieval; CQADupStack and Quora for duplicate-question retrieval; DBPedia for entity retrieval; SCIDOCS for citation prediction; and FEVER, Climate-FEVER, and SciFact for fact checking. Corpus sizes run from 5,183 documents (SciFact) to 14.9 million (BioASQ).

Does BM25 beat dense retrieval on BEIR?

On average, yes, for the dense models of 2021. The BEIR paper reports each system's average nDCG@10 relative to BM25 across the 18 datasets: DPR minus 47.7 percent, ANCE minus 7.4 percent, TAS-B minus 2.8 percent, GenQ minus 3.6 percent. Three systems beat BM25 on average: docT5query, a document-expansion method, by 1.6 percent; ColBERT, a late-interaction model, by 2.5 percent; and BM25 followed by a cross-encoder reranker by 11 percent. The paper's abstract calls BM25 a robust baseline.

Why does BEIR use nDCG@10?

Precision and recall at a cutoff ignore rank order, and mean reciprocal rank and mean average precision cannot use graded relevance, where a document can be partly relevant. Normalized discounted cumulative gain at rank 10 rewards relevant documents more when they appear higher and accepts both binary and graded judgments, which the authors say gives a good balance for both kinds of dataset in the benchmark. The paper's main results are nDCG@10, with Recall@100 in a separate table, and the beir package also reports MAP, recall, and precision.

Is BEIR part of MTEB?

Yes, in two versions. The Massive Text Embedding Benchmark paper (Muennighoff, Tazi, Magne, and Reimers, 2022) states that MTEB reuses datasets and evaluation from BEIR for its retrieval task and lists 15 publicly available BEIR datasets, including MSMARCO and NQ, scored by nDCG@10. The 2025 MMTEB paper's MTEB(eng, v2) is meant as a zero-shot benchmark: it excludes MS MARCO and Natural Questions and keeps 10 retrieval tasks drawn from BEIR datasets, several in hard-negative versions.

What are the limits of BEIR?

The paper states that all datasets in the benchmark are currently English and names multilingual and cross-lingual tasks as future work. Most tasks have average document lengths of up to a few hundred words, and the neural systems were run on only the first 512 word pieces. The authors also show lexical bias: many datasets were annotated from candidates found by keyword systems, so a relevant passage without word overlap counts as irrelevant, which disfavors dense retrieval. The domains are web, science, news, and forums, not technical manuals.

Is there a multilingual BEIR?

Not under that name, but two resources fill the gap. MIRACL (Zhang, Thakur, and colleagues, 2022) is a retrieval benchmark over Wikipedia in 18 languages with about 77,000 queries and more than 700,000 relevance judgments made by native speakers, scored by nDCG@10 and Recall@100, and released under Apache 2.0. mMARCO (Bonifacio and colleagues, 2021) is the MS MARCO passage ranking dataset machine-translated into 13 languages, intended for training multilingual retrievers. The BEIR authors list multilingual tasks as future work.

How do I run BEIR offline?

The beir package installs with pip install beir and is Apache 2.0 licensed. Each public dataset is a zip file downloaded once from the project's public server and unpacked into a folder holding corpus.jsonl, queries.jsonl, and a qrels TSV; the GenericDataLoader reads that folder with no network access. A retriever runs through EvaluateRetrieval and the evaluate call returns nDCG, MAP, recall, and precision at k of 1, 3, 5, 10, 100, and 1,000. A private collection in the same three-file format runs the same way.

What is the BEIR leaderboard?

The BEIR wiki says the current leaderboard is hosted on Eval AI, where anyone can submit models and runs for automatic evaluation, and that the earlier leaderboard kept in two Google Sheets was deprecated on January 10, 2023. The paper Resources for Brewing BEIR (Kamalloo and colleagues, 2023) explains why: the original BM25 baseline was run with Elasticsearch and never packaged reproducibly, and the stale spreadsheet led researchers to report results on partially overlapping subsets of the 18 datasets. That paper supplies reference runs through Pyserini and Anserini.

Sources

1. 1BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models · arXiv (Thakur, Reimers, Rücklé, Srivastava, and Gurevych; NeurIPS 2021 Datasets and Benchmarks)
2. 2beir-cellar/beir README · BEIR project (GitHub)
3. 3Load your custom dataset · BEIR project wiki (GitHub)
4. 4Leaderboard · BEIR project wiki (GitHub)
5. 5BeIR organization · Hugging Face
6. 6MTEB: Massive Text Embedding Benchmark · arXiv (Muennighoff, Tazi, Magne, and Reimers)
7. 7embeddings-benchmark/mteb README · MTEB project (GitHub)
8. 8MMTEB: Massive Multilingual Text Embedding Benchmark · arXiv (Enevoldsen et al.)
9. 9Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard · arXiv (Kamalloo, Thakur, Lassance, Ma, Yang, and Lin)
10. 10Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages · arXiv (Zhang, Thakur, Ogundepo, Kamalloo, Alfonso-Hermelo, Li, Liu, Rezagholizadeh, and Lin)
11. 11mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset · arXiv (Bonifacio, Jeronymo, Abonizio, Campiotti, Fadaee, Lotufo, and Nogueira)

Where this shows up in our work

• TRACE, the offline S1000D and IETM viewer opens S1000D, MIL-STD-3001 and PDF technical manuals on a tablet with every marking intact.
• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.