Dense vs sparse retrieval, explained

Sparse retrieval and dense retrieval are the two ways a search system turns text into something it can compare. A sparse retriever represents a document as a vector with one position for every word in the vocabulary, almost all of them zero, and scores it by the query words it contains; BM25 is the standard example. A dense retriever represents a query and a passage as short vectors of learned numbers and scores them by how close those vectors are, so two texts can match with no word in common. The Dense Passage Retrieval (DPR) paper by Karpukhin and colleagues, published at EMNLP 2020, drew the line this way, calling TF-IDF and BM25 the traditional sparse vector space models and showing that retrieval could be done from dense representations alone. Which one finds the right passage depends on the query, the documents, and what the dense model was trained on.

What sparse retrieval is

A sparse retriever stores an inverted index: for each term, the list of documents containing it. A query is answered by looking up its terms and scoring the documents that appear. The score comes from a weighting function, and the one in common use is BM25. Robertson and Zaragoza's 2009 survey, The Probabilistic Relevance Framework: BM25 and Beyond, gives each query term a weight that grows with how rare the term is across the collection, an inverse document frequency (IDF) of log((N - n + 0.5) / (n + 0.5)), grows with how often it appears in the document but flattens toward a ceiling the authors call saturation, controlled by the parameter k1, and shrinks for documents longer than average, controlled by b. The document score is the sum over query terms. Lucene's BM25Similarity class defaults to k1 = 1.2 and b = 0.75; the DPR paper tuned k1 = 0.9 and b = 0.4 for its Lucene baseline. A fuller treatment is in the post on BM25 and hybrid search.

The vector is sparse because any one document uses a small fraction of the vocabulary. That sparsity is what makes the inverted index fast and what sets the method's limit: the Sentence Transformers documentation describes lexical search as looking for literal matches of the query words and notes that it "will not recognize synonyms, acronyms or spelling variations."

What dense retrieval is

A dense retriever replaces the vocabulary vector with the output of a neural encoder. In DPR, two independent BERT-base encoders, one for questions and one for passages, each produce a 768-dimensional vector from the output of the [CLS] token, and the similarity between a question and a passage is the dot product of the two vectors. The passage vectors are computed once and stored in a vector index; at query time only the question is encoded, and the index returns the nearest passages. A separate post explains what an embedding model is and how its vectors are trained.

The encoder is trained on pairs. DPR learned from question-passage pairs drawn from open-domain question answering datasets, and the authors found that a retriever trained on only 1,000 examples already outperformed BM25, with accuracy continuing to rise from 1,000 to 59,000 examples on the Natural Questions development set. The corpus was the English Wikipedia dump of December 20, 2018, split into disjoint blocks of 100 words, 21,015,324 passages in all.

Because the vectors come from a model, the method inherits the model's knowledge. Two passages that use different words for the same thing land near each other if the training data taught the encoder that they mean the same thing; a string the encoder rarely saw has no reliable place in the space.

Where dense retrieval wins

In the domain it was trained on, dense retrieval won clearly. DPR's Table 2 gives top-20 and top-100 retrieval accuracy on the test sets. With single-dataset training, DPR against BM25 was 78.4% against 59.1% top-20 on Natural Questions, 79.4% against 66.9% on TriviaQA, 73.2% against 55.0% on WebQuestions, and 79.8% against 70.9% on CuratedTREC. The abstract summarizes the gap as 9% to 19% absolute, and the introduction gives 65.2% against 42.9% in top-5 accuracy. End-to-end answer accuracy followed: exact match on Natural Questions rose from 32.6 with BM25 retrieval to 41.5 with DPR.

The BEIR benchmark, by Thakur and colleagues at the NeurIPS 2021 Datasets and Benchmarks track, shows the same pattern on MS MARCO, the dataset most of its neural models were trained on. Normalized discounted cumulative gain at rank 10 (nDCG@10) was 0.228 for BM25, 0.388 for the dense model ANCE, and 0.408 for TAS-B. On Quora, BM25 scored 0.789 and ANCE 0.852. Dense retrieval wins when the query and the passage say the same thing in different words, and when the encoder has seen enough examples of that domain to know which words those are.

Where sparse retrieval wins

Sparse retrieval wins on rare exact strings and on unfamiliar domains. The DPR authors write that term-matching methods like BM25 are sensitive to highly selective keywords and phrases, and give the question "Who plays Thoros of Myr in Game of Thrones?" as a case where the salient phrase is critical and DPR could not capture it. On SQuAD, BM25 beat single-dataset DPR 68.8% to 63.2% top-20.

Sciavolino and colleagues, at EMNLP 2021, built EntityQuestions from 24 common Wikidata relations with up to 1,000 facts each, in templates such as "Where was [E] born?". DPR trained on Natural Questions found the answer passage in its top 20 for 49.7% of questions, DPR trained on several datasets for 56.7%, and BM25 for 72.0%. On place of birth, DPR scored 25.4% and BM25 75.3%. DPR did well on the most common entities and degraded quickly on rarer ones; BM25 was much less sensitive to entity frequency. Fine-tuning the passage encoder helped far more than fine-tuning the question encoder, which the authors read as pointing to passage representations as the bottleneck.

BEIR generalizes the finding. Evaluating 10 systems on 18 datasets without training on the target data, a setting the authors call zero-shot, the paper concludes that BM25 remains a strong baseline. DPR, the only model in the study not trained on MS MARCO, performed worst, 47.7% below BM25 on average; ANCE was 7.4% below, TAS-B 2.8% below, and GenQ 3.6% below. Dense retrievers underperformed on datasets with a large domain shift from their training data, such as BioASQ, where BM25 scored 0.465 and DPR 0.127, or a task shift, such as Touché-2020, where BM25 scored 0.367 and the best dense model 0.240. Part numbers, clause numbers, and equipment designations are rare, selective strings of the Thoros of Myr kind, though none of these benchmarks tests them directly.

BEIR's zero-shot results side by side

A selection of BEIR's nDCG@10 figures, with the learned sparse models included:

Averaged over all 18 datasets relative to BM25: DeepCT -27.9%, SPARTA -20.3%, docT5query +1.6%, DPR -47.7%, ANCE -7.4%, TAS-B -2.8%. docT5query, which BEIR describes as a document expansion technique that uses a T5 model to generate 40 queries for each document before indexing with BM25, beat BM25 on 11 of 18 datasets and was the only sparse or dense model in the study to beat it on average; the paper says the expansion adds new relevant keywords to a document. The two methods that scored highest overall, ColBERT at +2.5% and BM25 followed by a cross-encoder reranker at +11%, cost far more per query. BEIR also cautions that many of its datasets were annotated from candidates found by keyword systems, which disfavors dense retrieval.

Learned sparse retrieval and SPLADE

Learned sparse retrieval keeps the vocabulary vector and the inverted index but lets a neural model set the weights. SPLADE, the Sparse Lexical and Expansion Model by Formal, Piwowarski, and Clinchant, submitted July 12, 2021 as a SIGIR 2021 short paper, is the clearest example. A BERT model's masked-language-model head produces, for every input token, a score over the full 30,522-term WordPiece vocabulary. SPLADE sums those scores over the input after passing each through log(1 + ReLU(w)), a log-saturation that keeps any one term from dominating and yields sparsity on its own, and adds a FLOPS regularizer, a smooth relaxation of the average number of floating-point operations needed to score a document, which pushes most weights to zero. The result is a sparse vector over the vocabulary in which a document can carry terms it never used, the "expansion" in the name, so a passage about a pump can score for "hydraulic" without containing the word.

The paper's MS MARCO development set results put SPLADE between the sparse and dense camps. Mean reciprocal rank at 10 (MRR@10) was 0.184 for BM25, 0.243 for DeepCT, 0.277 for doc2query-T5, 0.322 for both SPLADE variants, 0.330 for the dense model ANCE, and 0.335 for TCT-ColBERT. Recall at 1,000 was 0.853 for BM25 and 0.954 to 0.955 for the two SPLADE variants. On TREC Deep Learning 2019, nDCG@10 was 0.506 for BM25, 0.665 to 0.667 for SPLADE, 0.648 for ANCE, and 0.670 for TCT-ColBERT. One of the most efficient configurations, at 0.296 MRR@10, kept an average of 18 non-zero terms per document and 6 per query, and the paper's conclusion states that SPLADE can operate on inverted indexes. On the same MS MARCO set the paper puts DeepCT at 0.243, so learned weights plus expansion closed most of the in-domain gap to dense retrieval while keeping the index structure of BM25. The paper reports no zero-shot BEIR results, the setting in which DeepCT and SPARTA lost to BM25 by 27.9% and 20.3%.

A fourth design sits beside these three. ColBERT, by Khattab and Zaharia at SIGIR 2020, encodes every token of the query and the document into its own 128-dimensional vector and scores by summing, for each query token, its maximum similarity to any document token. End to end on MS MARCO it reached 36.0 MRR@10 and 96.8% recall at 1,000 against 18.7 and 85.7% for BM25, at 458 milliseconds per query and 154 GiB of index for 8.8 million passages.

Hybrid combinations

Because the two methods fail on different queries, most systems run both. The DPR paper reranked the union of each method's top 2,000 passages by the BM25 score plus 1.1 times the dense score. Top-20 accuracy on CuratedTREC rose from 79.8% to 85.2% and on SQuAD from 63.2% to 71.5%, but on Natural Questions it fell from 78.4% to 76.6% and on WebQuestions from 73.2% to 71.0%. The combination helped most where BM25 was already strong and cost a little where DPR was. BEIR found no single approach consistently best on all datasets, which is the practical argument for fusion. A linear sum of scores needs a weight and comparable scales; reciprocal rank fusion, covered in the BM25 post linked above, merges the two ranked lists by rank alone, and a cross-encoder reranker can rescore the union of both lists, as BEIR's best configuration did for BM25.

The offline cost of each

The two methods spend their cost at different times. DPR reports the figures for its 21 million Wikipedia passages. Building the Lucene inverted index took about 30 minutes. Encoding the passages with the trained encoder took roughly 8.8 hours on 8 GPUs, and building the FAISS vector index on a single server took another 8.5 hours. At query time the order reversed: with the index in memory, DPR processed 995.0 questions per second returning the top 100 passages, while BM25 in Lucene processed 23.7 questions per second per CPU thread.

BEIR's cost table, measured on the 1 million document DBPedia corpus, gives the storage side. The BM25 index was 0.4 GB; the 768-dimensional dense indexes for DPR and ANCE were 3 GB; ColBERT's token-level index was 20 GB. Query latency was about 20 milliseconds for BM25 on a CPU, 19 to 20 milliseconds for DPR and ANCE on a GPU and 230 to 275 milliseconds on a CPU, 30 milliseconds for docT5query, and 350 milliseconds for ColBERT on a GPU. docT5query's 0.4 GB index matches BM25's, and its 30 milliseconds sits close to BM25's 20, because it searches a BM25 index; its extra cost, generating 40 queries per document, falls at indexing time.

Two consequences follow from where the cost sits. A dense index depends on one encoder, so every passage vector has to be recomputed if the encoder changes, where a BM25 index is rebuilt only when the documents change. And a dense retriever trained on one domain carries that domain into every collection it indexes, where BM25 carries only the collection's own term statistics. The 8.8 hours on 8 GPUs is a one-time cost on a server; on a laptop or an air-gapped machine with a large manual set, it is the number that decides whether dense indexing is feasible at all.

What to check

• Does the test set include the exact identifiers the users search for, such as part numbers and clause numbers, and the same items described in words?
• Was the dense encoder trained or fine-tuned on documents like these, and what did it score on the collection itself rather than on MS MARCO?
• Is BM25 run as a baseline on the same queries, and are results reported per query type? BEIR found no single approach best everywhere.
• If both retrievers run, how are the lists merged, and were the weights or the fusion constant tuned on labeled queries?
• What does re-encoding the collection cost on the delivered hardware when the embedding model changes, and how large is the vector index compared with the inverted index?

Azcuna's TRACE is one offline viewer for S1000D, MIL-STD-3001 XML, and PDF technical manuals. It keeps the source location and the markings of everything it shows and never modifies the source. It was proposed to NAVAIR under Navy SBIR Phase I topic DON26BZ05-NV078, and the proposal was submitted on September 23, 2026.

Questions people ask

What is the difference between dense and sparse retrieval?

Sparse retrieval represents a document as a vector with one slot per vocabulary word, almost all zero, and scores it by the query words it shares, as BM25 does through an inverted index. Dense retrieval represents a query and a passage as short learned vectors, 768 numbers in the Dense Passage Retrieval paper, and scores by the dot product, so two texts can match with no word in common. The DPR paper calls TF-IDF and BM25 the traditional sparse vector space models and shows retrieval working from dense representations alone.

What is dense retrieval?

Dense retrieval encodes a query and each passage into fixed-length vectors with a trained model and returns the passages whose vectors are nearest the query vector. The Dense Passage Retrieval (DPR) paper by Karpukhin and colleagues, at EMNLP 2020, used two independent BERT-base encoders, the 768-dimensional output of the [CLS] token, and the dot product as the similarity. Trained on question-passage pairs, it outperformed Lucene BM25 by 9% to 19% absolute in top-20 passage retrieval accuracy on open-domain question answering datasets.

What is sparse retrieval?

Sparse retrieval scores documents by the query terms they contain, with weights for how rare and how frequent each term is. BM25 is the standard example. Robertson and Zaragoza's 2009 survey gives its term weight as a saturating function of term frequency, adjusted for document length through the parameter b, multiplied by an inverse document frequency weight. The vector is sparse because a document contains only a few of the vocabulary's words, which is what lets an inverted index find candidates by looking up the query terms.

Is BM25 dense or sparse?

Sparse. The Dense Passage Retrieval paper names TF-IDF and BM25 as the traditional sparse vector space models, and the BEIR benchmark puts BM25 in its lexical category, next to a separate sparse category of learned models such as DeepCT, SPARTA, and docT5query. BM25 looks only at terms the query and the document share, through an inverted index; a dense retriever compares learned vectors and can match texts with no shared word. Lucene's BM25Similarity class, with k1 = 1.2 and b = 0.75 by default, is the common implementation.

What is SPLADE?

SPLADE, the Sparse Lexical and Expansion Model, is a learned sparse retriever from Formal, Piwowarski, and Clinchant, presented as a SIGIR 2021 short paper. A BERT model's masked-language-model head produces a weight for every term in the 30,522-word WordPiece vocabulary, passed through log(1 + ReLU) and pushed toward zero by a FLOPS regularizer, so each document keeps a few weighted terms, including expansion terms not in its text. On the MS MARCO development set it reached 0.322 MRR@10 against 0.184 for BM25, and it runs on an inverted index.

What is learned sparse retrieval?

Learned sparse retrieval uses a neural model to produce the term weights that BM25 computes from counts, while keeping the sparse vocabulary vector and the inverted index. The SPLADE paper describes learning sparse representations for queries and documents that keep exact term matching, inverted-index efficiency, and interpretability, and adds expansion terms the document never used. BEIR groups earlier methods of this kind, DeepCT, SPARTA, and docT5query, as sparse; docT5query was the only one to beat BM25 on average, by 1.6%, with DeepCT 27.9% below.

When does BM25 beat embeddings?

On rare exact strings and on data unlike the embedding model's training set. The DPR paper notes that term-matching methods are sensitive to highly selective keywords and gives the name Thoros of Myr as a phrase its dense retriever could not capture. Sciavolino and colleagues' EntityQuestions set put DPR's top-20 accuracy at 49.7% against 72.0% for BM25. BEIR found dense retrievers underperforming on datasets with a large domain shift, such as BioASQ, or a task shift, such as Touché-2020, and concluded that BM25 remains a strong zero-shot baseline.

Which costs more to index, dense or sparse retrieval?

Dense. The DPR paper reports that building the Lucene inverted index for 21 million Wikipedia passages took about 30 minutes, while encoding the same passages took roughly 8.8 hours on 8 GPUs and building the FAISS vector index another 8.5 hours. BEIR's cost table for a 1 million document corpus gives a 0.4 GB index for BM25 and 3 GB for the 768-dimensional dense retrievers DPR and ANCE. At query time the balance reverses: DPR answered 995 questions per second from memory against 23.7 per CPU thread for BM25.

Should you combine dense and sparse retrieval?

Usually, because each misses what the other finds. The DPR paper tested a weighted sum, reranking the union of each method's top 2,000 passages by the BM25 score plus 1.1 times the dense score: top-20 accuracy on TREC rose from 79.8% to 85.2% and on SQuAD from 63.2% to 71.5%, but on Natural Questions fell from 78.4% to 76.6%. BEIR found no single approach best on all 18 datasets. Reciprocal rank fusion merges the two lists without comparing their scores and is the common default in hybrid search.

Sources

1. 1Dense Passage Retrieval for Open-Domain Question Answering · arXiv (Karpukhin et al., EMNLP 2020)
2. 2BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models · arXiv (Thakur et al., NeurIPS 2021 Datasets and Benchmarks track)
3. 3SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking · arXiv (Formal, Piwowarski, and Clinchant, SIGIR 2021)
4. 4The Probabilistic Relevance Framework: BM25 and Beyond · Foundations and Trends in Information Retrieval, vol. 3, no. 4 (2009); author copy at City University
5. 5Simple Entity-Centric Questions Challenge Dense Retrievers · arXiv (Sciavolino et al., EMNLP 2021)
6. 6ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT · arXiv (Khattab and Zaharia, SIGIR 2020)
7. 7Retrieve & Re-Rank · Sentence Transformers documentation
8. 8Class BM25Similarity (Lucene 9.12.0 API) · Apache Lucene

Where this shows up in our work

• TRACE, the offline S1000D and IETM viewer opens S1000D, MIL-STD-3001 and PDF technical manuals on a tablet with every marking intact.
• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.