What is a reranker?

A reranker is the second stage of a search. A fast first stage pulls a list of candidate passages, and the reranker reads the query and each candidate together and puts them in a new order. In January 2019, Nogueira and Cho showed that BERT, a pretrained transformer language model, used this way became the top entry on the MS MARCO passage ranking leaderboard, beating the previous state of the art by 27% (relative) in mean reciprocal rank at 10 (MRR@10). The price is computation. In a retrieval-augmented system, the reranker also decides which passage an answer cites.

Bi-encoders and cross-encoders

The two designs differ in when the query meets the passage. A bi-encoder turns the query and each passage into vectors separately, so passage vectors can be computed in advance and indexed. A cross-encoder, as the Sentence Transformers documentation puts it, takes the query and a document "simultaneously" and outputs one relevance score between 0 and 1. The cross-encoder attends across query and document together, so every candidate must pass through the model with the query, and scoring millions of pairs would be "rather slow."

The Sentence-BERT paper by Reimers and Gurevych gives the scale. Finding the most similar pair among 10,000 sentences with a BERT cross-encoder takes n(n-1)/2 = 49,995,000 inference computations, about 65 hours on a V100 graphics processing unit (GPU). Their bi-encoder computed the 10,000 sentence embeddings in about 5 seconds. Accuracy runs the other way. On the semantic textual similarity (STS) benchmark test set, a large BERT cross-encoder scored 88.77 in Spearman rank correlation, against 86.10 for the comparable bi-encoder.

The common answer is to use both. The documentation describes a pipeline in which lexical search or a bi-encoder retrieves, for example, 100 candidates, and a cross-encoder rescores all of them. The retriever must be efficient over collections with millions of entries but may return irrelevant candidates, and the documentation says a cross-encoder reranker can substantially improve the final results.

What Nogueira and Cho reported

Nogueira and Cho, at New York University, describe a three-stage pipeline. A standard method such as BM25 retrieves a large number of candidates, for example a thousand. A more computationally intensive model scores and reorders them. The top ten or fifty then feed an answer module.

Their reranker is BERT-large used as a binary classifier. The query, truncated to 64 tokens, goes in with the passage, up to 512 tokens in total, and the model outputs the probability that the passage is relevant. Each passage is scored independently, and the list is sorted by that probability.

Their main test was MS MARCO, a MAchine Reading COmprehension dataset. The dataset paper describes 1,010,916 anonymized questions sampled from Bing search logs and 8,841,823 passages from web documents. The development set paired each of about 6,900 queries with the top 1,000 passages from BM25, with one relevant passage per query on average.

MRR@10 on that set was 16.7 for untuned BM25, 27.8 for the previous best leaderboard entry, 34.7 for BERT-base, and 36.5 for BERT-large. On the evaluation set, the paper reports 35.8 for BERT-large; the ColBERT paper, citing the same work, lists 35.9. Training took about 30 hours and covered 12.8 million query-passage pairs, less than 2% of the training set.

The cost of reading every pair

The ColBERT paper by Khattab and Zaharia, presented in July 2020, timed rerankers on MS MARCO's official top 1,000 BM25 results per query with one Tesla V100 GPU. Reranking took 10,700 milliseconds per query for BERT-base, at 97 trillion floating-point operations (FLOPs), and 32,900 milliseconds for BERT-large. ColBERT, which encodes query and passage separately and compares them token by token in a cheap final step, took 61 milliseconds, with MRR@10 of 34.9 against 36.0 for the authors' own BERT-base. They report over 170 times the speed with 14,000 times fewer FLOPs.

Cost grows with the number of candidates, because each one is scored separately. The BEIR benchmark reranked only the top 100 BM25 hits, with a 6-layer cross-encoder. That combination performed best in the study and beat BM25 on 16 of 18 datasets, failing on two tasks very different from its MS MARCO training data. The reranking latency was above 350 milliseconds, while dense retrievers were 20 to 30 times faster.

A reranker can only reorder what the first stage found. Nogueira and Cho note that some relevant MS MARCO passages might not be retrieved by BM25 at all.

How a reranker changes which passage gets cited

In a retrieval-augmented system, the generator sees only the top few passages, so the reranker's order decides which ones an answer can cite. Re2G, by Glass and colleagues at the 2022 conference of the North American Chapter of the Association for Computational Linguistics (NAACL), puts a reranker between retrieval and a BART-based generator. Because the reranker scores every passage itself, Re2G could merge BM25 and dense results whose own scores are not comparable.

Re2G was tested on the KILT benchmark of knowledge-intensive tasks, which expects systems to justify each output with provenance. KILT scores provenance with R-Precision: the fraction of the R ground-truth provenance documents that appear in the system's top R. The benchmark's headline answer metrics count an answer only when R-Precision is one.

On the Natural Questions development set, R-Precision was 64.65 for the dense retriever alone and 70.78 after reranker training. On the T-REx slot-filling set it rose from 65.02 to 81.22. The gains were not uniform. On TriviaQA, R-Precision fell from 71.80 to 60.37 after end-to-end training even as answers improved, which the authors attribute to incomplete provenance ground truth. They add that at least a third of the gain in output quality did not come from better recall. Answer quality and source quality moved separately.

What to check

• How many candidates reach the reranker, and what is the latency per query at that depth on the delivered hardware?
• What was the reranker trained on, and was it tested on documents like these? BEIR's reranker failed on two tasks very different from its training data.
• Can a relevant passage that the first stage missed ever reach the reranker?
• Is source correctness measured separately from answer accuracy, as KILT's R-Precision does?
• When the reranker or the list depth changes, are the cited passages checked again?

Azcuna's TRACE is one offline viewer for S1000D, MIL-STD-3001 XML, and PDF technical manuals. It keeps the source location and the markings of everything it shows and never modifies the source. It was proposed to NAVAIR under Navy SBIR Phase I topic DON26BZ05-NV078, and the proposal was submitted on September 23, 2026.

Sources

1. 1Passage Re-ranking with BERT · arXiv (Nogueira and Cho, 2019)
2. 2MS MARCO: A Human Generated MAchine Reading COmprehension Dataset · arXiv (Bajaj et al.)
3. 3Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks · arXiv (Reimers and Gurevych, 2019)
4. 4Retrieve & Re-Rank · Sentence Transformers documentation
5. 5ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT · arXiv (Khattab and Zaharia, SIGIR 2020)
6. 6BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models · arXiv (Thakur et al., NeurIPS 2021 Datasets and Benchmarks track)
7. 7Re2G: Retrieve, Rerank, Generate · ACL Anthology (Glass et al., NAACL 2022)

Where this shows up in our work

• TRACE, the offline S1000D and IETM viewer opens S1000D, MIL-STD-3001 and PDF technical manuals on a tablet with every marking intact.
• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.