Retrieval metrics, explained: precision, recall, MRR and nDCG

Retrieval metrics score a ranked list of search results against human judgments of which documents are relevant. The Stanford online edition of Introduction to Information Retrieval, by Manning, Raghavan, and Schütze, defines the two basic measures: precision is the fraction of retrieved documents that are relevant, and recall is the fraction of relevant documents that are retrieved. Mean reciprocal rank (MRR), from TREC question answering in 1999, and normalized discounted cumulative gain (nDCG), from 2002 work on graded relevance, extend them. Each condenses a ranking into one number, and each leaves something out. That matters when the ranking decides which passages an answering system reads.

Precision and recall at a cutoff

The textbook calls precision and recall set-based measures. For a ranked list, the natural set is the top k results, which gives precision at k, such as precision at 10. The book lists its strength and its weakness. Precision at k needs no estimate of how many relevant documents exist. It is also the least stable of the common measures and does not average well, because the number of relevant documents for a query strongly affects it. With only 8 relevant documents in a collection, even a perfect system scores 0.4 at precision at 20. R-precision, the precision of the top R results where R is the number of relevant documents, adjusts for this.

Recall at k divides the relevant documents in the top k by all relevant documents in the collection, the definition TREC's common measures appendix gives. That denominator has to be known. Recall alone is easy to max out: retrieving every document gives a recall of 1.

As an illustration only: a query has 4 relevant documents, and a system places them at ranks 1, 3, and 8 of its top 10. Precision at 5 is 2/5, or 0.4, and recall at 5 is 2/4, or 0.5. At 10, precision is 0.3 and recall is 0.75.

Mean reciprocal rank

MRR comes from the TREC-8 Question Answering track of 1999. The track report by Ellen Voorhees of the National Institute of Standards and Technology (NIST) describes 200 fact-based, short-answer questions. Systems returned five ranked answer strings per question, and human assessors judged each string right or wrong. A question scored the reciprocal of the rank of the first correct answer, or 0 if none of the five was correct; MRR is the mean over questions.

The report lists the advantages: MRR is bounded between 0 and 1, averages well, and is closely related to average precision. It also lists the drawbacks. A question can take only six values: 0, .2, .25, .33, .5, and 1. A system gets no credit for finding several different correct answers. Because every question required a response, a system also got no credit for recognizing that it did not know. The top 50-byte run scored .660 over the 198 official questions, with no correct answer found for 54.

As an illustration only: first correct answers at ranks 3 and 1, and none in the top five for a third question, give (1/3 + 1 + 0) / 3, which is 4/9, about 0.44.

Graded relevance and nDCG

Järvelin and Kekäläinen proposed cumulated gain in ACM Transactions on Information Systems in October 2002. They note that binary judgments give equal credit for highly and marginally relevant documents, and that TREC counted a document relevant if at least one sentence pertained to the request. Cumulated gain adds up relevance scores from 0 to 3 down the list; in their example, gains of 3, 2, 3, 0, 0, 1, 2 reach 11 at rank 7. Discounted cumulated gain divides each score by the logarithm, base b, of its rank, starting at rank b. The normalized form divides by the ideal list, built from every known relevant document sorted by grade, so 1 means ideal.

Their case study rejudged documents for 20 TREC-7 topics on a four-point scale. Of 2,772 documents TREC had judged relevant, 691, or 25.0 percent, were rated not relevant, and about half the relevant documents were marginal. The authors call the gain values and the discount somewhat arbitrary, and note that the measures ignore overlap between documents.

The textbook's nDCG formula differs. The gain is 2 raised to the relevance score, minus 1, divided by the base-2 logarithm of the rank plus 1, normalized so a perfect ranking scores 1. Gain and discount both differ from the 2002 form, so scores from the two do not compare.

How TREC applies them

NIST has run the Text REtrieval Conference (TREC) since 1992. The textbook's page on test collections explains that judgments are not exhaustive: NIST assessors judged only documents in the top k returned by some system in the evaluation. Recall measured against such a pool counts only the relevant documents someone found. The 2006 appendix lists the standard trec_eval report, written by Chris Buckley, with precision at nine cutoffs from 5 to 1,000 documents and R-precision.

The TREC 2019 Deep Learning track made nDCG at 10 its main metric, because it uses the track's four-level judgments and focuses on the first results users see. The track's training labels were sparse, often with one positive label per query, and each task's reusable test set had 43 queries, drawn from 200 test queries. Where reciprocal rank on the sparse labels disagreed with nDCG, the organizers considered the nDCG results more valid.

What to ask

• Which cutoff k was used, and how many relevant documents does each test query have?
• For recall, were relevant documents found by exhaustive judgment or from a pool of systems' top results?
• Were judgments binary or graded, and on what scale?
• Which nDCG formula was used: the 2002 log-base discount or the textbook's 2-to-the-relevance gain?
• Does an MRR figure come with the share of queries where nothing relevant was found?
• Were the test queries and judgments drawn from the manuals the system will search?

Azcuna's TRACE is one offline viewer for S1000D, MIL-STD-3001 XML, and PDF technical manuals. It keeps the source location and the markings of everything it shows and never modifies the source. It was proposed to NAVAIR under Navy SBIR Phase I topic DON26BZ05-NV078, and the proposal was submitted on September 23, 2026.

Sources

1. 1Evaluation of unranked retrieval sets · Introduction to Information Retrieval, Manning, Raghavan, and Schütze (Cambridge University Press, 2008), Stanford online edition
2. 2Evaluation of ranked retrieval results · Introduction to Information Retrieval, Manning, Raghavan, and Schütze (Cambridge University Press, 2008), Stanford online edition
3. 3Standard test collections · Introduction to Information Retrieval, Manning, Raghavan, and Schütze (Cambridge University Press, 2008), Stanford online edition
4. 4Cumulated Gain-Based Evaluation of IR Techniques · ACM Transactions on Information Systems 20(4), October 2002 (Järvelin and Kekäläinen)
5. 5The TREC-8 Question Answering Track Report · National Institute of Standards and Technology (Voorhees), TREC-8 proceedings
6. 6Common Evaluation Measures · National Institute of Standards and Technology, TREC 2006 proceedings appendix
7. 7Overview of the TREC 2019 Deep Learning Track · National Institute of Standards and Technology, TREC 2019 proceedings (Craswell, Mitra, Yilmaz, Campos, and Voorhees)

Where this shows up in our work

• TRACE, the offline S1000D and IETM viewer opens S1000D, MIL-STD-3001 and PDF technical manuals on a tablet with every marking intact.
• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.