Measuring search relevance: precision, recall, reciprocal rank and NDCG

Rafael

• 2025-11-11

A change to a ranking always feels like an improvement when you try three queries by hand. To know whether it is one, you need a measure that turns a ranked list into a number and a set of queries to compute it on. This post covers four measures: precision and recall at k, reciprocal rank, and NDCG, the normalized discounted cumulative gain. It ends with the part that matters more than the choice of measure, the set of judged queries behind it.

It is part 3 of 3. Part 1 covered how vector search defines "close", and part 2 how approximate indexes find it quickly. Both can lose a good result. This part measures whether the list that comes out is any good.

What you need first: judgments

Every measure below compares a ranked list with judgments: for a query, which documents are relevant, and how relevant. A judgment is a label from a person who knows the domain. For the query "cracked bracket", a supplier notice about that bracket might be a 3 and a maintenance log that mentions it a 2. A drawing of it might be a 1, and an unrelated work order a 0.

Binary judgments, relevant or not, are cheaper. Graded judgments, such as 0 to 3, capture that some results are better than others. NDCG uses grades. The others use the binary version.

Precision and recall at k

Take the top k results. Precision@k is the share of them that are relevant. Recall@k is the share of all relevant documents that made it into the top k. The book Introduction to Information Retrieval, by Christopher Manning, Prabhakar Raghavan and Hinrich Schütze, calls precision at a fixed number of results the least stable common measure. Unlike recall, it does not need the total number of relevant documents.

Both ignore order inside the top k. Moving a relevant result from rank 1 to rank 5 changes neither. Reciprocal rank, RR, adds order for one thing: 1 divided by the rank of the first relevant result. Averaged over queries it is the mean reciprocal rank, MRR. Average precision adds up the precision at each rank where a relevant result appears and divides by the number of relevant documents. Averaged over queries it is MAP.

Figure 1. Twelve ranked results, six relevant results in the collection, one of them outside the list. The marker moves down the list. Click a result to flip it.

In the figure's default list, ranks 2, 3, 5, 8 and 11 are relevant. At k = 5 there are 3 hits, so precision@5 is 0.60 and recall@5 is 3 of 6, 0.50. The first hit is at rank 2, so the reciprocal rank is 0.50. Average precision over the 12 results is 0.45.

NDCG: graded, and order matters

NDCG answers a question the others do not: given that some results are better than others, is the best one on top?

It has three steps. First, turn each result's grade into a gain. The usual choice, which the Manning book's formula uses, is 2 to the power of the grade, minus 1. Grades 0, 1, 2 and 3 give gains 0, 1, 3 and 7. Second, discount each gain by its position, dividing by log base 2 of (rank + 1). Rank 1 counts in full, rank 2 counts 0.63 and rank 3 counts 0.50. Add the discounted gains and you have DCG. Third, divide by the DCG of the best possible order, so a perfect list scores 1.00. In the book's notation, with a normalizing factor Z chosen to make a perfect ranking at k score 1:

\(\mathrm{NDCG}(Q, k) = \frac{1}{|Q|} \sum_{j=1}^{|Q|} Z_{kj} \sum_{m=1}^{k} \frac{2^{R(j,m)} - 1}{\log_2(1 + m)}\)

Here R(j, m) is the grade the assessors gave the document at rank m for query j.

The figure below runs this on eight results for "cracked bracket". It adds one rank at a time, then moves better results up one swap at a time while NDCG rises. Click any row to move it up yourself, and switch the gain to the plain grade to see how much the exponential gain favors the top grades.

Figure 2. NDCG@8 for eight judged results. The default order scores 0.70. After the swaps it reaches 1.00.

The default order has grades 1, 3, 0, 2, 0, 3, 1, 2. Its DCG@8 is 10.48. The ideal order, 3, 3, 2, 2, 1, 1, 0, 0, has a DCG@8 of 14.95. NDCG@8 is 10.48 divided by 14.95, which is 0.70. At k = 3 the same list scores 0.42. The other grade 3 result, at rank 6, and both grade 2 results fall outside the cutoff, while a grade 0 result sits at rank 3.

You can compute all of it in a few lines. This is the version used for the figure:

from math import log2

def dcg(grades, k):
    return sum((2 ** g - 1) / log2(i + 2) for i, g in enumerate(grades[:k]))

def ndcg(grades, k):
    ideal = dcg(sorted(grades, reverse=True), k)
    return dcg(grades, k) / ideal if ideal else 0.0

ndcg([1, 3, 0, 2, 0, 3, 1, 2], 8)   # 0.70

Which measure for which question

Measure	Question it answers	Blind to
Precision@k	How much of what I show is relevant?	Order inside the top k, and relevant results you did not show
Recall@k	How much of what exists did I show?	Order inside the top k
Reciprocal rank	How far down is the first good result?	Everything after it
NDCG@k	Is the best material near the top?	Documents nobody judged

BEIR, a benchmark of 18 retrieval datasets by Nandan Thakur and colleagues, picked a single measure to compare systems across tasks and chose nDCG@10. Its authors argue that precision and recall ignore rank, and that reciprocal rank and average precision cannot grade relevance, while NDCG handles both binary and graded judgments.

Unjudged documents count as not relevant

All four measures treat a document with no judgment as not relevant. That is where measurement goes wrong. BEIR looked at TREC-COVID and found that dense retrieval systems returned many hits that nobody had judged: 14.4% of the top 10 for ANCE and 31.8% for TAS-B. The authors had 980 query and document pairs judged. Lexical systems barely moved. ANCE went from an nDCG@10 of 0.654 to 0.735.

The lesson is not specific to that benchmark. A system that retrieves differently from the one that produced your judgments gets scored too low. When you change the ranking, look at the unjudged results near the top and judge them before you trust the comparison.

The golden set

A measure needs queries to run on. The relevance evaluation page in our documentation recommends a golden set of 80 to 120 queries with labeled correct results. The queries come from top queries by volume, queries that returned nothing, jargon, the long tail and known failures from users. Three people label in parallel, and the labels are refreshed quarterly. The set is run after every change, with a script and a gate that ships only what beats the baseline. The page tracks precision@1, precision@3, NDCG@10, recall@50 and the rate of queries with no results.

Two details from that page carry over. Label by document ID, not by text, because text drifts. And note the difference in definitions. The documentation's RecallAtK returns 1 when any expected result is in the top k, which is a hit rate. Its NDCG uses a binary gain on a ranked list of expected results. The figures in this post use the textbook definitions, so numbers from the two will not match unless you convert.

Do not mix up two kinds of recall

Part 2 used recall too, and it means something else there. ANN recall compares an approximate index with exact search over the same vectors. Relevance recall compares a result list with human judgments. An index can have recall@10 of 1.00 against exact search and still return a poor list, if the embedding model matched the wrong kind of text. Measure both, and keep the names apart in your reports.

Where this shows up in Curiosity Studio

The relevance evaluation guide in our documentation stores the golden set in the graph, as GoldenQuery nodes with Expects edges to the labeled results. It also gives the endpoint code for the metrics. The hybrid search page uses it to sweep the weight between keyword and vector search and re-run the set weekly, to catch drift after a model change. What none of this does is tell you the right grades: those come from people who know the parts, the tickets and the suppliers.