A search score is an order, not a grade

Leon

• 2026-01-21

You search for a supplier notice about a cracked bracket. The top result has a score of 0.84 and a full bar next to it. The next one has 0.79. It is hard not to read that as "84% relevant" and "almost as good". A maintenance planner asks whether 0.6 is good enough to trust. An engineer asks why yesterday's search scored higher than today's.

These are fair questions, and none of them can be answered from the number. A search score puts results in order. It does not say how good any one result is. We wrote a short paper about this, What a search score means, for anyone who reads ranked results in Curiosity Studio, not only for engineers. This post gives its argument, its best examples, and some of the wider discussion it sits in.

Think of a soup contest

The paper opens with a soup contest. Five cooks each make a soup. A judge tastes them, puts them in order and gives each one a number. From that you can say which soup came first, whether soup B beat soup C, and where the list drops off. You cannot say whether soup A is tasty, whether it is twice as good as soup B, or whether it is good enough to serve.

All five soups can be bad. Then the winner is only the least bad one. A search works the same way. It can return a first result even when nothing in your data answers the question.

Why the top bar is always full

Result lists in Studio often draw a bar for each score. The bar is as long as the score, compared with the best score in that list. So the top result always has a full bar.

The paper shows two searches side by side. In search A the best score is 0.84 and the next ones are 0.79, 0.66, 0.41 and 0.38. In search B the best is 0.29, then 0.27, 0.22, 0.12 and 0.11. Both lists start with the same full bar. Search A found strong matches. Search B did not. The bar tells you about the order inside one list, and nothing about the list compared with another one.

What goes into the number

A score in Studio is built from several parts, each with its own scale:

Part	What it looks at	How it changes the score
Search terms	Your words, and near matches	Rare words count more
Text similarity	The meaning of the text	Closer meaning scores higher
Graph links	Connections shared by two items	Adds a signal you can weight
Recency	The age of the item	Last 14 days keep full score
Boosts	Weights an admin sets	Chosen words or sources count more

Search adds the parts up, with weights, into one number. That number has no fixed top. Which parts count also depends on the search. A plain keyword search mainly uses search terms and recency. A similarity view can add meaning and shared links. So a score of 0.6 from one kind of search and 0.6 from another are not the same 0.6.

Teams that build search systems run into the same problem. John Wang, CTO of Assembled, describes how his team tried to combine keyword and vector search by weighting their scores and found the weights hard to set, because the distribution of vector scores was unknown and varied widely from one customer to the next. They moved to reciprocal rank fusion, which throws the raw scores away and combines the lists by rank alone. That is a strong vote for the paper's point: the order carries the information, and the raw number on its own scale does not.

Why it is not a percentage

The paper gives four reasons, each with an everyday example.

The scale is not fixed. The same number can mean different things in different searches. In German schools 1 is the best mark. In some other countries 1 is the worst.
A score only means something inside its list. The tallest child in a class is not always a tall person.
There is no whole. A percentage is a share of a whole, like 50% of a pizza. A score has no pizza. No number means "perfect".
A percentage looks more exact than it is. "87%" sounds like "87% sure". Imagine an app that says "Your day: 87% nice". Nobody knows what that measures.

Even the parts that come with a familiar range are not percentages. In many search systems, text similarity is measured as cosine similarity, which lands between -1 and 1. Piotr Migdał makes the point in Don't use cosine similarity carelessly: values that usually fall between 0 and 1 do not represent probabilities, and "the value 0.6 tells little if we have something really similar, or not so much". He also shows that the closest match to a question is often another question, not an answer. Harald Steck, Chaitanya Ekanadham and Nathan Kallus go further in Is cosine-similarity of embeddings really about similarity?. For regularized linear models they derive analytically that cosine similarity can give arbitrary, and so meaningless, similarities. If one ingredient of a score has no fixed meaning, the sum cannot have one either.

The counterpoint: then show a probability

The obvious objection is that we should simply calibrate. Turn the score into "how likely is this the right document" and show that. Some models are trained for exactly that, and Migdał's post describes this kind of setup, where a sigmoid over a dot product predicts a probability for a specific question such as "is B a correct answer to A?".

The answer lies in what such a number could mean in a list like Studio's. A score here is a weighted sum of keyword match, meaning, shared links, age and admin boosts. "Relevant" in that sum means relevant to a question the search engine guessed from your words. Only you know what you need. The paper's glossary puts it plainly: relevance is how well a result fits what you need, and only you can judge it. A percentage on the result would claim to know that, and it would be trusted more than it should be.

Several commenters in the HN thread under Migdał's post suggest a different fix: rerank the top results with a model that reads the question and each candidate together. That improves the order. It still gives you an order.

What a score can tell you

The paper sorts the questions people ask into those a score answers and those it does not:

Your question	Use instead
Which result should I read first?	The order
Is result A ranked above B?	The order
How far apart are A and B?	The gap, not a ratio
Is this result relevant to me?	Open it and read
Is 0.8 a good score?	Nothing: there is no such value
Which of two searches did better?	Read both top results

The gap is the part that is easy to skip. In search A above, the drop from 0.66 to 0.41 is larger than any other step in the list. A big gap between two results often means a change in kind. Above it you may have the maintenance logs for that bracket. Below it, field reports that only share a few words with your question.

Three habits for reading a list

The paper closes with three habits that take a few seconds each.

Read from the top. The order is the signal. Start at result 1 and go down.
Look for a drop. A large gap between neighbors usually marks where the strong matches end.
Judge by content. Open the result and read it. Only the content shows whether it fits.

And three checks before you act on a score: Are you comparing results from one search only? Are you using the gap between results, not the size of the number? Have you read the top results, not only looked at the bars?

The same holds when software does the reading. When an assistant in Studio answers from your part records and tickets, its answer traces back to a record you can open. The number decided what to read first. The record decides what is true.

Read the paper

The paper is ten pages, written in plain language, with the soup contest, the two searches and the tables above, plus a short glossary of terms like score, rank, signal and gap. It is a good thing to hand to the people on your team who read ranked results all day and wonder what the bar means. Read it on its page, or download the PDF directly.

Read the whitepaper (PDF)