Cosine, dot product and Euclidean distance: how vector search decides what is close

Rafael

• 2025-07-15

Vector search turns every record into a list of numbers and every question into another list. Then it asks which lists are close. "Close" needs a definition, and three are in common use: cosine similarity, dot product and Euclidean distance. This post explains what each one measures, when they give the same order, when they do not, and why a single similarity value says little on its own.

It is the first of three posts on how vector search works underneath. The next one covers approximate nearest neighbor indexes, and the one after that covers how to measure whether the results are any good.

A vector is an arrow

An embedding model reads a ticket, a part record or a question and returns a vector: a list of numbers, usually a few hundred to a few thousand of them. Think of it as an arrow from the origin. Two texts about the same thing should produce arrows that point roughly the same way.

The figure below uses two numbers per vector so you can see the arrows. A query and four documents, A to D. Switch between the three measures and watch the ranking change. Drag in the plot to turn the query yourself.

What each measure sees

Cosine ignores length. A and B in the figure have almost the same angle, so cosine keeps them together. The dot product multiplies by length, so a long vector outranks a short one that points the same way. Euclidean distance measures the straight line between the tips, so length counts there too.

The three agree in one case. When every vector has length 1, cosine similarity equals the dot product, and the squared Euclidean distance equals 2 − 2 × cosine. Tick "normalize vectors" in the figure and the three orders collapse into one. This is why many systems normalize once when a record is indexed and then use the cheapest measure, the dot product.

Which one to use

Use the one the embedding model was trained with. Piotr Migdał points out in Don't use cosine similarity carelessly that some models are trained on dot products of unnormalized vectors and others on Euclidean distance. Cosine may not match what those models learned. If the model card does not say, normalize and use cosine.

The tools make the choice explicit:

• pgvector has one operator per measure: <=> for cosine distance, <-> for Euclidean and <#> for the negative inner product. The inner product is negated because the index sorts ascending.
• Faiss has IndexFlatL2 and IndexFlatIP for inner product. Its wiki says cosine works with the inner product index after you normalize the vectors beforehand.
• HNSW-Sharp, our open source HNSW library, ships four cosine variants. Two assume unit vectors and do not check. Its documentation warns that passing non-unit vectors to them gives wrong distances silently, and says normalizing once and using the unit variant is typically 1.5 to 3 times faster.

Why 0.6 tells you little

A similarity value looks like a grade, but nothing fixes its scale. The second figure shows one reason. It draws random vectors in 2, 4, 8 and on up to 1,024 dimensions and plots the cosine of every pair.

For random vectors the spread of the cosine is about 1 divided by the square root of the number of dimensions. At 1,024 dimensions that is about 0.03, so nearly every random pair lands between -0.06 and 0.06. Real embeddings are not random, and the models spread them differently. The lesson carries over anyway: what a value means depends on the dimensions and on the model, so a 0.6 from one model is not a 0.6 from another.

Two papers go further. Migdał notes that a cosine between -1 and 1 should not be mistaken for a probability. He also shows a closest match of the wrong kind: a question matched to another question instead of an answer. Harald Steck, Chaitanya Ekanadham and Nathan Kallus show in Is cosine-similarity of embeddings really about similarity? that for some regularized linear models the cosine similarities can be arbitrary, and that regularization in deeper models can have effects nobody intended. Their advice, and ours, is to look at the results before trusting the number. We wrote about the reading side in A search score is an order, not a grade.

Two more you will meet

Binary codes, produced by hashing or quantization, are compared with Hamming distance: XOR the two codes and count the bits that differ. Sets, such as the tags on a record, are compared with Jaccard similarity: the size of the overlap divided by the size of the union. pgvector supports both, and the HNSW-Sharp documentation shows a Hamming distance in a few lines of C#.

A short checklist

1. Find the measure the model was trained with, and use that one.
2. If you can, normalize once at indexing time and store unit vectors.
3. Never mix vectors from different models in one index. They are not comparable, and switching the model means embedding the corpus again.
4. Compare scores only inside one query's result list. Read the order and the gaps, not the value.

Where this shows up in Curiosity Studio

Curiosity Studio normalizes similarity scores to roughly 0 to 1, according to the vector search documentation, so a score is comparable inside one result list and nowhere else. EmitWithScores() keeps it on the query so you can rank and explain with it. What this post does not tell you is which measure is best: for a given model, that is the one it was trained with.

Further reading

• Don't use cosine similarity carelessly, Piotr Migdał. Cosine as a dot product of normalized vectors, and where it matches the wrong kind of text.
• Is cosine-similarity of embeddings really about similarity?, Harald Steck, Chaitanya Ekanadham and Nathan Kallus. The formal case that cosine similarities can be arbitrary for some models.
• Faiss indexes, the Faiss wiki. Which index uses which measure, and how to get cosine from the inner product.
• pgvector, Andrew Kane. The distance operators in a database you may already run.