Which search do you mean? Six layers, and where each one belongs

Leon
•

Ask five engineers what search means and you get five systems. One means the box that finds a part number. One means a ranked list. One means a chat window that answers in sentences. Each of them is right about a different problem, so the meeting ends without a decision.

That is the starting point of Modern enterprise search, a whitepaper we wrote for the architects who have to choose. Its claim is short. Search is not one thing, and most arguments about it are about which layer is meant. Name the layer first, and most of the disagreement goes away.

Six layers, one question each

The paper splits search into six layers. Each one builds on the one before it.

Layer Finds Misses
Keyword Exact ids, part numbers Synonyms, wording
Fuzzy Typos, variants Meaning
Vector Meaning, descriptions Exact ids
Hybrid Both, in one list Relations between records
Graph How records relate Free text alone
Agents Multi-step questions Nothing, if built on the rest

The table is the paper's, condensed. No row covers everything. Each layer fills a gap the one above it leaves, and that is the case for running them together instead of picking one. The layers also depend on each other: an agent is only as good as the search it calls, and a hybrid ranking is only as good as the two lists it merges. The paper's advice is to list the questions people ask today, mark each with the layer it needs, and build the layers you marked most. Which product to buy comes later.

Where keyword still wins

A technician types A320-2741-08. There is one right answer. A near miss is the wrong part on the line.

Vector search is built to find what is close. For an identifier, close is the failure. Part 2741-08 and part 2741-09 sit next to each other in any space of meaning, and they are different parts. That is why a production system keeps a keyword index beside the vector one. The paper's rule: keep keyword for identifiers, add vectors for meaning, and never replace one with the other.

The same chapter has a design rule I would put on the wall. When the exact match is missing, return nothing and say so. Then offer the close matches as a separate list, marked as close. A list of near misses with no warning invites someone to pick the wrong part.

What vectors add

Keyword search has a vocabulary problem. A field report says the aft door seal was weeping fluid. The ticket about the same fault says hydraulic leak, door 4. No word is shared, so keyword search treats them as unrelated records.

Vector search turns each record into a point in a space where distance stands for meaning. Records about the same fault land close together, whatever words their authors chose. The cost is explanation. A vector match is a distance, and a reviewer has to trust it or open the record. Hence the paper's split: use vectors where people describe a thing, and keep keyword where they name it.

Merging two lists

Running two indexes leaves the question of order. The paper's answer is a hybrid ranking: exact matches first, close ones after, permissions applied before anything is ranked. The merge is its own problem, and John Wang at Assembled describes why weighting raw scores failed and merging by rank held up. Score numbers from different indexes do not compare on their own, which is the subject of a separate paper, What a search score means.

Graph retrieval: how records relate

Hybrid search still treats records as separate documents. Many industrial questions are about the links between them. "Which A320 parts failed after the March notice?" needs the notice, the parts it covers and the tickets those parts caused.

The paper walks through that question step by step. Permissions come first, synced from each source system. Keyword search finds the exact terms, the aircraft type and the notice id. Vector search finds the close ones: failed, fault, removed. The hybrid ranking merges them. Then the graph follows part to notice to ticket, and the answer is a list of parts, each with its source record. A reviewer can open every line.

Figure 1. The paper's walk through one question, step by step. Permissions come first, then keyword and vector search, the hybrid ranking and the graph.

Agents that search, under the user's permissions

Ask an agent which suppliers are behind the delays on the A320 line. It does not search once. It finds the delayed orders, then the parts on those orders, then the supplier notices about those parts. Each step is a search.

The paper's argument is that each of those steps has to run as the person who asked. If one step ignores permissions, the answer can contain a record the user could never open. That is why permissions belong in the search layer and not in the agent. An agent that filters afterward has already read what it should not have seen. This does not make an agent immune to prompt injection. It bounds the damage: an agent that searches as its user can only reach what that user could already open, and every step leaves a trail a reviewer can open.

How we built it

The last chapter describes how Curiosity puts all six layers in one system on your infrastructure. Its customer story is Airbus. The Services Innovation Team in Airline Services had forty years of technical data across CRM systems, network drives and engineering databases, written in acronyms and reference numbers that ordinary search could not handle. In the team's words: "By intelligently enhancing our search efficiency, Curiosity lets Airbus technical support quickly find information across millions of documents." The change was one entry point over every source, with the engineering references extracted and joined in a graph, and filters by aircraft type, program and reference number.

The paper recommends three steps, in this order:

  1. Index every identifier field for exact match first.
  2. Add vectors where people describe a thing instead of naming it.
  3. Rank both together, then follow the graph.

Before you choose anything, it asks four questions. Which fields hold identifiers? Where do people write in their own words? Who may see which record, and in which source system is that set? Who needs to check why a result was returned? If you can answer those, you know which layers you need.

The 21 page paper also covers BM25 scoring term by term, the query path as a diagram, and a glossary. Read it on its page or download the PDF directly.

Read the whitepaper (PDF)

Further reading

Read next

Articles on context graphs, enterprise search and industrial AI

Connected knowledge for AI systems