Where your data goes, step by step, inside Curiosity
Leon
Sooner or later, someone in your IT or security team asks the plain question. You connect the ticket tool, the file share and the maintenance logs. What happens to them after that? Where are they copied, who can see them, what reads the text, and what does the language model get?
We wrote the answer down. How Curiosity processes your data is a 13 page paper for business and IT professionals. It follows a record through seven steps, from the connector that loads it to the interface your users see. Each step links to its page in the documentation. Below is the paper's argument.
One idea: connect once, read the same result everywhere
Curiosity connects your data once and loads it into one graph, together with its permissions. It enriches the text, indexes it for keywords and for meaning, and serves the result to search, to AI agents and to your own applications. Each of them reads the same permission-aware data.
Steps 1 and 2: the data arrives with its access rights
A connector copies what a source system holds into Curiosity, and it copies who may see it too. It runs on a schedule. Each item keeps the ID it has in the source, so a rerun updates items instead of duplicating them. On Monday the first run copies every ticket and document. On Tuesday the next run adds only the tickets that changed. One sharp edge: data deleted in the source is not deleted in Curiosity automatically, so the connector has to remove it.
Permissions are written at the same moment as the data. The connector marks a record private, or limits it to a team or a person, as the source does. Users arrive with the groups your identity provider gives them, refreshed at each sign in. From there, every part of the system checks the same rules. In the graph, a record is visible only if the user has a route to it. Every search is filtered by the user's teams. Endpoints and AI run as the signed-in user.
Two lines in this chapter are for your security reviewer. A record with no restriction is visible to everyone, so mirror the source's rules rather than trusting the default. And the language model never decides who may read a record. It only receives what the user may already see. We wrote about why the check has to happen before the model reads anything in Permission-aware retrieval.
Step 3: text becomes linked objects
Enrichment finds the things a text mentions, such as part numbers and process references, and links each mention to the matching object in the graph. A support ticket mentions part 2741-08. Enrichment links that mention to the part, so the ticket shows up whenever someone looks at the part. A new part number found by a pattern can create its own object.
Built-in readers extract the text from files first, and scans and photos go through text recognition (OCR). Then you choose how to capture mentions, and the paper lists four ways, cheapest first:
| Approach | Best for | Speed | Cost |
|---|---|---|---|
| Word list | Fixed lists: products, customers | Instant | Negligible |
| Pattern | Codes with a fixed format: part numbers | Instant | Negligible |
| Trained model | People, organizations, places in prose | Fast | Computing power |
| Language model | Open-ended text, summaries | Seconds | Per use |
Its advice is to start with the simple, predictable methods and add the others only where gaps remain. Cost is one reason. Predictability is the other. A pattern for part numbers either matches or it does not. A language model asked to read a scanned page makes guesses that look right, and with part records a plausible wrong character is worse than an obvious miss.
Steps 4 and 5: findable by keyword, by meaning and by link
You decide which fields are findable, and how. A keyword index matches exact words, which is what you want for identifiers: searching 2741-08 finds records with that part number. A vector index matches meaning, so "hydraulic leak" also finds a report that says "fluid loss". Hybrid search switches on both and returns one list, with exact matches first and similar reports after. An adjustable balance sets the mix: more keyword for identifiers, more meaning for narrative text. Facets let users narrow by status, priority, a linked customer or a time range.
At query time there is a third route that a search index alone does not have: following links in the graph. From part 2741-08, find the service notices that mention it and the tickets that cite those notices. That answers questions about how records relate, which neither keyword nor meaning can do. Every route checks the user's permissions. What the resulting score means is the subject of the search score paper.
Step 6: the model asks, Curiosity answers as the user
AI orchestration is a loop. A person asks a question. Curiosity sends it, with your instructions and a list of tools, to the language model you chose: OpenAI, Anthropic, Azure or a local model. The model asks for a search or an action. Curiosity runs it as the signed-in user and returns only what that user may see. The model writes a reply that cites its sources, and each turn is logged for audit. A skill is a written guide to how your business works, a tool is a function that searches or acts, and an agent saves instructions, tools and a model as one reusable assistant.
Running retrieval as the user does not make an agent immune to a hostile ticket that carries injected text; Simon Willison's lethal trifecta talk is the clearest statement of that risk. What it does is bound the damage. The model only ever holds what that user could already open, and every turn leaves an audit record.
Step 7: your own logic on the same data
The last step is where teams build. Endpoints add your own logic, such as a supplier risk check. Interfaces put it in front of your users, built with Tesserae and Transpose. The paper ends the chapter with a short guide to which tool does which job:
- Load data from a system? Use a connector.
- Run something on a schedule? Use a scheduled task.
- Let chat trigger your logic? Use an AI tool.
- Add permission-aware logic or an action? Use an endpoint.
Sudo, the admin assistant in Curiosity Studio, works on this configuration. Ask for "a supplier risk check for the A320 line" and Sudo drafts and builds what it needs; you review the complete change, then approve or discard it.
Read the paper
If you have to sign off on connecting a system, start with chapters 01 and 02. If you have to make search good, start with 03 to 05.
Read the paper online or download the PDF directly. When you are ready, ask us to walk through the step that matters most to you on your own systems.
Further reading
- Zanzibar: Google's Consistent, Global Authorization System, Pang et al., USENIX ATC 2019. One data model for access control across Google's services, evaluated on every request, and how it holds up at scale.
- Better RAG results with Reciprocal Rank Fusion and Hybrid Search, John Wang, Assembled. Why vector search alone missed keyword queries, and how merging by rank fixed it.
- Why LLMs suck at OCR, Pulse. A vendor's catalog of how language models misread documents, with examples of plausible wrong output.
- My Lethal Trifecta talk at the Bay Area AI Security Meetup, Simon Willison. The data leak risk for agents with tools, and which defenses do not work.
Read next
Articles on context graphs, enterprise search and industrial AI