Reading a file safely: the rules every X-Ray package keeps

Rafael

• 2026-10-23

X-Ray, part 2 of 5. Part 1 listed the packages. This part is about the rules they share. Part 3 is the Office port.

A file extraction library has one input nobody controls, the file. A customer's share holds a PDF whose index was damaged by a bad save, a ZIP built to expand to a gigabyte, a spreadsheet with four hundred thousand rows, and a few thousand ordinary documents. The library has to read all of them, stop when told to, and never let one file take the process with it.

The X-Ray packages come from different upstreams and have different authors. They behave alike because we wrote the same few rules into the family and tested each package against them.

Saying and looking are different jobs

X-Ray.Content answers what a document says. X-Ray.Office answers what it looks like. They are separate packages because the costs are different. Text extraction needs a parser. Rendering needs fonts, shaping, line breaking, a layout engine and a rasteriser. The Office README states the design rule directly: extraction "must not pay for fonts, layout or a rasteriser".

// Says: text and Markdown, no fonts, no layout
ContentDocument content = OfficeDocument.Extract("report.docx");

// Looks like: layout is a separate, deferred step
using IDocument doc = OfficeDocument.Open("report.docx");
IPageSequence pages = ((IPaginatedDocument)doc).Layout();
using var pdf = File.Create("report.pdf");
new PdfRenderer().Render(doc, pages, pdf);

A format that both packages read, such as .docx, is read twice, independently, and the result is two different things. The documentation says so plainly in its package chooser.

One extraction, two shapes

Curiosity stores a document twice: as text for the indexes and as Markdown for the AI assistant, which reads headings, lists and tables better than flat text. Producing both naively means running the whole pipeline twice per file. That is wasteful for a parser and expensive when OCR is on, because the recogniser would run twice on the same page.

X-Ray.Content now returns both from one pass. PlainContent is the plain render that the pipeline produces on the way to every other format, so asking for Markdown costs nothing extra. RenderPagesInOutputFormat does the same per page:

var config = new ExtractionConfig
{
    OutputFormat = OutputFormat.Markdown,
    RenderPagesInOutputFormat = true,
};

var doc = new Extractor()
    .Extract(ExtractInput.FromUri("contract.pdf"), config)
    .Results[0];

string markdown = doc.Content;                       // headings, lists, pipe tables
string text     = doc.PlainContent;                  // what the index reads
string page1    = doc.Pages[0].FormattedContent;     // page 1 as Markdown

The per-page option exists because a page's plain Content is the concatenation of its element texts. A heading loses its hashes and a table, which has no text of its own, vanishes. That is what upstream produces and what the golden files pin, so it stays the default. To store a page number next to a table, the page has to be rendered by the same renderer the document used.

Hostile input

Every reader in the family follows three rules.

1. Reads are bounds-checked and recursion is bounded. A malformed length is an exception, not a read past the end.
2. Only the package's own exception types leave a load, so callers have one thing to catch.
3. Each package has truncation and mutation tests. They cut every fixture short and flip bytes in it, and fail if anything other than the package's own exceptions comes out.

For archives and expansion there are explicit ceilings, most of them in SecurityLimits. Those defaults are upstream's, value for value:

Real files also break the specification, which is the other half of hostile. Eliot Jones's So you want to parse a PDF? tests the claim on 3,977 PDFs from Common Crawl. Twenty-three of them, about half a percent, had a bad cross-reference declaration. Roughly half of those failures were junk bytes before the header, which shifts every offset. Viewers open these files because they recover, and a parser that only follows the specification rejects them.

Encrypted PDFs got the same treatment. The reader implements the standard security handler for RC4 and AES, tries the user password, the owner password and the empty password, and verifies the authentication instead of decrypting and hoping. A wrong password is a clear failure, where an unverified decryption returns garbage text and an extraction that looks like it worked. The tests write encrypted PDFs themselves, from the specification's algorithms, and also read files encrypted by qpdf.

Cancellation is part of the contract

Three rules, on every package that does real work, from the cancellation and streaming page: the cancellation token comes last, cancelling throws OperationCanceledException and nothing else, and large outputs go to a stream you provide.

The second rule needs the most care. A recovery path that catches everything swallows a cancellation and carries on with the document, so every best-effort path in the library lets a cancellation through.

We test it against inputs built to be slow: a 60,000-entry ZIP, a four-million-row CSV, a 6,000-page PDF, 1.5 million HTML paragraphs. Each is cancelled after 300 ms and must throw within five seconds. The test fails if the extraction finishes first, so a faster machine means a bigger input.

Some calls cannot be interrupted, such as decoding one large image, and the documentation says which.

X-Ray.Office adds limits for layout. LayoutOptions takes MaxPages, Timeout, MaxImagePixels and what to do with an oversized image. A layout that outruns its timeout throws LayoutTimeoutException, which derives from OperationCanceledException. One catch covers both, and the caller can still tell a slow document from a cancelled request.

Tests that can fail

A fix is proven by reverting it and watching the specific test fail. For one OCR test this found that the check for the disabled mode used an image document, so it passed with the mode gate removed, while a scanned PDF with OCR disabled would have been recognised. Another suite ran green for a whole session because about thirty tests began with if (path is null) return; and the fixture corpus was missing from the container. There is now a test that fails when the corpus is absent.

Further reading

• So you want to parse a PDF?, Eliot Jones. A catalogue of the ways real PDFs break the cross-reference rules, with counts from a random sample. It covers the file structure and leaves out text positioning. HN discussion.
• Cancellation and streaming across packages, the family's convention in full, with the list of calls that cannot be interrupted.