Porting a Rust document-extraction engine to C#, and ending up faster

Rafael

• 2026-09-07

Every system that indexes documents needs the same unglamorous component: something that takes an arbitrary file (a PDF, a .docx, a 1997 .doc, an email with attachments, a spreadsheet, a ZIP containing all of the above) and returns text, tables, metadata and structure. Curiosity needs one for every connector we ship.

For years we did that with a mix of commercial libraries and open-source ones. Each covered part of the format matrix, none covered all of it, and the seams were where our bugs lived. Licensing terms had to be checked against every deployment shape. When a customer sent us a file that extracted badly, the first question was which library had read it, and the answer determined whether we could do anything about it. It worked well enough to ship, but we kept watching the space for something that covered the whole matrix in one place.

The project we ended up taking was xberg. We first noticed it when it was still called Kreuzberg, after the Berlin district, which Germans routinely abbreviate to X-Berg, and that abbreviation is where the current name comes from. We followed it for a long time before doing anything with it: the format coverage was unusually broad, the architecture looked like something you could reason about, and it kept getting better.

It is also Rust, and C# is our favourite language. There is already an official .NET binding (XbergIo.Xberg on NuGet, with the source in the same repository), so the glue was not the problem. That binding reaches the Rust through P/Invoke, and P/Invoke was the part we wanted to avoid. We ported the engine instead: every extractor, every renderer, and most of the dependency stack underneath them. The result is X-Ray.Content: 48 extractors, entirely managed, no P/Invoke and no native binaries. We renamed it to X-Ray rather than keep any variation on xberg, so that nobody mistakes the port for the original. They are separate projects with separate bugs, and a shared name would have made every issue report ambiguous.

On the same corpus and the same machine it now extracts faster than the Rust it came from, which was not the goal and took some explaining.

Why not a binding

For a lot of libraries a binding is the right call, and we have shipped plenty of them. Since xberg already publishes one, choosing to port instead needs a reason, and ours are specific to how we deploy rather than a general objection to interop.

A native dependency has to be built for every runtime identifier you target. Curiosity runs on Windows and Linux, x64 and arm64, in containers, on developer laptops and inside a desktop app. Each native artifact multiplies that matrix, and each one is something that can fail to load on a customer's machine, where we cannot attach a debugger.

There is also the data path. Extraction is not a small-payload API: it moves megabytes per call and hands back deep object graphs of elements, tables, per-page content and images. Getting that across an FFI boundary means copying it wholesale or writing careful unsafe interop and then owning the lifetime bugs that come with it.

There is a security argument too, and it is the one we would make first to anyone running this on untrusted input — which document extraction always is. Every native dependency is memory-unsafe code parsing hostile bytes, and a malformed PDF or a hand-crafted ZIP is exactly the shape of input that finds buffer overruns. A managed parser gets a bounds check and an exception where a native one gets undefined behaviour. We keep native dependencies to a minimum for that reason, and C# being type- and memory-safe is a large part of why the extraction path is safe to point at a customer's file share.

The thing that decided it, though, was debuggability. When a table comes out of a PDF with the column boundaries subtly wrong, we want to set a breakpoint in the function that chose them. Through a binding, the best you can do is file an issue and wait.

What the engine looks like

The architecture is worth a paragraph first, because it is the reason the port was tractable at all.

Every extractor produces the same intermediate document, and the output format is applied afterwards by a renderer. A .docx extractor and a PDF extractor share nothing except the shape they fill in: a flat list of elements in reading order, plus tables, images and metadata. Nothing in an extractor knows what Markdown is.

For porting, that seam is everything. The 48 extractors are independent of each other and of the output side, so they can be done one at a time, in any order, and each one can be checked on its own. Without it this would have been a rewrite, and we would probably have abandoned it.

The oracle mattered more than the model

If there is one thing to take from this, it is that we never asked an agent to translate Rust by reading it and calling the result done. Every step was checked against a machine-generated answer.

xberg ships a fixture corpus of 2,498 files, from ordinary Word documents to deliberately malformed archives. We built two things around it:

• xberg-reference-gen, a Rust binary that links the original crate, walks the corpus, and writes {filename}-results-rust.json beside every fixture: plain text, Markdown, HTML and the JSON tree, plus metadata and tables.
• XRay.Content.TestRunner, a C# CLI that runs the port over those same fixtures and diffs against the generated files, per format.

That turns "port this extractor" into a number that either moves or does not, which is what makes the work suitable for an agent in the first place. It also means an upstream sync is mechanical: regenerate the goldens against the new Rust, and the diff is the list of changes still to port. We do not commit the goldens for that reason: they describe upstream at one revision and nothing else.

One flag on the test runner did more for our throughput than any prompt engineering. --cluster groups plain-text mismatches by the text at their first divergence, which turns "410 Markdown fixtures fail" into "395 of them diverge at the same smart-quote character". A failure count tells you nothing about what to do next. A failure cluster usually names the rule you got wrong.

What "porting a Rust crate" actually meant

We underestimated this by roughly an order of magnitude. The rule we settled on was that if a Rust dependency had no faithful managed equivalent, we ported the dependency too, and that turned out to be most of them.

The last row still looks unreasonable written down. Layout detection runs five ONNX graphs, and the Rust build reaches ONNX Runtime through ort, which links native code — exactly what we were trying to avoid. So the port carries its own ONNX runtime: a protobuf wire reader, a graph executor with liveness-based buffer release, and the 80 operators those five graphs need, vectorised through System.Numerics.Tensors with convolution lowered to GEMM.

It is checked layer by layer against real ONNX Runtime. Every operator instance matches in isolation, and on the document-layout model every page agrees in detection count and class, with a worst confidence delta of 2.5e-3 and a worst box delta of 0.17 px. It currently runs about 1.8x slower than ONNX Runtime, down from 16x when it first worked. That is the price of not shipping a native dependency, and we think it is worth paying.

BLAKE3 is the opposite kind of problem: small, and completely unforgiving. Element IDs are the first six bytes of a BLAKE3 hash over the element's discriminant, text, page and index. If the hash is off by a bit, every structured-output golden fails for reasons unrelated to extraction. It is 283 lines and it had to be right before anything else could be measured.

Where this method works, and where it doesn't

Three failure modes came up often enough to be worth naming, and they share a cause: the model reasoning about code instead of running it.

A test written from reading the source is a guess. One of our tests asserted that upstream emits no table for a particular right-to-left PDF form. The reasoning was careful: word merging fuses each label with its value, three of four columns end up empty, the validity check rejects it. It had been written while the fixture corpus was missing from the container, so it had never actually executed. When we fetched the corpus it failed on its first run: upstream emits exactly one table for that file, and our port had been producing that table all along, same page, same bounding box, same Markdown. Building the reference generator takes about six minutes and would have answered the question outright.

A rule derived from a minimal reduction can be backwards. A nested list inside a list item was recording more text in our port than upstream. Reduced to minified markup the fix looked obvious, and it matched the reduction exactly. Then the corpus dropped: HTML matches from 74 to 69, plain text from 97 to 93. Written the way an actual document is written, with the line breaks and indentation an author leaves in, upstream keeps the nested Markdown, which is what we were already doing. Every fixture in the corpus is formatted that way. The reduction was the outlier and we reverted the change.

A structural diff between two trees mostly measures naming. An audit of the PDF table detector reported that we had 23 of the Rust's 73 functions, which sounded alarming. Taking the call closure from the entry point xberg actually uses gives 41 reachable functions; the rest belong to a configuration xberg never selects. Of the ones still reported missing, a naive snake_case-to-PascalCase match had got twelve wrong, eleven of which existed under different names.

What the method is good at is the inverse of all three. Given a corpus, a diff tool and a way to cluster failures, an agent will work through an enormous amount of mechanical translation without losing interest, and that is most of this problem: 48 extractors, 80 ONNX operators, five model wrappers, a MIME table. Very little of it is deep. Almost all of it is volume.

Matching a bug on purpose

Parity against a golden file means reproducing upstream exactly, and upstream is sometimes wrong. We needed a policy, and it ended up being: match upstream unless it is demonstrably defective, and write down every divergence next to the line that causes it.

Some are easy calls. A dBASE reader builds its column headers in declared order and then fills each row from a hash map's iteration order, so values land under the wrong headers, and differently on every row. The golden file faithfully records the scrambled output. We emit declared order, and the fixture is marked as an expected divergence.

Others cost us before we understood them. Two places in the PDF engine's path handling drop operators they should keep, and our port had quietly fixed both while translating them. Each fix cost us matching fixtures until we stopped being right and started being faithful, which is a strange note to leave in a commit message.

Excel number formats went the other way, and there we chose to diverge. Upstream prints a cell's raw f64, so a cell displaying 3.2% arrives as 0.032280358222708555 and a date arrives as a serial number. We render the cell through its own format code, so the extracted text reads the way the spreadsheet looks. It is a changed default rather than a fix, and one flag puts it back for anyone comparing against the Rust goldens.

Past parity: a strikethrough is not a ruling line

Once a format matches, the goldens stop being useful. They can only confirm that you agree with upstream, including where upstream is weak. The interesting work starts after that.

A ten-page landscape report with four ruled tables (orange header rows, grey shaded rows, tracked-changes strikethroughs, vertically merged cells, wrapped text in every description column) came out with three of its four tables missing and the fourth broken into nine fragments, one of which had swallowed the header row. Both implementations do this. It is not a porting defect, it is what the current state of the art does to a document that a person would find unremarkable.

Six causes, of which two are worth repeating because the fixes are geometric rather than heuristic.

Strikethroughs were being read as table rules. A tracked-changes document strikes each deleted cell through, and those strokes reach the edge list looking like any other line. Where one overruns a column rule it raises an intersection, the grid gains a row boundary that should not exist, the rows it invents come out empty, and the emptiness filter then discards the whole table. That one cause accounted for six of the nine fragments.

The discriminator needs no notion of what a table looks like: a ruling line does not pass through glyphs. A stroke at least half of whose length runs over words, at a height inside those words' own boxes rather than at their edge, is decoration. On page 6 of that document the strikethroughs cover 0.90–1.00 of their length at relative heights of 0.25–0.35, while real row rules cover at most 0.45, at 0.96 and above. Underlines are deliberately exempt: an underline sits exactly where a rule sits, and dropping a real rule costs more than keeping a stray underline.

Wrapped cells were being read as several rows. The pass that splits bands into rows exists for grids whose row rules were never drawn, and it was splitting on any band with text on more than one baseline, which is what a description column wrapping to three lines looks like. The guard we added is that at least two of a band's baselines must span two or more columns. A wrapped cell only has one, because its extra lines all belong to a single column.

Each of the six fixes has a unit test, and each test was mutation-proved: revert the fix, watch that specific test fail. With generated code that matters more than usual, because a test written in the same session as the change it covers will happily pass for the wrong reason.

The shape of the work

Nobody on the team hand-wrote an extractor. Humans reviewed and merged (the merge commits are ours), but every commit under dotnet/ was produced by an agent working against the corpus.

The eight days are not eight days of steady work; they are a small number of long sessions with review in between. The two largest, at 34 and 32 commits, were an upstream sync and the layout-detection phase.

The first two rows are the part we would tell anyone estimating a port like this. The extractors, the part that sounds like the whole project, are 13% of the code. The rest is the dependency stack underneath, which is invisible until you start.

What it cost

Ports done with AI agents have had some attention lately, most visibly Bun's rewrite of its codebase in Rust, and the discussion of what that took has been mostly about the bill. Ours cost far less: no four-figure token spend, and nobody watching a run overnight.

We think the target language explains much of that, though we cannot prove it. A typed language answers in seconds. Most of the mistakes an agent makes while translating (a field that does not exist, a missing enum case, two int arguments swapped) are compiler errors rather than test failures. The agent gets a file and a line immediately and fixes it in the same turn, instead of working backwards from an assertion that failed minutes later. Cheap iterations are what keep the bill small.

The ecosystem matters for the same reason, and bob1029 put it better than we would:

The need to select an appropriate 3rd party library represents an entire dimension of the search space that can be eliminated. Imagine having to make this choice multiple times per day when your competition is just mindlessly using System.* types.

That is our experience exactly. Almost nothing in this port needed a package decision, because the BCL already had the thing: ZipArchive for every OOXML and ODF container, XmlReader for their parts, System.Text.Json with custom converters for serde's tagged enums, System.Numerics.Tensors for the ONNX kernels. Each of those is a question that never had to be asked, and a question not asked is tokens not spent.

Using it

using XRay.Content.Core;

var extractor = new Extractor();

var result = extractor.Extract(
    ExtractInput.FromUri("quarterly-report.pdf"),
    new ExtractionConfig { OutputFormat = OutputFormat.Markdown });

var doc = result.Results[0];

Console.WriteLine(doc.MimeType);        // application/pdf
Console.WriteLine(doc.Content);         // the rendered Markdown
Console.WriteLine(doc.Metadata.Title);
Console.WriteLine(doc.Tables.Count);
Console.WriteLine(doc.Pages.Count);

Extract does not throw on a bad document. Failures land in result.Errors with a type and a code, and an unsupported MIME type comes back as an empty document carrying a ProcessingWarning. That distinction matters when you are pointed at a customer's file share and a fraction of a percent of it is corrupt.

The benchmark

Both implementations were measured in a single run, on the same idle machine, over the same 2,498-file corpus, with plain-text output.

• Protocol: each side extracts the whole corpus twice as warm-up, then three timed passes. The figure reported per file is the best of the three. Warm-up matters on both sides: .NET needs it to get the extractor graph out of the JIT's quick tier, and Rust needs it because the PDF engine keeps a process-global font cache that would otherwise charge the first documents for work the later ones get free.
• What is compared: only files both sides extracted successfully. A file one side refuses and the other parses is a correctness difference, and folding it into a speed ratio would reward whichever implementation did less.
• Builds: Rust --release with opt-level = 3, lto = "thin", codegen-units = 1. C# Release, server GC, tiered compilation left at its shipping default.
• Machine: 4-core Intel Xeon at 2.10 GHz, 15 GiB RAM, .NET 10.0.112, rustc 1.94.1.

Of the 2,498 files, 2,414 were extracted by both and are what the numbers below describe. The C# port extracted 64 files the Rust did not; the Rust extracted 4 the port did not; 16 defeated both.

One asymmetry is deliberate: the C# side ran with Excel number formatting on, which is its default and is strictly more work than the Rust does. Turning it off for the benchmark would have been tuning the port to its own test.

Overall

Per file, the speed-up has a median of 2.93x and a geometric mean of 3.22x; the middle 80% of files land between 0.71x and 15.9x. The port is faster on 2,022 of 2,414 files, or 83.8%.

Per format

The headline number is a PDF number. PDFs are 309 of the files and 397 of the 498 MiB, and they account for 191.7 s of the Rust total of 193.9 s, or 98.9% of it. Everything else is rounding on the total, which is why the per-format view is the more useful one.

Both ratio columns are C# speed-ups: above 1 the port is faster, below 1 it is slower.

Where the two columns disagree (html, epub, md, json), the total is being set by a few large files while the median file is much faster in C#. HTML is the clearest case: a median speed-up of 2.96x against a total of 0.71x means most HTML documents extract about three times faster here and a handful of big ones drag the total the other way. Those are worth chasing and we have not yet.

json is the only format that is slower on both measures, and it is also the format where both implementations are doing almost nothing per file.

Is it doing the same work?

A speed number is worthless without a correctness number next to it, and "faster" is the easiest thing in the world to achieve by quietly extracting less.

The benchmark itself answers part of this. Only files both implementations extracted are counted, so nothing is gained by refusing a document. Of the 84 files that did not qualify, 64 were extracted by the port and not by the Rust, 4 the other way, and 16 by neither.

The broader answer comes from the golden corpus rather than the benchmark. Every fixture is compared on five dimensions (plain text, Markdown, HTML, the JSON tree, metadata and tables), and the port is considered done for a format when its fixtures match. The last full-corpus run we have recorded had 2,926 of 3,007 comparable fixtures matching on every dimension, with the remaining 81 concentrated in HTML (71) and PDF (8); every other format in the corpus was at full parity. That figure was measured against goldens generated on 2026-08-24, and upstream has had two syncs merged since, so it is the last audited number rather than today's.

The two documents that dominate the corpus timing are also the ones where "doing less" would be easiest to hide, and they go the other way: on the 50 MiB Intel manual the port produces 12,335,597 characters against the Rust's 12,269,452, and on a 34 MiB statistics textbook 2,156,892 against 2,128,319. Slightly more text, in roughly half the time.

Why it is faster, as far as we can tell

We did not set out to beat the Rust and we were not expecting to, so the first thing we did with this result was try to find the mistake in it. A few things are worth separating.

Most of the per-file ratios are measuring call overhead, not parsing. The formats at the top of that chart (.opml at 68x, .typ at 17x, .doc at 8x) are files where both implementations finish in well under a millisecond. What is being compared there is the fixed cost of one extraction call. The Rust entry point is async and the benchmark drives it through a task per file; the C# entry point is a synchronous method call. That difference is real if you are extracting millions of small files, but it is not a claim about either parser, and we would not present it as one.

The number that carries weight is PDF, because PDF is 98.9% of the corpus time. There the port is 1.52x faster on the total and 1.80x on the median file, over 309 files and 397 MiB, and the gap holds at the top end: the largest fixture in the corpus, a 50 MiB Intel architecture manual, takes 56.6 s in Rust and 30.7 s here.

We do not have a profile that decomposes that gap, and we are not going to invent one. What we can point at is a structural difference. The Rust PDF engine is about 275k lines; our port of it is about 36k. That is not because we dropped work from the extraction path; it is because the parts we did not port are unreachable from text extraction. Roughly 81k lines are CID mapping tables for CJK encodings, a further 27k are a page renderer we have no equivalent of, and a large fraction of the table detector belongs to a strategy xberg never selects. Less machinery gets carried through each page.

One hypothesis we have not confirmed: PDF text assembly does an unusual amount of character-level work (measuring gaps between glyph runs, deciding where spaces belong, repairing ligatures and hyphens), and .NET's UTF-16 strings make random indexing into that text a constant-time operation where UTF-8 does not. That would fit the shape of the result, but we have not instrumented it, so treat it as a guess.

Where the port is slower, the totals are misleading and the medians are the honest number. .html is 0.71x on the total but 2.96x on the median file, which means most HTML documents extract about three times faster here and a handful of large ones do not. Same story for .epub (0.56x total, 1.79x median) and .md (0.86x total, 3.69x median). Those tails are real work we have not done. .json is the one format that is slower on both measures, and also one where both implementations are doing very little per file.

OCR, and another port for the same reason

Upstream reaches OCR through Tesseract, or through candle-hosted vision-language models via ort. Both are native, so neither was available to us on the terms we had set. We had the same decision to make as with the engine itself, and we made it the same way.

The recognizer is PaddleOCR-VL-1.6, and what we actually ship is our own C# port of it, published as PaddleOCR on NuGet: the SigLIP vision tower, the ERNIE-4.5 decoder, the BPE tokenizer, the Paddle graph interpreter and the image pipeline, all in managed C#. No ONNX Runtime, no Paddle Inference, no libtorch.

It is not quite free of native code, and we would rather say so than round it off. Image decoding goes through SkiaSharp, and rasterising a PDF page goes through PDFium in the companion PaddleOCR.Pdf package. PDFium is unavoidable for scanned PDFs specifically: our PDF reader extracts text and geometry but does not render, and the text on a scanned page exists only as pixels. That is why OCR is opt-in and off by default — a consumer who never sets it never loads either library, and the extraction path stays purely managed.

Three modes, with Disabled the default:

The pass runs on the element stream, between extraction and the derive step, rather than over rendered text. That ordering is what lets recognised text be inserted directly after the image element it came from, so every renderer places it inline without knowing OCR exists. A pass over rendered output could only append to the end.

Recognised tables get the same treatment. PaddleOCR-VL reads a table region as OTSL and its pipeline turns that into an HTML <table>, which would be wrong in a different way in every output format: raw HTML where Markdown wants a pipe table, angle brackets in plain text. So the recognition is split into text runs and tables before any of it becomes an element, and each table is pushed onto the document's table collection behind an ordinary table element. After that nothing about it is OCR-specific.

Two rules the implementation holds to: the pass is additive, so native text is never replaced and a page that already has text keeps it; and it is never fatal, so a missing checkpoint, an undecodable image or a recognition that outruns its timeout becomes a processing warning and the document still comes back. Nothing downloads on its own either — a missing checkpoint raises, rather than pulling two gigabytes of weights as a side effect of an extraction call.

What we did not port

The port is deliberately narrower than the crate it came from. Content extraction is the whole of it. Audio and video transcription, embeddings, reranking, NER, keyword extraction, chunking for RAG, LLM-driven structured extraction and the server mode are all out of scope, and where a Rust code path branches into one of them the port takes the native-extraction branch and drops the other. Code intelligence, the tree-sitter grammars for 300-odd languages, is on the list but at the bottom of it.

There is one gap inside the PDF work worth naming, because it constrains a feature rather than removing it. Layout detection needs a rendered page bitmap, and our PDF reader does not render. Upstream reaches a renderer through the same crate it parses with (about 21k lines of rendering plus a software rasteriser plus font outlines), and since the model's output depends on the pixels, an approximate renderer would not produce matching detections. So the layout models are reachable only from images a caller supplies, not from a PDF directly. The OCR path sidesteps this by rasterising through PDFium, which is exactly the native dependency the rest of the library avoids, and is why that feature is opt-in.

Licensing

X-Ray.Content is MIT, with one qualification we would rather state than bury. Three of the Rust crates it derives from are Apache-2.0 only, so the files translated from them remain under Apache-2.0 and the package declares MIT AND Apache-2.0. Each derived file carries a notice naming its source and stating that it was modified, and the full text ships in the repository. Every other crate the port derives from is MIT or dual-licensed where MIT can be taken.

Where to find it

dotnet add package X-Ray.Content

• X-Ray.Content — the extraction package; the API lives under the XRay.Content namespace, since a hyphen is not valid in a C# identifier. Targets net10.0.
• X-Ray — an umbrella package with no code, which pulls in every member of the family. Take X-Ray.Content directly if it is all you need.
• PaddleOCR — the OCR port, usable on its own.
• xberg — the Rust original, and XbergIo.Xberg if a P/Invoke binding suits you better than a port. For a lot of projects it will.

If you try it on documents that matter to you and it reads one of them badly, that is the useful bug report. The corpus is broad but it is still somebody else's idea of what documents look like.