OCR with three engines, and the 6 MB model that chooses between them

Rafael

• 2026-11-04

X-Ray, part 5 of 5. The last post of the series. Part 1 has the full map, and part 2 explains why the OCR pass in X-Ray.Content is optional and off by default.

A scanned contract, a photographed receipt and a page of handwriting are three different problems, and no single recogniser we tried was the best at all three. Fast engines read clean print in a fraction of a second and fail on a handwritten page. Vision-language models read the handwriting and take minutes per page. A reader trained on photographs finds words on a shop sign that the other two mostly miss.

An HN commenter put the state of things well in the thread under an OCR benchmark: OCR is "mostly solved for 'normal' text laid out according to Latin alphabet norms", and they had seen regressions "around faxed or scanned documents where the text boxes may be slightly rotated", not to mention handwriting (comment). The counterpoint is that frontier cloud models have become strong on hard handwriting: Mark Humphries reports a character error rate of 1.7% on five difficult historical documents from an unnamed Google model in testing, on a sample he calls not definitive (article). Our constraint is that pages are read where the data lives, on CPU, so the question for us is which local engine to use for which page.

The three engines

The NanoOCR timings are from its README on a shared four-vCPU Xeon at 2.10 GHz, best of seven warm runs, against the Python original at 281 to 290 ms on one thread. The PaddleOCR figures are from that port's notes on the same kind of machine, where it runs 1.9 to 2.0 times faster than the upstream pipeline on the same pages with identical Markdown.

A few things behind that table:

• NanoOCR reproduces Indra's pipeline stage by stage and matches the Python output on the test corpus.
• Florence2 is a wrapper we wrote around Microsoft's Florence-2 model. Besides OCR it does captioning, detection and phrase grounding.
• PaddleOCR-VL is a vision-language model, so it reads tables and forms as structure. It is also the slow one. We port it because the upstream pipeline needs Python and Paddle, and we kept every upstream file in the fork as the specification.

Thank you to Datalab (Indra OCR is their work, and the repository history begins with their code), to PaddlePaddle and to Microsoft's Florence-2 team.

The classifier

NanoOCR is fast enough to read every page first. So the router does: it reads the page, then classifies it, and keeps NanoOCR's reading when the page is printed text.

The classifier is a small MobileNetV3 network with four outputs: printed, table, handwritten and photo. It has 1.5 million parameters, is 6 MB, ships inside the NanoOCR package and classifies a page in 22 to 31 ms. Its input is the upright grayscale page, scaled to 512 pixels on the long side.

The training data is 3,800 pages, with about 12% held out for validation.

Training was 8 epochs on CPU, with classes weighted inversely to their frequency. The rendered pages were degraded to look like a fax, a photocopy or a noisy scan two times in three, and training adds a scanner or fax look to non-photo classes at random.

The rendered office pages came from a failure. Before they were added, the model called 98 of 200 ordinary memos, letters and e-mails tables, and scored 74.3% overall. Adding 1,000 rendered pages fixed that.

The evaluation set is 610 pages that the model never saw: 325 from RVL-CDIP's test split, 20 pages composed from IAM test lines, and 240 photos from the test splits of CORD and Total-Text and the validation split of COCO-Text. RVL-CDIP and IAM are used for scoring only, never for training.

Those are the model alone. The router adds NanoOCR's own reading as a check: if the model says "printed" but NanoOCR read the page with low confidence, the page is treated as unsure. Through the router, 83.1% of pages reach the engine for their class, against 42.5% when we classify with reading rules and no model. Pages left as "unsure" fall from 28.7% to 5.9%, all 240 photos are recognised as photos where the rules found none, and no handwritten page goes to NanoOCR where the rules sent 8 of 45.

Two caveats from the repository's notes. The 0.4 confidence threshold was tuned on the same 610 pages, so 83.1% is slightly optimistic (a threshold of 0.6 gave 80.0%). And the weak spot is old fax-quality printed pages: of 200, only 54% reach NanoOCR and 30% are taken for tables. Those pages go to the slow engine, which costs time rather than accuracy. Hand-filled forms are confused with handwriting.

Which engine gets which page

The mapping was decided by a benchmark on held-out pages, scored as the share of words found:

On clean print NanoOCR is as good as the others and by far the fastest. The sets are small, the metric is words found and not character accuracy, and we have not published the benchmark.

The router in the product is a few lines. This is the same shape as the example in the NanoOCR README, simplified:

using var nano = new NanoOCR(new NanoOcrOptions { ClassifyPages = true, KeepImages = true });

var router = new PageRouter<string>(nano)
    .When(PageCategory.PrintedText, page => page.Ocr.Markdown)       // NanoOCR's own reading
    .When(PageCategory.Photo,       page => Florence2Text(page))     // scene text
    .Otherwise(page => PaddleVlMarkdown(page));                      // tables, handwriting, unsure

Three rules sit around it in the product:

• NanoOCR's reading is always the fallback: Florence-2 and PaddleOCR-VL are downloads of about 3 GB together. Until they are installed, or when one fails or finds nothing on a page, the page keeps NanoOCR's reading. The desktop app never downloads them and runs on NanoOCR alone.
• The expensive engine is bounded: a page sent to PaddleOCR-VL can take a minute or two on a few cores, so one page is capped at five minutes, and the page limit for a PDF drops from 100 to 20 once the engine is installed.
• Every page says who read it: the log records the category, the reason for it and the engine.

X-Ray.Content has its own, separate OCR pass, which reads pictures inside documents (a screenshot on a slide, a scanned table pasted into a report) and puts the text back where the picture was. It uses PaddleOCR-VL only and does not route, which is a gap.

Limits

The classifier has four classes, and a page can be two of them, such as a typed form filled in by hand. Printed pages are 64% right on RVL-CDIP, a set that includes many old fax-quality scans. Prompt injection is untested: Simon Willison's wish for an OCR benchmark that measures whether a model "might OCR a page and then accidentally act on instructions in that content" applies to any vision-language reader, ours included.

Further reading

• Show HN: Benchmarking VLMs vs. Traditional OCR, with the two comments quoted above. The thread is where the rotated-fax and prompt-injection objections come up.
• A new Google model is nearly perfect on automated handwriting recognition, Mark Humphries. Five documents, a small sample, as he says himself. HN discussion.
• OCR and file extraction in the Curiosity docs, for the user-facing side.