The first automated read looked perfect. Every field that mattered was empty
An extraction that returns a complete-looking result with nothing important in it is worse than one that fails.
A Dutch sun-shading installer
Installers fill in measurement sheets by hand on a tablet. Getting those into an order meant retyping them. The first automated attempt produced output that looked complete and contained none of the measurements, colours or instructions.
The challenge
The sheets are photographs of installation sites with handwritten annotations on top - dimensions, colour codes, motor types, mounting instructions. Everything that determines what gets manufactured is in the handwriting.
The first attempt extracted images from the documents and produced a tidy, plausible-looking result. It contained every photograph and not one instruction.
Worse, the standard diagnostic agreed with it. Running a conventional text extractor over the documents returned zero characters, which looks like clear evidence that there is no text to find. It reinforced exactly the wrong conclusion, which is the most expensive kind of evidence.
What we did
Distrust the confident empty result
Zero characters extracted is not proof that there is nothing there. It is proof that this reader found nothing. The investigation started by treating the tidy result as a symptom.
Find the actual structure
The photograph and the handwriting are stored as separate layers with different encodings. Extracting only the photographic layer recovers every image and no annotation - which is precisely what had happened.
Composite, then read
Rendering the full page - all layers together, at higher resolution - produces what a human actually sees, and only then is there anything worth reading.
Make uncertainty a type, not a comment
Every reading is clear, uncertain or unreadable. The type system requires a note whenever a value is not clear, and the verbatim original text is always retained alongside the interpretation.
Test that blanks stay blank
The specific failure to guard against is a blank field quietly inheriting a neighbour's value. Tests assert that uncertainty survives the whole pipeline rather than being smoothed away.
How it works
Two layers, one page
One sheet contained twenty-two images, of which eight were photographs. The remaining layers carried the annotations - the part that determines the order.
The misleading diagnostic
A conventional text extractor returned nothing, which read as confirmation that the documents held no text. It was confirming its own limitation.
Render, do not extract
Compositing the page at higher resolution recovered complete situations: dimensions, colour codes, motor type, mounting instructions and a full specification.
Read what a person sees
The correct input to the model is the rendered page, not the document's internal object graph.
Three states, not two
Clear, uncertain, unreadable. Collapsing this to a value-or-null loses the distinction between 'nothing was written' and 'something was written that I could not read'.
Enforced by the compiler
A reading that is not clear cannot be constructed without a note. The guarantee is structural rather than a convention someone has to remember.
The original is kept
The raw text sits alongside every interpretation, so a human reviewer can always check the machine's reading against what was actually written.
No helpful guessing
Thirteen tests assert that a blank field stays blank rather than inheriting a plausible value from a sibling. Helpfulness here is indistinguishable from fabrication.
Technologies
The result
After compositing, a single sheet yielded six complete situations with measurements, colour codes, motor type, mounting instruction and a full specification - from documents a first pass had classified as containing no readable text.
The more durable outcome is the uncertainty model. Thirteen tests exist purely to assert that the system does not fill in gaps. An extraction pipeline that guesses is not a time-saver; it is a source of manufacturing errors with a very convincing user interface.
The general lesson transfers well beyond measurement sheets: when an automated read looks complete, the first question is what it would have looked like if it had failed.
Measurement sheets contain customer names, addresses and site photographs. No sheet, fixture or extract from one is reproduced here.
Related work
Three percent looks completely normal
A supplier's price list was three percent too high. That is exactly the kind of error nobody catches.
Before the invoice button went live, we built the thing that stops it
The system was about to start emailing customers. Nothing prevented it from emailing all of them.