A composite engagement: real patterns, real numbers from our audit work, details merged and anonymised so no client is identifiable.
OCR at 99% in the Demo, 71% in Reality
The situation
A 120-person building-supplies distributor processes roughly 3,000 supplier invoices a month. An OCR-plus-AI pilot promised to end manual keying. The vendor demo scored 99% accuracy. The accounts team ran it for a quarter, then went back to typing — and kept paying for it. It had been "still bedding in" for eleven months when we arrived.
What the Reality Check found
The demo was run on the vendor's clean sample PDFs. The distributor's real inbox was different: faxed invoices from two of its biggest suppliers, phone-photographed delivery notes, carbon-copy triplicates, and seventeen different invoice layouts from the top twenty suppliers alone. Real-world extraction accuracy: 71% — and, worse, the failures were silent. Wrong quantities went into the ERP looking exactly like right ones. The team's trust was gone, so they re-keyed everything, including the 71% the system got right. Cost of the stalled pilot: $1,900/month in licences plus the entire manual process it was meant to replace.
The verdict
Fix — but re-scoped. Not "automate all invoices". Automate the top 20 suppliers (82% of volume), and make every failure loud.
The fix (Recovery Sprint, 6 weeks)
Per-supplier extraction templates for the top 20 layouts. Three validation gates before anything touches the ERP: line totals must sum, PO numbers must match an open PO, unit prices must sit within an agreed tolerance of the price file. Anything that fails a gate lands in a review queue with the failed field highlighted — a 20-second human check instead of 4 minutes of keying. Weekly accuracy dashboard, per supplier, visible to the team that had lost faith.
The numbers
- —Straight-through processing on top-20 suppliers: 0% (nobody trusted it) → 96.5%
- —Keying time: ~210 hours/month → ~38 hours/month — roughly $8,200/month back at loaded cost
- —Silent errors reaching the ERP: unknown (that was the problem) → zero; every exception is now visible and logged
- —Payback on audit + sprint: inside 9 weeks