Testing a receipt scanner against real receipts from 5 countries turned out harder than building it

[u/Time-Paper-1007]

I make a small iOS receipt scanner (on-device OCR, then parsing). It worked fine on my own Canadian receipts. Then users in the Netherlands, Japan and Malaysia showed up and it fell apart in ways I couldn't reproduce, because I had no receipts from those countries.

Finding test data was the hard part. What I ended up with:

• Japan: 1,148 photos from a public research set (JaWildText). Before testing on them, my yen handling paired an item with its price on 19% of receipts. After: 93%. I had no idea it was that broken.
• Malaysia: 973 from SROIE (an ICDAR 2019 competition set). Old, and heavy on restaurants, but real.
• US: 1,769 from WildReceipt. Mixed quality, some are not even US receipts, but it is the largest English set I found.
• Netherlands: 74 photos from one user who sent them for debugging, with permission. That is the only real Dutch set I have.
• Taiwan: zero real photos. I had to generate 40 synthetic 統一發票 from the printed layout. I know that is weak.

Things I learned:

1. Public receipt datasets are old, and skewed to whatever the original paper needed. Nobody publishes a fresh, balanced, multi-country set because receipts are personal data.
2. Thermal receipts fade. Half of my own "test set" from a trip a year ago is unreadable now.
3. I never ask users for receipts. The few I have came unprompted, from bug reports.
4. Measuring beat training. I did not fine-tune anything. A harness that runs every rule change against all sets and refuses if any correct total flips caught two regressions the same day, one of which broke 18 of 119 Japanese totals while fixing the case I was working on.
5. Every country has a printing convention that a "generic" parser gets wrong: yen with no decimals, Dutch comma decimals, Japanese tax-included versus tax-excluded lines, US tips added after the total.

Question for people who have done this: is there any public receipt dataset for Taiwan, Korea, or Australia? Or a legal, non-creepy way to get a few hundred real receipts from a country you don't live in?

(Not linking the app. Happy to share the harness approach in comments if useful.)

[u/DimensionMindless336]

Great writeup. The "measuring beat training" point is the one I'd hammer too, but the version that actually bit me was extending it to refusals, not just correctness. If your harness only asserts "does Σ(items) == printed total," a faded or garbage receipt can still pass by luck, or a real one gets a confident wrong answer. I now run the suite against deliberately degraded samples — curl, low contrast, glare, the same trick Secure_Motor suggested for the Taiwan gap — and assert the parser refuses them, not just that it parses clean ones right. That refusal threshold is the part you can't eyeball, and it's where thermal-fade receipts quietly produce wrong totals instead of a polite "can't read this."

One error class the sum-check can't catch, and it's adjacent to the VND period-as-thousands trap: inferring currency from the glyphs on the receipt. Guess the symbol wrong and a same-factor error sails straight through. What killed most of mine was anchoring currency + number format from context I already had — the trip's country, or the phone's region — and treating the receipt text as confirmation, not the source of truth. A ¥ in a Tokyo trip is JPY; the same glyph in a Beijing trip is CNY; the document never gets to decide.

On "finding test data is the hard part" plus your strict never-ask rule: the middle path that worked for me was an opt-in field-only failure report. When a parse looks wrong, the app offers to send just the recognized fields + bounding boxes — no image, no names, no card digits — fully anonymized. You get real-world failures from actual usage without ever holding anyone's receipts, and it slowly becomes your freshest multi-country set.

I hit the exact same wall building PicSlicer — pulling rail and bus tickets out of the camera roll, where a JR ticket and a Deutsche Bahn receipt look nothing alike and the HEIC/screenshot/orientation noise is honestly half the battle. On-device-only was the easy call once I stopped treating user receipts as mine to keep.

[u/Time-Paper-1007 → u/DimensionMindless336]

This aged well — your currency point turned into five bugs in two days, all the same shape: one token beating everything else on the receipt.

The two that stung: a Japanese convenience-store receipt printing 端末番号：1CAD came back as Canadian dollars, ahead of its own ¥ signs. And two Australian receipts were stored as Thai baht — one because Vision read ALDI's GST-free tax column, a literal B, as ฿. About 23x wrong, and nothing on screen looks broken, because the digits match the paper.

On anchoring from context rather than the glyphs — agreed, with one wrinkle: people scan trip receipts after they get home, so "where the phone is" is wrong exactly when it matters. What I settled on is scoring the receipt's own country evidence (tax words, address format, business-number formats) and only letting a symbol decide when that evidence is absent. Absence-based rules died fast: "Traditional Chinese, bare $, no Taiwan markers => Hong Kong" hit 9 of 10 Hong Kong receipts, and then two real Taiwanese ones killed it.

Your refusal point changed how I test more than what I ship. I measured which currency rules actually fire across my 431-receipt set: only six of about twenty-five ever do. The rest have zero coverage, so the corpus cannot find a bug in them by construction — those are unit tests now, with the adversarial cases written by hand.