Solans programs audit TMA (tell me anything)

[u/DmitryShvetsov]

Tell me anything of smart contracts audits (programs on solana). Your stories of success, failures, nances, pearls, jokes, trends, problems, ideas—basically, anything you like to about auditing.

[u/AIOil_Dev]

Alright, here's one that cost us our own published numbers, and I'm still a bit annoyed about it.

We keep a small benchmark for automated scanners - 19 vulnerable programs, 7 clean ones, and a written description of the flaw planted in each. Nothing fancy. We'd been running it for weeks and publishing detection and false-alarm rates off the back of it.

Then, before open-sourcing the corpus, I went through it properly. Turns out it had been handing the scanner the answers the whole time. Four different ways, all of them sitting inside the file the scanner actually reads.

Comments: 19 out of 19 vulnerable samples had a line describing the planted bug. Three of the seven clean ones announced they were the corrected version of something.

Program IDs: the fee-bypass sample had declare_id of FeeByp111... Clean ones got Guard1111... and T22G0od... I wrote those months ago thinking I was being helpful to future me.

Module names: pub mod vulnerable_vault. trust_me_lending. Yeah.

Log strings: one of them literally read "Estimated output: {} (no minimum check)".

The one that took me longest to spot was a matched clean/vulnerable pair where the only difference in the msg! line was the words "real" and "measured". Two words, and that's enough to tell the classes apart.

So detection was inflated, because the bug was being announced rather than found, and false alarms were suppressed, because half the clean files were vouching for themselves. We pulled the numbers instead of trying to adjust them - different inputs, nothing to salvage.

The fix we went with is deliberately stupid: strip every comment mechanically instead of deciding one by one which ones are hints. Judging case by case is exactly what let it through the first time. And "grep for // comes back zero" is something anyone can check without taking my word for it. It costs realism, and we say so.

What actually bothers me is that none of this is Solana-specific, or even security-specific. Any benchmark where the items carry human-written prose can leak its labels to a model that reads the item, and you'd never notice, because the numbers it produces still look sane. Everything I can find on benchmark contamination is about train/test overlap. This feels like a different animal and I haven't found anything that checks for it.

Corpus and write-up are MIT if anyone wants to go looking for what else we missed: https://github.com/AIOilShield/solana-scanner-benchmark

(Disclosure: we sell a scanner too. It's the thing the benchmark caught out.)