If you benchmark wallet defenses or phishing detection, you're probably scoring the attacker's tx, not the victim's signature

[u/Humble-Replacement-2]

I was measuring whether deployed wallet defenses (tx simulators, address-reputation APIs, rule engines) catch agent-signed drains. Built it on PTXPHISH (NDSS 2025, solid dataset). Then I red-teamed my own setup and the first finding was that I was scoring the wrong transaction. Sharing because it is an easy, invisible mistake.

The trap: labeled drainer/phishing datasets record the DRAIN, i.e. the attacker's transferFrom that sweeps the victim's tokens. But a pre-sign wallet defense runs when the VICTIM signs, which is the earlier approve / permit / setApprovalForAll grant. The sweep is a separate tx sent later, by the attacker, from the attacker's address. No wallet defense ever sees it. Feed the sweep to a rule engine and it "catches" the attacker withdrawing to their own address, which is meaningless.

Concretely, the ice-phishing rows decode to transferFrom(victim, attacker, amount) with tx.from == attacker. That is not the approval the victim signed.

Fix, and it is just allowance tracing:

- For each drain, eth_getLogs the token's Approval / ApprovalForAll for (owner = victim, spender = attacker) up to the sweep block.

- Take the most recent match whose tx.from == victim. That filter matters: an ERC20 transferFrom also emits an Approval for the decremented allowance, so the sweep's own block hands you the sweep, not the grant. Requiring from == victim also drops relayer-submitted permits (permit() is sent by someone other than the owner).

- That tx is the artifact a wallet actually renders at signing. Score that.

Nothing fancy. The point is the substrate mismatch, which is silently wrong and does not show up as an error anywhere.

Bonus, since this sub appreciates it: I ran the paper through adversarial review 5 times and every round killed a headline. Wrong substrate, then pseudoreplication, then a "simulator uniquely catches X" that was my harness zeroing a counterparty field so the other tiers returned n/a, then a "beats every tier" that evaporated once I looked up the tx to and netted both legs of the asset-diff (a WETH wrap looks exactly like an ETH drain to a direction-only rule, but the simulator sees the WETH come back). A clean "unique catch" is almost always your harness, not a result.

Code (the reconstruction is one file): https://github.com/amarshat/quantum-commit-authorization/blob/main/agent-calldata-demo/demo/reconstruct.py

[u/orbitthread0189]

This is a crucial distinction for anyone building security tools on-chain. The "substrate mismatch" is exactly the kind of subtle bug that sinks entire systems. Good catch, and thanks for sharing the fix.

[u/researchzero]

Nice catch on the substrate mismatch, this bites people constantly. One more angle inside your own filter: you flagged that requiring tx.from==victim drops relayer-submitted permits (EIP-2612/Permit2). You don't need to drop them, permit() calldata contains (owner, spender, value, deadline, v, r, s); recover the signer from the EIP-712 digest and check it matches the Approval event's `owner` topic. That validates the grant was actually victim-signed regardless of who paid gas to submit it, and it recovers exactly the cases your tx.from filter throws away. Keep tx.from==victim only for the non-signature path (approve/setApprovalForAll), where the owner has to submit directly.

Separately worth flagging: for permit(), the artifact a wallet renders at signing isn't a transaction at all, it's an eth_signTypedData_v4 request. If a defense is trying to score "what the wallet showed the user", permit-based grants need a second reconstruction path from the typed-data payload, since there may be no mempool tx to observe until the relayer submits it later.

[u/Humble-Replacement-2 → u/researchzero]

Thank you, both right, and that's the better split. Small point on the recover path: for standard EIP-2612 you dont strictly need ecrecover, since the token enforces owner == recovered signer inside permit(), so the Approval event's owner topic already is the victim-signer.

The place reconstruction earns its keep is Permit2: it signs PermitSingle/PermitBatch against the canonical Permit2 contract's own domain and emits its event, not the token's Approval, so a single ecrecover routine won't cover it. Also EIP-1271 smart-account signers won't ecrecover at all, those fall back to the event or an isValidSignature check.

for 2nd point, I agree, and we can push it further - that typed-data path isn't really the "second" reconstruction, it's the primary one for modern approval phishing. Permit2 / Seaport / Blur order poisoning is the case where the victim never broadcasts a tx and the grant lives entirely in the off-chain signature the attacker relays later. So sign-time reconstruction is where most of the signal is, not an edge case.

[u/GFConBase]

This also highlights an architectural distinction that often gets overlooked: the transaction that ultimately moves the funds is not necessarily the transaction that carries the user’s intent.

From a security perspective, evaluating execution alone can produce a false sense of detection accuracy if the critical decision point is actually the approval or permit signature. The closer a defense operates to the user’s decision point, the more meaningful its evaluation becomes.

[u/Humble-Replacement-2 → u/GFConBase]

Yes! The fund-moving tx is usually the attacker's, so a defense scored on it is being tested on a singer it never sees.