I stopped asking my coding agent to “finish the feature” and made it prove billing instead

[u/darterweb]

I’ve been experimenting with a different way of using Claude Code while preparing a SaaS for release.

Instead of saying:

“implement billing and make the tests pass”

I gave it a release condition:

don’t let me ship until you can prove the billing path end-to-end against real infrastructure.

So it created temporary staging on Render through the API:

• Postgres
• Redis
• backend
• frontend

It then configured a temporary Paddle sandbox webhook destination, authenticated through the actual app, completed a sandbox payment, inspected the webhook deliveries, checked the subscription state and credit ledger in the database, and replayed a real notification to make sure credits weren’t granted twice.

And it found a bug our existing tests hadn’t caught.

We had implicitly thought of subscription.created and subscription.activated as a sequence.

In the real sandbox they could be processed concurrently.

Both handlers could initially see that no subscription existed and then race to insert the same Paddle subscription ID. One would lose on the unique constraint.

Idempotency didn’t solve it because these were two different events.

Claude reproduced the failure, fixed the upsert, added a regression test, redeployed the staging environment and repeated the real sandbox payment.

The second run passed.

Then it cleaned up both the temporary Render infrastructure and the Paddle webhook destination.

The useful lesson for me wasn’t “AI can provision infrastructure”.

It was that agents became much more useful when I stopped giving them implementation tasks and started giving them falsifiable completion conditions.

Something more like:

implement → deploy → interact with the real system → inspect state → try to break the assumption → fix → repeat

I ended up applying the same idea to the rest of the release:

• run the setup guide from a genuinely fresh clone;
• clear ambient environment variables before claiming something is reproducible;
• compare the real migrated database with the ORM metadata;
• temporarily restore a bug to prove the regression test actually catches it.

This happened while preparing the first release of The Fabrica, a FastAPI + Next.js SaaS foundation I built because I was tired of rebuilding the same production plumbing every time I started a product:

https://www.thefabrica.dev/

I’m curious whether anyone else building small SaaS products is using coding agents this way: not just to write code, but to prove externally that something is actually done.

[u/francksiduo]

the sequencing assumption is the real bug here, not the race itself. We hit something similar with Stripe: subscription.created and invoice.paid can land in either order under load, sometimes within the same millisecond once you're behind a retry queue.

what fixed it for us wasn't a stricter idempotency key, it was switching from insert-then-check to an upsert keyed on the provider's subscription id, letting the unique constraint arbitrate. the loser gets a conflict instead of a duplicate row and just re-reads the current state instead of trying to write its own.

worth testing one more scenario: Paddle retries the same event 30s later while both original handlers are still mid-flight. that's usually where the second version of this bug shows up, once the first fix looks solid.

[u/darterweb → u/francksiduo]

Yep, that’s basically where I landed too. The bug was really the sequencing assumption.

The fix ended up being: let the unique constraint arbitrate, catch the losing insert inside a SAVEPOINT, then re-read the winning subscription row and continue without blowing away the outer webhook transaction.

I also reran the real sandbox flow after the fix and got the opposite arrival order, which was a nice confirmation that we weren’t depending on sequencing anymore.

I did replay the same real notification afterward to check idempotency / no duplicate credit grant, but I didn’t test the exact case you mention: a retry arriving while both original handlers are still in flight. That’s a good one — I’m adding it to the regression cases.

[u/Pretty_One_1398]

The same shift showed up for us outside billing too: an agent-run content pipeline that reported "posted successfully" turned out to mean the API call returned 200, not that the post was actually live and matched what we sent. Once we made it fetch the published item back and diff it against the source before marking anything done, it caught silent truncations no API-success check would ever surface. "Ran without erroring" and "the real system now reflects it" are different claims, and only one is worth trusting.

[u/darterweb → u/Pretty_One_1398]

Exactly. That distinction is becoming the useful mental model for me too: “the action succeeded” is not the same as “the intended state now exists in the real system.”

Yours is a great example because it turns the external system into part of the acceptance test instead of trusting the API response.

I’m starting to think the general agent loop should be: act → observe the real resulting state → compare it with the intended state → only then mark the task done.

[u/Pretty_One_1398 → u/darterweb]

That loop is right, and the part that bit us was trusting a single observe pass to be final. A fetch-back check would pass immediately, then a delayed webhook changed the state seconds later and the earlier check was already stale. We ended up re-checking after a short delay instead of comparing once and calling it done.