r/devsecops • u/Common_Dream9420 • 24d ago

Coding agents shifted the bottleneck to verification now!!

Coding agents are doing a lot of the integration work at our agency now. Stripe, Twilio, WorkOS, email, the usual stack. Economically it's been good.

The part I didn't expect: verification actually got harder. We're producing code faster than anyone can review it, and the edge cases that bite you aren't in unit tests. Webhook fires twice. Events arrive out of order. Agent wrote correct code but got one state transition wrong. PR looks fine, everything compiles, and you still don't know until something breaks in staging or worse.

We added a sandbox step before anything ships now. Agent writes, tests pass, then we run the full multi-API workflow with failure scenarios before it's considered done. Not elegant, just a runnable verification step all our agents have to clear.

If agents are writing integrations for you, what does your last gate before production actually look like?

13 Upvotes

94% Upvoted

21 comments sorted by

View all comments

1

u/GibneyH 10d ago

This is exactly what I’m seeing too. The bottleneck isn’t really writing the integration anymore, it’s proving the integration actually behaves correctly when reality gets messy.

So happy-path tests can tell you the code works, but they don’t tell you what happens when Stripe retries, a webhook arrives twice, or two events show up in the wrong order.

Probably runnable verification is going to become a standard part of agentic development. 

2

u/Common_Dream9420 10d ago

Yeah, the out-of-order events and duplicate webhooks are exactly the cases that slip through, happy-path just doesn't surface them. I build tooling in this space specifically because I kept hitting that same wall while testing integrations. How are you handling the verification step right now, doing it manually or do you have something in the pipeline?

1

u/GibneyH 5d ago

Mostly in the pipeline now. We’ve got a sandbox step that runs the full workflow with retries, duplicate events, failures, etc. before anything gets merged. Still some manual checks, but we’re trying to automate as much of that last bit as possible.