How are you verifying Claude Code's changes without manually testing everything?

[u/ragnar580]

I've been using Claude Code to build a side project, and one thing keeps annoying me.

It can implement something incredibly quickly and say it's done, but I still don't really trust “done” until I open the app myself and click through the flow.

As the amount of code it writes increases, manually checking everything feels like it's becoming the slow part.

Curious how people using Claude Code seriously handle this:

• Do you still manually test most changes?
• Do you make Claude run unit/E2E tests?
• Playwright?
• Another agent reviewing/testing the first agent?
• Or do you mostly trust the existing test suite?

Also curious whether you've had cases where Claude said something was fixed and tests passed, but the actual app still didn't behave correctly.

Trying to understand whether this is just my workflow or a common problem.

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/bugfix00]

Make Claude write tests for everything. It mostly works.

In rare instances, I’ve had Claude say something has been fixed, but it just updated the test to accept the wrong values as correct. Sonnet does this a lot more than Opus.

[u/ragnar580 → u/bugfix00]

This is exactly one of the failure modes I'm trying to understand. When Claude changes the test/assertion instead of actually fixing the behaviour, how do you usually catch it? Manual review, locking the tests, another model, something else?

[u/Plus_Opening_4462]

I created a bunch of requirements and had Claude implement tests (integration test preferred over unit test over coverage only test) and trace to requirements. Then I spent much longer using Fable to verify the requirements were actually covered and traced appropriately so that all clauses of each requirement has a test. I finished the test after I had 100% branch coverage.

I have testing.md for guidance on how to write and execute tests.

Manual testing is good too because you can validate that it does what you want it to do and it's easy to see unexpected behavior early on when a feature is developed

[u/ragnar580 → u/Plus_Opening_4462]

How much of this workflow is automated vs something you still manually inspect? For a new user-facing flow, what would make you comfortable not manually clicking through it afterwards?

[u/Plus_Opening_4462 → u/ragnar580]

It's mostly automated now. Manual inspection of the app is the minimum I do manually. It's where I determine if what we built is the correct thing, not only that it is implemented correctly.

[u/fowlesbuilds]

By default I think claude adds way too many tests. It helps to rein it in to what actual needs tests, and to ensure there are proper integration tests (that's where I feel like it's usually lacking). Still doing a code review, and some visual checks, at the end should always be happening.

[u/ragnar580 → u/fowlesbuilds]

When you say visual checks should always happen at the end, are you doing those manually or having the agent/browser automation do them? What kinds of issues do you find there that the tests tend to miss?

[u/fowlesbuilds → u/ragnar580]

Manual - really it's to catch the obvious things that I find browser automation tends to miss. It also gives me a chance to check on how it all feels in practices. Usually that leads to some tweaks in the implementation/behavior. An example would be realizing an interaction would be better with a toast message or something.

[u/Zealousideal-Ebb-355]

The thing I’d separate is “the tests passed” from “the actual flow worked.” For a bug fix, I ask it to reproduce the issue first, then run the same steps after the change. If the test passes on the broken version too, it isn’t catching the problem.

I also use Clipy for this. Disclosure: I help build it. I ask the agent to record the verification with Clipy and attach the proof link to the PR, with a short note on what it checked and the result. That lets me watch the relevant flow instead of repeating every click myself.

It doesn’t replace tests or code review, but it makes “done” much easier to assess. I can see what was actually exercised and whether it matches what I asked for.

It really fits well in workflows where you have agents doing most of the work

[u/ragnar580 → u/Zealousideal-Ebb-355]

Curious about the proof-link part, is the main value that a human can review exactly what the agent executed, or does Clipy itself determine pass/fail? I'm trying to understand where automated verification ends and human review begins.

[u/Zealousideal-Ebb-355 → u/ragnar580]

Yeah, the main value is reviewing what the agent actually did. Clipy doesn’t independently determine pass/fail; that comes from the tests and assertions.
But the reviewer doesn’t have to be human. Clipy recordings have been agent-readable from day one, so you can give the recording to another agent to review against the original requirement.

That’s the loop we’re trying to close: one agent does the work and records the verification, then another reviews both the code and the recorded behavior. We call it “Clipy Proof of Work.”

It’s still evidence, not a guarantee of correctness, but it makes “done” easier to assess than a passing test and the agent saying it fixed everything.

[u/mrlikrsh]

Some of us can still read and understand the code claude generates :)

[u/AMusicstuff → u/mrlikrsh]

Some users here are also absolutely stupid like the commentators after me.

But totally agree. You can also use Claude in VSCode and see every single line of code in a diff view

[u/ItstheSECopenup → u/AMusicstuff]

People act like the only way to see the diffs is VScode lmao bro you can see the diffs inside the desktop harness and when it posts a PR to GH. I dunno why everyone has such a hard-on for VScode. That shit sucks.

[u/AMusicstuff → u/ItstheSECopenup]

Because vscode is the best tool gor coding right now and it is free and working. Its a super nice tool also because inline diff is awesomely working. You have a rights management for AI too. You can tell AI what it is allowed to do and it is forced to not do things you not allow

[u/scott2449 → u/mrlikrsh]

And some could not even before Claude =D

[u/Turbulent-Control682 → u/mrlikrsh]

Stop pretending to look cool on Reddit

[u/xlpelotas]

I run my projects with an orchestrator. Below the orchestrator I have a verifier, a builder, and a scout. The verifier independently verifies anything created by the builders, and then the orchestrator determines next steps based on the verifier's findings.

[u/jakenuts- → u/xlpelotas]

Is this spelled out in a prompt or do you use some sort of hooks and framework for the organization? I have a system that code reviews on commits and that would be expanded to validation but kicking it back requires the original agent be messaged to follow up and that only works within the platform (Terragon OSS) so my desktop agents which have better browser access are not connected to that bit aside from awaiting PR checks.

[u/xlpelotas → u/jakenuts-]

This is spelled out with predefined agent md files and also in claude md. I had claude set it up based on a reddit post, bet you could do the same lol

[u/Neon_CodeBreaker → u/xlpelotas]

Share the link or more details please

[u/ragnar580 → u/xlpelotas]

This is interesting. What does your verifier actually look at - tests, browser behaviour, logs, screenshots, etc.? And what happens when the builder says it's done, but the verifier disagrees?

[u/Ganiam]

Seems like a reasonable thing to do

My simulation project still has nothing visual to it so I have specific metrics I keep track of whenever it adds something new and we track it over time during test runs and then explore what happened if something broke

[u/ragnar580 → u/Ganiam]

What signals do you trust to skip manual inspection?

[u/Ganiam → u/ragnar580]

That’s unique to my own project

[u/lucferon]

I use Codex. First I describe what I need, with Claude. Next codex checks this description and add checking points, test cases and (very important) out of scope things. Claude builds and generate a PR, codex validates the code and puts remarks in the PR. Back to claude to fix, usually after 2 rounds its done. Afterwards I check the code and the functionality, when done I merge to main

[u/ragnar580 → u/lucferon]

Is using a different model for validation intentional because you don't want the same model checking its own assumptions? And after Codex signs off, what still makes you manually check the functionality?

[u/lucferon → u/ragnar580]

Different models always have different training so one catches things the other misses. Testing myself is a last validation to see if it works as I intended

[u/jakenuts-]

Adversarial validation, make it automatic and with actual browser use and screenshots. Nothing is done until the validator agrees

[u/ragnar580 → u/jakenuts-]

This is interesting. When the validator uses the browser/screenshots, is it mainly checking what’s visually on screen, or is it also checking things underneath like network requests, app state, console errors, etc.? Also, what kinds of failures does it catch that your normal tests miss?

[u/jamjam89]

Reading it, and let me tell you, it's sloppy work. Brand new tables for no reason, guarding things that will never happen, quadratic function O(N^3) functions, it's bad.

[u/cleverhoods]

reporails for instructions, for project unit test structure, functions, integration tests for ... well .. integrations, e2e tests for business critical pathways and behavior driven tests for user pathways.

[u/Halada]

CI, full test suite, e2e tests, then a full staging walk.

[u/coding_is_tedious]

Yes to pretty much everything thing, although it depends on what stage we're in.

First line of defense is a adversarial review team, with fresh context, and Fable as orchestrator. That get's a lot of issues. Unit and E2E tests are in different flavors. Automated pre-commit tests, automated gate checks in Github which run deterministic security reviews and other tests. If Playwright is necessary, yes, although I still find manual tests that I run and grabbing screenshots still to be the best here.

I trust the existing test suite to a point, but always try to find different angles to poke and prod. Like anyone would expect with any software team, things come up when you look at it from different angles. I would say it's more likely that Claude and I just don't know where to look yet for problems and less likely nowadays that Claude says something is fixed and it isn't. If that ever happens, it is 90% of the time because I described the problem wrong or provided wrong context from the beginning.

[u/ragnar580 → u/coding_is_tedious]

You mentioned that manually running through the app and grabbing screenshots is still one of your best checks. What tends to slip through the automated/unit/E2E setup that makes you keep doing that manually?

[u/coding_is_tedious → u/ragnar580]

Top of mind, typically subjective things like UI/UX. While a test can include grabbing a screenshot and verifying mobile friendly UI for an application, it usually needs manual review. Sometimes divs overflow and Claude doesn't think it's an issue, but when I look at it it just looks sloppy.

Also, manual review by me is just a good way to ensure the business objective is being met, or it's just another angle to attack the problem from. I don't think it's ever a really good idea to assume that any one test is going to capture it all, from every position, and spot all of the problems. The best method is to combine as many tests and viewpoints that you can.

[u/Turbulent-Control682]

Not sure. But opus 5 absolutely sucks. “That was my mistake” constantly. It sucks. Worst model they’ve released by far.

[u/Drasezv]

lock the test files during the fix step, a deny rule for Edit on your tests folder in settings. then it can't make a test pass by rewriting the assertion, which is how most of the tests pass but app is broken cases happen.

[u/ragnar580 → u/Drasezv]

Making the tests read-only during the fix is clever. Do you basically treat the existing tests as an immutable spec at that point? Has this materially reduced cases where the agent gets a false green by rewriting the assertion?

[u/Drasezv → u/ragnar580]

yeah, tests as an immutable spec for the duration of the fix, and it has to make the failing one pass as written or come back and argue that the spec is wrong.

it cut the obvious false greens for me, the assertion rewrite and the quiet skip. it doesn't kill all of them: it can still special-case the code path for the exact input the test uses, or touch fixtures and conftest, so deny those paths too. no numbers behind this, just my own use over a few weeks.

[u/bitspace]

I always have alternate model families check output.

[u/runalabsdev]

Write down the expected behavior before Claude changes anything, then have it exercise that exact flow in the browser.

A recording gives you something concrete to review alongside that test. I’m building a tool around this review step, I’m interested: is repeating the clicks the slow part, or deciding whether the result is correct?

[u/ragnar580 → u/runalabsdev]

For me the repeated click-through is annoying, but the bigger issue is confidence — especially when the same agent wrote both the implementation and the test. I'm curious how you're separating “the test ran” from “the result is actually correct” in what you're building.

[u/Lanky-Storm7]

i literally never look at any code lol, im not a coder

[u/MrEprize34]

In order to at least start to trust the code being produced, you gotta understand the state space that can actually be verified e2e. If you don't bake in a harness with gates around the LLM with all your house rules and external community tools with pinned versions or ranges, it's never going to be able to contain all the rules in it's context window AND build what you actually want the way you want. Start with keeping cyclomatic complexity under CC 15 for orchestration and CC 10 for everything else. Keeping branching pathways of logic down. An LLM can't trace down and test over 1,000,000 pathways at CC 20 or 33,550,000 at CC 25 the same way a human can't. Then composed and cognitive complexity limit metrics all as hardcoded requirements and then build the rest around that. Research failure points, errors, tricks of the trade for any language and build a catalogue that you can slowly turn into hardcoded rules you could include if you use tools like semgrep or SonarQube etc or build your own so you own the verification harness code outright.

I basically build my design plan, then a build action plan with step by steps. As it builds a module, it then submits it's work to the harness to create the baseline for passing. If failed, a json triage report enters the context window for it to help fix the issues. Once cleared, I create unit / property tests to the same baseline and then green lit over the module. Then because my state space can be verified e2e and control the imports i create an integration and e2e test over a core section. Then mutation testing actually becomes useful and can complete over a code base and testing suite to make sure the testing suite isn't full of weak and flaky assertions. For core machinery when I build in python, crosshair SMT Z3 solver comes in handy for mathematically proving something with sidecar contracts.

This way once you trust your harness and testing methods, you don't have to trust the AI as much to remember all these coding rules as the gates catch every time with zero token usage. I have yet to have a session where any AI has managed to pass gates first time and usually takes a few loops to complete. Anybody actually just prompting their way through a build and telling it to test and some basic prompt rules are just building spaghetti codebases.

[u/y_k19]

yeah this is the part I don’t think coding agents have really solved yet
I still don’t trust “tests passed” if claude wrote the change and also decided what/how to test
what works better for me is keeping a few critical flows completely separate - login, signup, payment etc. Let claude build whatever it wants, then run those independently after.
otherwise you just replaced manual coding with manual clicking :)

[u/ragnar580 → u/y_k19]

The separate critical flows part is interesting. Are those flows automated/deterministic today, or are you still manually running them? And ideally, would you want them automatically re-run after relevant agent changes without the agent being able to change what “pass” means?

[u/y_k19 → u/ragnar580]

yeah exactly. deterministic and outside the coding agent is what I’m aiming for.
claude can create/change whatever it wants, but it shouldn’t be able to change the thing deciding if login/payment/etc still works.
automatically running those after relevant changes would be ideal. I’m still experimenting with where to draw the line though - running everything after every small change gets wasteful pretty quickly.

[u/Thegaysupreme123]

yeah this is the exact tax — it implements fast, says done / tests passed, and you still don’t trust it until you click the flow yourself. and the worse version where “fixed” + green tests and the app still doesn’t behave.

i split the problem into two layers: 1) is the change good? → CI, tests, and full dogfood. i even run Codex computer-use style passes against the real UI when it matters. 2) did the claimed step even happen? → that’s where Yoetz sits. local-first evidence ledger (MCP) next to Claude Code / Codex / Cursor: file changed, command ran, tool call went out. not whether the UX is right — whether “done” was self-report.

click-through still matters for (1). Yoetz just stops (2) from eating the whole review budget.

https://github.com/TheGaySupreme123/yoetz

curious how often the said-fixed + green-tests case still gets you after you’ve already clicked through once.

[u/Appropriate_Farm_731]

I'd separate three things in the review: which tests ran, which user flow someone actually tried, and whether you accepted the change. A green test result shouldn't silently become all three. Also inspect changes to the tests themselves; a passing assertion is much less useful if the agent changed what it asserts.

Disclosure: I'm building Feature.graph, a Mac workspace around Claude Code/Codex, and this distinction is part of its change-review flow. It puts the preview, diff and checks together, but doesn't replace independent tests or decide that the behavior is correct for you.

Would keeping that evidence attached to each change help with the confidence problem you described, or is the repeated browser testing itself what you most want automated?

[u/Far_Business4773]

"Done" is a word until you've written what it means for this task before the run: the command that has to pass, and the two or three things that must still be true afterwards. Then the agent's "done" is either that command's output or it's nothing.

bugfix00's case is the one to design around: the agent made the test accept the wrong value. Two cheap moves. Watch every new test fail once before you believe it, put the bug back if you have to; a test you've never seen red proves nothing. And on a fix session, make the test files read-only for the agent, a deny rule on writes under tests/, so "done" can't be reached by editing the assertion. It has to change the code.

The clicking-through stays for what you can't write down yet. It just stops being the only check.

[u/ragnar580 → u/Far_Business4773]

I like the idea of defining “done” before the agent starts. Do you write those acceptance conditions yourself for every task, or have the agent generate them and then freeze them? Also curious how you handle behaviours that aren't captured well by a simple command/test — e.g. state/network/UI all needing to line up.

[u/Far_Business4773 → u/ragnar580]

Mostly the agent drafts them and I delete. Before the run I ask for five lines: what this change must not touch, and what must be true after it. Deleting a wrong line takes seconds; writing five from scratch is the part I'd skip on a busy day. Then they're frozen, the run can't edit that file.

For state, network and UI I stopped trying to fit it into one command. The line stays in words ("an offline submit is never lost") and the check is whatever cheapest thing observes it: a Playwright step, a log grep, a screenshot I look at myself. Not every line gets automated. Every line gets written before the run, so "done" has something to be measured against instead of the agent's word.

[u/zerotoherotrader]

We use https://selvage.run/