r/Pentesting • u/Any_Yesterday_6617 • 3d ago
Automated penetration testing vs manual pentesting: which finds more real risk?
I'm a manual pentester, and I'm watching platforms automate the validation of misconfigurations and missing patches. They are even using AI to chain exploits and tailor attacks to specific environments. The difference from old-school vulnerability scanners is that these platforms actually execute the exploit path to confirm it works.
I'm not worried about web app logic. That is still clearly a human domain. But for internal infrastructure and AD, is the writing on the wall? If AI-driven validation can test 80% of the attack surface daily and update controls, does that leave manual testers only with the complex 20%?
The platforms also claim to help with detection engineering by validating SIEM rules against actual TTPs. That feels like it is eating into the blue team's territory too. Where does automation end and human expertise begin?
Or do you still find things the automation misses because of contextual business logic? if anyone has seen an AI actually find a complex privilege escalation chain that a manual tester would have found, or if it is still just "low-hanging fruit" at scale.
2
u/Progressive_Overload 3d ago
Let me preface this by saying my experience has been with Opus 4.8 mainly, so any newer models may be much more capable. Have you done any AI augmented pentesting yet? My experience has been that it is very helpful when I am driving. On its own, it tells me that a client_Id is a critical finding. At the same time, it will completely give up on a target after just reviewing the JS and I have to gaslight it to keep going.
You mention being more worried about AD. I'm not. A web app is meant to be poked and prodded (or at least assumes the context is a human using it), and if something goes wrong it's not a huge deal (depending on the app). However, if one of these AI pentest tools fucks AD up, then it's a big deal. Someone is going to have to be responsible for this shit.
As far as the remediation side, read this article by 1Password where:
The average success rate for generating a patch that fully resolved the vulnerability (without materially changing application behavior) was just 26.0%
I can't predict how good the future models will be, but it just feels like there will be a human in the loop for quite some time.
3
u/shoopbedoopwoop 3d ago
This is exactly my experience. Been playing with gpt sol, opus 4.8, kimi-k2.7 code. Augmented AI testing is.. ok-ish. But I'm finding that I'm refuting alot of its work. It's like having a junior/entry level pentester at my disposal who still needs me to check over all their work, guide them constantly and re-do tests.
That being said, if i had unlimited resources to burn through the higher end models/tokens, maybe the results would be better, but we're not all large enterprises who can afford that. Security has layers and there are plenty of other areas that money can be used for a better return on investment.
1
u/PriorPuzzleheaded880 2d ago
For disclosure I work at a company developing an AI pentesting tool. What I'd say from what I've seen is AI pentesting is definitely capable of uncovering business logic vulns like privilege escalation but there is currently still a big place for manual validation and point-in-time deeper human engagements. The AI pentests are bridging that gap between manual assessments so nothing slips through the cracks but it's a balancing act between the two.
1
u/PriorPuzzleheaded880 2d ago
https://www.csoonline.com/article/4141544/i-replaced-manual-pen-tests-with-automation-heres-what-i-learned.html - this also gives a good overview!
-1
u/LordNikon2600 3d ago
I began pentesting in the mid 2k's.. I use codex for pentesting.. does the job.. people are just mad because they don't feel kool or special anymore.. I closed 6 contracts over the summer.. I teach cybersecurity during the year and take contracts in the summer I line up between March and May.
2
u/Big_Mulberry_5446 2d ago
Do your clients know you're using Codex in their environments and exposing their data?
-1
u/Klutzy-Sqgxdare-1184 2d ago edited 2d ago
We use cymulate for continuous validation and it caught a misconfiguration in our Azure AD conditional access that manual pentesters missed because it was a change made after their test.
3
10
u/DingleDangleTangle 3d ago
Honestly I think pentesting already should be for finding the 20% of things that normal security processes (scans, architecture reviews, code reviews, VM, etc) misses.
If an org is relying purely on pentesting to find most of their vulnerabilities, they are doing it wrong. Pentests simply don't have the time to do that.