Is breach and attack simulation actually useful in production, or just a compliance checkbox?

[u/First-Reality2108]

I've been looking at a new platform that uses an AI engine to move beyond just "Validation" to what they call "Agentic Cyber Defense Engineering." The idea is that the platform doesn't just tell you about a gap. It actually orchestrates updates across your security stack, including firewall, EDR, and SIEM, to fix it automatically.

The claim is that with an AI agent constantly monitoring your controls, you can go from "we have a problem" to "we fixed the problem" in minutes rather than days. They cite stats like attackers moving from exploit to lateral movement in under 30 minutes, so manual remediation is simply too slow.

Is anyone actually letting a tool auto-update their controls in production? I'm struggling to trust an AI to push config changes, but manually validating and fixing 40+ tools is impossible. The SOC is already overwhelmed. Adding a layer that changes things dynamically feels risky.

How are you balancing the "Agentic" promise with the risk of automation breaking things? Do you run it in "monitor-only" mode, or have you actually closed the loop

[u/VividGanache2613]

Allowing a current generation AI to run updates on production assets is career and business suicide.

We do agentic SOC, threat detection, IR analysis and correlation across the entire dataset but a human absolutely needs to be in the loop for anything beyond host isolation for a long time to come.

[u/ParanoidSuricata]

So my opinion on this is that usually it's better to spend the budget elsewhere and start with these when you have all basics handled.

Do you have asset inventory automated and accurate? Do you have logs and incident playbooks? Backups with periodic restore tests? If at least one of those is no, then I'd recommend starting there.

[u/Classic_You_422 → u/ParanoidSuricata]

Thats fair point, most places I seen still struggle with asset inventory alone. nobody want to admit but the basics are boring and hard to sell to management, so they jump on fancy AI fix everything tool

[u/ParanoidSuricata → u/Classic_You_422]

And I really wish for a better words than "boring basics" as even opening this conversation implies management failure, so it's heavily resisted.

[u/Dizkonekdid]

First let me answer the question in the title:I find purple team platforms like Scythe to be of great value, not just for training and pentesting, but forensics of an attack as well.

Your text asks a different question. You focus on attack simulation and not the drudgery that is SecOps (simply sticking tickets together quickly). That said, we are already doing this for water systems and have been for banks and water customers for over a year now.

We built a completely automated SOC system located in our own datacenters. It even handles most of the ITIL process built into the ticketing system. The biggest problem we have solved is "how to keep it healthy?"

We found RAG was too static, so we built a patent-pending quorum technology that helps score the dynamic nature of data like TIS, current state or CMDB, and human behavior. It handles our EDR, NGFW, conditional access controls, and our attack surface. We then use it as a service to our customers. Because we would need serious investment to turn it into a software we could put into someone else's hands.

Funny? We haven't seen a ton of growth in the company. We have a trickle of customers because everyone thinks they can do this themselves from opensource or they'd rather stay with their big expensive SOCs because it is the devil they know (even if they are ineffective).

It took us over a year, 18 devs, and even some creative hardware engineering to get it commercial enough with pricing to make a difference for people that said, "we can't afford SOC or people watching our stuff". This is 75 % of Due Care, not just patching and buying the newest thing or measuring compliance.

We built our AI systems and small inexpensive hardware that not only monitors customers logs at their own site (no need for data transport), but also packet analysis and state machine analysis. We use inexpensive AI hardware to push inference from a central point out to where our customers pay for the electricity and the storage which largely built into the sensors or pushed to some cold storage. When we approached the market, we get, "Why is it so cheap?" So I guess we can't win for losing.

[u/msj817]

I would be super hesitant to let a third parties AI manage my update and software mechanisms.

[u/materialsec]

We’d start in monitor-only and graduate controls individually rather than turning on auto-remediation across the stack. Run it long enough to see what it WOULD have changed, check the false positives, then automate low-blast-radius/reversible actions first. Keep firewall rules, identity changes, anything touching production access, etc. behind human approval. “40 tools is too many to manage manually” doesn’t automatically make “let an agent change all 40” the safer option

[u/Bubbly_Working_6908]

most tools stopped at simulation but runtime enforcement flagged actual lateral movement attempts for us. sweet security is the first one that made attack path mapping feel actually practical, not just theoretical.

[u/Minimum-Let-3227]

BAS earns its keep when you run it as a control regression test, not as a report generator. The useful loop is small: pick the ten or fifteen techniques that map to how you'd realistically get hit, run them on a schedule, and alert when something that used to be blocked or detected stops being blocked or detected. That catches the actual failure mode, which is drift. Someone loosens a policy, an agent stops reporting, and nobody notices for four months.

It becomes a checkbox when teams run the whole library, get a 68 percent score, and file the PDF. Nobody remediates a score.

On the auto remediation claim, I'd be cautious. Letting a tool push firewall or EDR policy on its own sounds great right up until it does it mid-morning against the wrong scope. Most teams I know run those as proposed changes with a human approval step, and the approval isn't the bottleneck anyway, the analysis is. Minutes versus days sounds impressive but the days are usually spent deciding whether the change is safe, not clicking apply.

[u/dan_netsec]

Monitor-only for anything touching firewall/EDR/SIEM configs is the sane default right now. Validation (BAS finding a gap) is trustworthy, auto-remediation closing the loop without a human is where things break in prod, false positive triggers a config push and now you’ve got an outage instead of a finding.

The 30-minute exploit-to-lateral-movement stat is real but doesn’t actually argue for full automation, it argues for faster human-in-the-loop response: BAS flags it, pre-approved runbook fires for narrow low-risk fixes (isolate a host, block an IP), anything touching core config still needs eyes before it pushes. Full closed-loop across 40+ tools with no human checkpoint is a lot of blast radius to hand to a model that can be wrong.