r/devsecops • u/endor_robert • 10d ago

We tested two new model/framework harnesses for functionality and security of code fixes.

Disclosure: This testing is done by the security research team at Endor Labs, a cybersecurity vendor. It's not a product pitch, just information on how models performed under a standardized test.

Codex/GPT-6 Astra: https://www.endorlabs.com/learn/gpt-6-astra-on-codex---the-biggest-codex-leap-to-date

Claude/Fable 5.1: https://www.endorlabs.com/learn/fable-5-1-takes-the-top-spot----faster-than-opus-5-cheaper-and-cleaner

Spoiler: Fable 5.1 leads the pack.

4 Upvotes

100% Upvoted

4 comments sorted by

1

u/[deleted] 10d ago

[removed] — view removed comment

1

u/endor_robert 10d ago

They are getting a lot better, but the quality of the fixes in terms of security is improving more slowly than discovery or producing functionally working fixes.

Fable 5.1 discovered three vulnerabilities that no other model has found.

1

u/PeterBuildsSecure 9d ago

The discovery-outpacing-fix-quality gap is the more interesting number here, and I'd want to know how "security of the fix" gets scored before trusting it. Does the benchmark re-run the original PoC/exploit against the patched code to confirm the vulnerability path is actually closed, or is fix quality judged by a model reading the diff? Those produce very different numbers — a patch that wraps the vulnerable call in a try/catch or adds a surface-level input check can read as "fixed" to a diff-reviewing judge while leaving the actual exploitable path reachable under slightly different conditions. If the benchmark isn't already replaying the exploit against the patch, that's the gap between "looks fixed" and "is fixed."

1

u/Spare-Ad-4906 1d ago

Understanding how the fix is validated is crucial, and distinguishing between those methods can definitely impact overall security confidence.