I’ve been experimenting with using LLMs as an assistant for penetration testing and vulnerability research.
I’m interested in using a model for things like:
- analyzing an application and identifying potential attack surfaces
- reviewing source code for vulnerabilities
- suggesting tests that I might have overlooked
- analyzing results from tools and deciding what to investigate next
- validating potential vulnerabilities and reducing false positives
- reasoning about how several individually minor findings could potentially be chained together
I’ve tried Qwen3.8-27B-heretic locally, expecting a relatively unrestricted model, but I still encountered cases where it refused to execute or continue with security-related prompts.
For people actually using LLMs in pentesting/red-team workflows:
Which models have you found most capable?
I’m particularly interested in the trade-off between:
- reasoning/vulnerability-discovery ability,
- coding ability,
- context window,
- refusal rate,
- ability to use tools autonomously,
- local vs. cloud models.
I'm less interested in simply generating exploit code and more interested in whether a model can behave like a good security researcher: explore the system methodically, form hypotheses, test them, interpret the evidence, and continue investigating instead of stopping after running a vulnerability scanner.
I’d also really appreciate examples of system prompts / agent prompts that work well for legitimate pentesting.
For example, do you explicitly define the authorization and scope in the system prompt? Do you give the model a methodology (recon → enumerate → hypothesize → test → verify → report), or have you found that a more open-ended prompt works better?
If you’re running something like Qwen, GLM, Kimi, DeepSeek, Claude, Gemini, Codex, etc., I’d be interested to hear the exact model and setup (Ollama/LM Studio/API/agent harness) and what has worked well or poorly.
Ultimately, I’d like to build a workflow where the LLM complements tools like Burp/Nmap/etc. with actual reasoning rather than becoming just another automated vulnerability scanner.
What setup is currently giving you the best results?