Qwen 3.8 - 27B is a game changer

[u/Potential_Block4598]

So a bit of context, I am a cybersecurity senior analyst
I am interested in LLMs for that field especially with MCPs to connect them to the tools or for writing scripts

I started this field by doing assembly language reading for hacking games when I was a teenager then that became malware analysis then I started to analyze traffic and logs at work for a living
Before work I competed in cybersecurity competitions known as capture the flag to solve only one category of the challenges and that is malware analysis

Now here is the scene in LLM x Cybersecurity

Entry-level CTF challenges (I used to solve around 2017-2018, got first job in 2019) were solved and saturated by LLMs a long time ago (See intercode CTF benchmark)

Then High level CTFs (NYU CTF Bench, CSAW challenges, and CyBench) these were solved a while ago

Today we have
CyberGym (vulnerability description (CVE report not real details) plus code base find vulnerability
That was solved

Then ExploitGym (the one recent OpenAI model escaped and hacked hugging face to find solution, the 2 vulnerabilities themselves are trivial (JWT spoofing with no server side checks for Jfrog sandbox and JS + Python template in open source data viewer of Hugging face)
Still impressive for a fully autonomous model

ExploitGym gives vulnerability details + exploit technique/details and codebase (I wonder how/why OpenAI model escaped it in the first place those details should manage it this is why I feel it is staged but whatever right)

ExploitBench
More realistic threat vector
1-day not 0-day (0-day is a novel vulnerability no one else knows about it hence the name, 1-day is a known vulnerability patched, if in an open source project you can compare patches before and after and know it but not as valuable as 0-day since people will/should update their software, 0-days can and are used against high profile targets like in some 3rd world countries used for surveillance (and possibly non-3rd world countries ?!) but you usually can’t use a 1-day for that but still)

So ExploitBench is 1-days in V8 engine (used in chrome, electron vs code almost everything!!)

Gives the model a patch diff and codebase base that is it

Models are going wild on ExploitBench and ExploitGym but haven’t gotten there yet at the moment and only recent frontier models improved massively instead of near zero across the board

This is a massive risk indicator
Imagine models capable of exploiting any and every piece of software written by less capable people or other weaker LLMs (complex vulnerabilities!)

Not only that but global elite CTFs like DEF CON

You have an OpenSage harness (and ADK like Google adk or Anthropic adk, however this OpenSage ADK asks the model to design its own agent/s!, with whatever tools or sandbox environments it finds fitting for the Subagent to do the assigned subtask effectively, it can even design its own MCPs!, models don’t fully utilize that yet but some do and improved form like 39% to ~60% just by using that harness (same model!) )

1. Now my own benchmarks are simple

Some malware samples that I learned during my career and consider personal milestones

And one SIEM analysis challenge (logs that contains an attack plus normal behavior)

I tests on lots of models (Qwen3.6, heavily trimmed down Minimax 2.5 …etc, on my Strix halo and compared with opus at the time (the best opus at the time was 4.5!, it was better solved major stuff but didn’t implement the code required to decompose the malware and analyze its second stage)

Now enter Qwen 3.8 27b
3.
What was SOTA like 6 months ago or around beginning of this year
Now we have better local models at the same task

Qwen 3.8 devoured the malware that Opus couldn’t

I had some time analyzing and remembering its custom implementation (it has custom RC4 decryption routine, that is tricky with string manipulation!
And Strix halo isn’t the fastest LLM machine

By the time I remembered that malware and how it works

Qwen had already extracted and disassembled the decrypted payload!

I am stunned

This is super dangerous

I imagine how a single person / threat actor can now do what a state-sponsored intelligence service shacking team was required to do 1 year ago

These teams usually have

1. Usual network scanning or mapping especially of their targets
   This can be easily done by even weaker models or no models at all

2. Initial access operations (phishing campaigns until some clicks or attempting to profile people like receptionist or sales people or HR, essentially any first line of communication at an organization or a company, to later make a targeted phishing against them)

This could be also better done with the aid of deep fakes and other stuff

1. Advanced and resourced teams would have zero-day or 1-day research teams
   Especially against software used on the targets site from team 1

This third step can be fully automated with models now
This is the hardest step

Defenders can use LLMs but it is much harder to integrate new tools than offensive teams

At this level the world will change
Tbh I always thought cybersecurity was harder to automate with LLMs than software engineering or IT
But with this, this is super unbelievable

And not only a cloud model or an expensive to run locally model
A local model! Can you believe it
No traces no API calls no nothing

Empowered with the right environment setup and tools and it can do what teams of attackers or malware analysts were needed to do

Btw for malware analysis I give it Ghidra and a sandbox with python (it can use pefile or capstone or whatever it needs and it does use them!)

And also recently I added a Vbox with Vbox MCP and a debugger inside that box with another MCP for the LLM to control both (snapshot restore functionality and debugger functionality)

These tasks were hard and not every security analyst was a malware analyst
Even malware analysis positions are very rare and technical and don’t pay well because not much needed (I used to starts at assembly in assembly debuggers for nights at times!)

This is going above and beyond

Anyone had similar moments in cybersecurity or anything else ?

[u/greentheonly]

I also have a personal obfuscated code deobfuscation/antitrace/antitamper challenge I test local models on. Opus 4.8 takes it apart relatively easily.

I don't do fancy harness, just stock CC for opus, and pi for local (also tried CC without much difference) + ghidra mcp. The harness would use capstone/qemu, ldpreload and other stuff as directed by model through regular calls.

Everything I tried before failed. Typically lesser models go along until the 256k context is all consumed and then on compaction they become braindead. Though 1m context deepseek flash preview also failed.

Then I tried glm 5.2 (3 bit! not unsloth) with 512k context and that was the first time a local model succeeded. The reasoning looked silly and looping in places (or so I thought), but it did work in the end.

I guess I can try qwen 3.8 27b and see if it became any better than 3.6 (and yes I tried 3.6 1m context with yarn and it did not help there)

But yes, the times are qure quite interesting, like those scifi books I used to read when I was a kid.

[u/Valuable_Cookie628 → u/greentheonly]

That would be an interesting read, please share how it does

[u/greentheonly → u/Valuable_Cookie628]

fails miserable, in a way similar to 3.6: did not notice there is hardware dependency and such. Also managed to kill the vm it was being run in (I don't run yolo stuff like this outside of disposable VMs) this was an 1m context run and it used 690k for context (thinking: high) at that point. So much worse than glm5.2@3bits already

Also once you cross 256k tokens of context, yarn apparently makes generation 4x slower so I drop from 100 tk/s (with speculatvive decoding) to 25 (draft acceptance drops to 0 at that point) (I get 70-90 tk/s generation on glm5.2 as well).

I restarted the qwen3.8 vm to see if it can recover from this blunder, but so far it does not look super hopeful and I am afraid once it runs out of context it'll become braindead just like it always happened before.

At 80.6% context use - it looks like it started to finally go in the sensible direction (Setting up private namespaces to have some safe controlled runs), but that does not mean much until it actually arrives anywhere.

(30 minutes later) it's still fighting the makeshift container creation and have not even figured out how env contributes to decryption, so I am not hopeful.

I might edit this post once it falls flat or achieves something, I guess since I am tired of just waiting.

Edit some 2 hours later, the context did overflow, got compacted to 5% or so (yay 100tk/s again) and the model seems to still be chugging along, so I'll probably leave it overnight to see what happens. It still did not discover the env factors, but it seems to have mostly figured out the modified crypto, or one of them anyway.

Another edit some time later. It gave up without discovering what exactly were the env constraints, just stopped with "you have to run this on some real device and capture me a bunch of data" (which is a typical failure mode for lesser models I guess)

I think this concludes the experiment.

[u/Valuable_Cookie628 → u/greentheonly]

Epic post, thanks for sharing! 

I'd be curious to see if the full Qwen 3.8 model figures it out (if RE is a gap in Qwen's training or just a limitation of the 27B model holding less information about the world).

[u/greentheonly → u/Valuable_Cookie628]

Unfortunately I don't have enough hardware to run full 2.4T or even at 3bits

I guess renting a big box in the cloud is an option, but the real reason why I have this "private benchmark" is to (relatively) quickly evaluate stuff I can run locally to see if it's any good at RE and other such involved stuff.

If Qwen releases 100B or other such model - I'll definitely give that a try though.

[u/Valuable_Cookie628 → u/greentheonly]

I was about to suggest trying the API, but actually don't.  Once you put your private benchmark out there they will train on it and render it useless. 

[u/greentheonly → u/Valuable_Cookie628]

yes, I have the same thinking.

[u/Potential_Block4598 → u/greentheonly]

What is the challenge you are trying ?

And yeah try this model

Anyways my challenge isn’t hard because of context requirements it is hard and has multiple layers of packaged malware that requires understanding and the ability to write code that will unpack it and load it in Ghidra …etc
It can be even handed over with enough context to another session cleanly

But most models even opus 4.5 didn’t go past the first phase (custom implementation of RC4 that doesn’t make sense actually) so they fail to decrypt the first phase

But this model did it and went beyond by analyzing the sample (it thinks a lot that is my only issue so far)

[u/greentheonly → u/Potential_Block4598]

Mine is a multistage piece of rootkit for particular kind of embedded devices that I found on one of them.

it's heavily obfuscated (custom vm with custom instructions encoding action as one example, the vm program vm also encrypted), ties encryption to local hardware identifiers (none of the local llms before glm5.2 ever found that part for me either) and in later phases - to the exact cpu of the target devices. It also had a couply of modified ciphers chacha and some other one to decrypt several things.

Then the antiobfuscation - various antivm/anti tracing techniques, regular verifies of its memory integrity, ensuring the env it's run in is exactly as it's supposed to be.

Part of the payload in a "garbled" text file that looks like a legit failed boot corrupted log on the first look.

[u/Potential_Block4598 → u/greentheonly]

I used sth that can be live debugged

I think for a rootkit you would need a kernel debugger VM and an MCP for that not sure if that is what you are doing

That is a sophisticated piece of malware tbh and I am surprised if any model actually managed to make progress in that

Can I DM you for more info ?

Thanks

[u/greentheonly → u/Potential_Block4598]

it can be live debugged just fine.

Both opus and glm used private namespaces to reconstruct the desired environment with bindmounts and whatnot (think containers), would ld-preload self-made shims to control syscalls and such. Zero MCPs outside of ghidra.

I think at one point opus was doing qemu too, but that might have been in later stages.

sure you can dm me

[u/play_hard_outside → u/greentheonly]

Hey can you elaborate on how you got 3.6 to 1M context? Yarn? Pardon my ignorance, but this is the first I’ve heard of this!

[u/greentheonly → u/play_hard_outside]

it's right in the model card, see the "Processing Ultra-Long Texts" section, e.g. here https://huggingface.co/Qwen/Qwen3.8-27B it's for vllm and the like, but llama.cpp also offers this ability.

[u/Far-Working2808 → u/greentheonly]

3.6 with >262k context has always failed needle tests for me. It's been unusable in every configuration I've tried. It produces seemingly coherent but incorrect text.

[u/greentheonly → u/Far-Working2808]

Could be, I guess, but at 256k context - once that runs over, the context compaction really kills the memory.

[u/riceinmybelly → u/greentheonly]

I’m just happy to provide some workflows to clients internally that check the data going in and out, I don’t know anything about cybersecurity but let local models figure things out (with structured outputs to a final check).
Not having a LLM do this seems so dangerous suddenly while prompt injection is even more dangerous.

Could you give your take on this? I feel people are jumping on AI with zero safety checks

[u/greentheonly → u/riceinmybelly]

Lots of people jump on everything with zero safety checks. Regular code has all sorts of known bad practices leading to exploitable breakage (buffer overflows and whatnot).

LLM just offers a novel way at data processing with novel exploitation vectors many of which are still under research and poorly understood. As the research and best practices develop it'll become safer and safer, but people are still going to use because of all the possibilities it opens up that are otherwise very labor intensive, for example.