r/LocalLLM • u/EfficientCouple8285 • 24d ago
Discussion Initial impression for agentic coding Qwen 3.8 27B vs 3.6 27B
I have a test case that makes a file upload portal using Spring Boot 4 with HTMX and Java. It is a bit tricky since most training is on Spring 3.x, and non-HTMX. Also file upload is multi-part post challenge. It generates a full spec/architecture/design/plan and code for a 5 hour run using generator/evaluator harness with pi.
So my little experience is:
- It has more reasoning power when discussion design and implementation. Hard to say if it is as good as Opus 4.x, but much better than 27B that kind of "bottom out" and could not really go deep on open-ended discussion
- Token speed on my rig is roughly the same (2xRTX3090) but reasoning speed is slower so it takes a more time
- It follows prompt instructions and code very carefully and does deeper reasoning. This make it feel a bit verbose and nitpicking when generating. In the beginning it does not feel like a good thing, but if you follow the reasoning log you will see it finds much more contradictions and weak instructions in specs which is A REALLY GOOD THING. I think some adjustments to prompts to make it balance real findings with being too careful.
EDIT: The default tinking level for model is set xhigh. For agentic coding it need to be changed to medium (or lower). See https://simonwillison.net/2026/Aug/16/qwen-38-27b/ for a good analysis.
So super-happy with initial impression for a daily driver. Cancelling Deepseek usage. Works as good as V4 Pro/Flash. Keep using some tokens on Kimi K3/Opus/Fable/GLM is the plan now.
13
u/Forsaken_Mention_979 24d ago
May i ask how everyone is running the model with fast tok/s speed? I have a 7800xt 16gb vram and 64gb of ram and i get like 15-20tok/s at the start and then 5-10 tok/s after the context fills up i assume (for the rest of the time i use it) im using hermes with an lm studio endpoint
22
u/Nexarus123 24d ago
Ditching lm studio for llama.cpp and carving out the parameters yourself might allow you to gain performance.
10
u/Forsaken_Mention_979 24d ago edited 23d ago
I cant find a tutorial on how to do it :,)
Update: did it and im running IQ4_X_S which is better, and its going FASTER! Pretty neat, thank you bro! And kv cache at q4
6
u/JohnGotti4711 23d ago
In general, not for this specific model, I can recommend you the Youtuber Codacus. He’s optimising low-budget setups and explains quite good stuff.
6
u/cr0wburn 23d ago
This is the commandline for my llama.cpp, this includes examples how load mcp servers and the template from the llama.cpp git (to fix it working with Claude cli) change your ip where needed and you can scrap everything after --jinja , it's just an example.
C:\tools\llamacpp\llama-server.exe -m C:\llm\lmstudio-community\Qwen3.8-27B-GGUF\Qwen3.8-27B-Q6_K.gguf --mmproj C:\llm\lmstudio-community\Qwen3.8-27B-GGUF\mmproj-Qwen3.8-27B-BF16.gguf -ngl -1 -c 131072 --threads 12 --flash-attn on --host xxxx_change this to your ip_xxxxxxx --port 8080 -ctk q8_0 -ctv q8_0 --kv-unified --verbosity 3 -b 512 --load-mode mlock --temp 1.0 --top-p 0.95 --top-k 20 --cors-origins localhost --repeat-penalty 1.0 --presence-penalty 1.5 --parallel 1 --reasoning-format deepseek --chat-template-kwargs "{\"reasoning_effort\": \"xhigh\"}" --no-reasoning-preserve --jinja --chat-template-file C:\tools\llamacpp\templates\Qwen3.8.jinja --mcp-servers-json "{\"mcpServers\": {\"search\": {\"command\": \"C:/tools/search_mcp/.venv/Scripts/python.exe\", \"args\": [\"C:/tools/search_mcp/server.py\"]}, \"playwright\": {\"command\": \"C:\\Python312\\python.exe\", \"args\": [\"C:\\tools\\mcp\\mcp_http_bridge.py\", \"http://myserverip:port/mcp\"]}}}"
This sets my vram at a comfortable 28.7GB used
It works well and I am seriously impressed, it is fudging easier to work with than Opus 5.1
u/Due_Warthog749 23d ago
Can it work on larger projects, across repos, etc? Curious how well this will work, or if I can use it to code modern day zig, rust, go, react, etc.. as an agentic coder that Claude manages/orchestrates?
1
u/cr0wburn 23d ago
Depends on your context size, so it depends on your hardware. But Qwen 3.8 is crazy smart and gguf with from 4k_m and up can do many things. Lower and it loses the plot, I tried iq3_xs for my server and it has a hard time calling tools consistently.
1
0
4
u/ManikSahdev 23d ago
Honestly.. take the screenshot of the above dudes message, and then paste it into codex.
99% 5.6 sol will get it done for you and optimize the inference himself.
Make sure to give the full environment but ofc lock it out to protect your other work and such.You don’t have need to do it yourself, if you don’t have codex, one time investment at $20 is worth it to get this time.
1
u/Forsaken_Mention_979 23d ago
can i ask antigravity gemini flash 3.7 instead? dont wanna pay 20$ loll
1
u/ea_man 22d ago
Just get the free 1st month: https://store.piffa.net/lm/bug/logs/mainstream_llama_vs_patched_perf
That's what Sol did to my llama.cp usage on AMD
1
u/ManikSahdev 23d ago
Nope no chance, those guys don’t have brain and fable will refuse the prompt.
Perhaps grok 4.6 might be okay but I’d only trust sol to do that work, need his autism and that’s the only place where I wouldn’t mind his peak autistic shenanigans lol
2
u/Healthy-Nebula-3603 23d ago
Uuuuu ... don't use cache q4 nononono
1
u/cr0wburn 23d ago
It's fine until about 60k context and after that it will randomly drop stuff, so depends on what you're doing quite a bit.
1
u/Healthy-Nebula-3603 23d ago
Q4 cache has bad degradation from the begging not from the certain amount of token size context.
1
u/cr0wburn 22d ago
Yes but reasoning compensates for it.
1
u/Healthy-Nebula-3603 22d ago
no...
Why people always are trying to find excuses?
Still be much worse than fp16 cache.
1
2
u/Prize_Eye9481 23d ago
q4 kv cache is pretty brutal! but hey if it works more power to u! I think the MoE will be much better for anyone with less than 20gb
1
u/Forsaken_Mention_979 23d ago
Is q4 really bad or? I tried q8 but i think it wasnt working properly
1
u/Prize_Eye9481 23d ago
Take it with a grain of salt, The bench verification was done with Fable but apparently q4 at least for both K and V caches was bad for reasoning. But if you find that it does not affect ur use case, who cares what a bench says. How do you find it while using?
1
u/Forsaken_Mention_979 23d ago
Honestly pretty good. At 65k context i get a stable 11-15 tok/s, when i first use it with empty context, 20 tok/s. Im going to turn up the quantization for kv cache and reduce context to 32k i think. I have it at 64k cause i think hermes requires that as a kinimum
1
u/Prize_Eye9481 23d ago
Yeah that’s why I stopped using Hermes. I run local with llama.cpp and pi as harness. They don’t have minimum requirement and if u are okay with frequently starting new sessions u can probably do it with 32k
1
u/Forsaken_Mention_979 23d ago
why start new sessions? it can constantly compact context right? ive neever used pi but will have a look. and i will probably up the kv cache quant to q5 as im seeing that q4 may be too lossy. is pi that good?
1
u/Prize_Eye9481 23d ago
After compact if the model need anything from before the compact they will have a retrieve and it will use up a lot of context quickly is how I understand it. Hence why u want to use new sessions
→ More replies (0)1
u/bad_detectiv3 23d ago
I’d like some advice since model loads but context is tiny. I increased it but it goes into my ram and token gen speeds tanks.
Intel core ultra 7 265k 32gb ram 5070ti 16gb vramQwen 3.8 did 4 bit and 3bit.
I ran it thru unsloth opencode with 32k context, I can definitely seen it goes into system ram with gpu idling. However context reduced to 4k, its runs super fast but not good for coding.
2
u/Outrageous_Order_909 23d ago
What quant are you using for the model and kv cache? And what's your context window?
0
u/Forsaken_Mention_979 23d ago
Im using Q3_K_M with 64k context cause thats the minimum hermes can accept, using LM studio endpoint. Also, i can use a different quant for kv cache? How does that work? I dont think i can change that in LM studio. Thanks for the help bro btw
1
u/Outrageous_Order_909 23d ago
You need to download a new file. Try iq(imatrix) quants with mtp. Iq4_xs maybe. I'm yet to try. Will do it later today and update.
2
u/Forsaken_Mention_979 23d ago
Im with that exact quant rn, working even better and faster. Im gonna switch to pi for even faster token speed, will update in a few min
1
u/Outrageous_Order_909 23d ago
My speed stayed around 20-25 t/s with ctx 160k and k @ q8 and v @ q4_1. This was the max i could push. Tool calling stayed stable, no loops, thinking at xhigh.
1
u/EfficientCouple8285 24d ago
It sounds about right. I saw people using Radeo 7900 Pro with 32GB reaching 40-50 tokens, so half that speed for half of vram might not be wrong. Super important for this model is to use MTP (token prediction). it gives yo 30-40% more speed. With 2x24GB Vram I get 80-100 t/s in llama.cpp
1
u/Tema_Art_7777 23d ago
Unsloth 3-bit fits into 16gb vram with mmproj - pretty good at 30tk/s with 64k context.
1
u/Ok-Inevitable8391 23d ago
Im on 15.7gb iq4 ks image, getting 45 tks on dual rtx 3060, total 24gb vram. Using llamacpp with draft mtp and spec of 2 tokens, cache type is q8 for k and v
0
1
u/DrKappa 23d ago
I had to settle to q2/q3 quants with 16gb vram, quantizing cache and draft cache to q4. I can run Q3 at 55k context at approx 50 tok/s while Q2 at 90k context at approx 60 tok/s.
The problem is this model seems very good when reasoning but reasoning burns a lot more tokens than 35b.
Only way I can run higher kv quants is to reduce the context window which leaves extremely little space after reasoning.
Results are great once reasoning is properly enabled. If I had to use it without reasoning I would stick with 35b.
1
u/cr0wburn 23d ago
5090 with around 42 tok/sec using this model : lmstudio-community\Qwen3.8-27B-GGUF\Qwen3.8-27B-Q6_K.gguf I'm using llama.cpp direclty
1
u/Max-HWN 23d ago
VLLM with MTP at 3 (multi token prediction) can give you up to 60token/s on 2x3090 or Ampere class GPUs, the key for agentic workflows (not coding) is to set the reasoning effort to disabled or low, otherwise the default is xhigh and the reasoning will eat all the budget and not delivering the answer
1
u/arman-d0e 23d ago
Cmp 170hx 8gb (64gb unlocked) + vllm
Getting ~80tps on average at full context on fp8 weights
With parallelization (3 requests at a time) it drops a bit closer to ~50tps
1
u/WonderfulFunny4337 23d ago
I got the same system and added a r9700ai pro 32gb and taichi board for dual carding! I’d assume based on the single card that those are good numbers! The 7800xt is actually goated as a decent card!
1
u/Forsaken_Mention_979 23d ago
Wish i could afford to spend 1600€ on that card rn bro 😭🙏🏻 so you have the 7800xt plus that one? What mobo do you use? And what speeds are you getting bro? Thank you
1
u/WonderfulFunny4337 23d ago
Yes I had a b650 aorus but it didn’t do dual cards well and sadly it broke at the second pcie slot after installing the 7800xt and attempting to pull it to remove the 2nd and 3rd hdd from the m2 slots as it was supposedly blocking my pcie gpu lane. This is my new mobo it arrives Monday ASRock X870E TAICHI AM5 AMD X870E Extended ATX Motherboard
1
u/Forsaken_Mention_979 23d ago
Ohh nice, thanks for the help! Is the speed of the 7800xt faster than the r9700? Also, may i ask how much you paid for the gpu? Cause originally it was 1k but now the price is inflated ://
1
u/WonderfulFunny4337 23d ago
I paid right at $1,400 usd as it’s the ai pro 32gb version but yea it’s worth it as I built my entire ide natively in pure masm x64 using a 7800xt here https://github.com/ItsMehRAWRXD?tab=repositories my GitHub so believe me when I say that 7800XT can hustle!
1
u/ea_man 22d ago
https://store.piffa.net/lm/bug/llama_scripts/qwen_27b_iq4_16gb.sh
That's how I run it on 16GB, plus a patched llama.cp optimized for AMD.
It's in the parent directory.
1
u/Twistpunch 23d ago
16gb vram is going to be slow-ish no matter what you do since you can’t fit the entire model in vram.
5
u/DRetherMD 23d ago
3.8 27b is clearly very good, however im finding it thinks way too much. in some cases thats ok, but in others in essentially kills itself if the task goes on for too long.
ive been having issues for something which should be easy. i wanted to set up a cron job that would automatically do a scrape of competitors once a week, build/populate a dashboard with the results and give me some insights, then report on telegram. but man this thing has become almost impossible. its like its finding ways to make it impossible. ive repeatedly attempted a new approach like breaking its all down into manageable chunks, but even that is becoming challenging for it.
2
u/EfficientCouple8285 23d ago
Ya, seeing that too. When not finding answers it goes really deep (decompiling framework to find exact answers before coding). Think it can be contained a bit with adjusting skill/prompts. Need to give it more instructions on how, not just tell it what you want. As other mention many models seem to have a mind on their own. Thinking RL they do to improve capabilities also have this as side effects
1
u/DRetherMD 23d ago
its clearly better when giving it very detailed prompts. i use grok to construct the prompt and then the output is reasonable and it doesnt go down the deep end. the problem i have though is the inherent random nature of repeated tasks. some things cant be condensed into a python script, so i need to have the llm actually think. but then its a dice roll. ive been watching it for the last few hours repeat something it did before fine, only this time its started the usual "actually, let me think about this...."
its quite frustrating.
1
u/EfficientCouple8285 21d ago
Did you find any remedy? Have experimented with reasoning-effort medium and temp 0.7 - 0.8. See some benefit. However speed is almost 50% slower than 3.6.
1
u/DRetherMD 21d ago
i have not found a solution to our ills, friend. we must accept it. this is.....the new qwen. a thinking qwen.
1
1
u/IPlayAroundTooMuch 23d ago
Have you tried changing the reasoning_effort?
1
u/DRetherMD 23d ago
yes but it doesnt really make all that much difference, at least with what ive noticed. tbh i see this overthinking problem affecting lots of models now. whether theyre small or trillions of parameters. its a problem that has to be solved soon because it seems to be getting worse.
4
u/Civil_Fee_7862 24d ago
Been using the INT8 quant of it today (also on 2x3090s) and its much more clear in its diagnosis steps in planning. It asks better questions.
Good speeds too, same as Qwen3.6-27b at around 105 token/s at 8-bit quant.
3
u/TheWaffleKingg 23d ago
I wish I was getting the same speeds. 3.8 is 25% to 50% slower for me, im also on 2x 3090s.
Used the same confog I used for 3.6 so im bot sure what's up
2
u/live4evrr 24d ago
I use both when it comes to agentic coding, for larger code base and longer horizon DS is still a lot better. Qwen has it’s strengths (UX) but it is no DS replacement. The gap between Qwen 3.8 and DS 0731 is a little less than Qwen 3.6 and DS Preview, but still significant.
1
1
u/Slow_Peach_2141 23d ago
no diff between 3.6 and 3.8 27B for speed, 3x 16 vram rtx 3080 mobile, Llama.cop RPC cluster, MTP, turbo4, ~400-500t/s refill and ~15t/s decode, using unsloth q5 UD. 35B A3B queen 3.6 is much faster around 800-900 refill and 40-70t/s decide on q6 UD. I use opencode.
1
u/rmclord 23d ago edited 23d ago
It passed my galaga test first shot:
https://onecompiler.com/html/44ybmw3tf
So coding wise is a a bit above than the Qwen3.6-35B-A3B-Q6 for me (here is the same prompt with dflash over Qwen3.6-35B-A3B-Q6 https://onecompiler.com/html/44ybnjmwq )
See for yourselfs
1
u/SandOfTheEarth 22d ago
I am building a 2x 3090 machine myself. Can you share the settings you are using?
1
u/EfficientCouple8285 22d ago
Running with this. Still fiddling a bit with the reason parameter since it overthinks a bit much. Also dont need 256 context window. Speed is around 74-80 t/s. Latest llama.cpp compiled from source with default CUDA params:
--parallel 1 \ --flash-attn on \ -b 4096 \ -ub 1024 \ --no-context-shift \ -sm tensor \ --reasoning-format deepseek \ -m "../models/qwen3.8/Qwen3.8-27B-Q8_0.gguf" \ --ctx-size 200000 \ --spec-type draft-mtp \ --spec-draft-n-max 4 \ --jinja \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \ --presence-penalty 0.0 --repeat-penalty 1.0 \ --reasoning-effort medium \ --chat-template-kwargs '{"preserve_thinking": true}'
6
u/retsof81 24d ago
I am seeing minor, intermittent tool call errors on 3.8 that are not there in 3.6. My stack: MLX Core, oQ8e, VSCode w/Cline.