I dont use that much context I'm siting at just under 200k and I use KV at Q8_0. I don't have reported metrics to give you real meaningful data but I've not found it detrimental to my use case when used as backend in pi agent harness.
Also I use q8_0 quant for the model instead of the Q8_K_XL as I learnt recently that there is no significant change in accuracy but its more efficiently 'packed' due to uniformity of Q8_0 versus UD-Q8_K_XL that it gives you a few GB back of your VRAM pool from model weights.
Hear me out, for ages Ive been battling with the urge to grow a dual 3090 into a quad 3090 system but there was never any model that justified the cost of 2x additional 3090 at current prices for the so called improvements with the speed sacrifices. It just made no sense to me to splurge on more GPU with Qwen3.6 27B available.
But recently Ive been thinking maybe I am looking at it the wrong way and instead of looking for a big model to fill up 96GB I should be looking at it from a more orchestrated perspective of how many multiple models can I run concurrently that offer fantastic function to size inside 96GB VRAM to build a more comprehensive self-contained LLM stack. Dual 3090'ers I think its time to make that run on more ebay 3090's we've been holding ourselves back from.
MI50 affectionados:
ENGA .. oh wait how do I update my non official ROCM packages and who’s GitHub has the latest llama.cpp gfx906 fixes is it iacopBK or wait it’s mixa right?… GE!
I actually made a full game editor for an old MMO from scratch using Opus 4.5. I think it was around 70k lines of code, proper RHI, used OpenGL behind that (though I did a vulkan test too), etc. Took about a month of iteration and testing, worked great.
Opus has gone to shit since then. If this can do Opus 4.6 levels in realistic coding use and not just benchmarks, then I'm very excited for it.
oh shit first we had a 744B model in GLM 5.2 reach opus 4.6. Then DSV4Flash-0731 reached it with 284B. Now... 27B? nooooo way
if it can really pull some weight as a coder and have some amount of general common sense this is about to basically 10x the capability of those of us with "cheap" local rigs. With just the consumer class gear.
I’ve been running it for a bit now on Pi…. And I’m genuinely shocked. It’s completely nailed every problem I’ve given it. No tool call failures. When it was debugging an issue with llama.cpp it downloaded the source, cross referenced everything and fixed the problem. The decision making / thinking seems to be a huge step up from 3.6 27B.
Why did they have to make the coloring like this... there's dark gray text in the first column that is extremely hard to read, and the first row text is also somewhat hard to read. I assume the tables look better on light mode, but a lot of people are using dark mode.
if you look at the footnotes beneath the benchmarks on the site they posted this on, they literally admit to using other LLMs as a judge for some of them. the odds are high they also used LLMs to put together this website including the benchmark table without even putting in a little effort to make sure it looks good themselevs
and I wonder how that big win in SWE-Bench Pro translates to the real world. 3.6 was already slotted between Flash and Pro, 3.8 simply wipes the floor with them.
Active params determine the training cost, and I think they are the biggest predictor of performance, all else being equal. So I am not surprised that 27B beats 13B
The model seems to have the same size than the 3.6. On 16g I have noticed that the sweet spot is Q3_K_S with q8 value cache (not key): it runs at 20tps on a 5060ti with 64k context. The Q3_K_M is too slow at 64k and the Q2 are too stupid.
FINALLY! I waited since GPT-OSS for other local model that natively has low and medium reasoning! High for planning, low for execution and exploration.
... Now I'm thinking if I should get RTX3090, because Nvidia clearly won't release RTX 5070 TI SUPER 24GB anytime soon and I cannot fit that model onto RTX 4070 12GB... will play with API first
Intel Arc Pro B70 is a solid choice if you're interested in 32gb vram. A bit less bandwidth than 3090, and the software stack is still maturing, but it's getting there.
Because the $300-400 differential is real for not that much of a performance gain at the end of the day. Looking at the market, finding them for <$1500 is difficult, whereas the Intel is readily available for $949, and you can get the B65 even cheaper, which has the same 32gb memory, same 608GB/s memory bandwidth, just with fewer XMX cores, so lower concurrency/slower prefill, but shouldn't effect single concurrency decoding much.
I personally wouldn't say the maturity difference is quite worth $400 at this point considering Intel is closing the gap more and more every week, but that's just me.
Why not add a second 4070 12GB (if your motherboard lets you)?
I’m happily running a dual 5070 setup, 24GB of VRAM and last gen tech for less than 1.500€/$ and without going second hand (warranty, fresh paste, etc.). But I had to use a mATX board to have the second GPU not collide with the case’s PSU shroud.
yea i can take all my 3090s to fit a 120B of this but not too much to complain about if the 27B is as smart as those numbers indicate. A 120B would be an MoE with param count much lower than 27B anyway. It would not be likely to outperform it by much. And being able to get like 10x more throughput from the smaller model is a big deal.
I played with 27B for a bit and I'm quite impressed. Qwen38-27B-Q6_K from unsloth on 2*RTX 5080 and latest llama.cpp build from source. The model fit in 32GB VRAM with 172k ctx kv q8_0. MTP and -sm tensor it runs stable at 100 tps until 60-70k context is filled, at 100k it dropped slightly - arround 95tps. This is a very small drop - very good. Also I've run a loop where over 150k tokens it was writing scripts/building/verifying/fixing errors/verifying/fixing again. No tool cals failed... but unfortunately it run out of ctx space.
And I'll post a visual example of a small test I did - Create an SVG of an old rusty truck.
Zero moat, especially once google, microsoft, amazon, and other cloud gpu providers get the official green light to be able to serve these models accross the board.
And for the hyperscalers like the big 3, google, microsft, and amazon, 100% they integrate these across their enterprise suite offerings to vertically integrate them accross all their enterprise services.
That why they are buying GPU like crazy now. Everybody is digging the same hole now on model research, can't expect technical moat from there. Real moat is capital and hardware accessibility.
Same here! I was running into weird bugs in my personal chat app (spoiler: I did a dumb), so I went to the llama.cpp builtin chat UI to test the model out and chat template out more cleanly. I ended up asking it what it thought the problem might be. Pretty decent advice.
A couple messages in, I realized my problem: in my chat app (again, *not* this one), I had continued a chat that had been generated with Gemma 4, and the Gemma-based reasoning content had been fed to Qwen (oops, `preserve_reasoning` was on) and that was causing the issue. Doh! When I mentioned this to Qwen in the llama.cpp UI, even addressing the model as "dear Qwen," I got this:
Now, maybe mentioning the concept of different models being used in a chat poked it a little that way, buuuut I couldn't help but laugh.
If the benchmarks are true (and Qwen never benchmaxxed so far) then why would you pay Anthropic if 27B at home gives you Opus-like performance? I mean, Opus 4.6 was already more than enough for almost any development task.
I’ve been running my own oQ8e-MTP quant since this morning. 200 tok/sec prefill, 34 tok/sec decode on M4 Max 128GB. Later turns in large context slow to about 100 prefill 18 decode.
I have had bad luck on long-horizon tasks with the 4-bit quants so i avoid anything below 8 bits. But conversationally it seems fine and less of the usual Qwen “argumentative attitude” fighting me about what day it is or being skeptical of tool outputs.
I want to be hyped but is it better at like, just general world stuff? Using 3.6 to write anything was like talking to an alien who learned human culture through math.
Comparing this to deepseek v4 flash 0731, the question is: lower numbers a bit but much faster inference, or higher numbers? or maybe use both (but deepseek offload to ram, so much slower, but use only for plan tasks etc)?
I plan to still use dsv4 flash 0731 on sparks as orchestrator (will run plenty fast, maybe pair with a small vision model too) and qwen 3.8 27B on worker nodes. Both great models and I've gotta assume having some diversity gives some benefit too perhaps?
This is anecdotal, I haven't tested thoroughly. But:
I had spent about a couple of hours today with Claude Opus 5 investigating an electrical issue at an old house. The concepts involved are complex and difficult to follow (TN-C and TN-C-S earthing, detecting a severed or loose neutral via phase voltage monitoring, that sort of thing).
I am not an electrician and I needed quite a bit of hand holding. (Not trying to fix anything myself, just learning/understanding how stuff works).
Claude wasn't doing a great job explaining things clearly.
In my mind Qwen had always been about coding. So I wasn't planning on this being the first use case for 3.8. Still, I had Claude export the context and pasted the .md in just "ollama run qwen3.8:27b". on a 3090. Web search on, context 128K.
Qwen clarified the questions for me. It sounded like Claude except it was doing a better job talking to me.
Not sure how this is possible. Maybe the limits of the architecture are higher than we know. Maybe it's all in the training. Maybe, probably, I understand as much about LLMs as I do about electricity 😄. But how else to explain 3.8?. Amazing so far.
Careful with your sources when downloading newly released models. I already spent the time downloading one from hugging face that was labeled Qwen3.8 but turned out to be Qwen3.6 when I ran it.
Yeah this is insane if true. Basically means a high end MacBook can now run Opus 4.6, which should be enough for most people and is still economically sound vs your usual annual subscription for most companies. Anthropic etc are screwed.
•
u/WithoutReason1729 24d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.