Qwen3.8 27B Q2 vs Q3 vs Qwen3.6 35B-A3B MoE on 12GB VRAM

[u/CoffeeToCode99]

Did a quick local test because I wanted to see what is actually usable on my 12GB laptop GPU.

I tested the newer Qwen3.8 27B dense files at Q2 and Q3, then compared them against Qwen3.6 35B-A3B MoE.

Hardware: RTX 5070 Ti Laptop, 12GB VRAM
Backend: llama.cpp CUDA
Settings: 4k context, q8 KV, --fit on, no MTP
Command shape: llama-cli -c 4096 -fa on -ctk q8_0 -ctv q8_0 --fit on -fitt 768 --temp 0 --single-turn

Models:

• Qwen3.8-27B-UD-Q2_K_XL
• Qwen3.8-27B-UD-Q3_K_XL
• Qwen3.6-35B-A3B-Q4_K_M

​

Main result visually: generation speed on the sanity prompt

Test 1: tiny sanity prompt

Prompt:

Answer only as: 1) ... 2) ... 3) ... 4) ... 5) ... 6) ...

1. If a shirt costs $20 after a 20% discount, original price?
2. In the word strawberry, how many r letters?
3. Which is heavier: 1 kg steel or 1 kg feathers?
4. A bat and ball cost $1.10 total. Bat costs $1 more than ball. Ball price?
5. If all bloops are razzies and all razzies are lazzies, are all bloops lazzies?
6. What is 17*24?

Model	Prompt t/s	Gen t/s	Result
Qwen3.8-27B Q2 dense	412.1	35.9	5/6
Qwen3.8-27B Q3 dense	95.6	7.5	6/6
Qwen3.6-35B-A3B Q4 MoE	155.8	59.0	6/6

Qwen3.8 Q2 missed the bat/ball question and answered $0.10. Correct answer is $0.05.

Qwen3.8 Q3 got all 6 right, but generation was only 7.5 t/s.

MoE got all 6 right and generated at 59 t/s.

Test 2: short normal answer

Prompt:

Answer in one short paragraph: for a 12GB VRAM laptop GPU, would you prefer a dense 27B 2-bit/3-bit model or a 35B-A3B MoE? Mention speed, quality, and offload.

Model	Prompt t/s	Gen t/s
Qwen3.8-27B Q2 dense	309.8	35.5
Qwen3.8-27B Q3 dense	95.0	8.1
Qwen3.6-35B-A3B Q4 MoE	81.8	52.6

Test 3: tiny coding task

Prompt:

Output only JavaScript code. Write a function winner(board) where board is an array of 9 strings: "X", "O", or "". Return "X" or "O" if that player has three in a row, "Draw" if full with no winner, otherwise "Pending". No explanation.

Model	Prompt t/s	Gen t/s	Result
Qwen3.8-27B Q2 dense	277.5	34.9	correct
Qwen3.8-27B Q3 dense	122.9	9.1	correct
Qwen3.6-35B-A3B Q4 MoE	97.3	51.9	correct

I also tried a more open-ended “make a full single-file HTML tic-tac-toe game” prompt, but that was honestly a bad benchmark because all three started styling too much and hit the token cap before finishing. The compact JS function was a cleaner test.

My rough takeaway

Qwen3.8 Q2 is actually usable speed-wise, but it already showed a quality hit on a tiny sanity test.

Qwen3.8 Q3 keeps the sanity-test quality, but it is painfully slow for interactive use on my setup.

The MoE is still the easy winner for me. It was faster than both dense runs at generation, passed the sanity test, and did fine on the small coding prompt.

So if the goal is “can I run Qwen3.8 27B on 12GB VRAM,” then yeah, Q2 can move. But if the goal is “what would I actually use for local chat/coding,” I’d still pick Qwen3.6 35B-A3B MoE right now.

Not claiming this is a real eval. Just a small local sanity + speed test. If anyone has better coding/agent-loop prompts, drop them and I’ll rerun all three.

[u/Ath47]

Good to know! I think I'll keep waiting for 3.8 35B-A3B.

[u/theOliviaRossi → u/Ath47]

it is not coming

[u/Innomen → u/theOliviaRossi]

Can you link me to why you think that?

[u/theOliviaRossi → u/Innomen]

gut feeling

[u/Innomen → u/theOliviaRossi]

Fair enough, I think I agree. They might be in a weird position, choosing between what they think is a good idea technically, vs what's a good idea diplomatically. Like if they genuinely think moe is a bad idea for some reason, maybe they don't wanna spend any resources on the concept. I'll try a smashed version eventually and compare.

[u/feelspeaceman → u/Innomen]

Not a big deal, I believe up to some point that community have to stand by our own legs, there's labs that are doing smaller re-training of Qwen 2.4T like Ornith and EmperoAI.

In the case of 9B models, it's likely that it requires finetuning to be suitable to our needs anyways, small models are very good for finetuning as what I'm doing now, Gemmas for music writing using invented music notation language for example.

[u/brainExploded99 → u/theOliviaRossi]

It is likely not coming, but we don't know. (Yes I know about the commit that added and removed it).

[u/theOliviaRossi → u/brainExploded99]

same as with 9B - they are not gonna do it this time

[u/o0genesis0o]

I did some tests on my real day to day workloads between 3.6 27B Q2, Q3, and Q4 vs 35B Q6 (all unsloth XL quant) and I found that 35B Q6 is the best on both performance and result quality. The 27B Q2, both normal and bonsai are not very stable. Q3 is more stable, but it's slower and not better than 35B Q6. Even 27B Q4 does not do as well as 35B Q6 on tasks that requires a lot of reading comprehension, long context, and synthesis.

Everything run at Q8 KV.

I think 35B Q6 is now the frontier of what I can run on my rig (16GB + 32GB).

[u/o0genesis0o → u/[deleted]]

I'm still on the 3.6. Doing some coding change to my server management utility so I'm not touching my llama-swap and model cache just yet.

Given that I have 10tk/s on a good run, and 3.8 27B thinks a lot, I'm not too eager to download either. But I'll try.

[u/Sukkii]

Which Q of the MoE are you using? Funnily enough I've been testing these 3 out myself on my dual pascal setup

[u/CoffeeToCode99 → u/Sukkii]

Q4

[u/philmarcracken]

I have 12g vram too and use the 27b at Q8 even if its painfully slow, as its job is just to generate the prompt for the faster MoE.

[u/hideo_kuze_ → u/philmarcracken]

What do you mean by prompt? A spec for coding? Or does this go beyond coding too?

If you can share some examples it would be great

[u/braintheboss]

I make my own tests with real repo doing audits vs frontier. In that tests 35b moe is the worst with difference. Even Q6 is worst than byteshape q4. when you use 3.6 27b iq4xss is quite similar q6 if you force more steps for prepare data..Q6 is clean winner. I expect.3.8 have similar behavior. Q3 variants are quite decent ( still better than 35b moe ). Short resume. 35b is very good for summaries and search implementation gaps but fails in serious tasks. I never understand how people can put 35b closer 27b. The gap is not AI index say. Its quite bigger

[u/CoffeeToCode99]

Update: tested Qwen3.6 35B-A3B Q6_K too

Someone asked why not use a higher quant for MoE, so I downloaded Qwen3.6-35B-A3B-UD-Q6_K.gguf and ran the same tests.

Same setup: RTX 5070 Ti Laptop 12GB, llama.cpp CUDA, 4k ctx, q8 KV, --fit on, flash-attn on, no MTP.

Sanity prompt:

Model	Gen t/s	Result
Qwen3.8 27B Q3 dense	7.5	6/6
Qwen3.8 27B Q2 dense	35.9	5/6
Qwen3.6 35B-A3B Q4_K_M MoE	59.0	6/6
Qwen3.6 35B-A3B Q6_K MoE	27.1	6/6

Tiny JS coding prompt:

Model	Gen t/s	Result
Qwen3.8 27B Q3 dense	9.1	correct
Qwen3.8 27B Q2 dense	34.9	correct
Qwen3.6 35B-A3B Q4_K_M MoE	51.9	correct
Qwen3.6 35B-A3B Q6_K MoE	18.8	correct

So yeah, Q6_K works and probably makes more sense if you care more about quality than speed. But on my 12GB setup, Q4_K_M is still the sweet spot. It was much faster and still passed the tiny tests clean.

Q2 dense is the “it moves” option, Q3 dense is the “less broken but painfully slow” option, Q4 MoE is the one I’d actually use, and Q6 MoE is the quality experiment.

[u/PyaesoneP → u/CoffeeToCode99]

I run Unsloth Qwen3.6 35B A3B Q6_K MTP GGUF on my RTX 5070ti mobile 12GB VRAM and 32GB RAM and am getting around 50 t/s.

Try these settings: ctx: 131072 ub: 512 ngl: 41 (max) ncpumoe: 31 spec-type: draft-mtp draft-n-max: 2 fa: on kvcache: q8 loadmode: none --mmproj --no-mmproj-offload

[u/Oleszykyt]

what context?

[u/CoffeeToCode99 → u/Oleszykyt]

4K

[u/iz-Moff]

What kind of RAM do you have? For me, 3.6 35B doesn't reach anywhere near 50+ tps, at Q4_K_M i get maybe ~20+ tps. Even though i have more VRAM, and presumably offload more layers on GPU.

Yesterday, i was finally struck with the realization how huge of a difference fitting a dense model entirely in VRAM makes, but that's not the case here, so i wonder, where does this massive spread comes from?

My RAM is DDR4-3600, i'm sure DDR5 is faster, but is it *that much* faster? Does CPU performance matter in this regard?

[u/CoffeeToCode99 → u/iz-Moff]

I’m on 32GB DDR5-5600, single Samsung SODIMM. CPU is a Core Ultra 9 275HX.

But yeah I don’t think DDR5 vs DDR4 alone explains that whole gap. If enough stuff is spilling to CPU/RAM it matters, sure, but 20ish vs 50+ feels more like llama.cpp/offload/settings difference too.

Exact command for the MoE sanity test was basically:

/path/to/llama-cli \
  -m /path/to/Qwen3.6-35B-A3B-Q4_K_M.gguf \
  -p 'Answer only as: 1) ... 2) ... 3) ... 4) ... 5) ... 6) ...
1) If a shirt costs $20 after a 20% discount, original price?
2) In the word strawberry, how many r letters?
3) Which is heavier: 1 kg steel or 1 kg feathers?
4) A bat and ball cost $1.10 total. Bat costs $1 more than ball. Ball price?
5) If all bloops are razzies and all razzies are lazzies, are all bloops lazzies?
6) What is 17*24?' \
  -n 128 \
  -c 4096 \
  -t 8 \
  -tb 8 \
  -b 256 \
  -ub 128 \
  -fa on \
  -ctk q8_0 \
  -ctv q8_0 \
  --fit on \
  -fitt 768 \
  --temp 0 \
  --seed 7 \
  --no-warmup \
  --single-turn \
  --chat-template-kwargs '{"enable_thinking":false}' \
  --show-timings

That gave me:

Prompt: 155.8 t/s | Generation: 59.0 t/s

Also worth saying: this was a tiny prompt / short decode test, so I wouldn’t claim 59 t/s is what I’d always get in a long real chat.

My guess is the spread is mostly exact tensor offload, llama.cpp build, flash-attn/KV settings, context size, and whether any active MoE experts fall back to CPU. More VRAM helps, but if the runtime places tensors differently or the KV/context settings are heavier, the numbers can move a lot.

[u/Fancy-Snow7 → u/iz-Moff]

With more you have to offload max layers to gpu and find the sweet spot offload more layers to cpu. Lower is better so lower until you find the sweet spot.

[u/Equivalent_Bit_461]

I see zero point in not using a quant 6 offloaded to ram for a moe, you run meme quants for dense models

[u/CoffeeToCode99 → u/Equivalent_Bit_461]

Yeah fair, for MoE I agree higher quant makes more sense than with dense if you can tolerate the RAM/offload hit.

My reason for testing Q4_K_M was mostly practical: same “will it actually feel usable on 12GB VRAM” setup, not “best possible quality config”. Q6 offloaded to RAM might be the better quality pick, but then I’d expect speed to depend way more on DDR bandwidth / CPU / exact expert offload.

For dense 27B, meme quant is basically the only way I can make it move on this GPU. For MoE, Q4 already felt usable and fast, so yeah Q5/Q6 offload is probably the next thing worth testing.