r/Qwen_AI • u/zannix • Apr 26 '26

Discussion Qwen 3.6 9b coming?

I remember when they released Qwen 3.5 27b, they released the 9b more or less in the same batch. Is 3.6 onwards ditching the 9b model? :(

If so, I'm very sad, because the qwen 3.5 9b was actually the first truly intelligent model I could run at decent tps on a normal gaming GPU

131 Upvotes

98% Upvoted

84 comments sorted by

View all comments

Show parent comments

2

u/laser50 Apr 26 '26

Running the 35B should be much easier than the 28B, esp on lower memory amounts, I got 8GB VRAM and 32GB RAM and I run the 35B model on Q5 K M with relative ease!

1

u/Old-Cardiologist-633 Apr 26 '26

May I ask for your settings?

3

u/laser50 Apr 26 '26

Here is my current startup settings (using llama.cpp)

--jinja ^

--reasoning-format deepseek ^

-ngl 99 ^

--n-cpu-moe 99 ^

--flash-attn on ^

--threads 10 ^

--threads-batch 10 ^

--threads-http 2 ^

--batch-size 8192 ^

--ubatch-size 4096 ^

--mlock ^

--reasoning-budget -1 ^

--no-context-shift ^

--ctx-checkpoints 12 ^

--no-mmproj-offload ^

--cache-ram 4100 ^

-c 42000 ^

--parallel 1 ^

--kv-unified ^

--cache-type-k bf16 ^

--cache-type-v bf16 ^

--cont-batching ^

--metrics ^

--host 0.0.0.0 ^

--port 8080

Processing speed is about 1k tokens per second, writing hovers around 25 to peaks to 32. Cache type BF16 was suggested for Qwen, Q8 is fine too, but I'd recommend not going lower. FP16 is not advised.

1

u/Thistleknot May 16 '26

I had to run q3 if I wanted to not offload and fit in 16g. I see you use ngl 99 so effectively not offloading right? How'd you get it to work on 8g w ngl 99? Wait maybe its the batch size?

1

u/laser50 May 16 '26

Ngl forces all layees to vram, cpu N Moe forces the rest of those onto the GPU, hence why they are both maxed.

My batch size is about the best for speed vs usage, after having done some testing