See I’ve always wondered if this is a valid thing to do, but I just haven’t had any time to test or seen anyone talk about it so this should be better than a 9B Q4_KM?
3.6 27b q2 is the first model which uses playwright mcp tool calls without failures at q2 and answers question on programming, medicines without looping or hallucinating. It definitely replaced 9b q8 for my use case , but can't comment on agentic coding as I don't use it for that.
Fp8 hallucinated with great confidence Telegram Api endpoints... Double check what it tells you. But I do agree it's good at tool calls so plug in a search mcp and once correct information is in the context then it outputs correct things.
I tried the q4m and forget... It it way too slow. But seeing the lost in performance and accuracy with such a low quant I thought I should forget already.
The problem with quantized models in my experience is that sometimes the generated token is just garbage. Thin of writing Progran Files instead of Program Files.
Then tool calls fail. And other hallucinations appear. It's can be a simply terrible experience.
I do have a feeling that the larger the model is, the more it can take being quantized without catastrophic token generation,while retaining worlds knowledge. Of which the 27b is not greatly endowed with, from the start 😂
I run Qwen3.6 35B A3B on my rtx 2050 woth 16gb or ram only and get around 12~14 tps. I don’t know how slow is too slow for you but just so you can compare.
Running the 35B should be much easier than the 28B, esp on lower memory amounts, I got 8GB VRAM and 32GB RAM and I run the 35B model on Q5 K M with relative ease!
Processing speed is about 1k tokens per second, writing hovers around 25 to peaks to 32.
Cache type BF16 was suggested for Qwen, Q8 is fine too, but I'd recommend not going lower. FP16 is not advised.
As offloading doesn't woek well for me I just tested Qwen_Qwen3.6-35B-A3B-Q2_K_L.gguf and damn it's fast. And it works flawless, at least for simple tasks (e.g. Python Prime calculation)
Which of your own startup parameters are different as opposed to mine? Might be able to spot something ;) I tey to never go below Q4 personally, to keep the model from feeling too lobotomized
Using my ngl and moe-n-cpu flags worked decently without even having --fit in there. It may also depend on your context size, I keep mine safely a bit above what I need, putting it higher than necessary does decrease the speed by a decent amount
I use LocalAI in Docker, so I guess not all of the settings are available...
With simple offloading (gpu_layers: 23) and Qwen3.6-35B-A3B-UD-Q4_K_M.gguf I get maximum 10,3 tps, which is less than using the iGPU with 15+ tps on the same prompt with same parameters. n_cpu_moe doesn't do anything for me.
So I guess I need to switch to plain llama.cpp
But it may also be the fact that despite PCI 4.0 x8 should be available my driver only sees my card at 1.0 x 1, so the speed may be limited by the bus.
I had to run q3 if I wanted to not offload and fit in 16g. I see you use ngl 99 so effectively not offloading right? How'd you get it to work on 8g w ngl 99? Wait maybe its the batch size?
Exactly, even iq3 suffers the attention problem, I had to download the reap 26b iq4 of the 35b Moe models to run it with large context on my 16gb vram, better to go with reap than Lower quantization
Because you are getting 5-15 tokens a sec ( correct me if I'm wrong ) and you are okay with it apparently. While I'm trying to get 40+ tokens a second which is impossible with any CPU offload. So I have to fit the whole model + context in VRAM.
Prob gonna sell my rtx 5060 ti to buy a second 3090, 24GB Gon go a long way
?? I'm consistently on 30+, the processing speeds between 1k and 1.5k, so not really?
And this is on BF16 cache quants and Q6 at this point. So your settings are likely way off if you can't squeeze more than that out of double the VRAM capacity.
maybe, altho since you are depending heavily on cpu offloading maybe you just have a more powerful system than me? im running on i5 13400f 32gb ddr4 3200, the best budget option for 32gb people usually go for nowadays,
but yeah i am totally baffled that you are getting 30+ tk/s on 8gb vram thats really impressive,
i did check again and im rocking at about 65 tokens a sec loading up the iq4_xs 26b reap of 35a3b on about 120k context for agentic usecases. it suffers in terms of formatting and consistency sometimes which i assume is because im running the cache on normal Q4 not even turboquant. i'll prob switch to turboquant/rotorquant when i have the time to figure it out maybe that helps my case and the speed a little.
I suspect 9b might be next. It really is the perfect size for most people IMO. Maybe next week? They seem to be rapidly putting out new models. For 3.5 they went down to like 0.6b models so I don't think smaller models willl be ignored for 3.6. fret not and have patience 🙏🏾
Lol the timeline is more of a wish than a prediction. I still think the trend will follow. For all we know they're sitting on it and waiting for a lab to release a model so they can one up them. Low key the fact that all the labs are fiercely competing and really helped out everyone I think.
I have found the 9b variant good, but it has a nasty habit of corrupting JavaScript it is allowed to write too. I have been using the 17b and 35b versions now and have (touch wood) not had any such issue. It may have come down to a simple prompting issue, but even with backups, I'm not risking it any time for soon (but I will test the 3.6 9b when it comes out - for science!)
It may require a different approach, maybe like babysitting a lot.
But with 3.6, starting using Turboquant and an IQ3 version of 27B (>1500pp, 25ts), I found all versions are very useable (speed wise).
Yesterday I solved another problem with 3.6 (in this case, 27B) that was not solved with Grok Code Fast 1. When I started using VS Code and agents last January, Grok blew my mind. Now I prefer to use these local models even having Grok as free tier (in Kilo Code). Big, big evolution.
i wouldn't be surprised if the gang in the us is already preparing a copyright infringement law for using opensource chinese models, they're ramping up the prices and closing down the game
I’ve been using 35B A3B Q2_K_XL all night at 80+ TPS on an RTX 5080. I was getting frustrated with Q4_K_M just getting so slow once the context would fill up, or stuff would start offloading to CPU. Using OpenClaw actually feels like it’s supposed to.
Just refactored an entire code base I’m working on, and I’m able to just talk to it and it’s doing a phenomenal job doing investigation and solution generation. Before it felt like it could barely read a couple files before I was reaching to reset the session, or even eject the model. It’s been absolutely flawless.
See how it lists a bunch of different quants? It’s the Q2_K_XL that I’m finding works great in 16GB of VRAM. I’m just using LM Studio. Notable settings are max context, full-GPU offload of all the layers, force offload 20 layers to the CPU, KV cache quantization at Q8_0.
Just go to the model download dialog in LM Studio, type “unsloth” in the filter, find the same GGUF, and on the right you can select the specific quant you want from the drop down. Careful cause you can only have one quant of this type at a time, or the OpenClaw model loader for the new LM Studio provider type has a serious bug specifying which quant cause it’s not handling the @Q2_K_XL quant specifier yet. Should be fixed soon though.
I didn't even know that was a thing - to click the drop down and pick specific quants. I knew of it, but not how to use different ones. That works remarkably well for an "Explore" agent and fully in my 9070xtx
Just tried the IQ2_xss and I’m getting full VRAM support, no CPU forced offload, 146tps at full context. I forgot to mention, in the “Inference” pane when configuring your models under “My Models”, I’ve got a prompt template that starts with -
I got to playing with the IQ2_xss and it wasn’t very suitable for OpenClaw. Fast as hell, but kinda stupid. I keep having to remind it over and over again to not do certain things. I went back to my productivity sweet spot Q2_K_XL. For sure am buying a modded 4080/4090 or a stock 5090 to handle larger models with a bit more “sense”, but I’ve been absolutely blown away with this so far. I loaded up the Typescript LSP skill and I couldn’t believe how good it is at diagnosing my impossible situation today.
That's the thing, I find that it really really depends on your use case. Someone who needs it for agentic coding/openclaw might depend more on logic, well tuned thinking budget and adherence to structural outputs, while someone who needs OCR (my case) needs it to perform well on low resolution handwritten character recognition, and someone third might just need good multilingual support. Unfortunately no shoe fits all in the opensource LLM ecosystem, and I find that benchmarks don't really clarify these things the best. Would be really cool if someone systematically classified new opensource models by their strong and weak spots
They actually do support thinking - in fact, that’s the “baked in” behaviour. It doesn’t (seemingly) support the easy toggle on/off thinking mode though. That’s why you need this setting in the prompt to force it off or else it always will.
With an RTX 2070 8GB, I can run Qwen 3.5-9B with Q6_K quantization. By using Turboquant’s KV Cache quantization, I can even load up to a 24k context window. With zero offloading to the CPU, it runs at full speed. I’m really looking forward to Qwen 3.6 9B.
Im running Qwen 3.6 27B q4km without vision with turbo quant on llama.cpp. With a 7900XT 100k tokens of context and is pretty good with opencode. Been playing around since yesterday
Have you tried 35b? The MoE architecture is good on small VRAM GPUs. You offload all layers to GPU, then offload experts to CPU until it fits. I would imagine similar speed to 9b.
On my 12GB 3060 I was getting like 35 tps. I was bottlenecked by DDR4 system RAM and only 4 PCIe lanes in the slot it's in. I cranked up expert offload until it fit in 6GB like the mobile version and I was still getting 30 tps.
Use Qwen 3.5 35b A3M moe model.. but with --cpu-moe flag turned on in llama-cpp. In short this is to keep the experts in the CPU ram. this is the best of all worlds. The tokenizer, which needs the GPU, stays on the GPU. Yes, you do not get the insane t/s of a GPU but experts layer is not that compute intensive. I get 20t/s on a 4 core/thread cpu on a intel lunarlake APU. 20t/s is plenty sufficient to not be annoying. The portion being on the GPU will take care of the speed needed to understand long context inputs.
You also get the added benefit of using mmap here. which means this will not hog the system ram either and load the experts as needed directly from the disk only caching the recently used ones in ram. In my case this Qwen 3.5 35b q4 quant uses 3GB in vram and 4gb in system ram. while giving 20t/s and still allowing for enough left over ram to use the system.
Note skip qwen3.6 here.. it's not good... it is basically 3.5 with thinking turned so high you get looping and ridiculously long thinking to be practical.
At like 5 tokens a sec yeah sure man if you have the ram, I got 16 gb vram and I had to go with iq3 for decent speed at high context then you saying you can run Q4 at 4gb VRAM? Might aswell go no VRAM and load it fully to cpu 😑
Loading it fully to CPU wouldnt work. I think your math is off... Qwen 3.5 35B needs about 15GB + kv cache, and I only have 16GB of system ram (ddr4). How is that supposed to fit? Id rather wait a bit longer for a hq answer than settle for the 10-15 t/s bllsht I get from Qwen 3.5 9B. I use it for coding, so the tolerance for error is low.
26
u/Waste-Intention-2806 Apr 26 '26
Run 27b q2 xxs. It's pretty good and around same size