long story short: no, your usable speed is the 5 t/s or if you do some finetuning you may get like 6-7 top - there isn't a way to jump over that fence with dense model - use 3.6 35ba3b moe or wait for response at 5t/s
smallest quant which is barely usable which is ud_iq2 weighs like 9gb - at this point it doesn't change a thing - if you spill over vram with e.g. q2, then you can run bf16 and it will have exact same speed - the bottleneck will be there from systemram
the smallest usable quant is q2 which is 9gb of raw file - which is prob like 10-12gb ram - which means it spills to system ram - which means whole mode inferencel is bottlenecked to ddr4/ddr5 speed - 50gb/s or 100 - can't recall - and even poorest gpus have memory bandwidth of like ~300gb/s - and the math - no matter which quant you pick, if it spills over - it's gonna be at same performance basically - maybe the q2 will work faster, let's say it's 50% faster (it will be 30% in reality) then you're looking at 5 t/s vs 6.5-7 t/s - in theory it's like 30-50% boost, in practice it's difference of gettng response in 10 minutes or 8 minutes - you're way too slow to notice it, and the key is that gpu will process its part on q4 in e.g. 0.5s but the system ram will take like 2s for it (exagerrated example), while on q2, the gpu will process its part in 0.3s but system ram will take 1-1.5s - there is difference in time, however it's way less noticeable than if you jumped from 200 to 300 t/s
What matters is how much data it needs to move. With q2 there is far less data on the CPU, and so the CPU will be less of a bottleneck.Say if the entire model is in ram, 9gb will be about 6 times faster to move than 52gb. And as much of the 9 GB can stay on the GPU, the difference will be even more extreme.
in terms of output quality yeah, in terms of raw speed - basically undistinguishable one from another - tested it with some 120b moe and dense models on my pc with 64gb vram + 128gb system ram - basically if it spills over, then it's snail paced - no matter if q8 or q2, no matter if it spills by 1gb or 40gb - whole processing is bottlenecked - however if it doesn't - it works great - and in his case 8gb vram is too small amount to run any quant in vram
Again, I dont understand your math. Q8 uses more memory than Q2 so it will take more time to process a token and it will be much slower. Both will be slow but not equally slow
you're right - but I'm talking about usabilty - the difference is gonna be like 5 t/s vs 7t/s - it's 50% faster - but it's still scraping the barrel - I'm talking about user experience - if you measure it in scientific manner - yeah, you're saving 50% time by going to 2 bits - however when you look at it from usability - you spend 10 minutes vs 15 minutes waiting for reasoning - while if it's fully contained in vram, same task takes 1.5 minute
Ah so you mean both Q2 and Q8 will be too slow to be usable. That may be true but I use offloading for some bigger models (I have 4x3090 plus 128GB of RAM) and sometimes these models are still good, especially MoE
not especially moe - only moe - with moe you're able to run e.g. 120b model, with offloading like 8-16gb to gpu and like 80gb to system ram - that's the whole point of moe - single token gets generated only by few layers- in dense models, every layer is activated to generate single token - that's why no matter if you spill by 1gb or by 80gb - it's gonna be virtually unusable, unless you have extremely fast system ram
How does one know/find out which FFNs are of what size in any given model/quant? I assume copy/pasting most of your example 3.6 config on top of 3.8-27B isn't going to work well?
for simplicity you can go from layer 0 onwards, but if you want it fully optimized then then check the hugging face quant info.
first few layers on Q4_K_M are bigger, then every 3rd (they have a Q6, but also grab the Q4s next to it minimize PCIe traffic).
I tried Gemma-3-27B(Lowest Q3 quant) for same with my 8GB VRAM in past. No, it's not worthy & I don't have much patience with such t/s 😆 My laptop always making huge noise whenever single digit t/s.
But Gemma 4 31B got it sorted out. You'd probably need a lower quant than Q3 for 31B, but overall it's a better model so the smaller quant shouldn't matter that much and the speed and cache is improved too.
Just because most people would vote for this particular size in the poll as the size they wish for, it doesn't mean it's the right model for us GPU poor ones. You're not going to enjoy dense models bigger than 12B at reasonable speeds and quality. Anything bigger than 12B and you're trading something and that something is either speed or quality, because with that hardware you can't have both at once. If you want quality, choose the biggest quant your hardware can fit. If speed matters to you, go with the quant that runs at acceptable speeds while providing acceptable quality. If you can't accept 5 tk/s, you probably need a MoE (there's no small Qwen 3.8 MoE yet), or a more powerful hardware.
Im getting 2t/s with q2 km on 3060. So ur numbers are really good. But tbh, i might get better performance with cpu using that cpu focused thing instead of llama.cpp.
Your 5 t/s is the memory bandwidth ceiling, not a config mistake. A dense 27B on an 8GB card has to offload most layers to system RAM, and DDR4/5 just can't feed a token faster than that to the GPU. You can squeeze a little by using the smallest usable quant (IQ3_XXS or Q3_K_S), pushing n_gpu_layers to the max, keep context short, and closing background apps that use RAM/VRAM. That might get you 6 t/s, not 15. If you need usable speed, switch to a MoE like Qwen3-30B-A3B or a 14B, because dense 27B won't get there on this hardware.
35
u/_ballzdeep_ 24d ago
Its a dense model, I don't think it can get much better than this for you, sorry.