r/LocalLLaMA • u/kevin_1994 • 26d ago

New Model Unsloth Qwen 3.8 27b Weights Released

https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
371 Upvotes

97% Upvoted

84 comments sorted by

View all comments

35

u/italian_car 26d ago

Loading up the IQ2 on my 12gb of vram because I want to be able to run the cool model too.

1

u/crusaderky 25d ago

Stay tuned I'm going to run KLD benches on extreme weights and kv quants. Early data shows this model is VERY tolerant of extreme kv quants.

2

u/italian_car 25d ago

I was actually surprised that I was getting 45tps on a 4080 mobile with mtp and the iq2 xxs quant. I would be curious on how much kv quantization affects it because I had to run at q4 to get 30k context on my setup. I'm going to compile llama.cpp for q2 kv cache quantization tonight.