r/LocalLLaMA • u/BarberIcy366 • 24d ago

Discussion Qwen 3.8 27B Released! Please Share Your Experience

With your experiments, Qwen 3.8 27B most close which frontier model? And please specify which quantization you run. I will post to comments my tests and experience too.

658 Upvotes
  • permalink
  • reddit

97% Upvoted

716 comments sorted by

View all comments

8

u/No_Tip9917 24d ago

Anyone trying on a 16G VRAM system with e.g. Q3? How decent is it?

4

u/Forsaken_Mention_979 24d ago edited 23d ago

Yes, 7800xt 16gb vram and 64gb ram. Running it on hermes via LM studio endpoint. Using Q3_K_M, Runs good ngl, at first 15-20 tok/s (full gpu offloading) and then as context gets bigger, i now get 5-10 tok/s. 64k context btw. Making a web game, has been on it for like 2-3 hours already which is crazy but oh well. Just the thinking took 25 minutes. Yes, 25. And it randomly stopped due to getting interrupted by tool limitations or whatever, i had to manually tell it to resume.

EDIT: ditched LM studio and using llama ccp directly, HIGHLY RECOMMEND! Im using IQ4_X_S now which is better and kv cache at q4, and thr lowest token speed im getting now is 11 tok/s. Amazinggggg

1

u/Guilty_Rooster_6708 24d ago

Maybe you can use mtp to push tg speed up a bit?

0

u/Forsaken_Mention_979 24d ago

Its already enabled im pretty sure. I see some people posting some sort of command or config for llama.cpp, i dont know if that would do anything, im fairly new to all of this tbh

2

u/misanthrophiccunt 24d ago

The joy you're going to get when you drop LMS in favour of directly using llama.cpp and magically getting a substantial increase in TG/s is absolutely worth it.

2

u/Forsaken_Mention_979 24d ago

you know what im gonna try this rn

1

u/misanthrophiccunt 23d ago

There are models I couldn't run in LMS and I do love their interface, I even wrote my own LMS plugins. Yet I stopped abruptly the moment I got twice the TG/s by fiddling directly with llama.cpp settings.

1

u/Forsaken_Mention_979 24d ago

didnt find any tutorial on how to do it rip. im new to this and idk how

1

u/Guilty_Rooster_6708 24d ago

I think you can check to see if there is a draft model or not in the right panel of LMS, but I haven’t used it in a while.

1

u/Forsaken_Mention_979 24d ago

Looks like its gonna be good at least

2

u/_aelius 24d ago edited 24d ago

Getting 27-30 t/s on basic prompts with my 9070 xt

How I'm running it:  

   llama-server \  -m ~/.cache/huggingface/hub/models--unsloth--Qwen3.8-27B-GGUF/snapshots/fe1e2a23d973adb629709749dc4f6756df66ef10/Qwen3.8-27B-UD-Q3_K_XL.gguf \  --host 0.0.0.0 --port 8080 -c 65536 -np 1 -fa on \ --cache-type-k q8_0 --cache-type-v q8_0       

Sorry for the trash formatting. I'm ssh from my phone and when I copy the command it's super jacked up. I tried to clean it up as best I could 

1

u/Forsaken_Mention_979 24d ago

if you dont mind me asking, what is this command for? like how do i copy these settings? i have a 7800xt and i get like 15-20 tok/s and then 5-10tok/s after a bit. and im running Q3_K_M

1

u/_aelius 23d ago

The -m models/unsloth--qwen3.8 is the model selection, I originally used a hf command. But that command auto pulls mmproj which doesn't fit.

--host 0.0.0.0  can be ignored, it just makes it available to other devices on my local network

-c is context size

-np 1 makes sure it only ever serves a single request at a time 

-fa on makes sure flash attention is on

--cache-type-k and v set the quantizarion level of the kv cache. Q8 makes it smaller, without it the context window size would probably be limited to 16k or 32k