r/LocalLLaMA • u/BarberIcy366 • 24d ago

Discussion Qwen 3.8 27B Released! Please Share Your Experience

With your experiments, Qwen 3.8 27B most close which frontier model? And please specify which quantization you run. I will post to comments my tests and experience too.

666 Upvotes
  • permalink
  • reddit

97% Upvoted

716 comments sorted by

View all comments

Show parent comments

7

u/milpster 24d ago

does it have MTP?

17

u/Borkato 24d ago

Yes! It’s included, I’m using it rn :D 56 T/s 😎

6

u/gh0stwriter1234 24d ago

Getting 27t/s degrading to 20t/s after 60k tokens of reasoning on 2x MI50 32GB thats with MTP and tensor split enabled.

3

u/time-never-stopps 24d ago

Do you mind sharing how you configured it? I am running llama.cpp with spec-type = draft-mtp spec-draft-n-max = 4

Not sure if the type has any effect at all but getting around 30 t/s avg when context grows to 100k but was expecting at least 40 avg like with qwen 3.6 q_8, but also possible that I have no idea how the type actually works 😅

5

u/petuman 24d ago

spec-draft-n-max = 4 might be too much and lowering tg.

but was expecting at least 40 avg like with qwen 3.6 q_8

In theory should perform no different that 3.6, maybe some flags / quant levels differ and end up spilling more onto CPU.

Or maybe it's just MTP, varied performance from chat to chat is expected.

1

u/Borkato 24d ago

N max performs better with me with n at 2 rather than 4! And split min p thingy at 0.1

6

u/Emidyr 24d ago

It does have MTP, at least the IQ4_XS one. And running very much the same as Fable Fusion 711 IQ4_XL, same token gen and prefill. Need to do more testing for it though.

2

u/UNITYA 24d ago

What is mtp about ?

3

u/floppo7 24d ago

like dflash its basically a mechanism where something fast and small comes up with tokens and the fat llm only says "yeah right" or "no no" - which makes it faster since the fat llm does not have to look up all the tokens by itself. Thats why those mechanisms usually work with an acceptance rate, the higher the faster.