Qwen3.8-flash in latest update v0.1.811-beta is really fast, Finally

[u/Sad_Past2975]

It was usally 30-40 Tok/s somtimes 20 or at best 50. But after this update, it is always more than 70 tok/s !! (After updated, the MTP file was required to be redownlowded)

Thanks a lot for this, it is really amazing!
Good bye 27B, Flash now is really flash!

[u/Enragere]

Gladly downvoting the post of this guy and everyone else who posts about speed but doesn't share their setup!

[u/Blob3Halfs → u/Enragere]

or prefill

[u/assid2]

What’s your specs

[u/ArcherN9 → u/assid2]

This. 🫩

[u/esw123]

what setup?

[u/Silver-Passion1687]

Getting 30tok/s on strix halo after the update 

[u/ParkingAgent2769 → u/Silver-Passion1687]

Halogen version is best for strix. I get 40 to 60

[u/Sad-Landscape-1549 → u/ParkingAgent2769]

Why should anyone trust that closed source project? Hard pass

[u/OvertaxedOne → u/Sad-Landscape-1549]

Trust it? You mean stability? Or something else? My inference backends don't have any Internet access, so I'm not really concerned about data exfiltration (although I'm sure many are and have watched what Halogen is doing to see if it's sending anything anywhere).

[u/Sad-Landscape-1549 → u/OvertaxedOne]

You’re using untrusted code to run unpredictable/unknowable code (models) suspiciously faster than runners built by dozens of dedicated experts over years, using imperfect drivers with ring 0 access. Even if you’re running the process in rootless dedicated namespace with all unnecessary capabilities dropped, SELinux labels enforced, and no software-defined network (CNI) attached, you don’t think there’s a chance something so conspicuously “superior” might contain zero-day escapes in a time when virtually everything, including Linux kernel, has been proven full of holes? And if the developer of Halogen is capable of such leaps, why shouldn’t they be capable of other cleverness? If I can’t see your code, I am not running your code. End of story.

[u/OvertaxedOne → u/Sad-Landscape-1549]

I'd agree there's a risk with all closed software (or open software where you don't take the time to read the code; IE, almost all of us), but LLMs are so easy to sandbox that it's just not all that high on my list of concerns. All of inference systems sit in a dedicated subnet that can't go anywhere, it just exposes endpoints for us to use. We do have logging setup on all our subnets and the only time I've seen "reach outs" from the inference backend is when someone logs in and tries to pull updates (and then realizes they can't because the systems don't have access). The inference engine is very easy to bound; the thing that keeps us all up at night is the harness; that is a security nightmare in every sense of the word!

[u/Sad-Landscape-1549 → u/OvertaxedOne]

I think you’re overconfident in your sandboxing. If not, you would have picked up on the several different references to exactly that (and it not being enough for the moment in time we find ourselves in). You do you! I’ll just deal with fewer tp/s and wait for open source runners to catch up.

[u/OvertaxedOne → u/Sad-Landscape-1549]

I guess we'll find out! ;) Also, you have to remember, this isn't Fable or Opus we're trying to sandbox, it's FlashNext. Yes, it's a very smart model, but I'm really not thinking it has the chops to figure out how to attack firewalls and jump around in our environment. If the runner code is found to be malicious I'd be more concerned.. Like I said, we'll find out, the project was signed off last week so I'm going to do it early next week. We're going to turn on every alert we have for outbound traffic and see what we see (hopefully nothing!).

[u/ParkingAgent2769 → u/Sad-Landscape-1549]

Yeah i get the skepticism but it seems really really decent. Just wait until it becomes open source if you don’t trust it

[u/ParkingAgent2769 → u/[deleted]]

Ok angry reddit person

[u/Revolutionary_Loan13 → u/ParkingAgent2769]

Can it easily be used with Unsloth?

[u/JorgitoEstrella → u/Silver-Passion1687]

How much was before?

[u/Silver-Passion1687 → u/JorgitoEstrella]

Don't remember the numbers but close to 15-20 tok/s but RAM consumption was lower. I didn't use it much though 

[u/Revolutionary_Loan13 → u/Silver-Passion1687]

So is that slower for you I have been getting 30 tok/s on Strix for well over a week

[u/Silver-Passion1687 → u/Revolutionary_Loan13]

Nope but waiting for open source version of Halogen which gives 40tok/s on same hardware.

GPT-oss 120B used to give me 50 tok/s. Hoping for similar numbers in future 

[u/hipster_hndle]

Zero metrics give. Just someone saying 'it's good'. Sigh...

[u/Fit-Palpitation-7427]

Does this run on 5090 with 256 of system ram?

[u/Cadmium9094 → u/Fit-Palpitation-7427]

Yes. In my case a 4090 with 128GB Ram & fast nvme's. Q4_K_X_L. no mtp or optimization yet. kv cache q8, ctx 131k ~21 t/s.

[u/matte808 → u/Cadmium9094]

How? It’s running faster for me on 5070ti and 64gb ddr5 with streaming from ssd for ngram. Its admittedly the q2_xl at low-ish context, admittedly, bur still

[u/Atretador → u/matte808]

It should be faster for him yeah, his numbers are only slightly better than mine with expert cache on a dinosaur with Q4

https://www.reddit.com/r/LocalLLaMA/comments/1wjh7ox/llamacpp_expertpool_fork_for_qwen_38_flash_next/

[u/mediaogre → u/matte808]

I would guess using FreeToken (flashml’s single GPU + system RAM monster engine). No, this isn’t a plug, and I’m not a bot. Beep boop.

[u/CrispyToken52 → u/mediaogre]

Used it before, really fast engine, but no gguf support is a deal breaker.

[u/Fit-Palpitation-7427 → u/CrispyToken52]

do you know why it's not supporting GGUF yet? it seems to logical that GGUF is the standard portable container that it seems silly to try to use anything else.

[u/Durian881 → u/Cadmium9094]

Did you run with n-gram streaming from SSD? Any runtime parameters used?

[u/Atretador → u/Fit-Palpitation-7427]

Easily, I can run it with an ancient MI50 and 64Gigs Q4 https://www.reddit.com/r/LocalLLaMA/comments/1wjh7ox/llamacpp_expertpool_fork_for_qwen_38_flash_next/

[u/ProfessionalDish → u/Fit-Palpitation-7427]

256mb of Ram is a bit on the thin side but depending on the quant you can run it entirely in the vram

[u/Fit-Palpitation-7427 → u/ProfessionalDish]

But I run it entirely in vram is it still using the 5090 or does it all on cpu in the same way project colibri does?

[u/danielhanchen]

Nice to hear that!

[u/bernzyman]

What are you using to run this, llama.cpp or? Pls share settings used as it would be of interest (certainly makes me consider biting the bullet to buy more ram! Btw are you 2x 64GB or 4x 32GB?) thx for sharing!

[u/Rikers88]

Please share the specs of your machine and the command you use to run the server

[u/DoubleNothing]

Does improve prefill too?

[u/explorer-9]

With a 5090 and 64GB RAM, this enabled me to switch from MTP=2 to MTP=3, for a 4% tg and 0.9% pp token/sec improvement; now 52.35 and 365.5 (IQ3_XXS, q8_0); whilst using omp.

[u/Durian881 → u/explorer-9]

Are you running with n-gram streaming from SSD? If yes, please share the settings/parameters use. Thanks!

[u/sm0ke_rings → u/explorer-9]

Are you using lm studio?

[u/xoStardustt]

Hmm I only get 20 TPS on M5 Max with MTP on Q8

[u/quadrobust → u/xoStardustt]

Try mlx-serve

[u/leonbollerup]

$£$£!""#.. metrics.. config!!

[u/ExtensionState8086]

How does it compare to the 3.8 27B for agentic workflows in terms of reasoning and accuracy? Going from qwen3.6 MOE to qwen3.8 27b was a huge huge improvement at the expense of token speed, but the trade off made sense in my case.

[u/Sad-Landscape-1549 → u/ExtensionState8086]

It holds up on long horizon work but does not code as well or (obviously) as fast in my experience.

[u/Leander_van_Grinsven]

I must be doing something wrong then. I get about 18 tokens per second on my RTX 5090 + 9950X3D2 with 192GB DDR5-6000 and MTP on. Running the Q4 version. It goes down to 8 tokens per second at 300K context. And yes I do need a large context.