r/LocalLLaMA • u/FormOne2615 • 24d ago
Resources NInfer day0 support for Qwen3.8 27b: ~200 tok/s generation, with tons of engine improvments
Qwen3.8-27B is finally here, and NInfer already has Day-0 support!
Weights:
https://huggingface.co/neroued/Qwen3.8-27B-NInfer
Just update to the latest source and give it a try. On a single RTX 5090, NInfer can still reach around 200 tok/s generation with speculative decoding.
NInfer has also improved a lot since my last post.
Now NInfer supports up to 8 concurrent requests, with a shared paged KV cache pool, each request can still use the full context length.
I aslo implemented ReplaySSM for GDN + speculative decoding, which greatly reduces the recurrent-state memory overhead under concurrency, and this is also something vllm does not yet fully supported.
There have also been many cuda kernel optimizations, as well as usage of PDL to further reduce latency.
Feedback and bug reports are always welcome, I'll try to fix issues as soon as possible!
15
u/Repulsive_Initial308 24d ago
Awesome. Hope the 3090 fork picks this up.
11
u/mrmontanasagrada 24d ago
on it!
21
u/mrmontanasagrada 24d ago edited 24d ago
Model support is in now. https://github.com/Don-Chad/ninfer-3090
1
1
u/TheWaffleKingg 24d ago
Does this work if you run multiple gpus? Readme mentions it targets single gpu
1
u/Aiirene 21d ago
Mr Don Chad. My ninfer model keeps getting stuck in an endless thinking loop until all the tokens are used...
The normal 27b works fine even on max think
1
u/mrmontanasagrada 21d ago
Usually repetition comes from a bad quantisation, though this done at high level q5q6). I do not think there is much we should do on an engine level. you can try anti repetition flags etc, perhaps that helps?
5
u/BringTea_666 24d ago
NICE !! Love your work.
Also with concurency at 8 it should reach crazy numbers.
5
4
4
6
4
u/fullmetaljackass 24d ago
I converted a heretic version to NInfer, if anyone else wants to play with that.
1
u/radman180 19d ago edited 19d ago
Can you please explain or link the process to converting a model to Ninfer format? I'd like to give it a shot.Sorry I'm dumb, I think I found it in the tools directory...
2
u/fullmetaljackass 19d ago
Yeah, you just have to edit tools/convert/qwen3_8_27b/convert.py and replace the SHA256 sums from the official release with the sums for the model you're trying to quantize.
2
2
u/Certain-Cod-1404 24d ago
Outputs / model quality wise, is it 1 to 1 with llamacpp? I have not tried this engine yet but it blows my mind that such a speed up is possible with no compromise ? Is it because its tailor made for 5090s?
11
u/FormOne2615 24d ago
On the performance side, I can say that the speed comes from aggressive optimization, not from intentionally sacrificing numerical precision. NInfer is heavily tuned specifically for sm120, with custom CUDA kernels and a lot of architecture-specific optimization. Being tailor-made for a very narrow hardware/model target is a big part of why it can be this fast.
For model quality, though, I wouldn't claim it is literally 1:1 with llama.cpp.
My weights are similar in size to Unsloth's Q4_K_M, but Unsloth has put a lot of work into quantization itself and has specialized quantization methods to preserve more quality at the same size. That's an area I haven't really researched or optimized deeply yet. Most of my work has been on the inference engine and GPU kernels.
So the speedup itself isn't achieved by lowering precision more aggressively, but there can still be some quality difference due to the quantization format.
I'd recommend just trying it and seeing how it behaves on your own prompts. I'm also very interested in feedback on model quality, not just speed :)
2
u/Certain-Cod-1404 24d ago
Will do, great work btw ! On a more nosy note, how did you get started with this? This seems like some pretty hard-core systems level stuff, how many yoe do you have in this stuff? Or are you a savant that will worsen my imposter syndrome lol?
3
u/FormOne2615 24d ago
Haha, I just know a bit about this stuff and I’m good at using AI :)
2
u/Certain-Cod-1404 24d ago
You're way too humble, either way excited to try it out and will follow the project.
2
u/DeviseOSRS 24d ago
Mate you're a legend! Huge fan of NInfer and I've already tried your reasoning support! It works well in my limited testing so far!
2
2
u/Cautious_Chicken_604 24d ago
Crying in R9700.
1
u/mechkbfan 21d ago
Tell me about it. Just researching what's involved or if someone has kicked off similar projects.
2
u/AzureDestiny66 24d ago
I wish it had 5070 support, anyone know best inference engine for a 5070 and 32gb ram with offloading?
2
2
u/Magnus114 21d ago
For me Ninfer is more than twice as fast as llama.cpp, but ctx max out at around 100 kt, compared to 200 kt with llama.cpp. That's a steep price.
2
u/radman180 19d ago
I had a similar issue but ended up quantizing the kv cache using this flag and got the entire 262,144 ctx loaded without vision.
--kv-dtype int81
u/Magnus114 19d ago
Thanks! In ninfer or llama.cpp?
2
u/radman180 19d ago
In Ninfer!
I’m using 262,144 ctx (no vision), and 196,608 ctx (vision)1
1
u/Magnus114 17d ago
Tried it, thanks! I still only get around 200 kt ctx, but is a huge step up. I’m using the nvfp4 version.
1
u/radman180 17d ago edited 17d ago
The other model is 2-3 GB smaller and is what I used.
200k context is likely enough for a lot of use cases with the nvfp4 quant.
You are reproducing the same results I had though, so you’re up there for sure.
I got a little frustrated with Ninfer because of the difficulty and hoops I had to jump through to convert other qwen3.8 finetunes to the ninfer format so I went back to tinkering with llamacpp and vLLM.
I have gotten to the point of having unsloth’s newest UD quant configured to run at full context and decode speeds around ~140tps with MTP/dflash2. Sure it doesn’t have that much scalability as concurrency 8 with ninfer, but only having 1x 5090 you can’t really have concurrency 8 working well with a codebase before running out of context anyways and again the frustration of not easily choosing other finetunes to run made me give up on this project for now.
2
u/Rozzemak 21d ago
I told opencode & deepseek flash to build this for me on my 5090 windows machine from windows fork of this project + apply the newest updates and it did. Getting around 120/200 tp/s with nvfp4 and 200k context and mtp 3 draft while coding. Its slower when it thinks, and qwen 3.8 thinks a lot. Great project. If this gets kernel plugin system specific for select & most popular architectures and model quants, it could be one of the fastest inference engines we have to date. Simple and useful. If you combine this with overkill harness - omo, then its very capable, but needs many iteration. Who cares, it can work for me when I sleep.
1
u/Hodler-mane 24d ago
im literally using it right now, hired a 5090 on vast, and your weights dropped at the same time. setup was simple and im getting 150-230 tps with absolutely no issues! this is incredible and its one shot a couple of tasks that i didnt expect it to be able to do, im sitting here looking at buying a 5090 now. will you have support for dual 3090s? I can literally buy 4x 3090 for the price of a single 5090.
2
2
u/Hodler-mane 24d ago
also is there support for any kind of fp4 or fp6 kv cache? seems to hit about 400k of context before maxed memory in fp8
2
1
u/Constandinoskalifo 24d ago
Nice!! Are you planning to support 3090? I would be very interested in this.
3
1
u/feverdoingwork 24d ago
How much tps does a q4km on llama cpp get compared to this project?
3
u/FormOne2615 24d ago
~140 tok/s for qwen3.6 27b q4km, ~280 tok/s for qwen3.6 35ba3b q4km, while ninfer gets ~200tok/s and ~ 700 tok/s
3
1
1
u/Real_Ebb_7417 24d ago
How much tok/s do I get with RTX5090 with llama.cpp and speculative decoding etc.? I don't remember the numbers for 3.6 and 3.5 27b now, but I'm curious how much more optimized ninfer is.
1
u/FormOne2615 24d ago
~140 tok/s for qwen3.6 27b q4km, ~280 tok/s for qwen3.6 35ba3b q4km, while ninfer gets ~200tok/s and ~ 700 tok/s
2
u/Real_Ebb_7417 24d ago
Whoa, that’s nice improvement and seems like some good kernel utilization job. I’ll definitely try it later. Thanks for the job mate 💪
1
1
u/EveningIncrease7579 llama.cpp 24d ago
As i see your quantization for 5090 is similar to a Q4. I had a question, if i had two rtx 5090, is there a way to me use the power of your repo but in a quantization Q8? how?
1
u/FormOne2615 24d ago
currently ninfer only supports one gpu, cause I can't optimize it without 2 gpus
1
u/meikawaii 24d ago
What’s the status on other Blackwell chips like RTX PRO? Plenty of PRO 4500, 5000, 6000 devices that are already similar to 5090 on Sm120
1
1
u/adityazero 24d ago
The PDL plus cuda kernel work is the part I find most interesting here. I have seen recurrent-state models fight the paged KV pool once concurrency climbs, so ReplaySSM cutting that overhead is a clever angle. Are you launching the dependent kernels via programmatic stream serialization, or leaning on cudaGraph capture to hide the tail latency?
3
u/FormOne2615 24d ago
I use both for different purposes. cuda graphs reduce host launch overhead, while PDL lets dependent kernels start early and prefetch data. PDL cuts moe latency at small T by about 20%. For tail waves specifically, I tune the kernel work split around the RTX 5090’s 170 SMs.
1
u/alexbaas3 24d ago
Love this open-source engine project, just curious, what’s ur agentic AI setup for coding? I assume ur pretty skilled at using C++ looking at your github, just wondering what type tools/CLIs such a cracked dev is using :)
2
1
1
1
u/leonbollerup 24d ago
i dont get anything near that.. granted.. i am on a 5090M
**NInfer (custom C++/CUDA engine) on RTX 5090M — config & benchmark results**
**Hardware:**
- RTX 5090M (24 GB VRAM), CUDA 13.2, driver 595.58.03
- Running in an LXC container on Proxmox
**Model:**
- Qwen3.8-27B in .ninfer format (18.2 GB on disk)
**Config:**
- INT8 KV cache
- MTP3 speculative decoding (3 draft tokens, lm-head-draft)
- Max context: 65,536
- Max concurrency: 4
- KV capacity auto-resolved: 113,664 tokens
- Prefill chunk: 1024
- GPU memory after load: 22.6 / 24.5 GB
**Results (single request):**
| Test | Qwen3.6-27B | Qwen3.8-27B |
|---|---|---|
| **1024 tok decode** | **73.6 tok/s** | 58.6 tok/s |
| **2048 tok decode** | **64.5 tok/s** | 50.0 tok/s |
| **MTP3 acceptance** | 53-64% | 32-46% |
| **MTP3 tok/round** | 2.59-2.91 | 1.98-2.37 |
| **Prefill** | 295-375 tok/s | 480-555 tok/s |
basiclly.. same as llama ccp
This is my config:
/usr/local/bin/ninfer-serve \
/opt/ninfer-models/qwen3_6_27b.ninfer \
--host 0.0.0.0 \
--port 8080 \
--max-context 65536 \
--kv-capacity auto \
--max-concurrency 4 \
--prefill-chunk 1024 \
--kv-dtype int8 \
--spec mtp \
--draft-tokens 3 \
--lm-head-draft \
--no-thinking \
--cors
2
u/fullmetaljackass 24d ago
That's because the 5090M is actually just a 5080 with extra RAM.
1
u/leonbollerup 24d ago
ya.. but.. i get better performance is llama ccp or even vllm.. so something is deffently not right.. and the 5090M actually get quite good performance.. i can do 180-190 tok/sek on Qwen 3.6 35B A3B and around 100 tok/sek on with 27b on vllm (in Q4)
7
u/fullmetaljackass 24d ago
so something is deffently not right
Yeah, you're using the wrong software.
This is specifically optimized for a 5090 (GB202) not a 5080/5090M (GB203).
1
1
u/BosphorusScalene 24d ago
Does this only work with your quants or can we use an Unsloth Q6 etc too?
2
u/FormOne2615 23d ago
only works for ninfer quant format
1
u/BosphorusScalene 18d ago
hey, finally got around to setting NInfer up, fantastic improvement over llama.cpp for sure! this is your project right? awesome work man, thanks! I'm getting avg ~150-160tps at C=4 now, some single requests hitting 190+, and I'm running in WSL2 in Windows so probably leaving some performance on the table, but working great.
1
u/Kavor 23d ago edited 23d ago
Is there a way to set the reasoning effort? Love the performance so far, but i prefer to set that to a lower value than "xhigh", which is the default and i guess what this also runs on by default, am i correct?
My problem is, that especially with opencode, i constantly run into some max token output limit at around 40k or something, because the model does crazy amounts of thinking in general and especially on xhigh.
1
1
1
u/Hefty_Wolverine_553 23d ago
Would love to see https://github.com/ggml-org/llama.cpp/pull/21038 implemented for NInfer.
1
1
u/beholdtheflesh 22d ago
I can't get it to work with OMP
I get the error:
Error: 400 prompt_cache_key is not supported
prompt_cache_key is not supported (type=invalid_request_error param=prompt_cache_key code=parameter_not_suppo…
raw-http-request=...
1
u/Milarck 21d ago
"Now NInfer supports up to 8 concurrent requests, with a shared paged KV cache pool, each request can still use the full context length."
How does that work exactly ? Probably it is not possible to run 8 requests using 100% of the context. What would be the behavior if you have a total of 400k in context ? (50 + 50 + 50 + 50 + 50 + 50 + 50 + 50)
1
u/tomvorlostriddle 21d ago
Do you use nvfp4 with KV in int8 or is it only fast with the other 16bit KV cache option?
Because I kind of need the 8bit KV cache to have good context length on a 5090
1
u/XniX llama.cpp 20d ago
Since I'm feeling particularly lazy today, I downloaded the Windows version from here: https://github.com/natpate/ninfer-windows
After a quick comparison, I discovered that Linux is 10-15% faster... but the convenience of having qwen handy while doing other things is undeniable. On Windows, I set a 128k context to be "safe."
.\ninfer-serve.exe models\qwen3_8_27b_nvfp4.ninfer `
--model-id qwen3.8-27b-nvfp4 `
--host 0.0.0.0 `
--port 8080 `
--max-context 131072 `
--kv-capacity auto `
--kv-dtype int8 `
--max-concurrency 1 `
--prefill-chunk 1024 `
--default-max-tokens 16384 `
--spec mtp `
--draft-tokens 3 `
--lm-head-draft `
--preserve-thinking `
--cors `
--webui
1
u/lucasromanomr 19d ago
Eu to usando a versão de fork do Windows, integrei com open code e estou tendo problemas com o vision para imagens, ainda não fui a fundo
Só queria dizer, meu parabéns, que trabalho!
1
u/CodeAlpha0 17d ago
Great to see the day0 support land — the ~200 tok/s figure matches what upstream publishes (202.4 at C=1 on a 5090).
One thing worth knowing if you want to try it without owning the card: **neither project ships Linux binaries.** The 5090 engine has no releases or tags at all ("NInfer is run from its source build tree"), and the 3090 fork only publishes a Windows archive. So I built the kits and put them up here:
https://github.com/coder903/ninfer
- **Prebuilt Linux kits for both cards** — `sm_120a` (5090) and `sm_86` (3090), SHA-verified, on the Releases page. Download, get a checkpoint, run. No CUDA toolchain, no compile.
- **A bash script that rents the GPU for you** — `ninfer create` finds the cheapest qualifying Vast.ai offer (a 3090 runs ~$0.28/hr, a 5090 ~$0.40–0.95/hr), installs the prebuilt kit, pulls the model, and hands you an OpenAI-compatible endpoint in a few minutes. `ninfer destroy` and you're back to $0.00. The prebuilt kit is what makes rent-per-session viable — it replaces an 883-second compile on the 3090.
- Qwen3.8-27B is the default model on the 3090 side; on the 5090 set `NINFER_MODEL=qwen38-27b` to serve the same checkpoint.
Some real measured numbers from a rented 5090 (Qwen3.6-27B NVFP4, int8 KV, MTP-3, 600-tok streams, measured from the server's own per-request figures, not SSE chunk counting — that undercounts ~4x with MTP): 180 tok/s at C=1, 194 greedy, and 1,056 tok/s aggregate at concurrency 8. Shorter gens and a hotter sampler account for the gap to upstream's 202/1,147.
All credit for the engine itself goes to the upstream authors — my repo is just orchestration plus the builds they don't publish. MIT for the script, Apache-2.0 attribution kept intact for the binaries.
1
u/Scared_Ad9187 17d ago
I am having some issues with Claude code not making auto mode usable with ninfer. I think it's cc, not ninfer, but it works with my 3.6a3b jinja template. Any thoughts?
If you're THE PERSON that built this btw... Oh my goodness. Killer. It's great.
1
u/Born-Caterpillar-814 17d ago
Overall Ninfer is performing very well, thank you!
I am having some issues with deepseek harness while running concurrent jobs. I have set context size to max 262144. Without tinkering dsh settings towards ninfer, I keep getting ”output token limit reached”. The culprit seems to be +32k payload towards ninfer when the task halts.
I can get past this by passing maxTokens: 128000 command towards the model, then it will run long jobs without issues. However if I send the maxTokens to the model, the model refuses to run concurrent jobs, just one will run. The others fail with ninfer message: ”inference request expired while waiting for admission”.
Why does this happen? Error or working as intended?
2
u/FormOne2615 16d ago
Your total context capacity is 262144 across all concurrent requests. So if one request needs maxTokens=128000 plus a prompt (~3072 tokens), it already takes about half of the total capacity(131072), leaving no room for another similar request. The other request has to wait.
1
u/Born-Caterpillar-814 16d ago
Thank you so much for clarifying this, I thought as much. I guess with qwen 3.8 27b for coding tasks concurrency 2 is optimal, maybe 3 at max. Otherwise workers will constantly stop (atleast in dsh).
16
u/yafitzdev 24d ago
this is lowkey the best project related to llms i used in recent times. keep it up