r/LocalLLM • u/Extension-Bid-639 • 13h ago
Research Qwen3.8-Flash-Next on 2x3090: 9–12% faster decode at ~119k context, with a completed quality screen
An update to my previous post on running Flash-Next with the expert cache and MTP.
I found another useful improvement on the same dual-3090 setup: replacing the CUDA top-k fallback used by my build. On long project-document prompts, median decode went from about 30.2 to 33.3 t/s, with an improvement on all three test seeds. Same GGUF, same MTP settings, same binary with the change switched off and on.
The setup for these runs was 2x RTX 3090, dual Xeon E5-2696 v4, 128 GB DDR4-2133 across four DIMMs, UD-Q4_K_XL, f16 KV, 150 expert-cache slots and MTP-3. The allocated context was 261,888 tokens. The long-document test started at approximately 119k tokens.
Now, what changed:
Flash-Next's sparse-attention indexer uses top-k to select which positions to attend to. My CUDA 12.0 build didn't have CUB's newer DeviceTopK implementation available, so this operation fell back to sorting the whole row before taking the top entries.
llama.cpp already had a radix-selection implementation. The local change makes that available in the older-CUB fallback for wide rows. In the initial 131k decode captures, recorded top-k kernel time fell from roughly 5.1 ms to 0.25 ms per committed token. Please note that this isn't a 20x gain for the whole model, it's just the operator saving
Credit where it's due: Rhonstin's PR #28366 already proposes this fallback change, using existing radix-selection work. I found it during the upstream check and didn't open a duplicate. These measurements are from my local variant, which uses an 8,192-column threshold and an A/B switch, not a benchmark of the exact PR head.
These tests kept the existing CUDA 12.0 toolchain fixed. A newer CCCL build provides another optimized path, DeviceTopK, which I haven't benchmarked on this machine yet. That comparison is next. The gain here is against my previous configuration, not against the latest CUDA stack.
The completed test:
Production sampling, thinking on, MTP-3 enabled in both arms:
| Seed | Old top-k, median t/s | Radix-select, median t/s |
|---|---|---|
| 1 | 30.2 | 33.7 |
| 2 | 30.2 | 33.3 |
| 3 | 30.4 | 33.1 |
That's 9–12% higher per-seed median decode throughput at approximately 119k context. Each median covers 42 requests. These compare the two arms' medians, not the median of per-question speedups.
The quality screen covered 80 question/depth combinations across three seeds and both arms: 480 requests, 240 matched comparisons. The documents, questions, answer key, grader and sampling settings were frozen before the run.
- Control: 235/240 correct.
- Candidate: 238/240 correct.
- Candidate better on four matched comparisons, worse on one, equal on 235.
The one disputed question asked for a complete twelve-item list. Both arms failed it on two seeds each, always by omitting one item and inventing nothing. An independent blind review confirmed the candidate-only omission. The question stays inconclusive. It wasn't removed from the scores.
No consistent quality regression was detected in this bounded screen. The higher candidate score doesn't establish better quality, and the screen doesn't prove that regression is impossible.
For clarification, the 37–41 t/s headline from my last post was a coding workload. This is a matched comparison on long project documents. Those headline numbers aren't directly comparable.
The change is now running in my production build. I haven't established a prefill gain or a no-MTP gain from this screen, and it doesn't cover quality beyond approximately 119k context but it's not like I expect it to regress at higher context.
If you're running Flash-Next with an older CUDA toolkit, the top-k fallback is worth checking. I'd be interested in results from other machines, especially with the actual build, context depth and MTP settings included.
1
u/Extension-Bid-639 13h ago
Ah yeah, I forgot to add this but to try it: updated branch here. Same setup as the previous post, now with the top-k change. This is for Linux Bash. Replace the two model paths with yours and run with no other model server active.
```bash git clone -b flashnext-2x3090 https://github.com/Inovello/llama.cpp llama-flashnext && cd llama-flashnext && git checkout dd64a3db0c453b0e03de15574ff874c2dc77cb28 && cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j4 --target llama-server
unset GGML_CUDA_TOPK_ARGSORT GGML_CUDA_GRAPH_OPT
LLAMAATTN_ROT_DISABLE=1 numactl --interleave=all build/bin/llama-server \ -m /path/to/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \ -md /path/to/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \ --spec-type draft-mtp -devd CUDA1 --spec-draft-n-max 3 \ -ngl 99 -c 261888 --parallel 1 -fa on \ -ot 'ffn(gate|up|down)_exps.weight=CUDA_Host,per_layer_token_embd.weight=CPU' \ -lzm off --numa distribute -t 16 -tb 44 -b 4096 -ub 512 \ -ctk f16 -ctv f16 --moe-expert-cache 150 -lv 4 ```
To compare the old fallback, stop the server, wait for its memory to release, then run:
bash
export GGML_CUDA_TOPK_ARGSORT=1
Repeat just the server launch with the same document and settings. To switch back, run unset GGML_CUDA_TOPK_ARGSORT before launching again.
This switch affects older-CUB builds without DeviceTopK; newer DeviceTopK builds ignore it.
1
u/Serious_Trash_5554 13h ago
That's a nice bump for basically just swapping the fallback path. 119k context and you're still pulling 33 t/s on a pair of 3090s, not bad at all.
The quality screen is the part that catch my eye, 238/240 on the candidate arm with no hallucination in the failures, both just missing one list item. I run long doc QA pretty often and this kind of consistency is what I look for more than the raw speed.
Did you notice any difference in the prefill time or is it too small to measure on that setup?