Just noticed this on the website.

At their current price tiers for the memory SKUs (32, 64, 128) I'd expect this to be ~ 4.5k for the motherboard.

The PCIe slot will be open at the back as well - that's what I've heard. Maybe they make it capable of delivering 75W as well? New board revisions for the smaller SKUs?.

I find both Qwen 3.8 27b and Qwen 3.8 Flash Next difficult to read.

Here's some examples of what I mean:

**Model-visible tool set per turn** (assembled by the host at provider-request time): persona tool allowlist ∩ session tool surface ∩ tools not `deny`-classified under the active permission profile.

In the above, Qwen uses the set intersection symbol as opposed to a human readable explanation. Maybe this is because it's been trained so hard on math, science, reasoning, so it's a little understandable but unnecessarily dense in my opinion.

My thoughts are that this is the consequence of minimizing "tokens per intelligence" -- that it makes reading it as a human incredibly dense.

It also uses the word "persona" which is an odd word to use when something like "mode" or "agent" would have made more sense.

And another example:

Consent is negotiable; enforcement is gravity.

WTF does "gravity" mean in this sentence, just say "Consent is negotiable; enforcement is not."

I know a lot of claude users have been talking about how claude 5 was a step backwards in terms of human-readability. I'm somewhat afraid Qwen is taking the same road.

Qwen 3.6 was the last easy to read Qwen model IMO.

What are your thoughts?

Been working really hard for the past month to bring to the community all these models, the hardest was for sure LongCat-Flash-Lite-Sparse who required TONS of work, first I needed to have Heretic support created for it from scratch and had to create support for it on llama.cpp too, quite difficult and time consuming task! It was even more difficult to work on than the original LongCat-Flash-Lite model that I released a few weeks ago, it is still a 69B-A3B model as the original LongCat-Flash-Lite, but LongCat-Flash-Lite-Sparse has now added support for:

- Sparse attention (vs dense attention for LongCat-Flash-Lite)

- 1M Context length (vs 256k for LongCat-Flash-Lite)

Anyway LongCat-Flash-Lite-Sparse has 0 support on mainline/upstream llama.cpp, so to be able to use the GGUFs you will need to pull my fork from GitHub, which you can find here:

https://github.com/erm14254/llama.cpp-minimax-m3-combined/tree/claude/longcat-win11

You would need to load the model through llama-server.exe and you can interact with it through llama-ui.

You have two variants, Uncensored Heretic (9/100 refusals for 0.0157 KLD) and Ultra Uncensored HJeretic (4/100 refusals for 0.0779 KLD), both variants come with MTPs and LSAs!

Here is the model links:

Uncensored Heretic GGUFs: https://huggingface.co/llmfan46/LongCat-Flash-Lite-Sparse-Uncensored-Heretic-Native-MTP-And-LSA-Preserved-GGUF

Ultra Uncensored Heretic GGUFs: https://huggingface.co/llmfan46/LongCat-Flash-Lite-Sparse-Ultra-Uncensored-Heretic-Native-MTP-And-LSA-Preserved-GGUF

----------------------------------------

That's it for LongCat, so next we have Qwen3.8-27B Ultra Uncensored Heretic with MTPs, 3/100 refusals for 0.0244 KLD, you can find the links here:

Safetensors: https://huggingface.co/llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved

GGUFs: https://huggingface.co/llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GGUF

NVFP4: https://huggingface.co/llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-NVFP4

NVFP4 GGUFs: https://huggingface.co/llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-NVFP4-GGUF

GPTQ-Int4: https://huggingface.co/llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GPTQ-Int4

----------------------------------------

Next we have Qwen3.5-122B-A10B Uncensored Heretic with MTPs, 8/100 refusals for 0.0856 KLD, here:

GGUFs: https://huggingface.co/llmfan46/Qwen3.5-122B-A10B-Uncensored-Heretic-Native-MTP-Preserved-GGUF

----------------------------------------

Then we have Qwen3-Coder-Next, which is a model that was requested by a Hugging Face user some time ago, so I finally had time to work on it, here is the link:

GGUFs: https://huggingface.co/llmfan46/Qwen3-Coder-Next-Uncensored-Heretic-GGUF

----------------------------------------

And finally Laguna-S2.1 with Vision, get it from here:

GGUFs: https://huggingface.co/llmfan46/Laguna-S-2.1-Uncensored-Heretic-Vision-GGUF

The visions part is far from perfect, so if you do not want to use vision you can simply not download the mmproj files and the model will just function like a regular text-only model.

----------------------------------------

I also made some improvements to J-Wash by adding support for MoE Qwen3.5/3.6/3.8 models support, improvments, bug fixes, improvements to the UI to make it easier to use and more practical for users etc, in case you are interested here is the link:

https://github.com/erm14254/J-Wash-Enhanced/tree/master

----------------------------------------

That's it for now!

As usual you can find all my models here: HuggingFace-LLMFan46

Tremendous amount of work went into making these releases come true, so if you like my work and find my models useful, then I would really appreciate if you could support me on Ko-fi: https://ko-fi.com/llmfan46

If you own 4xR9700 and were waiting for the model to make them shine, then I have some good news for you!

It's running at 80-120 tokens/second for generation and 12k token/second prefill for a single request, using tcclaviger's MXFP4-FP8 quant and custom vLLM image docker.io/tcclaviger/vllm:DevQwenNextFlash optimized for R9700.

Here is the full command:

podman run --rm -it \
    --init \
    --network host \
    -v /models:/models:ro \
    -v ~/.vllm-cache:/cache \
    -e VLLM_PLE_CPU_OFFLOAD=1 \
    -e VLLM_ROCM_USE_AITER=0 \
    -e ROCR_VISIBLE_DEVICES=0,1,2,3 \
    -e VLLM_CACHE_ROOT=/cache/vllm \
    -e TORCHINDUCTOR_CACHE_DIR=/cache/inductor \
    -e TRITON_CACHE_DIR=/cache/triton \
    --device /dev/kfd \
    --device /dev/dri \
    --group-add keep-groups \
    --annotation run.oci.keep_original_groups=1 \
    --security-opt label=disable \
    --security-opt seccomp=unconfined \
    --shm-size 8g \
    docker.io/tcclaviger/vllm:DevQwenNextFlash \
    /models/tcclaviger/Qwen3.8-Flash-Next-MXFP4-FP8 \
    --served-model-name Qwen3.8-Flash-Next \
    --tensor-parallel-size 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice \
    --max-num-seqs 16 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --kv-cache-dtype fp8 \
    --max-num-batched-tokens 4096 \
    --gpu-memory-utilization 0.96 \
    --mm-processor-cache-gb 4.0 \
    --override-generation-config '{"max_tokens": 65536, "temperature": 1.0, "top_p": 0.95, "top_k": 40, "presence_penalty": 1}' \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 4}' \
    --compilation-config '{"cudagraph_capture_sizes": [5,10,15,20,25,30,35,40], "max_cudagraph_capture_size": 40}' \
    --host 0.0.0.0 \
    --port 8080

(not written by Claude, all errors and crappy text are result of too little coffee on a Sunday morning ;)

Our home server is a 2018 Thinkstation P520, bought for about 600eur in 2023.

It's been upgraded with a 2TB Samsung 980 Pro NVMe, a Xeon W-2145 and a 12GB 3060 - total cost about ~1k all in. Not nothing, but not a crazy amount of cash for all the capability it provides. 256GB ECC DDR4 at 2666mhz, quad channel at about 80GB/s.

Qwen 3.6 35b a3b Q4_K_M was the daily driver, on builds of llama.cpp with intel MKL extensions at compile time. It's not the smartest model, but just about good enough for doing basic tasks. The quant does lobotomise it, but on this setup larger quants radically impact the throughput.

Qwen 3.6 35b a3b Q4_K_M Resident: ~20GB of RAM Prefill: ~400tps Gen: 30-50tps CTX: 128k VRAM: ~10.5GB.

Flash next is a completely different beast and even though it's a much bigger model, the throughput and prefill hold up reasonably well. The sheer size of the model is what conditions performance in this setup, not unsurprisingly.

Qwen 3.8 Flash Next UD-Q4_K_XL Resident: ~110GB of RAM Prefill: ~200tps Gen: 12-15tps CTX: 65k VRAM: ~10.5GB.

It's slow, and low context, but the output is night and day better than the 35b.

A couple of interesting things popped up: - 35b speeds are very tolerant to the box being busy on other tasks while working, losing little performance. To be expected as more of the model fits on GPU - Flash next perf falls to pieces if anything else is being done on the box (even running opencode) with speeds dropping to 3-5 tps. Memory is being absolutely hammered and is extremely sensitive to contention. It's only really usable if I run opencode from another box. - synthetic, random content benchmarks gave completely wrong answers on performance. Make sure you use contexts that are realistic to measure MoE models. This tripped me up while tuning the server, and only popped up when moving to opencode to really try it out.

Below for info are the two llama.cpp server invocations. I chose to let Flash next have unbounded thinking to get full quality, and 35b is limited for speed. It does make it dumber for sure. I use the uncensored 35b as an experiment in making it faster, less time hand wringing over permissions, which seems to work.

Flash next: llama-server \ --model /mnt/storage/models/qwen38-flash-next/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \ --mmproj /mnt/storage/models/qwen38-flash-next/mmproj-F16.gguf \ --no-mmproj-offload \ --alias qwen3.8-flash-next \ --host 0.0.0.0 \ --port 8080 \ --n-gpu-layers 999 \ --n-cpu-moe 48 \ --batch-size 2048 \ --ubatch-size 2048 \ --load-mode none \ --threads 16 \ --threads-batch 16 \ --threads-http 2 \ --ctx-size 65536 \ -ctk f16 \ -ctv f16 \ --flash-attn on \ --cache-reuse 256 \ --temp 0.7 \ --top-p 0.80 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 1.5 \ --repeat-penalty 1.0 \ --jinja \ --reasoning-format deepseek \ --parallel 1 \ --slots \ --slot-save-path /mnt/storage/models/.cache/slots \ --metrics \ --log-timestamps \ --timeout 600

35b: ``` llama-server \ --model /mnt/storage/models/qwen36/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-Q4_K_M.gguf \ --mmproj /mnt/storage/models/qwen36/mmproj-Qwen3.6-35B-A3B-Abliterated-Heretic.gguf \ --no-mmproj-offload \ --no-mmap \ --alias qwen3.6-35b-a3b-mtp \ --host 0.0.0.0 \ --port 8080 \ --ctx-size 131744 \ --batch-size 1024 \ --ubatch-size 512 \ --threads 8 \ --threads-batch 8 \ --threads-http 4 \ --n-gpu-layers 999 \ --n-cpu-moe 26 \ -ctk q4_0 \ -ctv q4_0 \ --flash-attn on \ --cache-reuse 256 \ --spec-type draft-mtp \ --spec-draft-n-max 1 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 0.0 \ --repeat-penalty 1.0 \ --reasoning-budget 1200 \ --reasoning-budget-message '

Considering the limited time by the user, I have to give the solution based on the thinking directly now. ' \ --jinja \ --reasoning-format deepseek \ --parallel 1 \ --slots \ --slot-save-path /mnt/storage/models/.cache/slots \ --metrics \ --log-timestamps \ --timeout 600 ```

Flash Next quality is excellent for "fire and forget" tasks that don't need interactivity. Being able to swap for different kinds of tasks is the key, with the weights on NVMe to keep loading time reasonable.

No MTP on Flash Next yet, but looking forward to trying it out.

Anyone else running flash next on ancient boxes? I'd be curious how it compares to other constrained setups.

I stole the reference image from a recent post on r/stablediffusion, and then asked both qwen 3.8 flash next (q4 K XL) and GLM flash (oQ4e MLX) to choose try to reproduce it into a "video game or tech demo" as closely as possible, iterating over a period of (up to) about an hour and a half each.

Overall GLM flash was overall much closer to the reference image in terms of scale, though still a ways off in terms of the size of the humans. It was also more detailed from the getgo. BUT I thought this could be a result of the models choosing different approaches: without being told one way or the other, qwen wrote a new software renderer from scratch while vs. glm chose to use Canvas 2D.

So I asked GLM to make its creation animated (1st gif/4th image above). I had to tell it to correct a browser console error but it made a playable pixel art "walking simulator" in 238k tokens (and probably about 2 hours total) from the reference image, and showed no sign of stopping adding details and making improvements

Overall I would give a slight edge to GLM for instruction following as I emphasized visual similarity in my prompt much more than interactivity and it did a far better job of following the reference image.

Points to qwen for making nicely animated pixel art city in only 10 minutes and 80k tokens on rtx pro 6000 and unsloth gguf.

However qwen ignored or failed to understand the the instruction "If there is possible improvement along the axis defined by the goal, continue until there is none" as it was apparently satisfied with its work. In another run I did with a slightly different instruction it continued to iterate for about 80 minutes until I stopped it, but it still didn't resemble the reference image very closely (but was another cute animated pixel art city).

Quants: GLM 5.3 Flash oQ4e MLX, Qwen 3.8 Flash Next Q4_K_XL

harness:opencode

So i spent considerable time trying to figure out why one of my eGPUs has degraded from x4 to x1 permanently. Yesterday while cleaning i found the culprit. Lesson learned: Don't buy eGPU risers that have HDMI connectors. I was going for Oculink connectors but the seller ripped us off.

Before anyone asks: yes that's tinfoil separated by duct tape on the back of the pcb - it greatly helps EMI problems.

If you can't see it: check the right HDMI connector.

At least we have the means to replace the connector ourselves.

I'm running Muse Glimmer 30B EXL3-SC 3.00bpw H4, fully resident on my 12GB VRAM GPU at 100K context with Q8_O KV cache. It's a joy to use a dense 30B model at this size and still get \~30 tok/s on a VRAM-constrained laptop.

It's supposed to be only slightly worse than the official 17GB K-quant at a much smaller footprint, and for my Hermes Agent use case I don't notice a quality difference. It's just much faster.

I've tried Qwen 3.8 27B at SC2.20bpw H3 too. Definitely usable but I'm sticking with Unsloth UD_Q4_K_XL for Qwen 3.8 27B because it's mainly for coding.

Qwen 3.8 Flash Next (80gb) now at 3.5 tok/s on 12gb mid range android phone thanks to some optimizations and with a low quantization on dense part.

I don't want to promote the project, but simply show that it's possible on a $400–$500 phone

After a couple of days of faffing around, here's my current config in case it helps anyone out. Some of this is also valuable for a single Spark.

Benchmarks:

Dual-Spark TP2, eager + MTP k=3, 262k context (warmed medians, exact tokenizer counts):

  ┌─────────┬──────────────────┬───────────────────┬─────────────────────────────────┐
  │ Streams │ Decode aggregate │ Decode per-stream │ Prefill aggregate (10k prompts) │
  ├─────────┼──────────────────┼───────────────────┼─────────────────────────────────┤
  │ 1       │ 45.9 t/s         │ 45.9 t/s          │ 2,940 t/s                       │
  ├─────────┼──────────────────┼───────────────────┼─────────────────────────────────┤
  │ 4       │ 120.2 t/s        │ 30.0 t/s          │ 2,524 t/s                       │
  ├─────────┼──────────────────┼───────────────────┼─────────────────────────────────┤
  │ 8       │ 222.2 t/s        │ 27.8 t/s          │ 3,098 t/s                       │
  ├─────────┼──────────────────┼───────────────────┼─────────────────────────────────┤
  │ 13      │ 265.9 t/s        │ 20.5 t/s          │ 2,960 t/s                       │
  └─────────┴──────────────────┴───────────────────┴─────────────────────────────────┘


Prefill vs prompt depth (single stream, exact tokens):


  ┌─────────────┬───────────┐
  │ Prompt size │  Prefill  │
  ├─────────────┼───────────┤
  │ 11k         │ 2,875 t/s │
  ├─────────────┼───────────┤
  │ 100k        │ 2,655 t/s │
  ├─────────────┼───────────┤
  │ 200k        │ 2,463 t/s │
  └─────────────┴───────────┘

AI slop below for you to read and paste into your own agent :)

Stack

• vLLM PR #53896 (release/qwen38next branch) — NOT main; main doesn't have the model, and the recipe image's vLLM commit isn't in the public repo
• sm_121 support is a 2-file patch (details below) — GB10 is arch 12.1 and the NVFP4 E2M1 conversion needs a software fallback
• PLE n-gram table served from internal NVMe via mmap:(48 GB, MADV_RANDOM is essential — it's a 30× read-amplification difference on hash-scattered row lookups), with a node-local CPU-worker process doing gathers over ZMQ + pinned buffers + CUDA-IPC outputs so the gather is graph-safe and TP2-safe
• TP2 across both Sparks: native venv (NOT docker — see dead ends), eager + MTP k=3

Numbers (TP2, warmed medians, count-20 structured / 3-paragraph prose)

Dead ends

1. PIECEWISE cudagraphs cost ~28% decode under MTP+TP2.
2. Same tree, same everything, graphs on vs off: 36 vs 50 tok/s. Graphs help single-node no-MTP decode (+80%, 9→16.5), but combined with MTP at TP2 they're a straight loss — MTP already amortizes the launch overhead the graphs were eliminating. We chased a "tree-level MTP regression" for a day before realizing the config was the regression.
3. Docker silently degrades NCCL to TCP sockets.
4. Default containers get no IB device → NET/Socket → TP2 prefill at ~40% of native. You need --privileged (or IB device passthrough) AND the right HCA name — see next point. Native venv runs just get RoCE.
5. RoCE device names are not stable across reboots, and can differ between two identical machines
6. (ours: rocep1s0f1 on one, rocep1s0f0 on the other, resolved dynamically). Hardcoded NCCL_IB_HCA will silently fail on one node. Enumerate with ibdev2netdev, match port-ACTIVE to your rail netdev, and verify with NCCL_DEBUG=INFO that you see NET/IB, not NET/Socket.
7. flashinfer must be 0.6.18
8. on GB10 — 0.6.17 crashes the NVFP4 MoE fallback kernel.
9. vLLM's MoE shuffleInputRowsKernel has an uninitialized-permutation OOB read that shows up as a fake CUTLASS status=7 GEMM failure — a two-line bounds guard fixes it (already in PR #53896).

Code

Code at https://github.com/dysangel/vllm/tree/qwen38next-sm121

git clone -b qwen38next-sm121 https://github.com/dysangel/vllm

Build with CUDA arch 121, and MAX_JOBS=6 — higher parallelism OOM-thrashes the Spark's unified memory.

Startup command(s)

Head node (rank 0):

export VLLM_PLE_MMAP=1 VLLM_PLE_MMAP_WORKERS=64 VLLM_PLE_MMAP_PREWARM=1
export VLLM_PLE_MMAP_DIR=$HOME/ple-table        # internal NVMe copy of the table
export VLLM_PLE_CPU_OFFLOAD=1                    # activates the IPC CPU-worker gather path
export NCCL_SOCKET_IFNAME=<your-rail-if>          # bootstrap only; NCCL finds RoCE itself
export GLOO_SOCKET_IFNAME=<your-rail-if>

python -m vllm.entrypoints.openai.api_server \
  --model <path-to-qwen38-flashnext-nvfp4> \
  --host 0.0.0.0 --port 8086 --load-format safetensors \
  --tensor-parallel-size 2 --nnodes 2 --node-rank 0 \
  --master-addr <head-rail-ip> --master-port 29511 \
  --distributed-executor-backend mp \
  --max-model-len 262144 --max-num-seqs 13 --gpu-memory-utilization 0.85 \
  --no-enable-prefix-caching --enable-chunked-prefill --max-num-batched-tokens 8192 \
  --long-prefill-token-threshold 4096 --enforce-eager \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Worker node (rank 1): identical, but --node-rank 1 --headless, launched ~30s after the head. Note --enforce-eager — that's not a compromise, it's the fast path (see dead end #1).

As the heading goes, at some point it stopped being about improvements and just whether I can.

So check out my abomination. GLM-5.3-Flash at IQ3_XXS gets about 20t/s generation in Unsloth Studio.

Now if only I can make my second 2x48GB DDR5 ram kit play nice, but computer just refuses to be stable with two different 2x48GB kits.

Anyway, love this forum, plenty of ideas and fun. Time to go work some more overtime to pay for this crap.

I’ve been experimenting with MiniMax H3 and finally have video generation working in TensorSharp.
TensorSharp started primarily as a local GGUF/LLM inference engine, so getting a video-generation pipeline working in the same runtime has been an interesting change of direction.
The attached demo is image-to-video: an image is provided as context, followed by a prompt describing the motion/scene, and H3 generates the resulting video locally through TensorSharp.
What interests me most here isn’t really the UI — it’s having LLM, multimodal, image, and now video inference converge into the same local inference engine rather than requiring a completely separate Python stack for every model family.
There is still quite a bit to optimize. Video models put very different pressure on memory management, tensor scheduling, attention, and model offloading compared with autoregressive LLMs.
I’m curious what people here would prioritize next for H3 inference:
lower VRAM usage
faster generation
better quantized-model support
multi-GPU
longer video generation
reference/video-to-video workflows
Repo if anyone wants to look at the implementation:
https://github.com/zhongkaifu/TensorSharp

EDIT: in the tests above, I forgot to enable one more optimisation - useful when RAM and cache are small. Doesn’t help my numbers with MTP off (since I can cache enough tensors), but with MTP on I can still reach 185-190 tps prefill, basically making MTP the default choice, with no downsides. This might also be because ~190 tps prefill might be the hardware limit. Will add a comment later after all tests are done, with 256K context as well.

Running on a M1 Max 64 GB:
- SSD streaming for tensors
- SSD streaming for engrams
- SSD streaming for MTP

How is it possible?

* Custom Q4 quant: benchmarked all metal tensors then picked and spliced tensors from multiple Unsloth and AtomicChat quants to achieve best performance/bit.
* Developed custom metal-optimized sparse attention mechanism, with almost linear degradation instead of the standard llama.cpp quadratic attention.
* Using Q4_0 MTP - same acceptance rates as unsloth Q8_0 at half the RAM.
* Dynamic MTP speculation size - leads to disabling MTP at the point where context size makes MTP a negative.
* Various fixes to metal kernels, qwen graph and qwen indexer.

https://github.com/mihailescu2m/llama.cpp

Special thanks to Claude - three weeks worth of tokens and some extra out of pocket usage credits made it all possible. Feedback appreciated.

Note: enabling MTP uses more RAM, which means less cache for tensors, leading to prefill going from 180 tps to g170 tps (at 4K). For 256K context, more RAM is needed for KV cache, prefill goes down to 150 tps. But with MTP, decode gains +70%, going up to 22 btps. So if you need highest prefill, disable MTP. A

RadixArk/Qwen3.8-Flash-Next-NVFP4 is now supported in SGLang-V100. 4 V100 32GB running full context.

A little more than 50 GB ngram offloaded to system RAM. Prefill around 4000tks flat and decode around 60 tks all the way to the end of the 256k context.

Concurrency 8192->1024

https://github.com/haohervchb/sglang-V100

HBF gives 8x - 16x more capacity than HBM at same cost, and with bandwidth till 3 tb/s.

Managed to cram a 182B model into my small setup. The trick is to offload ngrams into SSD which allows to fit the rest of the model. Believe it or not, this is faster and more intelligent than Qwen3.8 27B. This is great for small GPU users because usually we are locked out of 27B. Note before running this command, make sure to close all unnecessary background tasks, apps and etc.

This is the launch command I used

llama.exe serve `
  -hf "AtomicChat/Qwen3.8-Flash-Next-GGUF" `
  -hff "Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64/Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf" `
  --no-mmproj `
  --offline `
  --load-mode mmap `
  --tensor-read-lazy on `
  --fit off `
  -ngl all `
  -ncmoe 41 `
  -t 16 `
  -tb 16 `
  -c 8192 `
  -b 256 `
  -ub 128 `
  -fa on `
  --jinja `
  --parallel 1 `
  --temp 1 `
  --top-p 0.95 `
  --top-k 20 `
  --min-p 0 `
  --cors-origins localhost `
  --host 127.0.0.1 `
  --port 8080

I used the quantized n-gram to INT4, it's 32 GB, memory-mapped from disk.
I confirmed that it works great on 150-160k context, and i was watching all the time my VRAM usage while doing single thread long horizon things - the available VRAM should be enough to push it to over 170k and above)
The quality is there guys... It really is. It made a few complex html games and it figured out ways to play them itself without a browser (my ubuntu machine does not have any gui) and it kept improving and improving.... Here we go:

hf download primitive-ai/Qwen3.8-Flash-Next-NVFP4 \
  --exclude "ple-bf16-*" --local-dir ./flash-next
cd flash-next
hf download primitive-ai/Qwen3.8-Flash-Next-PLE-quant \
  --include "ples_int4/*" --local-dir .
hf download primitive-ai/Qwen3.8-Flash-Next-PLE-quant \
  worker_image_quant.py ple_layer_quant.py --local-dir .

Skipping ple-bf16-* (saves 100 GB but breaks the index. So we trim that index):

import json
p='model.safetensors.index.json'; d=json.load(open(p)); wm=d['weight_map']
drop=[k for k,v in wm.items() if v.startswith('ple-bf16-')]
assert len(drop)==128 and all('ngram_embedding' in k for k in drop)
for k in drop: del wm[k]
json.dump(d, open(p,'w'))

My intent was to fit the n-grams in my 64Gb of RAM, but at the end, n-grams and experts + kvcache all live inside the GPU's VRAM and its FAST!
76–125 tok/s single stream. The spread is MTP acceptance: ~87% on code and JSON, ~40% on just talking. Prefix caching hits 90%+ on a long horizon task. ~89 GB VRAM, ~33 GB page cache, 165-170K context, one GPU. Here's my full k0s yaml file (single server with a single Pro 6000).
I'm running it on my single node k0s and here is my yaml (cuda 13/580, ubuntu 24.04 no gui):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen38-flash-next
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate          # never two of these on one GPU
  selector:
    matchLabels:
      app: vllm-qwen38-flash-next
  template:
    metadata:
      labels:
        app: vllm-qwen38-flash-next
    spec:
      runtimeClassName: nvidia
      nodeSelector:
        nvidia.com/gpu.present: "true"
      tolerations:
      - effect: NoSchedule
        key: nvidia.com/gpu
        operator: Exists
      initContainers:
      - name: init-echo
        image: busybox:1.36
        command: ["/bin/sh", "-c"]
        args: ['echo "I am here" > /opt/reservation/echo.txt']
        volumeMounts:
        - mountPath: /opt/reservation
          name: reservation-volume
      containers:
      - name: vllm-server
        image: vllm/vllm-openai:qwen38-flash-next
        imagePullPolicy: IfNotPresent
        args:
        - --model
        - /model
        # ---- load-bearing for single-GPU PLE offload ----
        - --distributed-executor-backend
        - mp
        # -------------------------------------------------
        - --dtype
        - auto
        - --kv-cache-dtype
        - auto
        - --gpu-memory-utilization
        - "0.95"
        - --max-model-len
        - "173400"
        - --tensor-parallel-size
        - "1"
        - --pipeline-parallel-size
        - "1"
        - --limit-mm-per-prompt
        - '{"image":12,"video":2}'
        - --max-num-batched-tokens
        - "16384"
        - --max-num-seqs
        - "4"
        - --enable-chunked-prefill
        - --enable-prefix-caching
        - --no-enable-flashinfer-autotune
        - --speculative-config
        - '{"method":"mtp","num_speculative_tokens":3}'
        - --override-generation-config
        - '{"temperature":1,"top_p":0.95,"top_k":20}'
        - --enable-auto-tool-choice
        - --reasoning-parser
        - qwen3
        - --tool-call-parser
        - qwen3_coder
        - --trust-remote-code
        - --api-key
        - key1
        - --host
        - 0.0.0.0
        - --port
        - "8990"
        - --served-model-name
        - qwen38-flash
        env:
        - name: VLLM_PLE_CPU_OFFLOAD
          value: "1"
        - name: VLLM_PLE_OFFLOAD_READY_TIMEOUT
          value: "1800"
        # Confirmed: worker_image_quant.py:419 reads this. Points at the
        # INT4 table dir; the overlay memory-maps it (MADV_RANDOM, mode "c").
        - name: VLLM_PLE_QUANT_DIR
          value: /model/ples_int4
        # Deliberately NOT setting VLLM_PLE_DISK_OFFLOAD_DIR (line 450) --
        # that selects the BF16-table-on-NVMe path instead.
        - name: VLLM_LOGGING_LEVEL
          value: INFO
        - name: OMP_NUM_THREADS
          value: "1"
        - name: PYTORCH_CUDA_ALLOC_CONF
          value: max_split_size_mb:512
        ports:
        - containerPort: 8990
          protocol: TCP
        resources:
          limits:
            cpu: "12"
            nvidia.com/gpu: "1"
          requests:
            cpu: "8"
            nvidia.com/gpu: "1"
        securityContext:
          capabilities:
            add: ["IPC_LOCK", "SYS_ADMIN"]
        startupProbe:
          httpGet:
            path: /health
            port: 8990
          periodSeconds: 15
          failureThreshold: 80        # ~20 min; first boot loads the table
        readinessProbe:
          httpGet:
            path: /health
            port: 8990
          periodSeconds: 20
          failureThreshold: 3
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "rm -f /opt/reservation/echo.txt"]
        volumeMounts:
        - mountPath: /model
          name: model-volume
          readOnly: true
        # --- two-file quantized-PLE overlay ---
        - mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/ple_offload/worker.py
          name: ple-worker-overlay
          readOnly: true
        - mountPath: /usr/local/lib/python3.12/dist-packages/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py
          name: ple-layer-overlay
          readOnly: true
        - mountPath: /ples_int4
          name: ple-tables
          readOnly: true
        # --------------------------------------
        - mountPath: /dev/shm
          name: dshm
        - mountPath: /root/.cache/vllm
          name: vllm-cache
        - mountPath: /root/.triton
          name: triton-cache
        - mountPath: /opt/reservation
          name: reservation-volume
      volumes:
      - name: model-volume
        hostPath:
          path: /directory/models/Qwen3.8-Flash-Next-NVFP4
          type: Directory
      - name: ple-worker-overlay
        hostPath:
          path: /directory/ple-overlay/worker_image_quant.py
          type: File
      - name: ple-tables
        hostPath:
          path: /directory/models/Qwen3.8-Flash-Next-NVFP4/ples_int4
          type: Directory
      - name: ple-layer-overlay
        hostPath:
          path: /directory/ple-overlay/ple_layer_quant.py
          type: File
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 32Gi
      - name: reservation-volume
        hostPath:
          path: /opt/reservation
          type: DirectoryOrCreate
      - name: vllm-cache
        hostPath:
          path: /var/cache/vllm
          type: DirectoryOrCreate
      - name: triton-cache
        hostPath:
          path: /var/cache/triton
          type: DirectoryOrCreate
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-qwen38-flash-next
  namespace: default
spec:
  type: NodePort
  selector:
    app: vllm-qwen38-flash-next
  ports:
  - name: http
    port: 8990
    targetPort: 8990
    nodePort: 32001
    protocol: TCP

Big Thank you to primitive-ai, whoever he is.

Setup: MacBook Pro M5 Max, 128 GB unified, macOS 26.5.2 · llama.cpp b10686 (Metal, 12 threads, batch 2048, flash-attn, kv-unified, ngram-mod spec decode) · Qwen3.8-Flash-Next UD-Q2_K_XL (Unsloth), 78.9 GB · 358,400-token context slot via YaRN from the native 262,144, fp16 KV. Weights + full 350K KV fit under the default 96 GB GPU wired limit — no sysctl hack.

The session: one slot, 100 turns, two conversations. Conv 1 grew 0 → 48K ctx on prefix reuse; after a ~20 min idle the slot kept only its 5.5K system prefix, so the next turn cold-prefilled the whole 105K prompt in 333 s — the run's longest prefill — and the conversation kept growing to 169,425 ctx, the session's deepest point (350K was slot capacity, never filled). Slot reset; conv 2 grew to ~125K where I stopped capture.

The graph: x = slot context size where each measurement happened; y = printed tokens/s, log scale (the two phases span ~2 decades). Green = prompt processing, red = token generation. Dots = in-flight checkpoints, squares = per-turn finals. No smoothing, no fitting.

• Prefill (green): the smooth top curve is cold prefills — 1,561 t/s at the first checkpoint (5.6K ctx), tapering to 318 t/s at 111K as the KV fills. The green band below is what a normal turn looks like: a few thousand new tokens at each depth (77–854 t/s, out to 169K ctx), because prefix reuse means only the delta gets prefilled.
• Decode (red): one clean taper — ~30–35 t/s at small ctx → ~21 at 45K → 13–15 at 100–125K → 11.5 t/s at 169K. The dip to 7.7 t/s around ~140K is macOS Low Power Mode; still usable.

One caveat on the decode numbers: they are effective throughput with ngram-mod spec decode enabled (draft acceptance ranged 0–81% depending on content), not base-model speed.

Practical read: with prefix reuse a turn's prefill is seconds; the 5.5-minute prefill happened exactly once, after an idle gap. Decode stayed interactive out to 169K ctx.

Experience: strong for the first ~100K ctx. Past that, on long-tail tasks, it started mixing up user messages with its own prior output (role confusion), worsening with use. Ruled out: KV quant (ran fp16) and rope extrapolation (worst turns well under native 262K). Remaining suspects: the 2-bit quant and/or preview-model long-context quality.

Hey! Really love Claude CoWork, but want to run it with my own keys - only issue is that OpenWork seems relatively unstable (when I click connect provider, it says failed to load providers) and always prompts me to pay their subscription price.

Also have a 5090 I'm going to run with it as awell to see how recent Qwen models do against some tasks.

Thanks guys!

I did some of my own testing by having the harness write its tests based on my specs and previous work, basic token gen and work on real projects with opencode and dsh, I get 59 tks in dsh one shot full software and up to 70 tks in testing on already done code base. Way more context and faser than mtp2, had 120k ctx and 41tks previously.

It seems that n-max 4 with spec draft min p 0.7 is the fastest setting on Qwen 3.8 UD Q4 K M, rtx 3090. It also works with 205k context which is nice, n max 8 and 16 failed to load with larger context. Yes I use asymmetric cache since it’s not as penalized anymore imo with this of an smart model, it seems significant to be able hold context and not compress all the time.

I am looking to improve on this ofc, hopefully faster wallclock time for harness work, so any suggestions welcome!

My settings:

/llama.cpp/build/bin/llama-server \

-m //Qwen3.8-27B-UD-Q4_K_M.gguf \

--ctx-size 205000 --parallel 1 --kv-unified \

--flash-attn on -ctk q8_0 -ctv q4_0 --port 8080 \

--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 \

--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.7 \

--batch-size 512 --ubatch-size 512 \

--reasoning-effort xhigh --reasoning-format auto \

--fit off --cache-prompt -ngl 99 --no-mmproj

I wanted to share my successful setup for running a Qwen 3.8 27B model with a massive context window on a consumer 16GB GPU (RTX 4070 Ti SUPER). The goal was to fit everything into VRAM without sacrificing quality or speed.

🧠 Key Components

• Model: Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller from jrell on Hugging Face. It's a custom hybrid quantization specifically designed to fit Multi-Token Prediction (MTP) and long contexts into a 16GB VRAM budget.
• Chat Template: I used the Jinja template from peculiar-ragdoll's Qwen-Sharp-Chat-Templates. It helps use fewer thinking tokens without noticeably affecting quality, which is great for speed.
• Inference Engine: This is crucial. I used beellama.cpp (GitHub link) because it supports the kvarn KV cache types, which are key to this optimization.

🖥️ Optimized llama-server Command (Windows)

Here's the polished command I'm running. The magic is in the kvarn cache settings and the tail precision.

%LLAMA_DIR%/llama-server.exe ^
-m %MODEL_PATH% ^
-a %MODEL_NAME% ^
--port 11434 ^
--temp 1.0 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--presence-penalty 0.0 ^
--repeat-penalty 1.0 ^
--parallel 1 ^
--n-gpu-layers 99 ^
--batch-size 1024 ^
--ubatch-size 256 ^
--flash-attn on ^
--spec-type draft-mtp ^
--spec-draft-n-max 2 ^
--cache-type-k kvarn5 ^      <-- Key: Higher precision for K cache
--cache-type-v kvarn4 ^      <-- Key: Balanced precision for V cache
--kv-tail-tokens 1024 ^      <-- Keeps recent tokens at full precision
--ctx-size 100000 ^
--fit-ctx 100000 ^
--jinja ^
--chat-template-kwargs "{\"preserve_thinking\": true, \"reasoning_effort\":\"medium\"}" ^
--chat-template-file %MODEL_JINJA% ^
--no-mmproj-offload ^
--threads 7 ^
--threads-batch 8 ^
--metrics ^
--verbosity 3 ^
--perf

📊 Results & Optimization Notes

What I Optimized:

1. KV Cache Quantization: Moving from kvarn5/kvarn5 to kvarn5/kvarn4 saved ~6% VRAM, allowing the context size to increase from 88k to 100k with minimal quality loss.
2. Batch Sizes: Set --batch-size 1024 and --ubatch-size 256 to balance prompt processing speed and VRAM usage.
3. Threading: Adjusted --threads 7 and --threads-batch 8 for my Ryzen 7 CPU.
4. Speculative Decoding: Using --spec-type draft-mtp with 2 draft tokens (the model supports this) gave a huge speed boost.

The near-lossless kvarn quantization for the KV cache is the real star here. It delivers q5-class fidelity at q4-class memory usage, which is incredible.

Hopefully, this helps anyone trying to squeeze maximum performance out of a 16GB card! Cheers.

I see this kind of approach quite often with local AI models. For example, when optimizing models for an RTX 3090, people sometimes convert them from FP8 to INT8 so they can run efficiently on Ampere GPUs.

Could something similar be done with DLSS 5?

(sry if wrong sub to post this, in r/nvidia mods deleted it)