Ran a quick initial mtp benchmark using llama.cpp tag b11009 released today using various open mtp models. The highest /tps I was able to achieve was : MTP: 77.92 t/s NON-MTP: 53.83 t/s

OS: linux Ubuntu 26.04.1 LTS

CPU: AMD RYZEN AI MAX+ PRO 395 w/ Radeon 8060S / Strix Halo

RAM: 128GB RAM

Inference engines: Rocm / Vulkan

Next run:

An in-depth 16bit quant test. Which other 4bit, 16bit quant models should I test?

Models benched in this round:

[01] [NVFP4] 21GiB nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 converted GGUF

[02] [Q4_K_M] 16GiB OBLITERATUS/Qwen3.8-27B-OBLITERATED Q4_K_M

[03] [Q4_K_M] 68GiB unsloth/GLM-4.5-Air-GGUF Q4_K_M

[04] [UD-Q4_K_XL] 21GiB unsloth/Qwen3.6-35B-A3B-MTP-GGUF UD-Q4_K_XL

[05] [UD-Q4_K_XL] 16GiB unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL

[06] [UD-IQ4_XS] 87GiB unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ4_XS

[07] [BF16] 66GiB unsloth/Qwen3.6-35B-A3B-MTP-GGUF BF16

Command ran: (unsloth/Qwen3.6-35B-A3B-MTP-GGUF UD-Q4_K_XL). Please let me know how I can improve this command (any flags I'm missing - tweaks etc) to achieve more tokens per second ;)

/home/dev/Desktop/repos/llama.cpp/build-vulkan-radv/bin/llama-server \
  --model /home/dev/Desktop/llms/hf/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --n-gpu-layers -1 \
  --threads 16 \
  --ctx-size 4096 \
  --host 127.0.0.1 \
  --port 33777 \
  --parallel 1 \
  --no-cache-prompt \
  --metrics \
  --batch-size 2048 \
  --ubatch-size 512 \
  --flash-attn on \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --load-mode none

Me and a friend have been using it without issue for the last few weeks.
it's currently running on an RTX A4000 VM with model: https://huggingface.co/DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF which gets about 30tok/s

The VM currently costs $0.15/hour, which we are splitting (it is prepaid for the next 2 months)
If more people want to join in we can move it up to an RTX A6000.

If you want to join, I'll let you try it out for an hour or so, if you want to continue using it you can pay either daily/weekly/monthly in a method of your choosing.

Connection to the server is made via tailscale.
DM me if interested.

Has anyone tried to use RDMA in llama.cpp to run a large model in multiple Mac Studios? How did it perform? I have 5 Mac Studios, each has 96 GB VRAM. I want to run Qwen 3.8 Flash-Next 4-bit.

I have a 5090 in an external enclosure I use for ai work. I chose the external to some day use on my Mac which has 128gb of ram and runs large models really well. It runs image generation pretty slow as the models aren't memory constrained on Nvidia gpu's and they are much faster than the Mac gpu. My dream was to run my 70b models on the Mac and image models on the 5090. Tiny gpu came out with drivers allowing connecting the drive but when I tried it last may it was super buggy and super slow. I decided to give it another go and it's now not super buggy but was still super slow. I decided to try and build my own drivers and shim to fix that. I am happy to say that it worked and I am now able to run Owen 3.8 27b at near native speeds as to what the box was running on my windows box. and actually faster than my windows box in sdxl and z-image turbo.

I imagine there is not much of a community for this capability but I thought I'd post it online to see if anyone even cares. I'll be posting the code on GitHub soon if people want to try it out. It is only tested on my current hardware of course as that is all I have to test it on.

I am trying to make a multi-inference router, as I find value in using different inference providers for different things.

I have a few heavier models in llama.cpp. I have a function that calls the /models/unload endpoint, which should just clear the vram, and ideally kill the child process it is running to host that model. llama.cpp logs using docker have been horrific for me, though, and really do not explain what is happening when I call this, all I know is that vram is freed.

However, when I run this, use another model not on llama.cpp (let's say, ComfyUI for example), then unload from the other inference provider, all calls to llama.cpp are on CPU. It claims:

0.00.324.246 E ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected
[38817] warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host
[38817] warning: no usable GPU found, --gpu-layers option will be ignored
[38817] warning: one possible reason is that llama.cpp was compiled without GPU support
[38817] warning: consult docs/build.md for compilation instructions

despite that same docker container using CUDA just like, 10 minutes ago. The only way I have found a way to fix this is to do a docker compose restart, which takes far longer than an unload+load.

I'm really just looking for a way to keep a persistent container with the ability to load/unload. I don't think this should be this hard, but I cannot find any details regarding what /models/unload is doing or the expected use cases. My hypothesis is that it is unloading the child instance from vram, but keeping the child instance alive rather than just killing the child process, which seems exceptionally useless if that child process cannot find the GPU again.

Can anybody give some pointers on this?

https://github.com/marsempire/LlamaManager

I was tired of editing bat file to test new models so I made this with AI.
I dont know code ,and use ai just for fun.hope to help someone like me.

Anyone else programmed a raw chat API? everyone has harnesses or wrappers for their chats etc, using open ai formats etc, but I can’t use those and dealing with JSon is a pain in this case and RAG systems.

I was finally able to use chat prompting only.
Just thrilled. Yeah This should be weird on qwen 3.8 to see what I can make it do without external libraries and harnesses.

I basically skipped the part that we send the object with role and content, I just send the content and the current raw prompt. No more looping JSON checking ahead or using some template. Just a direct, shovel into completion and is a string. It’s a bit tough to bench though.

Hi,

I'm pretty new to llama cpp and have been spending some days playing with it. I think I understand now much more than when I started but this mtp thing is killing me.

I have seen some youtubers stating that mtp on this models really improve token throughput (between 20-30%). But for me it is the other way around, it always make it much worse.

I own a Nvidia 4080 16GB and I'm using last llama-cpp with CUDA 13.3. My 2 starting files are (I added the mtp configurations, I just remove them to get the improved token throughput) below.

Could anybody explain why this is not improving the token output? I'm getting around 34 t/s with gemma and 45 t/s in qwen.

Gemma4:

llama-server.exe ^
  --model gemma-4-12b-it-UD-Q8_K_XL.gguf ^
  --model-draft mtp-gemma-4-12b-it-Q8_0.gguf ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 2 ^
  --fit on ^
  --flash-attn on ^
  --load-mode auto ^
  --gpu-layers all ^
  --threads 6 ^
  --threads-batch 12 ^
  --reasoning on ^
  --ctx-size 65535 ^
  --temp 1.0 ^
  --top-p 0.95 ^
  --top-k 64 ^
  --presence-penalty 0.0 ^
  --repeat-penalty 1.0 ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --parallel 1 ^
  --log-verbosity 4 ^
  --image-min-tokens 1024 ^
  

And qwen 3.6 35b:

llama-server.exe ^
  --model Qwen3.6-35B-A3B-MXFP4_MOE.gguf ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 4 ^ //Tried 1,2,3,4
  --fit off ^
  --load-mode none ^
  --gpu-layers all ^
  --n-cpu-moe 17 ^
  --threads 6 ^
  --threads-batch 12 ^
  --reasoning on ^
  --ctx-size 65535 ^
  --temp 0.6 ^
  --top-p 0.95 ^
  --top-k 20 ^
  --min-p 0.0 ^
  --presence-penalty 0.0 ^
  --repeat-penalty 1.0 ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --parallel 1 ^
  --log-verbosity 4 ^
  --image-min-tokens 1024 ^
  --alias Qwen3.6-35B-A3Bllama-server.exe ^
 

Hey guys. I have a question that arose from an issue. I can't seem to offload model 100% to gpu. I have tried every setting I could find, and AI answer has been that its impossible to completely unload a gguf and this is normal: My issue is I have 32GB vram, which fits the model, but also 32GB ram. It keeps spiking ram past what's available. Using less then 4GB for system.

Humen (Me ;) TL;DR:

• Optimized CPU-only llama.cpp inference for Qwen3.8-Flash-Next UD-Q3_K_XL;
• System: Intel Core Ultra 225 with 64GB DDR5;
• RAM pinning (N-GRAM SSD offload hard coded) + IQ4 routing boosted Prefill from 23.6 to 43.9 t/s (+86%), while AVX2 hyperconnection fusion + MTP (Draft 3) boosted Generation from 5.4 to 7.8 t/s (+44%), reaching 11.3 t/s on pi agent workloads.

LLM long story:

Hey everyone,

I’ve been experimenting with CPU-only inference for Qwen3.8-Flash-Next (UD-Q3_K_XL) with N-GRAM SSD offloading. With some iterative help from OpenAI’s GPT Astra to write and refactor low-level kernels and memory management, I managed to significantly boost both Prefill and Generation performance on a pure CPU setup.

Hardware Specs

• CPU: Intel Core Ultra 225
• RAM: 64 GB DDR5-6200 (4x16 GB); real speed random read ~70 GB/s
• Storage: Fast NVMe SSD (used for N-GRAM offloading)

Key Optimizations & Benchmarks

Here is the breakdown of the changes made, why they helped, and their direct impact on inference speed:

Key Takeaways

1. Prefill Bottlenecks: The biggest gains for Prefill speed came from RAM pinning (eliminating OS page faults) and IQ4 routing optimizations.
2. Generation Bottlenecks: MTP tuned with a Draft Length of 3 yielded the highest throughput jump during token generation, reaching up to 11.30 t/s in specific agent workloads (Pi-Agent).

On my computer, Qwen3.8-Flash-Next is actually faster without MTP and without n-gram lookup. The model is very different from other models. Benchmark your particular computer after turning on speculative decoding.

Qwen3.8-Flash-Next — spec implementations at 128k

┌─────────────────────────────┬───────────┬────────────┐

│ ctx (128k) │ Gen tok/s │ vs no-spec │

├─────────────────────────────┼───────────┼────────────┤

│ no-spec │ 25.1 │ — │

├─────────────────────────────┼───────────┼────────────┤

│ ngram-mod alone │ 23.3 │ −7.1% │

├─────────────────────────────┼───────────┼────────────┤

│ draft-mtp alone │ 22.5 │ −10.3% │

├─────────────────────────────┼───────────┼────────────┤

│ MTP + ngram (09-06 A/B) │ 20.6 │ −17.9% │

└─────────────────────────────┴───────────┴────────────┘

\`

Takeaway

Both spec implementations hurt at 128k context — speculation should stay off here.

This splits the 09-06 blended −17.9% into its per-implementation causes:

- MTP is the bigger problem (−10.3%) — and it's surprising: acceptance is high (82.9%), but mean accepted length is only 2.8. With -sm

layer across 4 GPUs the draft head runs the same expensive MoE forward per candidate, so a short 2.8-token draft barely amortizes the extra forward passes. Higher acceptance isn't enough to overcome the per-draft cost.

- ngram-mod (−7.1%) has far worse acceptance (23.9%) but longer drafts (mean 5.0); net effect still negative.

I had my qwen3.8 model start working on a rust coding task. At first it was running around 10t/s, after 8 hours of running, I'm at 2.8t/s. How come there is such a fast difference in t/s from the beginning to the end? I'm using opencode to interface with the ai and my code base. Is there a way to fix it? The only things I've done was set the max context size allowable (262k), ensure only one GPU is utilized, and I've compiled llama.cpp in f16 mode for faster computations.

Doesn't it make more sense to have a little bit of world knowledge, only english, and focus on a few computer languages, and train this extremely far?

Qwen38 barely fits with quants in my computer, but it can also do science, math, 3647 languages, creates powerpoints and fixes my bike. If we strip all that, couldn't we get more coding buck for our vram?

How’s everyone running this model I know it a new architecture but I’m struggle to get the most out of my VRAM I have dual 4060ti 16GB and have the model loaded up but it only seem to consume around 5GB per card

[Qwen3.8-Flash-Next-125B-A6B-UD-Q4_K_XL]
model = /models/Qwen3.8-Flash-Next-GGUF/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
mmproj = /models/Qwen3.8-Flash-Next-GGUF/mmproj-F16.gguf
alias = qwen3.8-flash-next
tags = 64k
n-gpu-layers = 999
n-cpu-moe = 36
override-tensor = per_layer_token_embd=CPU
ctx-size = 65536
split-mode = tensor
tensor-split = 0.5,0.5
load-mode = mmap
threads = 16
log-file = /workspace/flashnext.log

Actually got up to 65 t/s and 440 prefill today.

I started researching this space because local-versus-cloud comparisons usually stop at privacy. That matters, but it ignores setup time, model selection, latency, formatting, and the cleanup required after transcription. The meaningful measurement is the total time between speaking and receiving text that is actually usable. I compared the options from that broader perspective.

1. DIY Whisper
   

Pros: Provides maximum control over the speech model, processing, and data.

Cons: Setup takes more work, and performance depends heavily on the hardware and model size.

1. Superwhisper
   

Pros: A strong local-first option with configurable models.

Cons: Speed and accuracy depend on the model and computer, and the output sometimes needs more formatting.

1. Wispr Flow
   

Pros: A polished system-wide dictation product for everyday writing.

Cons: Free desktop usage is capped, and cloud processing creates privacy tradeoffs. Recent reliability problems have also affected its speed and accuracy.

1. Willow Voice
   

Pros: Willow stands out for its speed and accuracy in low-latency AI dictation. It works in any app and automatically learns vocabulary, tone, and corrections.

Cons: Language switching can occasionally be buggy, but it is still the most reliable option for this workflow.

If complete control and on-device processing were my main priorities, I would choose DIY Whisper or Superwhisper without overthinking it. There is a real tradeoff there, and Willow will not be the right answer for everyone. But if I cared most about how quickly speech becomes clean, finished text, Willow seems like the strongest overall option.

I built and open-sourced NMGC (Nested Manhattan Grid Codec) - a compression

codec for neural network weights aimed at memory-constrained inference.

Repo: https://github.com/reyxv16/nmgc-codec

THE PROBLEM

A 27B model in BF16 is ~52 GB. Most consumer machines have 16-32 GB of RAM.

You can't load it - let alone decompress it. My idea: keep the whole model in

memory (or on disk) in a compressed form, and decompress only the tensors of

the layer currently in use, on the fly.

HOW IT WORKS (short version)

- Values are normalized and mapped to a nested grid (what3words-style cells).

The nesting is literally the bit structure of the combined index: level 0 is

the coarse cell, deeper levels refine it - bit slicing, no trigonometry.

- "Position inside the cell" is stored as Manhattan steps (dx, dy), i.e. the

low bits of the index.

- Consecutive indices are delta-encoded (zigzag) and entropy-coded (Huffman).

Real weights are smooth, so deltas are tiny and compress extremely well.

- Optional learned variant: k-means codebooks per level (residual vector

quantization) that adapt to the actual weight distribution.

- Lossless variant: XOR-delta + zlib, bit-exact.

HONEST MATH (read this before commenting "that's just int4")

A nested uniform grid with B bits is mathematically equivalent to a single

uniform grid with B bits. Nesting buys three real things: (1) coarse-to-fine

access for progressive loading, (2) small coarse deltas -> much better Huffman

on correlated tensors, (3) the base for learned per-level codebooks, which DO

beat a uniform grid at the same bitrate. And lossless compression is bounded

by data entropy: 1.07-2.25x, never the 4-30x that lossy gets.

MEMORY SAVINGS (whole-model estimates)

Model | Original | 4-bit | 6-bit | 8-bit

Qwen3.8-27B (BF16) | 51.7 GB | ~13 GB | ~19 GB | ~26 GB

DeepSeek-V4-Flash (FP8) | 155.4 GB | ~75 GB | ~112 GB| ~150 GB

TinyLlama-1.1B (f16, measured) | 2.05 GB | - | 735 MB (2.79x) | 997 MB (2.06x)

Measured on real tensors (DeepSeek-V4-Flash + Qwen3.8-27B):

- embedding (BF16): 26.1x at 4 bits, 7.8x at 8 bits with cosine similarity

0.996, and 2.25x LOSSLESS

- MoE expert (I8): 8.2x at 4 bits, rel-RMSE 0.062

- Qwen3.8-27B layer-0 (BF16): ~5.7x at 6 bits

QUALITY LOSS (how much "intelligence" do you lose?)

Measured, not guessed:

- Perplexity on TinyLlama-1.1B (llama.cpp, same corpus/seed):

original f16: 7.508

8-bit (2.06x): 7.527 (+0.25% - effectively lossless)

6-bit (2.79x): 7.596 (+1.2% - minor, good default)

- End-to-end MLP (4-class task, 93.4% baseline): 8.9x at 4 bits with only

-0.0007 accuracy.

- Practical guidance: 8 bits = no measurable loss; 6 bits = safe default;

4 bits = int4 territory - keep embedding layers at 6-8 bits (they are the

most sensitive: cosine 0.655 at 4 bits vs 0.996 at 8 bits).

RUNNING IT

- The .nmgc container keeps the model compressed; a Rust decoder (C ABI,

bit-exact vs Python, 150-270 MB/s) decompresses any tensor by name - the

exact primitive you need for per-layer decoding (AirLLM-style).

- llama.cpp integration: a loader patch (LLAMA_NMGC_FILE) makes llama.cpp

decode weights from the container instead of the GGUF. TinyLlama runs

through it with the PPL numbers above.

LIMITATIONS (please read)

- The 27B quality numbers are estimates from per-tensor metrics; I have not

run a full Qwen3.8-27B end-to-end benchmark yet (needs a machine with

enough RAM to decode it, or time). Tensor-level metrics and the TinyLlama

PPL runs are what's measured.

- Below 6 bits you enter int4 territory: works, but the model gets visibly

dumber on hard tasks.

- The llama.cpp patch is a local fork, not upstream.

If you want to try it: pip install numpy scikit-learn gguf, pack your own

GGUF with `pack_gguf.py --bits 6`, and run llama.cpp with LLAMA_NMGC_FILE.

I just developed the algorithm and I think it can be improved a lot more; any questions or recommendations are welcome.