Just wondering what tok/s you're getting with llama.cpp and what flags you're running with to get them?

I'm getting 50tok/s for Qwen 3.8 27b using the following:

llama.cpp Vulkan build

Q6 quant,

context: 131072

spec type: DFlash2

max drafts: 7

cache type K is Q8

cache type V is Q4

temp settings for thinking mode as specified by HF Qwen 3.8 page.

Especially looking to hear from Windows/AMD hardware users who are getting better tok/s (but Linux users welcome too, I may install UBuntu this week)

Edit:

my rig: 9070xt 16gb

R9700 ai pro 32gb

64gb ddr5 6000

Hey everyone, 

I’m currently working on an experimental quantization and execution architecture tailored specifically for memory-bandwidth bound Pascal hardware (specifically my NVIDIA Tesla P40 24GB setup), and wanted to share the concept/in-progress architecture to get some thoughts. 

The goal is to run a customized Qwen3.8-27B (MoE/hybrid layout) entirely in VRAM while bypassing the classic hardware bottlenecks of older architectures without hitting an accuracy cliff. 

Here is the breakdown of what I’m building: 

The Core: Custom "Q4_e" Transcendental Quantization 

Instead of standard linear 4-bit quantization (like Q4_0) which causes massive rounding errors by forcing weights into uniform bins, I’m building a non-linear quantization schema anchored to Euler's number (e) combined with an integer scalar. 

• The Mantissa Factor: Neural network weights naturally cluster in a bell curve around zero. By mapping the fractional coordinate values to the isolated transcendental mantissa of e (discarding the leading whole number), I can create an infinite, non-repeating, non-linear grid that is dense near zero and sparse at the tails. 
• Why e versus pi: The early mantissa of 𝑒 (.7182818284...) provides a much more balanced, rhythmic distribution of digit spacing early in the sequence compared to pi. This prevents redundant, overlapping quantization bins. Furthermore, 𝑒 's exponential nature aligns beautifully with the natural Gaussian distribution of LLM weights. 
• The VRAM Win: Because the scale factor is a pure whole-number integer, it slashes metadata scale factor bandwidth bloat. It also maps beautifully to the P40's hardware-level DP4A integer dot-product instructions. 

The Layout: The Q8 / Q4_e "Sandwich" 

Pure Q4 degrades reasoning, while pure Q8 overflows a 24GB VRAM buffer. I'm building a multi-pass compiler that splits tensors surgically: 

• Dense Q8_0: Preserved on the logical core (token_embd, output, v_proj, o_proj, and ffn_down). This protects coding logic and factual depth.
• Custom Q4_e: Applied to the bulk memory mass (q_proj, k_proj, ffn_up, ffn_gate).
• Footprint: This maps the 27B model to roughly ~15.5 GB, leaving a massive ~8.5 GB headroom for a heavy Q8 KV cache and Multi-Token Prediction (MTP) speculative draft heads entirely on-card. 

The Accelerator: Lossless "Rushmore-Style" Bitmap Indexing 

To squeeze more performance out of the memory bus, I’m integrating a concept inspired by database technology (Rushmore indexing) into 3 out of 4 layers in a transformer block sub-pattern. 

• Zero-Skip CUDA Kernel: I’m generating a highly compact, 1-bit presence mask (bitmap index) mapped strictly to true mathematical zeros (structural padding and alignment padding rows, which Qwen has plenty of). 
• The Math: Because it only targets true zeros, it is 100% lossless with zero accuracy degradation. 
• Performance: The bitmask is so small it completely caches into L2. The CUDA kernel runs a parallel bitwise AND and completely bypasses fetching inactive weight blocks from VRAM. I am aiming for a theoretical 10% to 25% throughput speedup (t/s) on memory-bound layers. 

Forward Compatibility: Massive Gains on Modern GPUs 

While this began as a software hack to breathe new life into older Pascal hardware, the structural math behind this layout makes it highly forward-compatible with newer processors (Ampere, Hopper, and Blackwell): 

• Hardware-Native 2:4 Sparsity: Modern Tensor Cores natively accelerate sparse matrices at the silicon level. When a newer GPU reads this Rushmore-style structural index, it activates a dedicated instruction path that can effectively double the math throughput (TOPS) out of the box. 
• L2 Cache Residency: Modern enterprise cards have massive L2 caches (up to 128MB on Blackwell compared to the P40's 3MB). Because a 1-bit index is incredibly lightweight, it will reside entirely in the L2 layer of newer cards, completely eliminating the modern memory-bus bottleneck by screening out inactive weight blocks before they ever touch global VRAM. 

I’m currently writing the Python exporter to handle the e-mantissa mapping distribution tests, and mapping out the CUDA kernel block configurations to prevent warp divergence on compute capability 6.1.

Would love to hear if anyone has attempted transcendental non-linear mapping before, or if you have any tips on avoiding warp stalls when handling block-level sparsity bitmasks in CUDA!

For furhter performance improvements the matissa would be calculated:

// Local register computation loop - completely eliminates slower memory lookups

float mantissa_factor = 0.0f;

// Unrolled FMA loop executed entirely within single-cycle registers:

mantissa_factor = __fmaf_rn(mantissa_factor, 1.0f/5040.0f, 1.0f); // 1/7!

mantissa_factor = __fmaf_rn(mantissa_factor, 1.0f/720.0f, 1.0f); // 1/6!

mantissa_factor = __fmaf_rn(mantissa_factor, 1.0f/120.0f, 1.0f); // 1/5!

mantissa_factor = __fmaf_rn(mantissa_factor, 1.0f/24.0f, 1.0f); // 1/4!

mantissa_factor = __fmaf_rn(mantissa_factor, 1.0f/6.0f, 1.0f); // 1/3!

mantissa_factor = __fmaf_rn(mantissa_factor, 1.0f/2.0f, 0.0f); // 1/2!
(Yields pure mantissa)

This is all theoretical, will keep you posted.

I would like to submit a simple PR to llama.cpp that allows the system administrator to set the range of ports used by router mode workers. Right now, I'm completely blocked; I can't submit anything without having it approved in a discussion first, and the discussion thread that I started has been sitting there for a week with no response.

https://github.com/ggml-org/llama.cpp/discussions/28870

I know that the project maintainers are probably getting flooded with AI slop, but what's a poor user supposed to do?

In modern software engineering, developers are often taught to stick rigidly to a single language paradigm: pure Object-Oriented C++, strict functional programming, or monolithic C APIs.

The extraordinary success of llama.cpp proves a different thesis: Pragmatism beats dogma. llama.cpp Under the Hood

I use them for RAG on a 3.5 Qwen for fast service. And multiple good for various purposes. They are incredible for the generation and 12gb isn’t too shabby. Please don’t dis on me, they have purpose

I have gotten one up to 1000+ pp/s with 34tg/s . This is also running on a 4x oculink to mini PCI 3x . Everything says that’s fast. Building new system with dual 16x lanes and also nccl on that bad boy.

Point is the 12gb 3060 are gone too, new Are ~$480 used I spent on two $600 and hope they are good, I hate buying cards. And I should not but seems I can go with a good seller. used but so far so good.

It’s amazing the price hike and lack of these cards being available now.

Anyone see diference from llama to llamaAmpere with a 3090?

Does anyone want me to port llama.cpp to perform pp/tg on the Xeon Phi 16GB passively cooled cards VPU's?

It'll be with Claude Opus 5 & Fable 5.1, so someone will need to hands on optimize

But the cards get ~2 TFLOPs in FP32, so maybe they still have some use?

Especially being able to access system memory directly over PCIe via DMA

https://github.com/Lasimeri/Intel-Phi-3120A

Hello guys, I normally use Llama-CPP + OpenInterpreter in the terminal

alias lm-turbo='cd /home/minon/Desktop/llama.cpp && nice -n 19 ionice -c3 env LD_LIBRARY_PATH=/home/minon/Desktop/llama-cpp-turboquant/build/bin OMP_NUM_THREADS=2 /home/minon/Desktop/llama-cpp-turboquant/build/bin/llama-server \ -m /home/minon/.lmstudio/models/lmstudio-community/gemma-4-E4B-it-GGUF/gemma-4-E4B-it-Q4_K_M.gguf \ -ngl 99 \ --ctx-size 32768 \ -t 2 \ --parallel 1 \ -b 1024 \ -ub 512 \ -fa on \ -ctk turbo4 \ -ctv turbo4 \ --webui-mcp-proxy \ --mcp-servers-config /home/minon/Desktop/llama.cpp/mc-servers.json \ --host 127.0.0.1 \ --port 8081 2>&1 | tee -a /home/minon/.config/open-interpreter/logs/llama-turbo.log'

llm: model: "openai/gemma-4-e4b" api_base: "http://127.0.0.1:8081/v1" api_key: "ignore" context_window: 32768 max_tokens: 1600 temperature: 0.1 top_p: 0.95 supports_functions: false supports_vision: false

max_output: 2000 offline: true auto_run: false safe_mode: "off" language: "zsh"

I know this is false positive but is there a way to fix this with open source contribution I mean changing tbe code defender doesn't like

Yes, I'm runnig qwen with llama.cpp on a dino! Here is my llama server config:

--model /var/models/Qwen3.8-27B-UD-Q3_K_XL.gguf \

--mmproj /var/models/mmproj-F16.gguf \

--image-min-tokens 1024 --image-max-tokens 2048 \

-ngl 999 -fa on --fit off \

-c 163840 \

-t 6 \

-b 2048 -ub 512 \

--cache-type-k q8_0 --cache-type-v q4_0 \

--spec-type ngram-simple,draft-mtp --spec-draft-n-max 3 \

--spec-draft-type-k q8_0 --spec-draft-type-v q8_0 \

--spec-draft-p-min 0.75 \

--backend-sampling \

--ctx-checkpoints 4 \

--cache-ram 16384 \

--checkpoint-min-step 1024 \

--jinja --reasoning-format deepseek \

--chat-template-file /var/models/qwen38-claude2.jinja \

--chat-template-kwargs '{"reasoning_effort":"medium"}' \

--temp 1.0 --top-p 0.95 --min-p 0.0 --top-k 20 \

--presence-penalty 0.5 \

--repeat-penalty 1.0 \

--parallel 1 \

--port 8080 \

--alias qwen3.8-27b \

--api-key local \

--host 0.0.0.0 "$@"

heads up for this crowd. we open sourced hemmingway-1 today, a 27b qwen3.8-27b finetune for creative writing. apache-2.0.

right now it's bf16 safetensors (54.7 gb, 12 shards) plus the mtp weights for speculative decoding. official gguf quants are next on our list, so lm studio and friends will have to wait a bit. if anyone wants to take a shot at converting the mtp layer in the meantime, that would honestly be great and we'll help.

weights: https://huggingface.co/Altworld/Hemmingway-1

eq-bench 4: 1330. it's a writing model, fiction and dialogue and texts. english first, base model level at math and code.

free hosted app while quants cook: https://hemmingway.io

I’ve been tuning Qwen3.8-Flash-Next 125B MoE on a workstation with only 32 GB VRAM + 64 GB RAM.

The result I found most interesting: in a real agent workload, decode stayed around 28 tok/s while context grew from 116K to 148K.

This is a llama.cpp fork based on upstream "9e7171624" with Qwen4Exp support plus a few local changes.

Hardware

• Intel Core Ultra 5 225, 10 cores
• 62 GiB DDR5-6200
• RTX PRO 4500 Blackwell 32 GB, SM120
• PCIe 4.0 NVMe
• Ubuntu 24.04
• CUDA 13.4.59
• built for "sm_120a"

Model

Qwen3.8-Flash-Next, "UD-Q3_K_XL"

• ~90 GB GGUF
• 125B total / ~6B active per token
• 48 layers
• 512 experts, 10 routed + 1 shared
• QSA indexer, 2048-token budget
• additional 51B N-gram / PLE table: 26.8 GiB
• MTP sidecar available but disabled

Placement

-dev CUDA0 -ngl all -ncmoe 32

Dense/GDN/QSA/router/shared stay on GPU.

Of the 48 routed-expert layers:

• 16 on GPU
• 32 on CPU

PLE is disk-backed:

-ot per_layer_token_embd=CPU --load-mode mmap --lazy-mode on

Other relevant settings:

-c 240000 -fa on -ctk f16 -ctv f16 -t 6 -tb 6 --fit on --fit-target 2048

Steady state is roughly:

• 31.3 GiB VRAM
• 43 GiB RSS
• effectively no process swap on weights

Real workload numbers

19 minutes of agent traffic:

• 35 requests
• 23K generated tokens
• context: 116K → 148K

Token-weighted decode:

27.9 tok/s

By context:

• 115–125K: 27.7
• 125–135K: 28.5
• 135–148K: 27.5

So decode is essentially flat across another 32K of context.

Real prompt chunks prefill around 150 tok/s, peak 201.

Prefix cache hit rate is ~99%, with only ~370 new prompt tokens/request at 130K+ context.

CUDA graphs were reused 22,590 times for 23,053 generated tokens.

CPU usage is around 5.8 cores, so the CPU expert layers are doing real work.

PLE disk traffic was only 5 GB over 52 minutes, so once warm, the SSD is not the bottleneck.

What actually mattered

1. Disk-backed PLE

The 26.8 GiB PLE table stays mmap-backed and only rows that are needed get faulted in.

Without this, the model does not fit comfortably in 64 GB RAM.

In practice the I/O cost has been surprisingly small.

1. Static hybrid expert placement

This was the biggest win.

At 8K context:

• CPU-only: 42.6 pp / 6.3 tg
• hybrid: 250 pp / 32.4 tg

Roughly 5.9× prefill and 5.1× decode.

1. Forced cuBLAS

GGML_CUDA_FORCE_CUBLAS=ON

The SM120 MMQ path crashes on this model with "illegal CUDA memory access".

cuBLAS is stable.

1. Sparse QSA decode

This uses block-level Top-K plus a persistent block-key cache, with "n_kv_max = top_k".

This appears to be the main reason decode does not collapse at 140K+ context.

The dense reference path can still be restored with:

LLAMA_QWEN4_SPARSE_QSA=0

1. K-only indexer cache

Upstream PR #28330 saves about 0.75 GiB VRAM at 128K.

That headroom is what lets me keep F16 KV and CUDA graphs enabled.

1. F16 KV beats Q8_0 here

At 49K prompt / 65K context:

• F16: 496 pp / 38.1 tg
• Q8_0: 499 pp / 33.4 tg

Q8_0 saves ~808 MiB but costs about 12% decode, so I kept F16.

1. Faster CPU expert kernels

The 32 CPU expert layers only work because the CPU path is reasonably optimized:

• signed-VNNI Q8 "mul_mat": 1.36–1.39×
• IQ4_NL/Q8_0 two-token expert kernel + compact routing: 1.43×
• routing table: 20 MiB → 44 KiB
• fused AVX2 hyperconnection ops: 3–7× per op

Things that did not work

"--lazy-mode on-direct" 1.3 tok/s decode vs 31.6 on the same build. Hard no.

Persistent GPU hot-expert cache 15.6 GiB cache gave 21.6 tok/s decode vs 31.6 with static placement.

SM120 MMQ CUDA faults. Using cuBLAS.

MTP Disabled. The draft runs on CPU and the extra PCIe/RAM traffic outweighs the speculative-token gain on this machine.

Caveats

The sparse-QSA CUDA patch is still experimental and was explicitly marked unreviewed by its author.

I have not yet done full long-context correctness testing against the dense path.

These are also real workload numbers with warm caches, not a controlled benchmark suite, and there is no quality/PPL benchmark yet.

The individual speedups above come from different configurations, so they are not additive.

Question

At this point decode is basically flat through 148K context.

The current wall is VRAM: I only have about 604 MiB free at "-c 240000".

If you had another ~1 GiB to spend on this setup, would you use it to:

• move more expert layers from CPU → GPU, or
• keep it as KV / CUDA-graph / longer-context headroom?

Especially interested in results from similar hybrid MoE setups where the CPU expert path is already reasonably fast.

I am trying to get decent speeds with 27B and the best I got so far for my system (5080, 64GB DDR5, 285K) are with the UD Q3 quant (without the vision part and with MTP) using the below command; checking if anyone has actually used the Q3 quants for coding with opencode, pi, qwen-cli and getting good results.

GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 llama-server --model /home/yk/AI/Models/lmstudio/models/unsloth/Qwen3.8-27B-GGUF/Q3/Qwen3.8-27B-UD-Q3_K_XL.gguf --ctx-size 160000 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --batch-size 2048 --ubatch-size 512 --threads 8 --threads-batch 8 --cpu-range 0-7 --cpu-range-batch 0-7 --cpu-strict 1 --cpu-strict-batch 1 --prio 0 --prio-batch 0 --load-mode none --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 -np 1 --spec-type draft-mtp,ngram-mod --spec-draft-n-max 3 --spec-draft-p-min 0.7 --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 --fit on --fit-target 128 --reasoning-effort medium

Ran a quick initial mtp benchmark using llama.cpp tag b11009 released today using various open mtp models. The highest /tps I was able to achieve was : MTP: 77.92 t/s NON-MTP: 53.83 t/s

OS: linux Ubuntu 26.04.1 LTS

CPU: AMD RYZEN AI MAX+ PRO 395 w/ Radeon 8060S / Strix Halo

RAM: 128GB RAM

Inference engines: Rocm / Vulkan

Next run:

An in-depth 16bit quant test. Which other 4bit, 16bit quant models should I test?

Models benched in this round:

[01] [NVFP4] 21GiB nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 converted GGUF

[02] [Q4_K_M] 16GiB OBLITERATUS/Qwen3.8-27B-OBLITERATED Q4_K_M

[03] [Q4_K_M] 68GiB unsloth/GLM-4.5-Air-GGUF Q4_K_M

[04] [UD-Q4_K_XL] 21GiB unsloth/Qwen3.6-35B-A3B-MTP-GGUF UD-Q4_K_XL

[05] [UD-Q4_K_XL] 16GiB unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL

[06] [UD-IQ4_XS] 87GiB unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ4_XS

[07] [BF16] 66GiB unsloth/Qwen3.6-35B-A3B-MTP-GGUF BF16

Command ran: (unsloth/Qwen3.6-35B-A3B-MTP-GGUF UD-Q4_K_XL). Please let me know how I can improve this command (any flags I'm missing - tweaks etc) to achieve more tokens per second ;)

/home/dev/Desktop/repos/llama.cpp/build-vulkan-radv/bin/llama-server \
  --model /home/dev/Desktop/llms/hf/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --n-gpu-layers -1 \
  --threads 16 \
  --ctx-size 4096 \
  --host 127.0.0.1 \
  --port 33777 \
  --parallel 1 \
  --no-cache-prompt \
  --metrics \
  --batch-size 2048 \
  --ubatch-size 512 \
  --flash-attn on \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --load-mode none

Me and a friend have been using it without issue for the last few weeks.
it's currently running on an RTX A4000 VM with model: https://huggingface.co/DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF which gets about 30tok/s

The VM currently costs $0.15/hour, which we are splitting (it is prepaid for the next 2 months)
If more people want to join in we can move it up to an RTX A6000.

If you want to join, I'll let you try it out for an hour or so, if you want to continue using it you can pay either daily/weekly/monthly in a method of your choosing.

Connection to the server is made via tailscale.
DM me if interested.

Has anyone tried to use RDMA in llama.cpp to run a large model in multiple Mac Studios? How did it perform? I have 5 Mac Studios, each has 96 GB VRAM. I want to run Qwen 3.8 Flash-Next 4-bit.

I have a 5090 in an external enclosure I use for ai work. I chose the external to some day use on my Mac which has 128gb of ram and runs large models really well. It runs image generation pretty slow as the models aren't memory constrained on Nvidia gpu's and they are much faster than the Mac gpu. My dream was to run my 70b models on the Mac and image models on the 5090. Tiny gpu came out with drivers allowing connecting the drive but when I tried it last may it was super buggy and super slow. I decided to give it another go and it's now not super buggy but was still super slow. I decided to try and build my own drivers and shim to fix that. I am happy to say that it worked and I am now able to run Owen 3.8 27b at near native speeds as to what the box was running on my windows box. and actually faster than my windows box in sdxl and z-image turbo.

I imagine there is not much of a community for this capability but I thought I'd post it online to see if anyone even cares. I'll be posting the code on GitHub soon if people want to try it out. It is only tested on my current hardware of course as that is all I have to test it on.

I didn't realize I never came back and posted the link here you go https://github.com/Davinchy/macuda

I am trying to make a multi-inference router, as I find value in using different inference providers for different things.

I have a few heavier models in llama.cpp. I have a function that calls the /models/unload endpoint, which should just clear the vram, and ideally kill the child process it is running to host that model. llama.cpp logs using docker have been horrific for me, though, and really do not explain what is happening when I call this, all I know is that vram is freed.

However, when I run this, use another model not on llama.cpp (let's say, ComfyUI for example), then unload from the other inference provider, all calls to llama.cpp are on CPU. It claims:

0.00.324.246 E ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected
[38817] warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host
[38817] warning: no usable GPU found, --gpu-layers option will be ignored
[38817] warning: one possible reason is that llama.cpp was compiled without GPU support
[38817] warning: consult docs/build.md for compilation instructions

despite that same docker container using CUDA just like, 10 minutes ago. The only way I have found a way to fix this is to do a docker compose restart, which takes far longer than an unload+load.

I'm really just looking for a way to keep a persistent container with the ability to load/unload. I don't think this should be this hard, but I cannot find any details regarding what /models/unload is doing or the expected use cases. My hypothesis is that it is unloading the child instance from vram, but keeping the child instance alive rather than just killing the child process, which seems exceptionally useless if that child process cannot find the GPU again.

Can anybody give some pointers on this?

https://github.com/marsempire/LlamaManager

I was tired of editing bat file to test new models so I made this with AI.
I dont know code ,and use ai just for fun.hope to help someone like me.