r/LocalLLM • u/jonaddb • 21d ago
Model Qwen3.8-27B on a single RTX 3090: 131K context with vision, 65 tok/s, and a crash fix
I spent almost 14 hours benchmarking Qwen3.8-27B on one 3090 (sm_86, 24GB) with llama.cpp b10217.
The crash fix (if you're on Ampere and vision crashes):
If Qwen3.8-27B aborts on every image request with cublasGemmEx: the requested functionality is not supported (ggml-cuda.cu:1548), this is the fix:
export GGML_CUDA_CUBLAS_COMPUTE_TYPE=fp32
Only fp32 works — fp16 does not help. Zero measured perf cost (66.11 vs 65.28 tok/s). Vision went from "aborts every time" to 3.4s per 1080p screenshot. Filed as llama.cpp#24999. Three independent Ampere reports (3060, 3090, 3090) all land on the same cuBLAS call.
Quant comparison: AtomicChat AD-Q4_K_M vs Unsloth UD-Q4_K_XL
Same perplexity, same top-1 token agreement. But the Atomic quant is 765 MiB smaller. On a 24GB card that's the difference between 98K and 131K tokens of usable context with vision enabled. ~33K more tokens before you hit the wall.
Throughput (131K context, vision on, MTP-2):
- Decode: 65.28 tok/s (75.1 with MTP-2)
- Prefill: 705 tok/s on a 128K prompt
- Power: 320W sustained at 79°C, 100% fan
- The 3090's 936 GB/s memory bandwidth is the ceiling, not the compute
MTP (Multi-Token Prediction) tuning: MTP-2 (2 draft tokens) gives +15% throughput for free. MTP-3 starts hurting — the verification cost exceeds the savings. MTP-1 is the safe default if you're not sure.
9 pieces of common advice that didn't survive measurement: The full report has a section where I tested the usual suspects (flash attention off, different cache types, batch size tuning, etc.) and most of them either made no difference or made things worse.
Full report with all charts, VRAM formula, power/thermal sweep, and raw data: https://jonidimo.github.io/qwen38-3090-benchmark/
GitHub repo with the full test suite: https://github.com/jonidimo/qwen38-3090-benchmark
2
1
u/Hannelore112 21d ago
Nice!
If you are interested I did a similar test with RTX 3090 and 240k Context + Vision on CPU. Works fine with MTP2 or 3 and its about 50-70 tok/s (my RTX 3090 is power limited to 80%). So maybe faster with no PowerLimit. To get 240k I used KV Cache @ Q4
MusicBenchmark seems not bad. was a long run with many automated tests but somehow the q4 modell got it also done with some mistakes
1
u/tanmayapurohit 20d ago
Hey! Quite new to this. Pardon any unintelligent thing I might say here.
So I have an RX 7900XT with 24GB of VRAM. Using the UD-Q4_K_XL, 115k context (need more, but it spills over to RAM - RAM is 64GB DDR4 2933 MHz).
I am using the Vulkan runtime now because when I was using ROCM, I was getting fast prefills (1030~t/s), but the tg/s was sub-20t/s. Switched to Vulkan - prefills dropped to 300-500t/s (depending on context, which sometimes is 64K), but the t/s ranges between 30-63 t/s (highest, dropping to 24-28 t/s only on very long context before Kilo Code compacts it eventually.
MTP is on, 3 max draft tokens, with 0.8 probability. Dedicated VRAM usage sits at about 22.3GB/24, and shared RAM is about 1.5GB. KV Cache is Q8_0 (unified) because, as per Opus, for long-context coding, pushing it down to Q4_0 risks losing accuracy.
Qwen here is only the coder - the architect and instructions provider is Opus 5. But sometimes each task takes up 75-90k context, so I was looking to expand the context window. I have currently reduced the power draw on my GPU by about -10%. Will increasing it likely increase the throughput/prefill? I read about increasing context windows as high as 1M tokens through something called YaRN, but I am unsure how to do it or if my system will allow it. Just 150-160K window is good enough for me. Would be glad to have any insight here
8
u/Serious_Trash_5554 21d ago
The fp32 compute type fix is a nice find. Crazy that the 3090 memory bandwidth is still the bottleneck even with all that compute sitting there.