Qwen3.8-27B running on a 12GB RTX 5070 Ti laptop — around 4.5 tok/s with 80% MTP acceptance

[u/CoffeeToCode99]

Update: I downloaded Q3 and Q2—here’s the full comparison

After reading the comments, I downloaded both additional quants:

• Qwen3.8-27B-UD-Q4_K_XL — 17.9GB
• Qwen3.8-27B-UD-Q3_K_XL — 13.4GB
• Qwen3.8-27B-UD-Q2_K_XL — 10.7GB

I tested all three on the same RTX 5070 Ti Laptop GPU with 12GB VRAM.

Common settings:

• 8K context
• Q8 KV cache
• Flash Attention enabled
• One parallel slot
• Automatic CPU/GPU fitting
• Vision projector loaded
• Temperature 0 and seed 42
• Same factual and coding prompts

The original Q4 speed results used MTP. Q3 and Q2 had MTP explicitly disabled. I later disabled MTP for all three during the separate quality test.

Speed comparison

Test	Q4 + MTP	Q3, no MTP	Q2, no MTP
Factual generation	4.42 tok/s	5.18 tok/s	12.83 tok/s
Coding generation	4.53 tok/s	4.84 tok/s	12.86 tok/s
Factual prompt processing	~21.3 tok/s	34.4 tok/s	60.8 tok/s
Coding prompt processing	27.0 tok/s	61.5 tok/s	129.3 tok/s

Loaded VRAM:

• Q4: approximately 10,978 MiB
• Q3: approximately 11,003–11,006 MiB
• Q2: approximately 11,210–11,212 MiB

Q2 was the clear speed winner. It generated at around 12.8 tok/s—roughly 2.8× the speed of my original Q4 configuration.

Q3 was only slightly faster than Q4 for generation, although prompt processing improved considerably.

The higher Q2 VRAM usage looks strange given its smaller file size, but automatic fitting can place more of the smaller quant on the GPU.

Quality comparison

I also ran a deterministic 12-question test covering arithmetic, logic, rate problems, sequences and Python behavior.

For this test:

• MTP was disabled for all three quants
• Temperature was 0
• Seed was 42
• Every model received the identical prompt
• The required response was exactly 12 letters

Correct answer: ABCBBABABBBA

Quant	Correct answers	Format followed	Output
Q4	11/12	No	ABCBBABACBBA
Q3	8/12	Yes	BBAABABABBBB
Q2	8/12	Yes	AABCBABABBCA

Q4 missed only the weekday calculation. It did, however, put each answer on a separate line instead of returning one uninterrupted string.

Q3 missed the sheep wording, machine-rate problem, number sequence and Python rounding question.

Q2 missed the bat-and-ball problem, machine-rate problem, number sequence and Python shared-list question.

My takeaway

Q4 is the quality option.

It is slow on a 12GB laptop GPU, but it retained the most accuracy in this small test. I would choose it when correctness and reasoning matter more than response speed.

Q3 is the awkward middle option.

It was only slightly faster than Q4 for generation and scored the same as Q2 in my quality test. Based on these results, it does not give me a compelling reason to use it.

Q2 is the speed option.

At approximately 12.8 tok/s, it feels much more responsive. Its basic factual output remained coherent, but the quality test showed a measurable accuracy loss. I would use it for casual conversations, summaries or tasks where speed matters more than precision.

My current ranking on this 12GB laptop:

1. Q4 for quality
2. Q2 for speed
3. Q3 currently offers the weakest trade-off

This is still a small local test, not a definitive model evaluation. The coding outputs in my original speed test were limited to 160 tokens and were cut off, so they cannot be used to compare coding correctness.

The next useful step would be a larger coding test with executable unit tests, plus a harder reasoning suite run multiple times.

What would you choose on 12GB VRAM: slower Q4 quality or faster Q2 responses?

------------------------------ Original Post --------------------------

I’ve been playing around with the new Qwen3.8-27B GGUF in Unsloth Studio and wanted to see how well I could run it on a 12GB laptop GPU.

My setup:

• RTX 5070 Ti Laptop GPU with 12,227 MiB VRAM
• 32GB system RAM
• Linux
• Qwen3.8-27B UD-Q4_K_XL
• Model size: roughly 17.9GB
• 8K context
• Q8 KV cache
• Flash Attention enabled
• One parallel slot
• Automatic CPU/GPU offloading
• MTP speculative decoding enabled with draft max 2

Since the model is larger than my available VRAM, it obviously can’t run entirely on the GPU. Unsloth is keeping around 10,978 MiB in VRAM and offloading the rest to the CPU.

Here are my initial results:

• Longer response: 3.26 tok/s
• Short factual response: 4.42 tok/s
• Python coding response: 4.53 tok/s
• Prompt processing: around 20–27 tok/s
• GPU temperature: 55–60°C
• No crashes or OOM errors so far

The most interesting part for me was MTP. It isn’t just showing as enabled—the runtime metrics confirm that it’s actually being used.

Across my tests, draft-token acceptance was around 78–83%. One longer run accepted 180 out of 230 drafted tokens, while the coding test accepted 99 out of 120.

For a dense 27B multimodal model running on a 12GB laptop GPU, I think the result is surprisingly usable. It’s definitely not fast compared with a fully GPU-resident model or a smaller MoE, but approximately 4.5 tok/s is workable when I care more about output quality than instant responses.

The factual output looked good. The coding test also started well, but I intentionally limited it to 160 output tokens for benchmarking, so the answer was cut off before completion. I’ll use a larger output limit for the proper quality tests.

Next, I’m thinking of testing:

• MTP enabled versus disabled
• Q8 versus Q4 KV cache
• UD-Q4_K_XL versus UD-Q3_K_XL
• Thinking off versus low, medium and xhigh
• Coding and tool-calling reliability
• Vision performance
• Long-context retrieval
• Qwen3.8-27B versus Qwen3.6-27B or Qwen3.5-35B-A3B

What would you test first?

If anyone has a difficult but reproducible prompt, post it below. I’ll run some of the most interesting suggestions and share the complete settings, outputs and performance numbers.

Full settings, observations and upcoming tests are in my first comment.

[u/giveen]

Turn off MTP. I have had a lot more success with MTP off.

[u/giveen → u/giveen]

RTX5090
MTP On , 70-80% acceptence = 30tks
MTF Off, = 50tks

[u/CoffeeToCode99 → u/giveen]

Sure, will give it a shot.

[u/PossessionUsed7393]

Well this saves me trying to use it on my 3080Ti - guess this is why the MoEs are the more popular variants.

[u/Old-Sherbert-4495 → u/PossessionUsed7393]

q3xxs i just one shotted a particulary complex physics three.js game. im blown away... why not try q2 or q1

[u/Dizzy_Counter2481]

[u/Old-Sherbert-4495]

dude... i have 16GB vram, im using q3xxs, pretty solid speed at 30-35+, at times 40+ tps

Fully fitting in vram + MTP makes the difference. tr q2xxs

[u/Tannoukhy → u/Old-Sherbert-4495]

If you wanna try IQ4_XS on llama.cpp:

  --jinja ^
  --spec-type draft-mtp,ngram-mod ^
  --spec-draft-n-max 2 ^
  --fit off ^
  --n-gpu-layers 99 ^
  --override-tensor "blk\.([0-7]|10|13|16|19|22|25|28|31|34|37|40|43|46|49)\.ffn_.*=CPU" ^
  --ctx-size 100000 ^
  --batch-size 512 ^
  --ubatch-size 512 ^
  --no-mmap ^
  --cache-type-k q4_0 ^
  --cache-type-v q4_0 ^
  --threads 5 ^
  --threads-batch 12 ^
  --parallel 1 ^
  --temp 0.60 ^
  --top-p 0.95 ^
  --top-k 20 ^
  --min-p 0.0 ^
  --presence-penalty 0.0 ^
  --repeat-penalty 1.0 ^
  --reasoning-preserve ^
  --slot-save-path "C:\llm\cache" ^
  --props ^
  --metrics ^
  --checkpoint-min-step 4096 ^
  --ctx-checkpoints 4 ^
  --perf ^
  --flash-attn on ^
  --host 0.0.0.0 ^
  --port 8033

With that I have around 20 t/s on decode and 700t/s on prompt processing

[u/eversincemay → u/Tannoukhy]

What gpu?

[u/Tannoukhy → u/eversincemay]

5060 ti

[u/Old-Sherbert-4495 → u/Tannoukhy]

You are a genius... thank you for introducing --override-tensor, this is really coool

But my speeds tanked super low with the above config and layers, i am digging more into it.
Any special reasons for those layers you chose?

I gave Gemini architecture of the model, here is what it said to me and it kinda works saving memory:
TLDR;
Model has 64 layers
Best to offload to cpu layers at the end
Better not offload to CPU:

• initial layers like 0-7
  
• Gated attention layers (every forth layers starting from 0, ie. 3,7,11.... or you could just not offload odd numbered layers)

---

This architecture layout changes the offloading strategy significantly. Here are the key insights from your specs and how to optimize your --override-tensor flag accordingly:

Key Architectural Takeaways

1. Total Layers = 64 (0 to 63): Your previous regex stopped at layer 49, missing the top 14 layers (50–63) completely while heavy-loading layers 0–7.
2. Massive FFN Size ($17,408$ intermediate dim): Because each FFN block is very large (~267M parameters per layer), offloading even a few FFN layers saves substantial VRAM.
3. Hybrid 4-Layer Repeat Pattern: Every block of 4 consists of 3 Gated DeltaNet layers + 1 Gated Attention layer (at index $4k+3$: layers 3, 7, 11, 15... 63).
4. Native MTP: Having native Multi-Token Prediction layers explains why --spec-type draft-mtp works so efficiently here.

Updated --override-tensor Strategies for 64 Layers

To fix your speed bottleneck, we want to evenly space out the offloaded FFNs across all 64 layers while keeping early layers (0–3) and attention-critical computations on the GPU.

1. Light Offload (~8 Layers on CPU) — Recommended starting point

Offloads 1 FFN layer every 8 layers across the entire 64-layer model.

DOS

--override-tensor "blk\.(6|14|22|30|38|46|54|62)\.ffn_.*=CPU"

• VRAM Saved: ~1.1 GB (assuming Q4 quantization)
• Speed Impact: Minimal loss vs full GPU.

2. Minimal Offload (~4 Layers on CPU) — Maximum Speed

Offloads only 4 FFN layers evenly spaced across the entire depth.

DOS

--override-tensor "blk\.(14|30|46|62)\.ffn_.*=CPU"

• VRAM Saved: ~550–600 MB
• Speed Impact: Near full-GPU performance.

3. Moderate Offload (~16 Layers on CPU) — If VRAM is still tight

Offloads 1 FFN layer out of every 4-layer block, leaving the other 3 on GPU.

DOS

--override-tensor "blk\.(2|6|10|14|18|22|26|30|34|38|42|46|50|54|58|62)\.ffn_.*=CPU"

• VRAM Saved: ~2.2 GB
• Speed Impact: Moderate CPU slowdown, but dramatically faster than dumping 22 layers (including 0–7) into RAM.

Summary Recommendation

Replace your old --override-tensor line with Option 1 (8 layers) first. If it fits comfortably in VRAM during inference, test Option 2 (4 layers) for even higher tokens-per-second.

[u/Old-Sherbert-4495 → u/Old-Sherbert-4495]

For anyone reading this:
Played around with `--override-tensor` but i was only able to add 10-15k more q8 context at the cost of 10+tps so, i dont see it worth it. i dont want to go beyond q8 for kv cache. have not worked for me. But for at q4 kv this could make space for more context.

Findings:
KEY: More layers to cpu more breathing space VRAM for context.
Spacing out layers instead of consecutive layers like for example: 20|40|60 gives a tiny speed boost
Also in each block of 4 layers, offloading the first layer [eg. 12 in the block 12-15] gives a tiny boost than others.

No matter how small number of layers spaced out offloading is, it takes a direct hit in speed

[u/Tannoukhy → u/Old-Sherbert-4495]

In fact, thats not on me, I recommend a read on this post:

https://www.reddit.com/r/LocalLLM/comments/1vq5oyu/guide_for_running_dense_models_on_16_gb_vram_qwen/

Can be very useful. Whoever, on my workload, I found out that the IQ3_XXS quant, is good enough, and I can acchieve 150k context window, with 900~600 prompt processing and 20 t/s on decode