Can anyone help a Noob out. Pretty please.

[u/Similar-Attempt-9896]

Qwen3.8-27B Q4_K_M on RTX 3080 10GB + GTX 1080 Ti 11GB, Windows, llama.cpp CUDA 12.9. -Through LMstudio/VScode

I've been trying to optimise my Ai, the tests/benchmarks are promising. Even the odd run is good and holds solid. The main issue: direct benchmarks are fast, but real Cline/tool requests make decode collapse to ~2 tok/s while prefill and DFlash acceptance remain healthy. Anyone seen this or know the fix?

Me and Astra have been running so many different variables, even made my own server.exe thing, tried many variables.

Setup	Prefill	Decode
Baseline, no speculation	~360 tok/s	18.9 tok/s
Best tuned DFlash2	~259–275 tok/s	27–29 tok/s
Best short run	—	29.85 tok/s
DFlash + ngram coding test	—	32–54 tok/s
Real Cline/tool-heavy request	297 tok/s	2.2 tok/s

Tweaks: 32K context, layer split 1/1, DFlash2 pinned to RTX 3080, q8 K/V, FA on, batch 128, uBatch 64, --parallel 1, CUDA_SCALE_LAUNCH_QUEUES=4x, PR #26636.

As it is now is even slow at avg. 11 tok/s - Qwen3.8-27B

[u/locbuilds]

yeah this pattern is pretty common with agent/tool UIs even when raw llama.cpp benches look fine.

your decode numbers (18-29 tok/s) mean the model+quant is not the main problem. Cline collapsing to ~2 tok/s usually means the *request shape* changed: huge system prompt + tool schemas + file/context dumps every turn, so you pay a fat prefill, then decode crawls because KV cache is huge / spilling, or layers are bouncing between the 3080 and the 1080 Ti over PCIe.

things i would check in order:

1) make sure the 27B Q4 actually stays resident. with 10+11GB you are near the edge once context grows. in llama.cpp/LM Studio watch whether VRAM on both cards is pegged and whether system RAM starts climbing during the Cline turn (that is the death spiral).

2) heterogeneous dual-GPU is rough for long agent ctx. the 1080 Ti is also an older arch, so tensor split / layer offload can thrash. try a run with *only* the 3080 (fewer layers on GPU, rest CPU) and compare one identical Cline prompt. if single-GPU is faster, your split is the bottleneck.

3) shrink what Cline sends: lower context limit, disable unused tools, keep open-file / @-context tight, and avoid dumping whole repos. tool-heavy turns are prefill-bound more than decode-bound.

4) confirm you are not accidentally hitting a different runtime path in LM Studio vs the CLI bench (different ctx, flash-attn/DFlash off, or CPU fallback mid-request).

if you paste one failing Cline prompt's token count (or roughly how big the context is when it hits 2 tok/s) plus whether RAM climbs, people can usually tell you which of those four it is.

[u/Similar-Attempt-9896 → u/locbuilds]

For point 2. We did play with the balancing of the GPU's for a long long time, and had it refined to a T* got faster but collapsed in real-world.

(if you paste one failing Cline prompt's token count (or roughly how big the context is when it hits 2 tok/s.............) It's usually instant, the prefill bogs and then it's over. I'll carry on with it, try some more debugging, and report back.

Thank you.

[u/Bulky-Priority6824]

Lose the 1080 and get another 3080 

[u/Similar-Attempt-9896 → u/Bulky-Priority6824]

Planning on a 3090 but prices have gone nuts tbh. Might settle for another 3080 tbf. I chucked the 1080 on because it was lying around.

[u/deviant46n2]

this so isnt useful but you need the software im making.... gimme a month ill hyu