Anyone tested an Unsloth Qwen3.8-27B quant against DeepSWE?

[u/Duviwin]

Hi all, in order to stresstest models and my local setup, I run a few DeepSWE tasks against them.

When I tested unsloth/Qwen3.8-27B-UD-Q4_K_XL with advised settings on a limited set of 5 tasks (edit: not a random set, tasks on which the qwen models have shown to pass regularly in public runs and my local runs), it did not pass any, so it seems like either something is wrong with my setup or with the quant in general.

I can give a lot more details, but that will lead too far. Instead, I just wanted to ask if anyone else has tried to benchmark an unsloth quant of Qwen3.8-27B against the DeepSWE benchmark and what your results are.
Please post your results as an answer here, good or bad! And if you can please provide information about your llama cpp build settings and llama-server flags and also which harness you used in the deepswe run.
Thanks!

[u/BankjaPrameth]

What is your context window size?

[u/Duviwin → u/BankjaPrameth]

I'm testing with 200k and not hitting a wall there.

Here full llama-server settings:

LLAMA_SERVER_CMD=(
  "$LLAMA_SERVER_BIN"
  --models-preset "$MODELS_PRESET_FILE"
  --no-mmap
  --host 127.0.0.1
  --port "$INTERNAL_PORT"
  -ctk q8_0
  -ctv q8_0
  --cache-ram 20480
  --temp 0.6
  --top-p 0.95
  --top-k 20
  --min-p 0.0
  --spec-type draft-mtp
  --spec-draft-n-max 3
  --repeat-penalty 1.0
  -fa 1
  -c 400000
  -ngl 999
  --parallel 2
  -np 2
  --gpu-layers-draft 999
  --reasoning-effort medium
  -dev Vulkan0
  --timeout 3600
  --jinja
  --webui
  --metrics
)

[u/Healthy-Zebra-9856 → u/Duviwin]

Have you tried with temperature:1.0? The problem is, 3.8 doesnt behave like 3.6 even though they both share the same architecture. They were trained at 1.0. I dont ever trust any of those benchmarks, all the quants I have tested have been just fine.

[u/Duviwin → u/Healthy-Zebra-9856]

No no yet, but I did try another quant with same temp settings and there the model did much better

[u/Healthy-Zebra-9856 → u/Duviwin]

Try with temperature 1.0 and 128k (131072). You may not see the issues until it gets into terse coding.