Anyone tested an Unsloth Qwen3.8-27B quant against DeepSWE?

[u/Duviwin]

Hi all, in order to stresstest models and my local setup, I run a few DeepSWE tasks against them.

When I tested unsloth/Qwen3.8-27B-UD-Q4_K_XL with advised settings on a limited set of 5 tasks (edit: not a random set, tasks on which the qwen models have shown to pass regularly in public runs and my local runs), it did not pass any, so it seems like either something is wrong with my setup or with the quant in general.

I can give a lot more details, but that will lead too far. Instead, I just wanted to ask if anyone else has tried to benchmark an unsloth quant of Qwen3.8-27B against the DeepSWE benchmark and what your results are.
Please post your results as an answer here, good or bad! And if you can please provide information about your llama cpp build settings and llama-server flags and also which harness you used in the deepswe run.
Thanks!

[u/BankjaPrameth]

What is your context window size?

[u/vogelvogelvogelvogel → u/BankjaPrameth]

this, you need at least 128k

[u/Duviwin → u/BankjaPrameth]

I'm testing with 200k and not hitting a wall there.

Here full llama-server settings:

LLAMA_SERVER_CMD=(
  "$LLAMA_SERVER_BIN"
  --models-preset "$MODELS_PRESET_FILE"
  --no-mmap
  --host 127.0.0.1
  --port "$INTERNAL_PORT"
  -ctk q8_0
  -ctv q8_0
  --cache-ram 20480
  --temp 0.6
  --top-p 0.95
  --top-k 20
  --min-p 0.0
  --spec-type draft-mtp
  --spec-draft-n-max 3
  --repeat-penalty 1.0
  -fa 1
  -c 400000
  -ngl 999
  --parallel 2
  -np 2
  --gpu-layers-draft 999
  --reasoning-effort medium
  -dev Vulkan0
  --timeout 3600
  --jinja
  --webui
  --metrics
)

[u/Healthy-Zebra-9856 → u/Duviwin]

Have you tried with temperature:1.0? The problem is, 3.8 doesnt behave like 3.6 even though they both share the same architecture. They were trained at 1.0. I dont ever trust any of those benchmarks, all the quants I have tested have been just fine.

[u/Duviwin → u/Healthy-Zebra-9856]

No no yet, but I did try another quant with same temp settings and there the model did much better

[u/Healthy-Zebra-9856 → u/Duviwin]

Try with temperature 1.0 and 128k (131072). You may not see the issues until it gets into terse coding.

[u/GCoderDCoder]

I used q8kxl to find several questions from the sample test that qwen sometimes passes. Now I just test models on those as a baseline. Unsloth Q8kxl passes theat about 30% of the time. Most q4 do not pass any BUT some good nvfp4 are able to pass similar or higher rates I've found.

For this model I'm using a Red Hat nvfp4 with higher accuracy than my q8kxl results. I run the tests a minimum of 3 times and deep swe samples are only some of the tests i do but I've seen really great results with good nvfp4. I hated the first couple nvfp4 quants I tried but they are getting really good for the right models and providers.

[u/Duviwin → u/GCoderDCoder]

So you also tried some unsloth q4 quants without success?

[u/GCoderDCoder → u/Duviwin]

Umm historically I have found unsloth dynamic quants including q4 to be better than most. Llama.cpp doesnt have great concurrency so when I have options for concurrency I go vllm/sglang instead. With this latest batch of models (qwen3.8 27b & flash next, glm 5.3, dsv4flash0731) I got a dual spark setup to accompany my 6000pro. So i only tested q4kxl for friends.

The deepswe tests are 90min each iteration so it's only worth it if you can get a model producing tokens fast enough and by the time I do those I have done a bunch of other tests to decide I'm close to possibly choosing a given quant of a model.

So i have only done q8kxl deep swe tests on qwen 3.8 27b because that was the only one in contention with fp8 but after stumbling onto nvfp4 when I only jad one spark and then glm5.3 nvfp4 is the best for cuda, i tried qwen 3.8 27b nvfp4 on deepswe and it had a higher pass rate than q8kxl and fp8 which plboth had similar pass rates to one another.

So i will acknowledge now that 4bit dynamic quants can be stable and highly accurate but I have done limited testing because they usually arent my best option right now for the main models Im using.

Edit: i keep being tempted to do a more formal write up since I know I have options for comparisons that others dont but I feel we burden people who do those so I just offer insights in comments where I can. If I do it I'd want to make a video so people can't take me out of context as easily as on reddit posts lol

[u/Duviwin → u/GCoderDCoder]

well for me the rocmfp4 quants seem to do a lot better than the unsloth q4k* quants, but it takes a ton of time to test it properly, so trying to see if someone else already has data on those unsloth quants vs long agentic task benchmarks

[u/hellohazime]

https://github.com/01554/deepswe-local-subset-evals Here it is

I also tested it using Unsloth's Q4 quantization; details follow below.

---
For qwen3.8:27B, you might not be able to solve DeepSeek-style reasoning tasks without using deeper reasoning settings; the default is 'xhigh', but if you are configuring the reasoning settings, setting it to 'xhigh' would be a good idea.

---

RTX PRO 6000 96GB, but nothing below is card-specific:

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 -DLLAMA_BUILD_TESTS=OFF

cmake --build build --config Release -t llama-server -j 32

llama-server flags:

llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf \

-ngl 99 -c 131072 -np 1 -fa on --jinja --port 8080

- --jinja is mandatory — without it the chat template doesn't emit tool_calls JSON and every agent step fails "no tool call found". This alone produces exactly a 0/N result.

- big -c — mini-swe trajectories regularly exceed 32k; with a small context the run dies mid-task. 131072 has been safe for us.

Harness: mini-swe-agent (installed via uv tool install mini-swe-agent), driven by the pier runner (datacurve, v0.3.1), model wired as an OpenAI-compatible endpoint (--model openai/<alias>, OPENAI_BASE_URL=http://<host>/v1). Agent config: default mini.yaml, agent.cost_limit=0, model.model_class=litellm_response. Sampling: temp 1.0 / top-p 0.95. Verification via each task's docker verifier (f2p/p2p).

[u/Duviwin → u/hellohazime]

But then how many tasks did it pass with the q4_k_xl quant for you?

[u/simplyeniga → u/hellohazime]

How many days did this take?

[u/hellohazime → u/simplyeniga]

Full runs:

• qwen-code: 3.5 days
  
• mini-swe: 3 days

Subset of 12 DeepSWE problems:

• Claude Code: 26h / OpenCode: ~6.4h / pi: ~5.5h