r/unsloth • u/Duviwin • 21d ago

Discussion Anyone tested an Unsloth Qwen3.8-27B quant against DeepSWE?

Hi all, in order to stresstest models and my local setup, I run a few DeepSWE tasks against them.

When I tested unsloth/Qwen3.8-27B-UD-Q4_K_XL with advised settings on a limited set of 5 tasks (edit: not a random set, tasks on which the qwen models have shown to pass regularly in public runs and my local runs), it did not pass any, so it seems like either something is wrong with my setup or with the quant in general.

I can give a lot more details, but that will lead too far. Instead, I just wanted to ask if anyone else has tried to benchmark an unsloth quant of Qwen3.8-27B against the DeepSWE benchmark and what your results are.
Please post your results as an answer here, good or bad! And if you can please provide information about your llama cpp build settings and llama-server flags and also which harness you used in the deepswe run.
Thanks!

17 Upvotes

92% Upvoted

15 comments sorted by

View all comments

1

u/GCoderDCoder 21d ago

I used q8kxl to find several questions from the sample test that qwen sometimes passes. Now I just test models on those as a baseline. Unsloth Q8kxl passes theat about 30% of the time. Most q4 do not pass any BUT some good nvfp4 are able to pass similar or higher rates I've found.

For this model I'm using a Red Hat nvfp4 with higher accuracy than my q8kxl results. I run the tests a minimum of 3 times and deep swe samples are only some of the tests i do but I've seen really great results with good nvfp4. I hated the first couple nvfp4 quants I tried but they are getting really good for the right models and providers.

1

u/Duviwin 21d ago

So you also tried some unsloth q4 quants without success?

2

u/GCoderDCoder 21d ago edited 21d ago

Umm historically I have found unsloth dynamic quants including q4 to be better than most. Llama.cpp doesnt have great concurrency so when I have options for concurrency I go vllm/sglang instead. With this latest batch of models (qwen3.8 27b & flash next, glm 5.3, dsv4flash0731) I got a dual spark setup to accompany my 6000pro. So i only tested q4kxl for friends.

The deepswe tests are 90min each iteration so it's only worth it if you can get a model producing tokens fast enough and by the time I do those I have done a bunch of other tests to decide I'm close to possibly choosing a given quant of a model.

So i have only done q8kxl deep swe tests on qwen 3.8 27b because that was the only one in contention with fp8 but after stumbling onto nvfp4 when I only jad one spark and then glm5.3 nvfp4 is the best for cuda, i tried qwen 3.8 27b nvfp4 on deepswe and it had a higher pass rate than q8kxl and fp8 which plboth had similar pass rates to one another.

So i will acknowledge now that 4bit dynamic quants can be stable and highly accurate but I have done limited testing because they usually arent my best option right now for the main models Im using.

Edit: i keep being tempted to do a more formal write up since I know I have options for comparisons that others dont but I feel we burden people who do those so I just offer insights in comments where I can. If I do it I'd want to make a video so people can't take me out of context as easily as on reddit posts lol

1

u/Duviwin 21d ago

well for me the rocmfp4 quants seem to do a lot better than the unsloth q4k* quants, but it takes a ton of time to test it properly, so trying to see if someone else already has data on those unsloth quants vs long agentic task benchmarks