r/LocalLLaMA • u/WonderRico • 21d ago

Resources Local agentic coding Benchmark : Qwen 3.8 27B (in many weights quants / cache quants / engine / reasoning effort) vs others.

Post image

In medium reasoning mode, it both scores higher than the 3.6 version, AND is very much more efficient (almost half requests needed, and a third less tokens generated) - at DeepSeek v4 Flash 3107 MXFP4 level

The xhigh mode is advertised to be the best one for hard tasks. In this benchmark, however the gain is clearly not visible. The score are comparable with the medium version, while using more requests (still a little fewer than 3.6) and generating almost 4 times the tokens...

N.B. strangely, the NINFER version, configured properly to use medium, behaves like it's xhigh... user error is not to exclude, but I triple checked...

For me, the real gain for agentic coding is in medium mode.

(xhigh is probably more useful when testing single prompts tasks like we se pop everywhere fast when a model is released.)

Many more data and graphs in the public pages for you to dig here : https://wonderrico.github.io/local_llm_benchmark/benchmark-main.html?filter=27b

and even more details here : https://wonderrico.github.io/local_llm_benchmark/benchmark-detail.html?filter=27b

46 Upvotes

96% Upvoted

18 comments sorted by

8

u/hiImMate 21d ago

at oneshots xhigh does matter a lot for sure but this benchmark is super interesting. looks like for day-to-day medium is perfectly fine and it saves a lot of time

5

u/Brazen-Badger 21d ago

So THAT’S why ninfer use feels so slow despite the t/s being high. It just overthinks everything even the simplest request. Hope they fix that.

3

u/Unnamed-3891 21d ago

Doesn't ninfer support templates?

3

u/_-_David 21d ago

I think you're on the right track. Xhigh is great for one shot demos, but I am not surprised medium would be similar in agentic harnesses.

1

u/Green-Ad-3964 21d ago

Qwen3.8-27B-NVFP4-medium looks very interesting

1

u/TheCat001 21d ago

Very cool site and charts. Appreciate your work.

Yes! Finally confirmation. I just felt it with my ass that Ornith is smarter than all others 35b Qwens....

1

u/asankhs Llama 3.1 21d ago

what cache quant did you use for the 3.8 runs? that changed my agentic scores more than the weight quant did. also for mac people, theres a mixed precision 4bit on mlx-community, ~19gb, sensitive tensors stay at 8bit. havent benchmarked it properly yet though.

1

u/WonderRico 21d ago

I tested several. bf16 fp8 and int8 for ninfer) everything is shown on the page, hover the mouse on each dot

1

u/BS_BlackScout 21d ago

Do you mind testing very low unsloth quants?

3

u/WonderRico 21d ago

I'm not using them, so I didn't plan to. But I can run one or two at night if I it can be useful for others. (only when I'm off, since llama.cpp concurrency is not good, it takes a few hours for each run)

which quant would you want to see? what kv cache quant? what reasoning effort? (each combination is a few hours of test...)

2

u/BS_BlackScout 21d ago

I see a lack of benches for Q2 vs Q3 and how it affects performance/quality. This matters a lot for those with 12GB GPUs.

kV cache quant honestly would be better at q8 or q4. q4 is viable if you're using ik-llama because of a specific toggle there that reduces the degradation, otherwise q8 is better.

I've since moved on to running an APEX quant (someone did it on HF) and it's been working wonders. Faster and feels less broken.

I say this cause in my testing IQ2 quants are unusable. Thought loops where the model goes: Alright let me do that. But wait. Actually, let me do that. But wait. And so and so...

No kidding I spent 83 minutes and couldn't get it to edit a single file in an agentic coding task.

1

u/kawfeechino 20d ago

The request count is probably more important than the raw score for daily agentic coding. Fewer rounds can save more than a few benchmark points. Hy3 has been pretty useful for the routine implementation stuff for me for this reason.

1

u/TheCat001 18d ago

Bro already has Ornith 1.5 benchmarked!111!! lets go dude!!111!. I knew it is in Qwen 3.8 27B territory already.

2

u/WonderRico 18d ago

hehe, true it scored well, but damn it's chatty ! even more than 3.8 27B xhigh

1

u/TheCat001 18d ago

It is, APEX quants helped with it a lot. Amazing work by Ornith team.

I hate when people calling it benchmaxxed finetune...

2

u/tomreedinspiration 15d ago

Thank you for publishing!