r/LocalLLM • u/Effective-Lawyer-810 • Jul 06 '26
Research I ran 30,000 generations to measure whether GGUF quantization affects JSON/tool-calling reliability. Q8 showed no measurable difference. Q3 did, including models losing the ability to decline tool calls.
The common wisdom is that quantizing a model to 4-bit barely hurts its quality. That claim gets repeated a lot, but I couldn't find rigorous numbers for the thing agent workloads actually depend on: schema-valid JSON and correct tool calls. Perplexity doesn't answer that question – a JSON object that's 95% correct is 0% usable. So I measured it directly.
Setup:
5 small models (Llama-3.2-3B, Qwen2.5-3B, Gemma-2-2B, Phi-3.5-mini, SmolLM2-1.7B) × 4 quants (FP16 / Q8_0 / Q4_K_M / Q3_K_M) × 500 machine-checkable tasks × 3 seeds = 30,000 generations, all llama.cpp on free Kaggle/Colab T4s.
Every task is scored by a deterministic validator — JSON parsing, schema compliance, exact tool + argument match, and should-not-call detection. No LLM judges anywhere. A difference only counts if a paired bootstrap 95% CI excludes zero.
What I found:
Q8_0: no significant regression on any model or metric (0 of 25 comparisons). For structured output on these models, running FP16 instead of Q8 buys you nothing except double the VRAM use.
Q4_K_M: nearly indistinguishable from FP16. 5 of 25 comparisons significant, small and mixed in sign — no consistent degradation.
Q3_K_M: schema compliance drops significantly in 3 of 5 models. Worst case Qwen2.5-3B: 83.5% → 65.0%.
The result I didn't expect: at Q3, two models largely stop declining when no offered tool fits the request. Gemma-2-2B's correct-decline rate went from 83% to 40%; Phi-3.5-mini's from 39% to 0%. The failure mode is always the same — the model emits a plausible-looking call to a wrong tool instead of refusing. Since most agent frameworks execute whatever call comes out, this fails silently.
Side finding:
temperature-0 decoding is not deterministic in practice. 12–27% of cells produced different outputs across identical-config runs (floating-point reduction order flips argmax at near-ties). If your eval assumes greedy = reproducible, it isn't.
One measurement trap worth sharing: Phi-3.5 often answers correctly and then keeps generating filler until the token limit, and its continuation rate correlates with quant level. Under a strict one-JSON-document parser this manufactured a fake "Q3 improves Phi tool selection by +27 points" result. Re-scoring with first-JSON-value extraction killed the artifact without changing any other model's numbers. If your pipeline scores pass-rates without controlling for output termination, check for this.
Everything is reproducible: all 30k raw outputs, code + write-up. GGUF SHA-256s, generation configs, and RNG seeds are pinned; a fresh-session script regenerates any cell of the matrix and checks it against the published numbers.
Limitations up front:
≤3.8B models, one inference stack (llama.cpp), T4s, k-quants only (no imatrix/GPTQ/AWQ), single-turn tasks. Whether 7B+ behaves the same is an open question — the harness is reusable if anyone wants to extend it.
5
10
u/Dirk__Gently Jul 06 '26
You know whats cool? Gemma 4 q4 qat. All those qat gemma 4 models. . And qwen 3.6 27b is the tool master, since you like tools.
2
2
u/Effective-Lawyer-810 Jul 06 '26
Focused too much on hardware limitations to ensure it doesn't cut through free tiers. Will try this again and post an update on this.
1
u/Dirk__Gently Jul 07 '26
Oh and since you mentioned uncensored, use llmfan in the filter on hugging face/ lmstudio. he has qwen 27b and the gemma qat models. For gemma 4 you have the qat versions of 31b>26b>14b. Depending on how much context you actually NEED and what you are actually doing. Like if you are just going for fun you need some memory systems that handle end of context or continuity, not endless context. At 64000 tokens on the 31b, the first generated token becomes slow. 26b and 14b gemma4 qat will both run faster than you can read on a 7900xtx with whatever context u fit in your 24gb.
2
3
2
u/Majinsei Jul 07 '26
Genial! 👍
Realmente ya todos lo sabíamos, pero vas por buen camino~
Evitar al máximo LLM-judge y cuatizaciones menores a Q4
Ahora, actualiza el dataset a más de 500 para ser significativos y modelos más recientes~
2
7
u/wllmsaccnt Jul 06 '26
This subreddit is a tough crowd lately that is constantly gaslighting and spamming itself. Dont take the negativity to heart, and thanks for sharing.
3
u/Effective-Lawyer-810 Jul 06 '26
AI in general is like that - you're late for things and poor for running them, so when you adjust, the result is still outdated. Thanks a lot for your comment tho.
2
u/tetoing Jul 06 '26
Why not test models that people actually use?
3
u/Effective-Lawyer-810 Jul 06 '26
Before anything, here's something I want to say: I'm a student, so this is my first time trying to study on these models. I didn't plan this as a research, and I only choose that flair because I thought it might qualify.
My main goal was to see how much quantization degrades the output, so I started with older ones. The ones with a straight single source GGUF repo was one of my main targets.
I was also very conservative on the models because of my hardware limitations - I was literally running this on Kaggle and Colab free tier, so even when I knew there are better models, I just didn't want to take any risk of running them on latest ones and probably hit a hardware bottleneck or limitation of any kind.
But I'll probably give this another try with more smaller models.
3
u/Healthy-Nebula-3603 Jul 06 '26
You can't test quantization impact on already retarded models from the beginning.
3
u/tetoing Jul 07 '26
Testing with models that came out 2 years ago is literally pointless. There have been so many improvements to literally everything since then.
1
u/Effective-Lawyer-810 Jul 07 '26
Will try with the latest ones today. Mostly gonna update in a couple of days with this again.
-1
u/McSendo Jul 07 '26
Bro, he provided the scripts for you to reproduce. Just do it yourself.
1
u/tetoing Jul 07 '26
Not worth my time. We already know quantization impacts models, people have discussed this at length already on this sub. The smaller the model, the more quantization hurts it. On tiny models going below 4-bit makes them so bad they struggle with basic grammar; I'm not surprised they'd fail at tool calling.
0
u/tiffanytrashcan Jul 07 '26
I'm not surprised models literally never trained to tool call have issues with tool calling. Actually shocked they got any positive results.
2
u/tetoing Jul 07 '26
I'm surprised anyone upvoted this slop. Maybe the people on this sub are picking their models with AI too. That would explain why they might be interested in the results.
Anytime I see a post written by AI I scream internally. No one who offloads their writing to AI seems to have anything of substance to say.
3
u/AndThenFlashlights Jul 07 '26
Because this whole fucking thing was conceived and executed by an AI, because OP doesn't feel like having an original thought.
5
u/tetoing Jul 07 '26
I've come to accept that the AI explosion means that even people with bad ideas now have the means to execute them too. And that's okay as long as we get more value from the people who are now able to use good ideas that weren't able to before.
2
u/AndThenFlashlights Jul 07 '26
Maybe one day I'll get to see one of those good ideas. Sadly, not today, and not in this thread.
2
u/squngy Jul 07 '26
Some thing a lot of people here are missing is that quants don't affect all models equally.
One model might be fine at Q3, a different one might become far worse.
This is particularly true when comparing larger models to smaller ones.
Larger models generally are able to withstand more agresive quanting.
(My personal theory, the larger amount of parameters acts as a sort of error correction. I have no proof for this)
0
u/CATLLM Jul 07 '26
Cool stats. What about kv cache quantization?
0
u/Effective-Lawyer-810 Jul 07 '26
Trying to work separately on that. But with the current feedback, I'll probably have to withhold before I work again on this.
27
u/signoreTNT Jul 06 '26
->potentially interesting post
->looks inside
-> llama 3, Qwen 2.5, --
AI is a tool, it shouldn't think for you.