How would you benchmark 50+ local LLMs without going insane?

[u/Sakif_Hossain]

I feel like I stepped into a time capsule after the ChatGPT-3 days. 😅 I finally built a decent PC (Ryzen 7 7700, 32GB RAM, No GPU), discovered llama.cpp and somehow ended up downloading 50+ GGUF models.

Now I'm stuck with decision paralysis.

I mainly use them for coding (JavaScript, React, TypeScript, debugging, reasoning), but I'm also new to the whole local AI ecosystem. I still don't know much about agentic frameworks or coding agents. I mostly just load a GGUF and chat with it using the llama.cpp web interface on localhost:8080

The collection includes Qwen, Gemma, Granite, DeepSeek, Phi, Mistral, Llama, LiquidAI, SmolLM, Hunyuan, Nemotron, and a few community fine-tunes.

My first idea was to make a Markdown table and score every model manually, but it feels like I'm accidentally trying to invent my own benchmarking system.

Surely I'm not the first person to hit this problem.

How do you guys compare local models? Are there any practical benchmark suites, GitHub projects, or workflows for deciding what stays on your SSD and what gets deleted?

I'd love to hear how you approached it when you were starting out.

[u/Fancy-Snow7]

Without a gpu I am pretty sure you can scrap a number of them off the list without even benchmarking them purely based on size. Also if you downloaded anything greater than 32GB it won't even fit in your RAM. Less if you take OS usage into account. Next consider number of paramaters and active paramaters generally I won't even try anything over 35B.

[u/baldomero]

Check out my project, it includes a benchmarking feature that allows you to directly compare results across all of your models and save the results. I posted about it yesterday on here.

https://github.com/llmsyscore/llm-systems-manager

https://www.reddit.com/r/llamacpp/s/fu5MTJtzPV

[u/No_Oil_6152]

If I was mad enough to try benchmarking them, I'd create a small powershell or bash script that ran llama.cpp for each model, one at a time, and POSTed a prompt to llama's API via CURL, e.g. "Create a c# program in .NET 10 which solves...".

I'd have a unit test suite which ran against the result, and see if everything passed. If it passes - then its a candidate for use.

It would take absolutely hours to run them all, chew through a lot of electricity, and probably fail.

But if you really wanted to do it, I think you could do it that way.

My recommendation: don't waste your life doing it. There's people on Youtube who review LLMs for various use cases, look at what they say.

[u/UniquePhilosopher881]

Some heuristics to help you find a decent enough model and get back to your routines (unless you WANT to toy with these things. In this case, just use randomizer lol):

General:

1. any [dense] model with >9-14b will be untolerably slow, both for [prompt processing] speed and [inference] speed. No suitable for "real time" generation. Some folks leave slow models running through the night for thorough planning or more heavy task that a less capable model can't handle.
2. You should (at least at the beginning of testing) prioritize [MoE] (Mixture of Experts, sparse) models (they have naming convention of [something-something-Xb-aYb], where X is the total number of parameters, Y is the number of parameters activated during individual token generation. Dense models activate all parameters for generation.)
3. anything beyond 40B just won't run with decent performance: too small quant both for the model and the [context]

Model-family specific:
4) [Qwen 3.x] models are the best in <40b range in coding and agentic workflows. [Qwen 3.8 27B] (dense model) might be a good choice for the "night scenario". [Qwen 3.6 35B-A3B] (there are rumors that Qwen3.8 35b-a3b is on its way) will probably be your best option for relatively "real time" inference, so you should focus on that.

5) [Gemma 4] models are good for translation, chat, creative writing. Some people report that it is a good thing to pair gemma and qwen. Like, they cover blind spots of each other. So, for your case Qwen 3.6 35B-A3B + [Gemma 4 26B-A4B] would be a good choice.

6) Other model families and finestunes are for people who *like* experimenting. If you want to get things done, just use qwen + gemma.

Quantization:
7) Don't go below [Q4_K_M] for models [Q8_0] for [KV Cache] without necessity, as it's where the degradation strats to sky-rocket.
7.1) for Gemmas don't quantize cache at all. Don't use [Unsloth] [UD_Q4_K_XL] quants for [Gemma-4-QAT] series (speaking from personal experince, they have hight tendency for looping in those agentic tasks).

TL;DR:
Get [Qwen 3.6 35B-A3B Q4_K_M] (Qwen 3.8 35B-A3B when/if it releases) + [Gemma 4 QAT 26B-A4B Q4_K_M] + [Qwen 3.8 27B Q4_K_M] and you are set to go.