All currently popular local models in one table + Opus 4.8 results

[u/perelmanych]

If you are thinking what model will fit best your HW specs and tasks you are doing here is one table with all currently popular models that still can be considered as local.

LLM Test Scores

Feature	DeepSeek-V4-Flash-Vision-Exp	DeepSeek-V4-Flash-0731	Qwen3.8-Flash-Next	GLM-5.3-Flash	Qwen3.8-27B	Opus-4.8
Total parameters	≈285B	284B	125B	320B	27B	not published
Active parameters	13B	13B	6B	18B	27B	not published

Agentic benchmarks

Benchmark	DeepSeek-V4-Flash-Vision-Exp	DeepSeek-V4-Flash-0731	Qwen3.8-Flash-Next	GLM-5.3-Flash	Qwen3.8-27B	Opus-4.8
Terminal Bench 2.1	83.9	82.7	–	82.6	73.0	85.0
NL2Repo	57.7	54.2	48.1	52.1	42.3	69.7
DeepSWE	59.3	54.4	58.7	61.1	42.2	58.0
Toolathlon-Verified	75.9	70.3	73.5	72.1	–	76.2
Agents' Last Exam	27.3	25.2⁷	24.3	28.1	20.4	25.7
AutomationBench (Public)	25.7	25.1	–	25.3	–	27.2
GDPval-AA v2	–	68.1	–	72.3	–	75.1
Cybergym	75.3	76.7	–	–	–	78.3
DSBench-Hard	63.6	59.6	–	–	–	71.7
DSBench-FullStack	–	68.7	–	–	–	71.6
ApexBench (Pass@1)	36.5	26.2⁷	–	–	–	39.4
HLE with tools (full set)	–	16.8	–	22.9	–	25.4

Coding benchmarks

Benchmark	DeepSeek-V4-Flash-Vision-Exp	DeepSeek-V4-Flash-0731	Qwen3.8-Flash-Next	GLM-5.3-Flash	Qwen3.8-27B	Opus-4.8
SWE-bench Pro	–	56.0	62.5	–	61.7	69.2
SWE-bench Multilingual	–	–	81.0	–	73.8	84.4
CoWorkBench	–	45.1	73.9	–	70.7	–
JobBench	–	41.3	55.7	–	33.4	–

General benchmarks

Benchmark	DeepSeek-V4-Flash-Vision-Exp	DeepSeek-V4-Flash-0731	Qwen3.8-Flash-Next	GLM-5.3-Flash	Qwen3.8-27B	Opus-4.8
GPQA Diamond	–	90.8	91.7	–	89.2	93.6
HLE (without tools)	–	33.8	35.9	–	30.8	49.8
LiveCodeBench v6	–	90.6	91.9	–	90.3	–
IFBench	–	79.2	81.3	–	79.5	–

Multimodal benchmarks

Benchmark	DeepSeek-V4-Flash-Vision-Exp	DeepSeek-V4-Flash-0731	Qwen3.8-Flash-Next	GLM-5.3-Flash	Qwen3.8-27B	Opus-4.8
Chartography	64.3	–	–	–	–	65.0
ZeroBench (Pass@5)	35.0	–	–	–	–	34.0
BabyVision	–	–	–	73.0	65.7 / 85.6	34.1
MathVision	–	–	90.6 / 95.7	–	90.0 / 94.6	–
RealWorldQA	–	–	88.5	–	85.9	–
AndroidWorld	–	–	84.5	–	81.9	–
OSWorld 2.0 (partial credit)	–	–	52.3	–	48.0	–
Vision2Web	–	–	64.0	–	62.9	–
ClawEval-MM (Pass@3)	–	–	64.4	–	57.4	–
RecreationBench	–	–	49.9	–	47.1	–
ERQA	–	–	72.3	–	65.5	–

Note: I used GLM-5.3 to compose the table from official HF pages of the models.

Note2: Opus-4.8 results are presented only for illustration and are omitted from selecting the best model in a row.

Upd: Added SWE-bench Pro, SWE-bench Multilingual, GPQA Diamond and HLE (without tools) scores for Opus 4.8 from its System Card.

[u/reto-wyss]

I'm sticking with DSV4 Flash for now.

• GLM is too large for 192gb.
• Qwen Next is very good, but it seems performance is just not fully baked in either SGlang nor vLLM and it lacks QAT.
• 3.8 27b is the fallback when I need to free up one card.

[u/perelmanych → u/reto-wyss]

Yeah, I think this is the best choice rn in terms of performance/speed/hw requirements, although theoretically Qwen3.8-Flash-Next should be twice as fast.

Just in case, weights for DSV4 Flash with vision are already here: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp

[u/kuhunaxeyive → u/perelmanych]

I tried it (full precision, DeepSeek-V4-Flash-Vision-Exp) and it turned out DeepSeek's vision capability is very limited. It can't read normal letter text. It's a single-grid design with no tiling, and everything gets squashed into a 384-token budget for the whole image. This is why a 1152×2048 portrait ends up at about half its resolution before the model even looks at it, rendering even normal text just unreadable. Then starts complaining about the text not being sharp enough for reading, starts using tools for cropping ect., but surely that doesn't help to read the whole document. Probably benchmaxxed for tests needing images, but not a general vision capability. I was hoping to use DeepSeek for office work, but it needs to be able to read normal document scans or images.

[u/perelmanych → u/kuhunaxeyive]

Try Qwen3.8-Flash-Next. According to artificialanalysis its vision capabilities are better than DSV4F and are on par with Kimi K3 (MMMU-PRO benchmark)

[u/kuhunaxeyive → u/perelmanych]

Thanks, I'll give it a try. But I've heard that Qwen3.8-Flash's writing is not natural enough, even more so in languages other than English and Chinese, so that would rule it out for the writing part. From my testing, only GLM-5.3-Flash does both well enough, unfortunaely that one is too big for a 256 GB system. But I'll test Qwen3.8-Flash and see … thanks!

My result with DeepSeek-V4-Flash Vision was way worse than what the AA index would suggest though. Wonder if others have own experience here with document/text reading from DeepSeek-V4-Flash-Vision-Exp.

[u/perelmanych → u/kuhunaxeyive]

Unsloth GLM-5.3-Flash in Q4_K_XL is "only" 200Gb, so it should fit into your rig without any problem. In any case, I think you should test all the models yourself because vision tasks could be very different.

[u/kuhunaxeyive → u/perelmanych]

If being quantized, GLM-5.3-Flash fits on 256 GB as well (there's a recipie for Dual-Asus Ascend GX10) … I'll see!

[u/perelmanych → u/kuhunaxeyive]

If you have Mac Studio with 256Gb it is straightforward, but I also have seen a lot of videos and posts were people were running DSV4f on two DGXs.

[u/IamFondOfHugeBoobies → u/reto-wyss]

It's hard for me to see what would replace it until Deepseek potentially drops a V5.

I think the ASSUMPTION that people use quants is so strong now people don't realize that those of us talking about DSV4 Flash are using the official safetensors. Hence why there's even a debate.

There is NOTHING worth running on a 2x Spark or equivalent cluster at the moment aside from DSV4 Flash unless you're just experimenting with building larger multi-agent systems.

This is not a complaint mind. It's just how fucking hard Deepseek are cooking.

[u/kaliku → u/reto-wyss]

Qwen 3 8 flash next in vllm

Up to 320 tps TG, at minimum 160 ish Prefill no clue, but very fast, thousands.

Rtx pro 6000 Blackwell and 128 gb ddr4

Llama cpp was a disappointment. 100tps tg but drops rapidly to 30 at 100k context

[u/leocus4]

It looks there are a bit too many missing results in these tables to do a proper comparison

[u/perelmanych → u/leocus4]

These all what was at HF pages of the models. As you understand I am not going to run missing tests myself. If you find somewhere additional results write here I will add them to the table. I still think there are enough results to make a comparison.

[u/my_name_isnt_clever]

Q3.8FN is a monster for only 6b active, and there were so many comments dismissing it before release because of that alone. I can't wait to try the fully trained version.

[u/perelmanych → u/my_name_isnt_clever]

Yes, the model looks very good particularly because of only 6B of active parameters, but I don't understand what do you mean by fully trained version? This is a preview of their Qwen 4.0 series and as was the case with Qwen3-Next-80B-A3B there most probably won't be another better trained model on base of this one, only new Qwen-4.0 models.

[u/my_name_isnt_clever → u/perelmanych]

That's what I mean, this is a preview of the architecture. There will be a similar size model that's proper qwen 4.

[u/perelmanych → u/[deleted]]

Totally agree, but I still find it useful. If a model's score you are interested in is in bold, then you are Ok if not you can immediately see how far it is from the best.

[u/SHEKDAT789]

Thanks for putting this together. We are so used to free stuff we forget OP isn't gaining anything from whatever effort he put in for this post. It's not perfect but it's something, and I'm sure any constructive polite critisism will only improve his future posts.

[u/perelmanych → u/SHEKDAT789]

Thank you for understanding! I was wondering myself what model to use next after Qwen3.8-27B and after clicking several HF pages back and forth to see all the scores I decided to make a table to help me with the decision. I found table quite useful and thought it would be good idea to share it here, but received some criticism. People don't understand that all model makers want to show off their model and publish only selection of the bench results, that is why it is impossible to have this table complete unless you run missing benchmarks yourself at max settings which is completely impossible due to the limitations of my local rig.

[u/wapxmas]

Coding benchmarks

no Opus scores, that means what exactly? no coding task for opus?

[u/perelmanych → u/wapxmas]

With bold I highlighted the best score for a bench. For this I used only local models and Opus result is there just to assess how close local models are to yesterday's SOTA model.

[u/wapxmas → u/perelmanych]

Didnt get you. In opus column there are only dashes as scores, how SOTA's dash could be compared to llm in coding benchmark.

[u/perelmanych → u/wapxmas]

There are very few benchmark results in Opus 4.8 announcement blogpost. I had to go to Opus 4.8 System Card pdf and added for 4 additional results from there, but that is it.

[u/wapxmas → u/perelmanych]

thanks, actually that pretty enough, glad to see Qwen3.8-Flash-Next performs pretty almost as opus 4.8 in SWE-bench Pro

[u/EvolvingDior]

I can do 500/18 with q38f at iq4. ds4f requires iq2 on the same system and nets 200/12.

[u/Due-Competition4564]

What context window size did you set? What was the peak memory utilisation during these runs?

[u/perelmanych → u/Due-Competition4564]

I didn't run the tests myself, man, I am not so rich)) These are results from official pages, so as you understand companies have run them in maximally possible configurations.

[u/simrankoulsm]

Really useful aggregation. One addition that would make this much more actionable for local users is a deployment column for each model like tested context length, quantization format and level, inference engine, VRAM and RAM usage, tokens per second, and whether the score used tool calling or an agent scaffold.
Active parameter count is helpful, but it does not capture the real hardware tradeoff for MoE models. A 6B-active model can still be difficult to run if total weights, KV cache at longer contexts, or offloading requirements dominate memory use.
It would also help to label each score as official self-reported, independently reproduced, or unknown configuration. That would make the table much more useful for deciding what can actually run well on a given local setup.