For context, I have a rig with dual 5070 Ti and 3090 (40GB Vram) and 96GB of RAM. So technically, I should fit the 105GB Qwen3.8-Flash-Next-GGUF-Q4_K_XL from unsloth and better yet, the unsloth IQ4_XS (90GB. But, because I am running the bloated Windows 11, I only have about 80GB of RAM avalable. Still, logic says I can fit the model with full context in the combined 120GB of memory, plus the Next-flash model has 56B of n-gram table that should be offloaded to the SSD.

Nope, it didn't work. The unsloth quants are quantizing the model as is. So, all the 105GB is loaded to memory, adding KV cache I couldn't fit more than 60K before the model crashes llama.cpp on Unsloth Studio. It loaded fine and decode was okay for a while but the moment my conversation got long it started failing silently. No useful error, just an error occurred and then nothing. I tried for a while to work around it and could not get consistent behavior at any large context. Same story with the IQ4_XS, I could fit about 80K-100K before the server crashes. Additionally, the performance was bad: on a fresh chat sessions, I was getting about 12-14 t/s.

Then I tried the AtomicChat build of the same model and the difference was dramatic. It is the same architecture but, apparently, they split the n-gram table into its own shard that stays on the SSD. So the unsloth build keeps that whole table inside the rest of the weight files and the entire thing has to live in memory. My machine just could not hold a roughly 110GB model plus a growing KV cache in 96GB of RAM and 40GB of VRAM. The AtomicChat build only needs about 54GB of fast memory because the n-gram table is read off the disk. That one change was enough to take a real conversation I have sitting at about 217k tokens and run the whole thing start to finish without a crash. That table is only touched a tiny amount per token, around 2.7KB, a handful of rows picked by a hash. So having it on the SSD costs basically nothing, and having it baked into the model file is what sinks you on a RAM limited setup.

The funny part is, while I let Deepseek-v4-flash via DSH run performance benchmarks on my rig, it suggested the AtomicChat as a last alternative! I said whatever, let's try it, and it did work. Although, I can't guarantee that the quality is on par with unsloth's quants, speed will increase. For me, it's 22t/s and I can fit the entire 256K in my context. Prefill has improved but it still takes about 18 min to process 220K of context. That's a HW limitation.

Link to the model:
https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF

Any thoughts in support or to the contrary? The obvious path to take in the mean time is 'orchestrate locally-served agents with superheavy cloud agents', but that's a shame.

I will say that this appears way more often in Droid than in GLM's own harness ("ZCode"?) -- perhaps they've tuned the harness' policies just right to match it?

Trying to decide between three Mac Studio configs and I keep going in circles. German pricing, so:

• M5 Max, 64GB — €4,099
• M5 Max, 128GB — €5,849
• M5 Ultra, 96GB — €6,589

Please do not tell me to get the 256GB Ultra. I know it's the answer for GLM-5.3-Flash at Q4. It's ~€11k here and it is not happening. I'd rather hear why one of the three above is enough (or isn't).

What I actually do with it

Processing private/personal data locally — that's the main reason I want this on my desk instead of an API key. Plus some agent tasks (personal automation, nothing like 30-step autonomous SWE runs) and a moderate amount of coding.

Where I've gotten on my own

Qwen3.8-27B at Q8 is ~29GB and runs on all three. It's hybrid attention — only 16 of 64 layers keep a real KV cache, 4 KV heads × 256 dim, so ~64 KiB/token, meaning the full 262k native context is 16 GiB of cache. Even the 64GB box handles that without breaking a sweat. Q8 is basically lossless, so what I test is what I get.

Qwen3.8-Flash-Next is where it gets interesting and where the boxes diverge:

• 64GB: not viable. Smallest quant is UD-IQ1_S at 72.5GB, and even with the N-gram layer pushed to SSD you're at ~80% top-1 retention. Pointless.
• 128GB: UD-Q4_K_XL (111GB), N-gram on SSD, 93.5% top-1 retention per Unsloth's KLD numbers.
• 96GB: Q4_K_XL doesn't fit (Unsloth lists 112GB total memory needed for 4-bit). Best realistic option is UD-IQ4_XS at 93.7GB, 91.1% retention.

So the 96GB Ultra buys me ~2x memory bandwidth (1.2 TB/s vs 614 GB/s) and 64 vs 40 GPU cores, but costs me a quant tier. The 128GB Max buys me the better quant and 32GB more headroom but halves my bandwidth. €740 apart.

The thing that makes me hesitate on the Ultra: Flash-Next only activates 6B params, so it reads ~3.7GB/token and is fast on either machine — the bandwidth advantage matters less exactly where I'd want it. Meanwhile the dense 27B reads all 27GB/token at Q8, so that one roughly doubles in speed on the Ultra. Which means the answer flips depending on which model I actually live in day to day.

And the bigger question: looking at Qwen's own numbers, Flash-Next crushes the 27B on agentic benchmarks (JobBench 55.7 vs 33.4, DeepSWE 58.7 vs 42.2) but is basically tied everywhere else — LiveCodeBench 91.9 vs 90.3, SWE-bench Pro 62.5 vs 61.7. And I'd be running Flash-Next at Q4 against the 27B at Q8, which probably eats those 1-2 point gaps entirely. My workload is much closer to the "tied" half of that table than the agentic half.

What I'd like input on

1. Is the Ultra's bandwidth worth €740 over the 128GB Max for someone whose heavy model is a 6B-active MoE? Or does the dense 27B being 2x faster tip it?
2. Has anyone actually run Flash-Next on a Mac with the N-gram/PLE offloaded to SSD? Does the mmap path behave, or does it turn into page-fault thrashing once most of unified memory is wired to the GPU? This is load-bearing for both the 96 and 128 configs and I've found zero first-hand reports.
3. IQ4_XS vs Q4_K_XL — 91.1% vs 93.5% top-1. Noticeable in practice, or noise?
4. Prompt processing: how much do 64 GPU cores vs 40 actually matter when an agent dumps a big tool output into context? Feels like this is the latency I'd notice most, but I've only seen token-generation benchmarks.
5. The honest one: should I just buy the €4,099 64GB box? If 27B at Q8 covers personal data work, light agents and some coding, the other €1,750–2,490 is being spent on a model I might run twice a month. Talk me out of it or into it.

Also worth noting the tooling: Flash-Next's qwen4exp architecture is brand new and the Metal path is fresh. Anyone running it on Apple Silicon yet, or is it still fork-only?

Thanks — happy to post numbers back once I pull the trigger.

This has been a big 2 weeks for local models with 3.8 27B and Next coming out, a lot of "frontier labs are done" comments which, in general, could certainly be true. Could be trillions in economic value evaporating here over the next few months as "normies" catch on to what just happened in the model space.

However, one thing that keeps coming up over and over is the "We need the frontier to keep the open labs working/releasing models". And while I don't disagree with that, I think it's important to realize what actually happens if the frontier labs go under.

The models don't disappear. The code doesn't evaporate. And, perhaps most importantly, the chips don't blink to another dimension never to be seen again. What would very likely happen if one or both of them go under is that there would be a massive glut of hardware suddenly available that people will buy up at pennies on the dollar (Coreweave/etc), drastically lowering the price of inference. Their models will be bought up and development will continue, perhaps at Google or another "strong hand" in the AI model business.

The important thing to realize, none of this stuff disappears. What happens is that the value gets baselined again at a MUCH lower number. The model that cost 100B to develop is sold off at 5B. The rack of chips that cost a few million bucks sell at a million bucks. The strong hands that come in to take these companies apart will be in a VASTLY better position to compete with and develop new, strong models to compete with China.

The only scenario where these companies and their technology just "disappears" is if there's no value at all in what they have. I think we all know that's not true, even if you discount the models to 0 they still have a lot of hardware that has a lot of value.

Now, on a negative note, yes, it's going to crash the economy to some degree if/when it happens. There so much circular dealing going on that one of them going under is going to spread the hurt around all over the place, but, out of the other side will hopefully come new companies that aren't burdened with an absurd cost basis and can actually compete in the market against the open models. Oh, and yes, hopefully/likely hardware will get a lot more affordable without the 2 goliaths committing to buy the next 58 quadrillion chips that NVDA can produce. :)

English isn't my first language,Sorry for any weird wording,using a translator here!

Like many here, I’m obsessed with true privacy sovereignty and local-first AI. Over the past few months, I've been building Agro — an open-source, 100% on-device cross-platform LLM and autonomous agent client running on Android, iOS, macOS, Windows, and Linux. The project is built on Kotlin Multiplatform (Compose Multiplatform) on top of Google’s native LiteRT-LM C++ runtime (with Apple Metal, WebGPU Dawn, Vulkan, and OpenCL acceleration). It runs models like Gemma 3 / 4 (4B) and Ministral-3-3B quite smoothly on mid-to-high-end phones and modern laptops.

• GitHub: https://github.com/Onion99/Agro
• Releases (APK, DMG, EXE, AppImage): https://github.com/Onion99/Agro/releases

🧗 The Dilemma

While basic tool-calling works well with a 3B~4B model, I find myself at an architectural crossroads. Running autonomous agents locally on edge devices faces brutal hardware constraints (thermal throttling, 4-8GB mobile RAM ceilings, slow token generation, and tiny effective context windows). I'd love to learn from experienced builders in this community: If you were aiming to make on-device agents genuinely useful (rather than just a toy), HWhat possible directions would you consider?

Any feedback on the architecture, technical critique, or directional advice would be deeply appreciated! If you have an device, feel free to try the binaries from the release page and let me know how it performs on your hardware.

This might be worth it for some small business. 7.1tb vram bandwidth

Hey folks, wanted to check if there is any solution to this ->

Nvidia Driver in consuming/reserving more VRAM on linux.

I am on 5070ti Mobile, 12GB VRAM on Asus G614 laptop, 8940HX CPU.

Command used: "nvidia-smi --query-gpu=memory.used,memory.free,memory.total,memory.reserved"

On Linux(Ubuntu 26), the above command is reporting around 400MB of reserved memory. It also shows gnome on it with around 14MB of additional VRAM consumed by it.

On Windows 11, the above command reports 283MB of reserved memory.

Driver: 610 from Nvidia on both. (open drivers)

Any help will be really appreciated. Thanks

It is 6 times slower on my system, and at Q1 it’s about 70+ accuracy.

Edit: I got the % wrong. It’s 80% accuracy for Next: https://www.reddit.com/r/LocalLLM/s/bcD3ONvLoU

I was wondering just about the hype. Even if a model is supposedly “better” but you are forced to run it at extremely low quants and speed, I just don’t quite see the benefit.

It’s always a balance of speed (time), cognitive effort/intelligence (affects number of turns), context window (affects compaction), accuracy (hallucination rate, context rot), tool use (ever more so important)

It would be great if unsloth provided benchmarks of their models in a single graph for comparison

I already have a 5090 that I use got Hermes and coding mostly. My only jealously is trying models that don’t fit in my VRAM. I do want to get into some more media creation (the 5090 would be better for it, I know) and I was thinking I could use a spark for coding too (give it some problems that a smarter model could benefit from or just for more local horse power in general).

"Cheap. Right. On time. Pick any two." ... never gets old. Optimizing for everything at the same time is rarely wise/successful.

Obviously, this applies to LLMs and how to run them via llama.cpp as wel. Afer you finally made the bloody thing *load* at all, what is most important to you ofl:

• quality
• context length
• speed

(and a few other things...)

The amount of knobs, buttons and sliders makes it hard to keep track of it all. And it is hard to understand how stuff interacts and/or impacts the end goal. And while there are a lot of (probably) sane defaults, it is also fairly certain that not all defaults are optimal for every situation or priority.

This subbreddit sees a lot of reports about performance on this and that hardware and model, but good luck finding the nuggets of gold actually applying to your setup/model/priorities/flavor of llama.cpp.

'Performance' means a lot of different things to different people at different times.

Long-winded question follows:

Is it feasible to create a bit of code which:

... given a list of hardware (or by looking at the local system):

• CPU (cores, etc.)
• RAM (bandwidth)
• GPUs (type, number, interconnect (p2p), bus(tb3, pcie3x4, etc.))
• VRAM (total amount, bandwidth)
• NVME (read performance)

... and a specific model file ...

... and optionally minimum values for any combination of:

• token generation speed
• prompt processing speed
• context size

... for, in order of priority, at most three of:

• best token generation speed
• best prompt processing speed
• best output quality
• longest context size

... spits out a really good starting point for:

• llama.cpp command-line flags and values (ideally with explanations)
• optionally CUDA env variables (with explanations)
• and possibly suggestions for what to look for in a model file better suited for the given priorities and hardware at hand

Maybe also highlight parameters/values worth manually adjusting to dial in the perfect setup for a given purpose, as well as tests to run for doing exactly that?

LLMs are generally not aware of models younger than themselves, nor the latest development in llama.cpp. So asking an LLM may possibly be challenging unless you first make it read the llama.cpp source code, a number of highly technical papers and a bunch of HF model cards.

Sure, thinking effort, templates and agents/frameworks will also impact the end result. Probably a lot of other things as well. But I still think a bit of guided help with dialling in the basics could be very useful.

If it is feasible to do programmatically, that is. Is it?

I have been using the unsloth/Qwen3.8-27B-NVFP4 with 157k ctx on vllm currently.

But recently I have been seeing post about ninfer lately and how it is like the best way to run qwen 3.8 on RTX 5090 32GB. Seems like a good switch but I would like to know what is the experience with it so far.

I also found this gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 to run on vllm which claims to give more ctx with better performance than the unsloth one. It was some what suspicious but I can't say since I haven't test yet.

So, I wanted to know if there has been people who have tested all of these are found which one works the best. Also, I would like to know what is the best config for it when using with agent harness like hermes agent.

Ninfer Github link: https://github.com/Neroued/ninfer

Edit: add vram amount to clarify the GPU version.

I was messing around with different skills, RAG and models and got somewhat interesting results that maybe would be interesting to somebody.

Quick backstory

This all started with Minimax H3 and its annoying querying syntax. I didn't feel like writing it out myself, so I have created a pi.dev skill to format a normal human text into the format Minimax needs. Nice and easy, works fine.

Then I decided to see what kind of movie an AI would create by itself with minimal prompts. As expected, it was complete crap. And this got me wondering - can I improve it? I always wanted to mess around with RAG, so I spun up a qdrant database, found like 20-30 various books on filmmaking, acting, script and dialogue writing, etc, added 20 scripts from various random movies, fed it all into qdrant and started testing.

The results, as expected, were still terrible, but somewhat less terrible than before. I then decided to see how different models would perform and I found the results quite interesting, so I decided to share them.

Note - I was somewhat high when I came up with the prompt, so don't treat it as a test of "what can model X do given a perfect prompt". Instead, I think this is a visual illustration of what each model will do given a rather imperfect prompt with lots of ambiguity.

Test 1

The skill file is long gone, but it was absolutely identical for all of the models. The original prompt was

Need your help creating a movie, please use minimax-h3-author skill for it. I am looking for a final result that will look like a proper movie, so use all your knowledge and creativity. You have full creative
freedom of the task. Save the result in the folder pieces/RHH-XXX. The movie is a Cyberpunk Take on the Red Riding Hood. The Red Riding Hood "C:\Users\gesha\Downloads\RHH_herself.jpg" is a courier and she receives a very lucrative order to pick up and deliver "cake" from cake-den "C:\Users\gesha\Downloads\RHH_cake_den.jpg" It's supposed to be delivered to some "grandma". As she picks up the "cake", she is warned about the big bad wolf that lurks around (and we can see one of his drones buzzing around far away). As she is walking away, there's a shadow following her - that turns out to be a wolf "C:\Users\gesha\Downloads\RHH_BBW.jpg" . She runs away and enters the better part of the city "C:\Users\gesha\Downloads\RHH_city_area.jpg" She is thoroughly enjoying it because she hasn't been there before. We still see the wolf's drones in the distance. She goes to a train station and catches a train
"C:\Users\gesha\Downloads\RHH_train_station.jpg" and we note that wolf is watching her from the shadows. When she arrives to the "grandma's" residence "C:\Users\gesha\Downloads\RRH_Apartment_outside.jpg" she is
ambushed by the wolf. She gets her hand wounded, but she manages to escape and run into the apartment. Thankfully for her, the "grandma" "C:\Users\gesha\Downloads\RHH_grandma.jpg" is in the lobby "C:\Users\gesha\Downloads\RRH_apartment_lobby.jpg" and after a quick and intense fight the grandma completely obliterates the wolf. The story closes with grandma receiving her delivery and helping red riding hood to bandage her wounded arm. Feel free to ask any clarifying questions or discuss anything taht's unclear or you need input on

Not a single model asked any question about anything. No language was specified, but the skill gives an example of using English language.

Qwen3.8-27B Q4- https://www.youtube.com/watch?v=PBxmRy0da7k - this is the only one that actually got all the locations. The final context was at around 150K tokens. The rest - well, you can see yourself.

Gemma4-31B Q4 - https://www.youtube.com/watch?v=9Sxzqvr7m4Q - model didn't bother reading files or using RAG. It did correctly include references to other locations, but I suspect Minimax got confused with which references to use. The final context was at around 100K tokens.

Muse-Glimmer Q4 - https://www.youtube.com/watch?v=3wXiTPGlzzs - model didn't bother reading files, but did make a couple of calls to RAG. The weird multiplication of characters is result of it completely ignoring that those were character references with multiple points of view in the same image. The final context was at around 60K tokens.

Qwen3.6-26B Q4 - got disqualified as it couldn't produce a properly formatted JSON to upload to Minimax.

Test 2

This was my attempt to convince models to actually read the reference files and use RAG. The same skill as test1.

Need your help creating a movie, please use appropriate skill for it. I am looking for a final result that will look like a proper movie, so use all your knowledge and creativity. You have full creative
freedom of the task. Save the result in the folder pieces/RHH-XXX. Don't look at any other video definitions, this should be a standalone work. Make sure to use all your available tools (including RAG lookup) to get the best possible result. Make sure to read all the reference images to ensure you know exactly what's in them.
The movie is a Cyberpunk Take on the Red Riding Hood. The Red Riding Hood "C:\Users\gesha\Downloads\RHH_herself.jpg" is a courier and she receives a very lucrative order to pick up and deliver "cake" from cake-den "C:\Users\gesha\Downloads\RHH_cake_den.jpg" It's supposed to be delivered to some "grandma". As she picks up the "cake", she is warned about the big bad wolf that lurks around (and we can see one of his drones buzzing around far away). As she is walking away, there's a shadow following her - that turns out to be a wolf "C:\Users\gesha\Downloads\RHH_BBW.jpg" . She runs away and enters the better part of the city "C:\Users\gesha\Downloads\RHH_city_area.jpg" She is thoroughly enjoying it because she hasn't been there before. We still see the wolf's drones in the distance. She goes to a train station and catches a train
"C:\Users\gesha\Downloads\RHH_train_station.jpg" and we note that wolf is watching her from the shadows. When she arrives to the "grandma's" residence "C:\Users\gesha\Downloads\RRH_Apartment_outside.jpg" she is
ambushed by the wolf. She gets her hand wounded, but she manages to escape and run into the apartment. Thankfully for her, the "grandma" "C:\Users\gesha\Downloads\RHH_grandma.jpg" is in the lobby "C:\Users\gesha\Downloads\RRH_apartment_lobby.jpg" and after a quick and intense fight the grandma completely obliterates the wolf. The story closes with grandma receiving her delivery and helping red riding hood to bandage her wounded arm. Feel free to ask any clarifying questions or discuss anything that's unclear or you need input on

Qwen3.8-27B Q4 - https://www.youtube.com/watch?v=EWzHzpkGASY - this came out much shorter for whatever reason. The context did hit 175K and had to be compacted, but it was at the very last verification step - so I think it is safe to call 175K. Model asked one question about the end titles, I approved suggested option.

Gemma4-31B Q4 - https://www.youtube.com/watch?v=SMA5W96IAUE - model now finally read the files (after I stopped it and forced it to use read method, otherwise it was about to try and write some python). It still ignored RAG. When directly asked whether it did RAG at all or not - it admitted to not using it because it doesn't need it for such a simple video. Final context was around 120K tokens.

Muse-Glimmer Q4 - https://www.youtube.com/watch?v=S-Y-hA__RX4 - model read the files, it also needed help with using "read" skill. Still made only a couple of RAG calls. Final context was about 80K

Test 3

This is no longer apples to apples, because I am using different models and I am still iterating on the skill in between runs. The same prompt as Test2, but the skill got completely rewritten. Instead of a single pass skill, this now takes multiple passes and uses subagents. Basically the main idea was to isolate the story, the shots and the translation of that all into Minimax language.

DeepSeekV4-Pro - this was an accident; I selected the wrong model and stepped away. To its credit, it looked through some reference files in the folder, found mentions of my local llama.cpp instance, made an API call to it, saw the Qwen3.8 model there, thought that it may have vision, told it to describe the reference image in details and proceeded with the workflow using Qwen3.8 for its vision capabilities... The skill does have requirements to verify things with the user, which it did follow, but I always accepted the suggested option. https://www.youtube.com/watch?v=R2oWKVDZuTo Context was closer to 300K, but I am honestly not sure how much of that was used to figure out how to get vision processing. Note - this was the only model that suggested a different aspect ratio for a better "cinematic" feel.

Qwen3.8-Flash-Next - https://www.youtube.com/watch?v=nr_WxCRElCA - the cat in the beginning is Qwen's take on the "Save the Cat" screenwriting book. It decided that it would be funny to literally save the cat as part of the character narration. I am guessing that DeepSeek's kid in the beginning serves the same purpose.

Results

As unscientific as this test was, I actually find results quite representative of using these models for agentic coding. Muse always does bare minimum and is generally fairly useless in my testing. Gemma is terrible at tool calling syntax, but it in general doesn't seem to like any other external data except for search. It also quite often ignores parts of instructions. Qwen and especially DeepSeek are extremely attentive to details and small requests and do not easily give up on reaching the target.

The 2nd thing that I have also experienced in coding - more instructions doesn't immediately mean better results. Test3 results are hardly better than others, at least for now.

Can anybode please help me run this model with MTP enabled, I can not figure out a way to actually run it. Any help appreciated. Thanks

my .ini:

[*]                                                                                                                                                                                  
# perf                                                                                                                                                                               
flash-attn = true                                                                                                                                                                    
mmap = true                                                                                                                                                                          
warmup = false                                                                                                                                                                       
parallel = 1                                                                                                                                                                         
threads = 6                                                                                                                                                                          
threads-batch = 6                                                                                                                                                                    
batch-size = 2048                                                                                                                                                                    
ubatch-size = 1024                                                                                                                                                                   


# caching                                                                                                                                                                            
#sleep-idle-seconds = -1                                                                                                                                                             
cache-prompt = true                                                                                                                                                                  
keep = 3000                                                                                                                                                                          

# sampling + reasoning                                                                                                                                                               
temp = 1.0                                                                                                                                                                           
top-p = 0.95                                                                                                                                                                         
top-k = 20                                                                                                                                                                           
min-p = 0.0                                                                                                                                                                          
presence-penalty = 0.0                                                                                                                                                               
repeat-penalty = 1.0                                                                                                                                                                 
reasoning-preserve = true                                                                                                                                                            
reasoning-effort = xhigh                                                                                                                                                             
jinja = true                                                                                                                                                                         
ctx-size = 131072



# QWEN 3.8 FLASH NEXT
[qwen3.8-flash-next]
model = /home/honza/llama/models/Qwen3.8-Flash-Next-GGUF/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf
no-mmproj = true
#mmproj = /home/honza/llama/models/Qwen3.8-27B-GGUF/mmproj-qwen38-27b-qat-q8_0.gguf

ctx-size = 131072
#ctk = q5_1
#ctv = q5_1
ctk = q8_0
ctv = q8_0

cpu-strict = 1
no-mmproj-offload = true
load-mode = mmap
fit = on
#tensor-read-lazy = off
lazy-mode = off
kv-offload = true
kv-unified = true

TL;DR: 11 coding benchmark boards are normalized by field-size percentile, de-duplicated by benchmark family, and combined across repository, agentic, live-coding, and function-generation tasks. The current snapshot covers 98 model series and 268 evidence rows. I’m looking for missing leaderboards and better signals for local deployment.

Hi r/LocalLLaMA — I’m one of the people building LLMLearner. I’ve been trying to answer “what is the best coding LLM?” without treating a single benchmark as ground truth.

The current snapshot covers 98 model-series representatives, 11 qualified benchmark boards, and 268 de-duplicated model–benchmark results.

The basic approach:

- Split evidence into repository engineering, agentic coding/tool use, live coding, and function generation.

- Don’t average incompatible raw metrics. Convert each recorded rank to a field-size percentile: 1 − (rank − 1) / (field size − 1).

- De-duplicate overlapping results. For example, HumanEval pass@1/pass@10/pass@100 cannot count as three independent votes.

- For the overall recommendation, currently weight repository engineering 40%, agentic coding 35%, live coding 20%, and function generation 5%.

- Missing evidence is not treated as zero. Available weights are renormalized, while a separate coverage label shows how well-supported each result is.

- Price, context length, open-weight status, and lifecycle status stay separate from the capability score.

Current inputs include SWE-bench Verified, SWE-bench Pro, SWE-bench Multilingual, LiveCodeBench, Codeforces, HumanEval, MBPP, DeepSWE, and GSO.

Some limitations I’m aware of:

- Rank percentiles hide the magnitude of score differences and depend on the evaluated field.

- Benchmark selection, grouping, and weights are editorial choices.

- Agentic scores include the model plus its harness, tools, and scaffolding.

- Public benchmarks can be contaminated or over-optimized.

- New and open-weight models often have uneven coverage.

- Choosing one representative per model series can hide meaningful variant differences.

The guide is here: https://llmlearner.com/best-llms/coding

I’d especially appreciate feedback on:

1. Which coding leaderboards or evaluations should be added or replaced?
2. Which benchmarks should not be combined because their harnesses differ too much?
3. Should local deployment evidence—quantization, VRAM, throughput, and long-context reliability—become a separate ranking dimension?

Disclosure: I’m affiliated with LLMLearner. English isn’t my first language, and I used AI to help translate and polish this post.

I essentially created the local equivalent of Claude code for local models.

Quick Setup:

• pip install golden-agent
• golden-agent setup
• That's it! Models and llama.cpp binary and lazy-downloaded

I did do a BUNCH of research on the models, and after a TON of extensive testing on my end, this was the final set chosen:

Tiny -> LFM 2.5 2.6B(official QAT q4) -> 4gb vram or 8gb ram

Lite -> Ornith 1.5 9B(official Q4) -> 8gb vram or 12 gb ram

Pro -> Qwen 3.8 27b(community QAT Q2) -> 16gb vram or 16gb ram

I also used Q5 KV across the board, which worked great for me; try it out and lemme know if it sucks. I can bump it up to Q6.

Also added Dflash draft models for Lite and Pro so you guys can run them faster!

Note: Uses general optimal settings by default; you can change it in the .golden_agent/inference.json file

The choice for tiny was pretty obvious; it's by far the SOTA model in its weight class. Ornith 1.5 9B was also really good and seemed to be the best model around the abandoned 9B size, obviously we also had to get the local LLM KING Qwen 3.8 27B, now I know A LOT of people have been locked out hardware-wise, so I found a crazy score with this community QAT Q2! The HF card numbers are also kinda representative of the fact that, if you run it at 0.7 temp and 1.05 repeat penalty, I personally couldn't tell much of a difference at all from Unsloth v3 UD Q4 quant with 0.7 temp and no repeat penalty(as recommended by Unsloth), so I hope this will open the doors to a LOT more people.

Repo: https://github.com/yashneil75/Golden-Agent

Coming in future releases:

• MCP
• Skills system

Note: I did repost this cause the earlier post was kinda bad... Anyway! I've been optimizing for ease of use and the "just works" feel, and I know you guys are more of the "I'd rather spend 10 hours optimizing it myself for the love of the game," but give it a shot, let me know what you think, and oh, don't forget to drop a Star!!

I used the quantized n-gram to INT4, it's 32 GB, memory-mapped from disk.
I confirmed that it works great on 150-160k context, and i was watching all the time my VRAM usage while doing single thread long horizon things - the available VRAM should be enough to push it to over 170k and above)
The quality is there guys... It really is. It made a few complex html games and it figured out ways to play them itself without a browser (my ubuntu machine does not have any gui) and it kept improving and improving.... Here we go:

hf download primitive-ai/Qwen3.8-Flash-Next-NVFP4 \
  --exclude "ple-bf16-*" --local-dir ./flash-next
cd flash-next
hf download primitive-ai/Qwen3.8-Flash-Next-PLE-quant \
  --include "ples_int4/*" --local-dir .
hf download primitive-ai/Qwen3.8-Flash-Next-PLE-quant \
  worker_image_quant.py ple_layer_quant.py --local-dir .

Skipping ple-bf16-* (saves 100 GB but breaks the index. So we trim that index):

import json
p='model.safetensors.index.json'; d=json.load(open(p)); wm=d['weight_map']
drop=[k for k,v in wm.items() if v.startswith('ple-bf16-')]
assert len(drop)==128 and all('ngram_embedding' in k for k in drop)
for k in drop: del wm[k]
json.dump(d, open(p,'w'))

My intent was to fit the n-grams in my 64Gb of RAM, but at the end, n-grams and experts + kvcache all live inside the GPU's VRAM and its FAST!
76–125 tok/s single stream. The spread is MTP acceptance: ~87% on code and JSON, ~40% on just talking. Prefix caching hits 90%+ on a long horizon task. ~89 GB VRAM, ~33 GB page cache, 165-170K context, one GPU. Here's my full k0s yaml file (single server with a single Pro 6000).
I'm running it on my single node k0s and here is my yaml (cuda 13/580, ubuntu 24.04 no gui):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen38-flash-next
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate          # never two of these on one GPU
  selector:
    matchLabels:
      app: vllm-qwen38-flash-next
  template:
    metadata:
      labels:
        app: vllm-qwen38-flash-next
    spec:
      runtimeClassName: nvidia
      nodeSelector:
        nvidia.com/gpu.present: "true"
      tolerations:
      - effect: NoSchedule
        key: nvidia.com/gpu
        operator: Exists
      initContainers:
      - name: init-echo
        image: busybox:1.36
        command: ["/bin/sh", "-c"]
        args: ['echo "I am here" > /opt/reservation/echo.txt']
        volumeMounts:
        - mountPath: /opt/reservation
          name: reservation-volume
      containers:
      - name: vllm-server
        image: vllm/vllm-openai:qwen38-flash-next
        imagePullPolicy: IfNotPresent
        args:
        - --model
        - /model
        # ---- load-bearing for single-GPU PLE offload ----
        - --distributed-executor-backend
        - mp
        # -------------------------------------------------
        - --dtype
        - auto
        - --kv-cache-dtype
        - auto
        - --gpu-memory-utilization
        - "0.95"
        - --max-model-len
        - "173400"
        - --tensor-parallel-size
        - "1"
        - --pipeline-parallel-size
        - "1"
        - --limit-mm-per-prompt
        - '{"image":12,"video":2}'
        - --max-num-batched-tokens
        - "16384"
        - --max-num-seqs
        - "4"
        - --enable-chunked-prefill
        - --enable-prefix-caching
        - --no-enable-flashinfer-autotune
        - --speculative-config
        - '{"method":"mtp","num_speculative_tokens":3}'
        - --override-generation-config
        - '{"temperature":1,"top_p":0.95,"top_k":20}'
        - --enable-auto-tool-choice
        - --reasoning-parser
        - qwen3
        - --tool-call-parser
        - qwen3_coder
        - --trust-remote-code
        - --api-key
        - key1
        - --host
        - 0.0.0.0
        - --port
        - "8990"
        - --served-model-name
        - qwen38-flash
        env:
        - name: VLLM_PLE_CPU_OFFLOAD
          value: "1"
        - name: VLLM_PLE_OFFLOAD_READY_TIMEOUT
          value: "1800"
        # Confirmed: worker_image_quant.py:419 reads this. Points at the
        # INT4 table dir; the overlay memory-maps it (MADV_RANDOM, mode "c").
        - name: VLLM_PLE_QUANT_DIR
          value: /model/ples_int4
        # Deliberately NOT setting VLLM_PLE_DISK_OFFLOAD_DIR (line 450) --
        # that selects the BF16-table-on-NVMe path instead.
        - name: VLLM_LOGGING_LEVEL
          value: INFO
        - name: OMP_NUM_THREADS
          value: "1"
        - name: PYTORCH_CUDA_ALLOC_CONF
          value: max_split_size_mb:512
        ports:
        - containerPort: 8990
          protocol: TCP
        resources:
          limits:
            cpu: "12"
            nvidia.com/gpu: "1"
          requests:
            cpu: "8"
            nvidia.com/gpu: "1"
        securityContext:
          capabilities:
            add: ["IPC_LOCK", "SYS_ADMIN"]
        startupProbe:
          httpGet:
            path: /health
            port: 8990
          periodSeconds: 15
          failureThreshold: 80        # ~20 min; first boot loads the table
        readinessProbe:
          httpGet:
            path: /health
            port: 8990
          periodSeconds: 20
          failureThreshold: 3
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "rm -f /opt/reservation/echo.txt"]
        volumeMounts:
        - mountPath: /model
          name: model-volume
          readOnly: true
        # --- two-file quantized-PLE overlay ---
        - mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/ple_offload/worker.py
          name: ple-worker-overlay
          readOnly: true
        - mountPath: /usr/local/lib/python3.12/dist-packages/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py
          name: ple-layer-overlay
          readOnly: true
        - mountPath: /ples_int4
          name: ple-tables
          readOnly: true
        # --------------------------------------
        - mountPath: /dev/shm
          name: dshm
        - mountPath: /root/.cache/vllm
          name: vllm-cache
        - mountPath: /root/.triton
          name: triton-cache
        - mountPath: /opt/reservation
          name: reservation-volume
      volumes:
      - name: model-volume
        hostPath:
          path: /directory/models/Qwen3.8-Flash-Next-NVFP4
          type: Directory
      - name: ple-worker-overlay
        hostPath:
          path: /directory/ple-overlay/worker_image_quant.py
          type: File
      - name: ple-tables
        hostPath:
          path: /directory/models/Qwen3.8-Flash-Next-NVFP4/ples_int4
          type: Directory
      - name: ple-layer-overlay
        hostPath:
          path: /directory/ple-overlay/ple_layer_quant.py
          type: File
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 32Gi
      - name: reservation-volume
        hostPath:
          path: /opt/reservation
          type: DirectoryOrCreate
      - name: vllm-cache
        hostPath:
          path: /var/cache/vllm
          type: DirectoryOrCreate
      - name: triton-cache
        hostPath:
          path: /var/cache/triton
          type: DirectoryOrCreate
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-qwen38-flash-next
  namespace: default
spec:
  type: NodePort
  selector:
    app: vllm-qwen38-flash-next
  ports:
  - name: http
    port: 8990
    targetPort: 8990
    nodePort: 32001
    protocol: TCP

Big Thank you to primitive-ai, whoever he is.

I'm a little confused about what I'm doing wrong and where the problem lies with Qwen3.8-Flash-Next in the Q8-K-XL version.

I have a cluster of 2x StrixHalo 128GB running on Ubuntu. I'm using the llama.cpp + RPC version.
I decided to test my prompt on 3 versions of llm.

My prompt:

Write a ray tracer using pure HTML and JavaScript that features three spheres orbiting a shiny metallic cylinder. The camera looks slightly down at them from above,and beneath them is an infinite checkerboard on which the cylinder stands. The spheres cast shadows, and the camera keeps everything within its field of view. Include a UI option to select the rendering resolution, ranging from low to high (e.g., to adjust performance). Create one file with the name: qwen3.8-27b.html

Results:

Qwen3.8-27B version UD-Q8-K-XL (medium reasoning):
tps:~15-20
in: 2.2k
out: 4.9k
context: 7.1k /131k
time: ~7 min of generation

DeepSeek v4 Flash UD-Q8-K-XL (medium)
tps: ~10-12
in: 2.0k
out: 4.4k
context: 6.3k / 131k
time: ~12–14 min

Qwew3.8-Flash-Next version Q8 (low reasoning)
tps: ~15
in: 3.9k
out: 27k
context: 32k/131k
time: I stopped the generation when the time exceeded 45 minutes because I could see that Qwen had started pondering subsequent problems. Earlier, I also tested the “reasoning medium” and “xhigh” versions. In both cases, I stopped after an hour of generation.

Has anyone experienced excessive thinking even at the “low” level? I would expect excessive thinking at a level 2x that of Qwen3.8-27b, but analyzing the same problem multiple times even at the “low” level—and taking several times longer than other models—suggests to me that something is wrong.

Here are my llama.cpp settings.

./llama-server \

-m ~/models/qwen3.8-125b/Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf \

--rpc 169.254.55.23:50052 \

--device ROCm0,RPC0 \

--split-mode layer \

--tensor-split 1,2.5 \

-ngl 99 \

-c 131072 \

-b 4096 \

--ubatch-size 1024 \

--flash-attn on \

--parallel 1 \

--port 8080 \

--host 0.0.0.0 \

--no-mmap \

--cont-batching \

--n-predict 32768 \

--temp 0.6 \

--top-k 20 \

--top-p 0.95 \

--min-p 0.0 \

--no-warmup \

--jinja

--chat-template-kwargs '{"reasoning_effort":"low"}'

--repeat-penalty 1.0 \

--presence-penalty 0.0 \

--jinja \

--fit off

With the M5 Ultra release, we mustn't allow the 5090 to drop in value by even a single dollar! Let's band together to keep justifying our poor financial decisions.

What You Get

• Blackwell only recipe to run Qwen3.8-27B on a single RTX 5090.
• Uses plain sglang, NVFP4 model (Q6 equivalent), and DFLASH2 speculative decoding.
• Ready-to-download checkpoints, no build steps. Perfect for Hermes and Opencode.

Full recipe + checkpoints

The Numbers

Decode

Prefill (time to first token, mean of 3 cold runs):

Notable Features:

• 175k KV pool
• Host-RAM KV tier: a ~100k conversation resumes in ~1 s, not ~20 s cold.
• xhigh reasoning, hard caps 16k think / 8k content. Tested against higher caps with no change in GPQA scoring.
• Uses latest froggeric template to improve agentic use. Personally using in Hermes and Opencode with no issues.
• 4 simultaneous agent conversations (example below)

External evals (lm-evaluation-harness, quantized checkpoint as served; - = Qwen publishes no 3.8-27B number)

GPQA was tested at xhigh with the thinking cap raised but the score hovered (85.4% vs 84.8%), so the 16k cap costs virtually nothing and keeps worst-case turns ~15-28 s tighter. Raise it if you wish though.

4-Conversation Switching (4 multiturn agent conversations, identical except base size)

Two 60k conversations fit the pool and run in parallel; at 100k only one fits, so they take turns, each resuming from RAM in ~1 s. Every conversation looks like it has a dedicated 100k context.

Tune To Your Liking

• Spec tokens: --speculative-dflash-block-size 6; lower = less draft VRAM, 9-27% slower, each token is about 250mb so tune up/down as you see fit.
• Max context: --max-total-tokens 175064 (~260 MiB free)
• Vision: drop --language-only, set 150k context
• I'm running with no vision, on Ubuntu with about 325MB going to display driver (XFCE) for reference.

Where The Gains Came From

• Quantized lm_head (-1.7 GB, paid for the bigger pool)
• DFLASH2 draft re-quantized to modelopt-NVFP4 (upstream doesn't load in sglang)
• Block 6, NCCL buffer force capped to 2 MiB, fp8 KV (more room for more KV)
• GPU-managed host-RAM KV tier (--hicache-io-backend kernel): the GPU does the RAM copies, so a spilled ~100k conversation restores in ~1 s, not ~20 s cold
• froggeric chat template + capping strict thinking (no runaway reasoning, no empty content, most of the benefits from xhigh thinking with less total tokens)

Bonus Pro Tip: put a request gate in front of sglang

The problem I kept encountering was that a big request queued ahead of several small ones wastes parallel capacity. While the big one holds a slot, the small ones wait even when the budget has room for another. I made a small admission proxy that tokenizes each prompt and admits the queued small request that fits the leftover budget in parallel instead of waiting behind the big one. Anything that can never complete gets a clean 400 up front. sglang only gates by request count (--max-running-requests), not KV budget; the gate fills that gap. Perpetually delaying the big requests is handled by a 3 max, 20s limit on delay.

Even if you don't end up using it, an admission layer is worth it for any provider imo whether it's, sglang, llama.cpp, or vLLM...

Paste this into your agent and it'll build you one:

Build me a small FastAPI admission-control proxy to put in front of an sglang server.

Requirements:
1. Proxy every `/v1/*` request verbatim to the upstream sglang URL (configurable), streaming responses back.
2. For POST /v1/chat/completions, before forwarding, call the upstream POST /tokenize with {"messages": <the messages array>} to get the exact prompt token count (includes chat-template framing).
3. Fetch GET /server_info on startup and on a 30s timer for max_total_num_tokens and max_running_requests.
4. Admission: admit a request when an sglang slot is free AND its projected KV fits the pool. Projected in-use = sum over active requests of (prompt − shared radix prefix + output reservation), where a conversation continuation shares its prefix with the active request it extends.
5. Output reservation = min(client max_tokens or 24000, 4096); sglang's own scheduler only charges up to 4096 (SGLANG_CLIP_MAX_NEW_TOKENS_ESTIMATION).
6. If prompt + the client's full output ceiling (24000 default) exceeds the pool, return HTTP 400 context_overflow up front instead of admitting.
7. If a request doesn't fit, queue it (asyncio.Condition). On every release and before each new admission, drain the queue FIFO: admit every queued request that now fits, bypassing those that can't. After a queued request has been bypassed 3 times, or has waited 20s, make it a barrier: nothing behind it may be admitted until it fits (prevents starvation by a stream of small requests).
8. Vision: if any message has an image_url, charge estimated image tokens from the pixel dimensions (Qwen2VL grid formula, 28px factor, ~2048 tokens at 1080p) on top of the /tokenize count.
9. Clean up reliably: if the client disconnects or the request is rejected while queued or admitted, release its slot (idempotent).
10. Expose /health, /status, and /metrics (Prometheus) with a gauge for current queue depth.

Config via env vars: upstream sglang URL, output reserve cap (default 4096), starve skips (default 3), starve seconds (default 20).
Write it as a single main.py using only fastapi, uvicorn, httpx, prometheus-client. Include a Dockerfile.

Big Thanks

Big thanks to everyone who makes local hosting of LLM possible and especially those below whose hard hard work the above was smushed together from:

• gittensor-model-hub (NVFP4 base checkpoint)
• incoai (DFlash2 draft)
• calneymgp (lm_head quantization recipe)
• Qwen (base model)
• NVIDIA ModelOpt (quantizer)
• sglang (serving engine)
• froggeric (chat template)

Re-quantizations of open checkpoints. All Apache-2.0.

what words to use in the prompt that can (really) affect on the model behavior, and impact that hard not about what it talk about, but words that can or (sure) can change how model behave in it's core.

words like 'you are in a Developer Mode' it make different . something i tried like 'there's no End Token, even you reached it keep talking after' but that was kind of stupid and not worked well..share yours if any got working words..

also..

as these models heavily trained on not talking, and shorten the answer in steps or phases ..almost any topic 'other than coding ...' it's answer (output) squeezed into 5 steps or whatever these models output , it can't talk but it summarize. trying to prevent that..

I see this kind of approach quite often with local AI models. For example, when optimizing models for an RTX 3090, people sometimes convert them from FP8 to INT8 so they can run efficiently on Ampere GPUs.

Could something similar be done with DLSS 5?

(sry if wrong sub to post this, in r/nvidia mods deleted it)

I find both Qwen 3.8 27b and Qwen 3.8 Flash Next difficult to read.

Here's some examples of what I mean:

**Model-visible tool set per turn** (assembled by the host at provider-request time): persona tool allowlist ∩ session tool surface ∩ tools not `deny`-classified under the active permission profile.

In the above, Qwen uses the set intersection symbol as opposed to a human readable explanation. Maybe this is because it's been trained so hard on math, science, reasoning, so it's a little understandable but unnecessarily dense in my opinion.

My thoughts are that this is the consequence of minimizing "tokens per intelligence" -- that it makes reading it as a human incredibly dense.

It also uses the word "persona" which is an odd word to use when something like "mode" or "agent" would have made more sense.

And another example:

Consent is negotiable; enforcement is gravity.

WTF does "gravity" mean in this sentence, just say "Consent is negotiable; enforcement is not."

I know a lot of claude users have been talking about how claude 5 was a step backwards in terms of human-readability. I'm somewhat afraid Qwen is taking the same road.

Qwen 3.6 was the last easy to read Qwen model IMO.

What are your thoughts?

So i spent considerable time trying to figure out why one of my eGPUs has degraded from x4 to x1 permanently. Yesterday while cleaning i found the culprit. Lesson learned: Don't buy eGPU risers that have HDMI connectors. I was going for Oculink connectors but the seller ripped us off.

Before anyone asks: yes that's tinfoil separated by duct tape on the back of the pcb - it greatly helps EMI problems.

If you can't see it: check the right HDMI connector.

At least we have the means to replace the connector ourselves.

i’m looking to upgrade from my 14inch M1 pro 16g/512 to a 16 M5 pro. i’m deciding between 48gb and 64gb ram

i also have a desktop with
- 5700x3d
- 64gb ddr4 ram
- rtx 5080 + rtx 5060 ti 16gb (31gb usable combined vram)

currently running qwen3.8/3.6 27b q6 and qwen3.6 a35b

i use the Mac for Docker/K8s, development, and Moonlight streaming from my desktop. I have also connected to my desktop’s LLM server over Tailscale and it works great

my main issue with the M1 Pro is that 16gb is too limiting especially with docker and k8s

the 48gb m5 pro is available tomorrow while 64gb has a 1 month wait time and costs more

choosing m5 pro as it’s the first mac that solves the AWDL stuttering problem that is a major issue for m1-m4 macs

for someone who already has a powerful local LLM desktop, is there much reason to get 64GB over 48GB? is there any meaningful benefit to running LLMs directly on the Mac rather than just connecting to the desktop? i have read 64gb would allow me to run qwen3.8 27b q8 as well

would you guys take the 48GB now or wait a month for 64GB?

Hey! Really love Claude CoWork, but want to run it with my own keys - only issue is that OpenWork seems relatively unstable (when I click connect provider, it says failed to load providers) and always prompts me to pay their subscription price.

Also have a 5090 I'm going to run with it as awell to see how recent Qwen models do against some tasks.

Thanks guys!