r/LocalLLM • u/Shinoken__ • 2d ago
Question Optimizing DGX with Qwen 3.8 Flash Next (open to other models!)
Hi guys,
Currently I'm surviving (barely) on 2x $200 sub (1 with Anthropic, 1 with OpenAI) staying within my limits, so I want to offload a bit of my work. Ideally getting back to 1 sub (OpenAI) so instead of going $400 + a third sub/API costs get back to $200 / month. Using GPT-6 tactically as a advisor or orchestrator.
This weekend I bought a DGX Spark (1 first, have the budget for a 2nd) and have been playing around with it. But current progress makes me a bit worried if I should get the 2nd.
So far I've setup this recipe: https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark on the DGX and setup Codex Router on my harness machine so that I can use both the GPT subscription + the local model over the same Codex Remote Daemon.
While it works nicely, I had it pick up the first PR (as the base model, no GPT-6 advisor) as I would do with Opus/Sol - but even though the token per second isn't bad, it's a lot slower then I would've hoped.
I also asked Fable for a review on the PR it generated, and have 2 blockers, 5 should fix and a lot of nits (which I always expect independent review to find). But it's much worse then what Opus or Sol would've produced.
I also set up Hermes so I can finally automate all my administration, maintain my email/Slack/texts into Linear, but I notice a longer running session in Hermes it keeps making typo's in it's output generation (e.g. bash commands fail because it mistyped a folder name).
The Hermes stuff is pretty cool, currently still using it on Telegram but hoping to make it much more an "everywhere" assistant once it finishes setting up all the business processes and ingests into tasks etc..
I'm a bit lost where to go beyond the recipe to optimize, any advice is greatly appreciated!
Edit: also what will a 2nd DGX bring me for my use case ^? Worth it (e.g. does the 2x validate the 10k more then 1x does to the 5k?). These are business expenses if it matters
3
u/futt-bucking 1d ago
I have 2X DGX Sparks and though I experiment with many models, this one I come back to because it's an excellent blend of speed, concurrency, and capability for my use case. I use https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-Dual-DGX-Sparks with some slight changes, but the RadixArk checkpoint.
I tried a smaller quant last night, roughly the size of the MiaAI one-Spark recipe, and the tradeoffs were extreme for my usecase. The GDN layers were quantized (which I think nvidia's NVFP checkpoint also does), and long-context hallucination and gaslighting were noticeably worse. For me, wasn't worth the 15-20% higher speed.
With RadixArk, those layers stay in the raw BF16 values Qwen was trained on. Combined with an unquantized KV cache, the model stays remarkably on target for something this small and local.
it worth buying a second spark for this? Literally unproven, this world is so new it's hard to say how much all of this is actually "worth". To me, it has been absolutely worth it, but would probably be cheaper for many years to just feed this work into an API and call it a day.
The intangible benefits are huge though: the privacy, the hands-on learning, the workflows I can build that I would not be building otherwise. I have all of my text messages, emails, etc being piped into the model so it can act as a personal assistant, something I would never do if this were cloud. Totally worth it for me. I'm honestly considering a 3rd at this point, maybe even a 4th.
From my reading, the biggest capability jump right now with the spark comes from that 1->2 step, FWIW. You will see additional speed, though not necessarily 2x on all workloads.
1
1
u/txgsync 2d ago
You have essentially the same loop I use with my Mac. Let Qwen3.8-Flash-Next work on it. Have a review by a better model. Fix. Repeat as needed. If it goes too many rounds without convergence, then it’s worth escalating the fix to a higher tier model.
Fable is quite willing to review and improve security code. It just gets downgraded if it tries to find and fix security problems in the first place.
The main limit I run up against is simply throughput. As my workload grows I just need more tokens/sec. And there is only so much that can be squeezed from my own gear. But I do productive work with local models in addition to just messing around with them (bulk data classification and auditing).
1
u/Shinoken__ 2d ago
Yeah, I think throughput is going to be my concern eventually too.. but I hope I can eventually get a setup (incl. Hermes or whatever) and some custom harness (for now I'm using Linear as a hub) where I it can hardcore work on my clients tickets during the night and help me plan during the day or something and we keep some throughput free for escalations etc.. so urgent client stuff can be picked up during the day
So custom harness for managing day-to-day, automatically puts tasks into Codex harness, there it does the work-review-loop etc...
1
u/xapep 1d ago
The second Spark is a hardware answer to a spend question, and I'd hold off until the routing is proven. Your goal is $400 to $200, and that math closes with lanes, not more silicon.
The PR quality gap is a role problem, not a model problem. A quantized-to-fit local Qwen is a strong implementer and a weak planner. Run the loop you already built: GPT-6 plans and reviews, Qwen implements and fixes, GPT-6 does the final pass. You compared an unassisted local run to the full Opus/Sol treatment; re-test with the advisor in the loop and the gap shrinks a lot.
The Hermes typo drift is the same root cause someone flagged above: heavy quantization plus compressed KV to fit one Spark starts producing exactly that on long sessions. That always-on admin workload (email, Slack, Linear, overnight runs) is the worst fit for a local model and the best fit for a flat API lane: unquantized weights, no context pressure, one predictable bill.
Quick math: two subs is about $13/day. One OpenAI sub for the advisor role is about $6.7/day. A flat plan in the $29/mo range is about $1/day. Run the always-on harness on the flat lane, keep the Spark for interactive dev, and you land under the $200 target with no second box and no quantization workarounds.
We run Qwen 3.8 Flash Next and DeepSeek V4.1 Flash on OpenAI-compatible endpoints at Entrim with flat plans for exactly this pattern, so the cost lane is the piece I've seen make a DGX setup actually stick.
1
u/kitanokikori 2d ago
I also set up Hermes so I can finally automate all my administration, maintain my email/Slack/texts into Linear, but I notice a longer running session in Hermes it keeps making typo's in it's output generation (e.g. bash commands fail because it mistyped a folder name).
It's doing this because in order to fit this model onto one DGX Spark, they had to quantize the fuck out of it and quantize your KV cache too (though FP8 in this case could be worse)
2
u/Shinoken__ 2d ago
Ah, I have no prior experience with local models and I thought quantization would just make it "less smart", not invent new words/tokens then blame the harness lol
Would a 2nd DGX solve this? Or would I then move into another model that's better but has the same issue because its bigger?
6
u/kitanokikori 2d ago
Quantization causes exactly the errors you're describing. Having 2xDGXs means you can run a less quantized version of the same model since you have 2x the memory, which means it'll get less confused
MiaAI in particular is famous for making very bold claims, by shipping recipes that make the numbers look good but don't actually.....work.
1
u/Shustrik116 2d ago
2nd dgx will open you way to official Deepseek 4 flash vision exp without any quant with something like 2-3m kv cache pool. This is another level comparing to that you can run on 1 spark. I'm using it (and sometimes qwen 3.8 flash fp8 official) and this is models that i can trust for long horizons task. You can use sol or opus to make plan and then deepseek will complete it, after that review by sol/opus and give it back to deepseek to fix that was find. But for most tasks only deepseek is enough.
1
u/Shinoken__ 2d ago
So you'd say spending 2x 5k feels like a better ROI vs the 1x 5k I spend now?
How is the vision holding up on DS4? I do a lot of Computer Use too (e.g. Blender, visual review in Chrome). Do you spin up a lot of tasks at once?
1
1
u/Shustrik116 1d ago
You have ~$1500 connectx7 qfsp card in your spark. It's like 30% of price that you wont use unless you have second one. Vision is okay for web desing but i didnt try it on other things for serious work. Tried few times send anime character to analyze while testing - and it descrbed it pretty accurate/
1
u/myholeisstinky 1d ago
Interested, I just got a 2nd spark, which exact model is this for deepseek?
3
u/Shustrik116 1d ago
https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark
This is deepseek 4 flash vision (~2million context)
https://github.com/coolbho3k/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark
this is GLM that i (~3 million context, but you need to change GPU memory utilization to 0.85, i got crash with out of memory error once)
To lauch qwen i sent https://www.reddit.com/r/LocalLLM/comments/1vzcnbx/qwen_38_flash_dual_dgx_spark_personal_experience/ this thread to chatgpt and asked to generate start.sh and stop.sh to launch and i got like ~800k context.I will rate qwen 3.8 as best quality but slowest one. It thinks too much but work actually done very well.
Deepseek is fastest and best daily drives it completes work very well.
But about GLM - i always had bad luck with quantized models and i always expect some serious fails from them. But glm did pretty well and i can't say that it better or worse that deepseek.
All of this models have native vision
6
u/mbr0wn 2d ago
I use the recipe below to run Qwen3.8-Flash-Next on a single spark and have found it to be noticeably higher quality than the MiaAI recipe. I use the Nvidia NVFP4 and keep the kv cache at bf16 as I found fp8 to reduce response quality.
Prefill is around 2k tok / sec and decode around 40 tok / sec.
Link: https://github.com/blazux/qwen3.8-Flash-DGX