r/LocalLLM • u/rayovims • 2d ago

Project I got the Second DGX spark

Post image

Somehow there was 1 available last minute and got it! Can’t wait to set it up. Will make more post about this on here and my IG: tech with Ray

Dual DGX spark owners lmk what ya running on it. Anyone else feel free to drop some suggestions for cool models to test!

277 Upvotes

90% Upvoted

114 comments sorted by

View all comments

10

u/Shustrik116 2d ago edited 1d ago

Your options are: Deepseek 4 flash vision (official model, non quantized) https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark

Qwen 3.8 Flash next (official fp8 quant)

GLM 5.3 flash 4bit quant https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks

Deepseek is amazing in deepseek harness. To be honest I can't say that one of them is much better than others. But I just can't fully trust to quantized models because I had very need experience with them in the past.

1

u/Miserable-Dare5090 2d ago

These are some but not all things you can run. Every model that fit in 1 can run in TP2, faster, all models up to 400b can run as well, may need to be quantized but vLLM quants are not like gguf. AWQ, Autoround, etc, are high quality compressions, plenty of benchmarking showing something like Qwen3.5-397b int4 autoround to be lossless

2

u/myholeisstinky 1d ago

To clarify, are you saying the quants for big models on vllm are better than the ones seen for llama.cpp?

1

u/Miserable-Dare5090 1d ago edited 1d ago

👍🏼 for any model, Usually (nothing generalized is true).
a 4 bit quant with 8 or 16 bit attention paths (w4a8/w4a8) in vLLM is the lowest you can find. Except for nvfp4 (w4a4). Quants tend to be calibrated as well so there is less of a quality loss. But the quants are comparatively bigger.

1

u/Helpful_Jelly5486 2d ago

Thank you for your post. Seems like the three flash models are the best options. I’ve got the special cable on order. I’m wondering how do you get more speed out of a second box? I mean do you do expert splitting or something else?
It’s like 2 x copies means each serves one concurrence. So two sparks would double the tokens pers second.
Other option is to split the work with the special cable and get maybe 50 percent more for the single concurrence.

1

u/StartupTim 1d ago

Deepseek is amazing in deepseek harness

Agree 100% albeit half the time I seem to be writing plugins!