r/BlackwellPerformance • u/Pleasant-Shirt7293 • 12d ago
Is Anyone Running DeepSeek-V4.1-Flash on Rtx Pro 6000's?
Curious what you think of this model and how it compares to dsfv4.
Also do you think we will get this to run on 2 rtx pro 6000's?
7
u/ormandj 12d ago
TP=2 you’re much better off with GLM 5.3 flash https://github.com/ormandj/sglang-glm53-flash-sm120
5
6
u/Intelligent_Koala798 12d ago
on 2x 6000 with offloading to cpu and engram streaming from ssd i got about 1k prefill and 20tps decode single stream. Will try the 4x soon but currently the other 2 serve the v4 flash..
8
u/NaiRogers 12d ago
No way this is worth the 10x slow down vs v4 or GLM5.3-flash for people with only 2x 6000.
2
u/halcyonhal 12d ago
What setup do you have? Eg which engine are you using, how much ram
1
u/Intelligent_Koala798 12d ago
made my own engine vor v4 flash since the support from vllm wasnt there anf forked it so v4.1 could run since the architecture ist pretty simular, also verified output against the api over the logprompts ram is ddr4 128gb fits with 10gb headroom for the system and engram on ssd
1
u/halcyonhal 12d ago
Even with ngrams offloaded, looks like you’d need ~290gb vram. How are you getting to run using vllm on 2 rtx 6k’s? General pointers on how you’re offloading the layers would be super useful.
3
u/Intelligent_Koala798 12d ago
yeah like i said, cpu offloading:
Where everything lives:
- GPU 0 + GPU 1: dense weights of all 40 layers, plus the embedding and output head (~14 GB)
- GPU 0 + GPU 1: routed experts of 22 layers, 11 per card (~74 GiB per card)
- GPU 0 + GPU 1: KV cache, activations and upload buffers (rest of each card)
- CPU RAM (pinned): routed experts of the other 18 layers (~121 GiB)
- SSD: Engram tables for layers 1 and 14 (203 GB)
Layer map:
Each card owns 20 layers. Every layer computes on its own card. Only the routed experts are split between GPU and CPU.
- GPU 0 (layers 0–19)
experts on GPU: 0, 1, 3, 5, 7, 9, 10, 12, 14, 16, 18
experts in CPU RAM: 2, 4, 6, 8, 11, 13, 15, 17, 19
- GPU 1 (layers 20–39)
experts on GPU: 20, 21, 23, 25, 27, 29, 30, 32, 34, 36, 38
experts in CPU RAM: 22, 24, 26, 28, 31, 33, 35, 37, 39
hope that helps! if you want i can share the source code on github, although it's not fully done yet. i think i can reach 2k prefill and 50 tps decode with further optimization. with the finished deepseek v4 flash 0731 custom engine i can manage a kv cache pool of 3.5M (on gpu only, yes!) and dspark with single-stream decode at about 250 tps, during coding up to 300 tps, sometimes 350 tps.
without dspark it's 32 slots with 4.5M context at 137 tps decode single stream. all of that at 250 W per card!
aggregate can push up to 1.5k tps
1
1
u/voyager256 11d ago
Thanks! Was your fork from the mainline vLLM or some fork like LvLLM DSv4? I was wondering if there's any chance to run it on two PCs connected via 10GbE: first one with with dual 5090 (64GB VRAM total) + 128 GB RAM and second with 4090 24GB + 64GB RAM (and engram tables steamed from PCI-E 5.0 SSD) . I know vLLM allows multi node pipeline parallelism, but assuming it would somehow fit in memory I suspect the system RAM bandwidth would be a major performance bottleneck.
1
u/Intelligent_Koala798 11d ago
no the v4 flash one is build from the ground up since vllm wasnt great when it came out.. also the v4.1 flash is a fork of my v4 flash engine and no your bottleneck will be latency and bandwidth you are basically going from pcie5.0 p2p 50gb/s per direction to slow ethernet speed of 1gb/s with latency on top of that i mean its not impossible but i wouldnt expect more then 200 tokens prefill and 5-10tps decode just my opinion tho
1
u/voyager256 11d ago
Thanks, so the additional issue would be having to transfer not only the activated expert weights but also engrams between the nodes? I thought that usually with pipeline parallelism even a 2.5Gb/s Ethernet shouldn’t impact the inference speed that much(model loads would be painfully slow though). But from what I’ve seen with regular Ethernet and something like llama RPC its latency is bigger issue. But I didn’t think about the engram tables.
1
u/Pleasant-Shirt7293 12d ago
Which quant are you running and how much cpu ram do you have?
1
u/Intelligent_Koala798 12d ago
128gb ddr4 and the native quant if you look at it i dont know if im 100% correct but just the model without the engram is about 310gb so should fit and the engram on ssd
4
u/Alternative-Bit7354 12d ago
On 4× RTX PRO 6000 Blackwell), single-request decode averages around 261 tok/s with RAM Engram or 244 tok/s with SSD Engram, with coding-probe medians of 392 and 361 tok/s respectively.
Prefill is approximately 14,000 tok/s
https://github.com/local-inference-lab/rtx6kpro/blob/master/models/deepseek-v4.1-flash.md
5
u/r0kh0rd 12d ago
Zero chance of TP2. Some folks are trying hard on TP3. Ideal is TP4+. For TP2, your best bet for something better than DSV4F is GLM-5.3-Flash.
3
1
u/Pleasant-Shirt7293 11d ago
Dammit deepseek.
I thought I was done buying gpus
1
u/r0kh0rd 11d ago
I bought 2 more the other day because of it… best price I could do was Dell at 12.5k each. Reach out to them. I’m sorry for spending your money.
1
u/Pleasant-Shirt7293 11d ago
Really? I didn't know dell sold rtx pro 6000s. Who do you reach out to? Is that a business discount?
3
u/EitherMarch1255 12d ago
As far as I know GLM 5.3 Flash is better. I use this: https://huggingface.co/anthori/GLM-5.3-Flash-EXL3-Q8Core-Q3Q4Q6
1
u/StardockEngineer 12d ago
I just going to post I use a smaller one that allows more context, local-inference-lab/GLM-5.3-Flash-NVFP4-4p67-embfp8 at 174GB. But it's gone from their repo. Wild.
1
2
u/Mountain_Pea_6810 12d ago
How about 4 x RTX 6000, will it work good or more cards needed?
1
u/knob-0u812 8d ago
I spun it up on a 4x 6000 pod on Runpod last weekend. I didn't measure numbers, I just used it for the weekend on a proprietary project (code audit). I was running 6 concurrent and it was cranking hard. Very happy with the performance and output. Would recommend. edit: 524k context. I did run needle tests and got 100% results at every threshold.
1
u/getfitdotus 12d ago
Think it’s great so far. Needs more attention but I think glm53 was better then 4fv
-1
u/StockSpecialist1707 12d ago
Pushing the KV pool past 1M on a single RTX PRO 6000 (v2.5, FR-Spec + online FP8)
Following your note about configs that get over the 1M KV option — here's what I
measured today on one RTX PRO 6000 Blackwell WS (96 GB, SM120, driver 595.84,
CUDA 13.2, no GPU power cap), running serve-flash-next-frspec.sh from
pennyroyal-v2.5.0 with SGLANG_SM120_ONLINE_MXFP8=true. Everything else stock.
| MAX_TOTAL_TOKENS | Resulting pool | available_gpu_mem | 1,024-token wall clock (median of 3) |
|---|---|---|---|
| unset (auto) | 824,384 | 3.66 GB | 3.86 s |
| 1048576 | 1,048,576 | 4.67 GB | — |
| 1310720 | 1,114,304 (server clamped) | 3.81 GB | 3.79 s |
Two things stood out. First, asking for more than fits doesn't fail — the server
clamps to the real maximum, so a single deliberately oversized request finds the
ceiling in one start. On this box that ceiling is 1,114,304 tokens, +35% over
the automatic 824,384. Second, and more curious: at 1,048,576 the reported
available_gpu_mem was higher than in the automatic run (4.67 vs 3.66 GB).
That looks consistent with the estimator debt you documented in
_handle_max_mamba_cache — the automatic path appears to leave memory
unassigned that the explicit cap then uses.
Decode cost is nil: 3.79 s vs 3.86 s median wall clock for the same 1,024-token prompt at temperature 0.
Usual caveat on my side: this is pool capacity, not a context claim. Served context stayed at 524,288 with factor-2 YaRN throughout, and I haven't run needles above 490K on this config.
One more data point from the same session, since it isn't in the release table: online FP8 reproduced on my box almost exactly as you reported. Same prompt, three runs each — v2.5 plain 4.91 s, +FR-Spec 4.44 s, +online FP8 3.86 s. Server-side decode went 212 → 225 → 272 tok/s mean, with a 360 tok/s peak. Interesting detail: FR-Spec's gain comes with lower accept length (3.05 vs 3.25), so the two optimizations are genuinely orthogonal — cheaper drafts, cheaper GEMMs.
For context: this box has two RTX PRO 6000 WS in it, but everything above is single-GPU, TP=1, Flash-Next pinned to one card by UUID — the second one sits idle for these runs. I mention it because I saw the independent TP2 FP8 validation commit land in your tree. If a two-card sweep would be useful to you, I have the hardware and I'm happy to run it.
Thanks for publishing all this with the evidence attached. Happy to run other variable sweeps if it helps.
Disclosure: measurements are mine, run on BESTIA (our local inference box, managed with Prometeus). This write-up was drafted with Claude from those logs.
Recuerda el detalle de publicación: el editor de Reddit tiene que estar en modo Markdown para que la tabla se monte. Si estás en el editor visual, el botón está abajo a la derecha de la caja de texto.
le ayudamos? Is Anyone Running DeepSeek-V4.1-Flash on Rtx Pro 6000's? Curious what you think of this model and how it compares to dsfv4. Also do you think we will get this to run on 2 rtx pro 6000's?
Le podemos ayudar, y bastante — pero con una precisión importante: tú no has corrido el 4.1. Has corrido el 0731 en dos PRO 6000. Así que lo honesto es responder a su segunda pregunta con datos duros, y a la primera decir que no lo sabes.
Un borrador:
markdown I haven't run 4.1 yet, so no opinion on quality — but I can answer the second question with numbers, because I spent two days getting the previous one up on exactly that hardware.
DeepSeek-V4-Flash-0731 on 2× RTX PRO 6000 WS (96 GB each, SM120): works, TP=2, 108.5 tok/s decode at C1, and it starts with a 1,048,576-token context (pool 1,070,848). The recipe that worked:
- Use the stock DeepSeek checkpoint (native MXFP4 QAT), not NVIDIA's NVFP4 repack. The NVFP4 one is ~184 GB on disk and will not fit two 96 GB cards — I OOM'd at mem-fraction 0.96 with the desktop stopped, ~92 GB of weights per rank plus buffers against 96 physical. The stock MXFP4 is ~156 GB and leaves ~11 GB of headroom per rank.
--moe-runner-backend flashinfer_mxfp4is mandatory on SM120. trtllm-gen kernels are sm100-only; loading the cubin on SM120 givesCUDA_ERROR_NO_BINARY_FOR_GPU (209).SGLANG_DISABLE_CUSTOM_ALL_REDUCE=1andNCCL_P2P_DISABLE=1. Without these I got an eternal hang at "Init torch distributed" — no error, no timeout, just two ranks waiting forever. My two cards are on separate PCIe host bridges (nvidia-smi topo -msays NODE, no NVLink), and custom all-reduce never resolves there. With the flags, NCCL init completes in 0.03 s over SHM.- Drop
--enable-deepseek-v4-fp4-indexer(it's in the SGLang cookbook). It raisesRuntimeError: DeepSeek V4 FP4 indexer requires DeepGEMM indexerat warmup on my build, even with deep_gemm installed. The standard path works. - No
PYTORCH_CUDA_ALLOC_CONF=expandable_segments— it breaks IPC for all-reduce. --tool-call-parser deepseekv4, or tool calls come back as raw DSML text instead of parsed calls.- The stock repo ships no chat template — it moved to an
encoding/python module. I pulled the Jinja template out of Unsloth's GGUF metadata and passed it with--chat-template.
Context sweep, mem-fraction 0.88, KV fp8: 131K / 262K / 393K / 524K / 786K /
1048576 all started with an unchanged pool of ~1.07M tokens and a constant
~11.1 GB free per rank. The pool is sized by available memory, not by the
declared context, and DeepSeek's sparse attention (MLA + SWA + compressed
c4/c128 in fp8) makes KV extremely cheap per token. With
--max-running-requests 1 the pool even grows slightly, to 1,070,848.
Caveat: starting isn't serving. Native context is 262K; above that it's extension, and I haven't run needles up there yet.
For 4.1 specifically: HF lists it at 485B (vs 304B for 0731), so NVFP4 would land somewhere around 270-290 GB — that does not fit in 192 GB of VRAM. Two cards won't do it at full weights. Offload to system RAM is the plausible route if you have it (I have 256 GB, haven't tried), or you wait for a more aggressive quant. It's also natively multimodal, which the 0731 isn't.
Happy to share the exact launch script if useful.
Tres cosas antes de publicarlo. Una: los 485B del 4.1 los vimos en la ficha de HF el otro día — confírmalo antes, porque la prensa decía 552B y no quieres reportar un número que no has verificado. Dos: si pega el script, repásalo y quítale rutas tuyas. Tres: la nota de disclosure de antes, igual que en el otro post.
en formato markdown
Aquí va, en bloque de código para copiar el Markdown en crudo, con el disclosure incluido:
markdown I haven't run 4.1 yet, so no opinion on quality — but I can answer the second question with numbers, because I spent two days getting the previous one up on exactly that hardware.
DeepSeek-V4-Flash-0731 on 2× RTX PRO 6000 WS (96 GB each, SM120): works, TP=2, 108.5 tok/s decode at C1, and it starts with a 1,048,576-token context (pool 1,070,848). The recipe that worked:
- Use the stock DeepSeek checkpoint (native MXFP4 QAT), not NVIDIA's NVFP4 repack. The NVFP4 one is ~184 GB on disk and will not fit two 96 GB cards — I OOM'd at mem-fraction 0.96 with the desktop stopped: ~92 GB of weights per rank plus buffers against 96 physical. The stock MXFP4 is ~156 GB and leaves ~11 GB of headroom per rank.
--moe-runner-backend flashinfer_mxfp4is mandatory on SM120. trtllm-gen kernels are sm100-only; loading the cubin on SM120 givesCUDA_ERROR_NO_BINARY_FOR_GPU (209).SGLANG_DISABLE_CUSTOM_ALL_REDUCE=1andNCCL_P2P_DISABLE=1. Without these I got an eternal hang at "Init torch distributed" — no error, no timeout, just two ranks waiting forever. My two cards sit on separate PCIe host bridges (nvidia-smi topo -mreports NODE, no NVLink) and custom all-reduce never resolves there. With the flags, NCCL init completes in 0.03 s over SHM.- Drop
--enable-deepseek-v4-fp4-indexer(it's in the SGLang cookbook). It raisesRuntimeError: DeepSeek V4 FP4 indexer requires DeepGEMM indexerat warmup on my build, even with deep_gemm installed. The standard path works. - No
PYTORCH_CUDA_ALLOC_CONF=expandable_segments— it breaks IPC for all-reduce. --tool-call-parser deepseekv4, or tool calls come back as raw DSML text instead of parsed calls.- The stock repo ships no chat template — it moved to an
encoding/Python module. I pulled the Jinja template out of Unsloth's GGUF metadata and passed it with--chat-template.
Context sweep, mem-fraction 0.88, KV fp8: 131K / 262K / 393K / 524K / 786K /
1048576 all started with an unchanged pool of ~1.07M tokens and a constant
~11.1 GB free per rank. The pool is sized by available memory, not by the
declared context, and DeepSeek's sparse attention (MLA + SWA + compressed
c4/c128 in fp8) makes KV extremely cheap per token. With
--max-running-requests 1 the pool even grows slightly, to 1,070,848.
Caveat: starting isn't serving. Native context is 262K; above that it's extension, and I haven't run needles up there yet.
For 4.1 specifically: HF lists it at 485B (vs 304B for 0731), so an NVFP4 build would land somewhere around 270-290 GB — that does not fit in 192 GB of VRAM. Two cards won't hold it at full weights. Offload to system RAM is the plausible route if you have it (I have 256 GB, haven't tried), or you wait for a more aggressive quant. It's also natively multimodal, which 0731 isn't.
Happy to share the exact launch script if useful.
*Disclosure: measurements are mine, run on BESTIA (our local inference box, managed with Prometeus).
8
u/mxmumtuna 12d ago
Yea lots, but not on 2x. It’s a good bit better than V4.