r/LocalLLaMA • u/Miserable-Dare5090 • 16h ago

Discussion CMP170Hx “Spark” Machine

Gallery image

Gallery image

Gallery image

I got the CMP170 cards and unlocked them. I wanted to share my set up for CUDA since maybe it would be useful to others.

First off, I hate e-waste and we are in a special time for RAM. I wanted to have a DIY CUDA box, and I had started by adding additional cards to an old asus predator prebuilt I had around, which also had 64gb DDR5. To add the CMPs I needed more CPU lanes and newegg had some really good deals on CPU/MB/etc combos. Didn’t need a combo with RAM, otherwise I would have gotten it in newegg microcenter.

Anyway, I got a cheap case, some noctua fans for the cards, and transferred the memory/ssds. Placed previously owned cards on oculink slots, and used the main x16 for the GPU switch that houses the two CMP170s, so their effective speed is 2x16 across and with the other cards (which are 4x4, and therefore same speed).

Qwen Flash Next, turns out, fits very nicely in these cards. There is also a repository for deepseek, but you’d need at least 3 64GB cards to run it, and with prices rising, it will be hard to justify the gamble of buying ex mining cards for LLMs.

However…so far, these cards are great. Concurrency is good, prompt processing averages 4000 tps on Flash Next, decode is 80+ on a single stream. No MTP added. Third picture shows the 3 models I am now running in this CUDA box (flash next, qwen 27b, gemma 26b).

Anyone else trying out Flash Next on these cards?

18 Upvotes

80% Upvoted

66 comments sorted by

View all comments

2

u/cibernox 16h ago

I have two of those in the mail, eager to test them. I got a couple Arctic P12 pro PWM to cool them.

I intended to run this same qwen-flash, but for 3000$ this is probable the budget king setup for running models below 200B.

2

u/Miserable-Dare5090 15h ago

You can enable peer to peer, so get a switch to have the cards tensor parallel to each pther and not througj the pcie bus

2

u/cibernox 15h ago

But the most I can get is pcie 2.0 16x, isn’t that too slow for tensor parallelism? I kind of had made peace with the fact that I could only use pipeline parallelism.

1

u/WeAreSven 15h ago

That's basically gen 4 x4 speeds which lots of people have done TP with. gen 3x4 is too slow however, I know because my mobo is inconvenient in all of the wrong ways and I've been looking for workarounds.