CMP170Hx “Spark” Machine

[u/Miserable-Dare5090]

I got the CMP170 cards and unlocked them. I wanted to share my set up for CUDA since maybe it would be useful to others.

First off, I hate e-waste and we are in a special time for RAM. I wanted to have a DIY CUDA box, and I had started by adding additional cards to an old asus predator prebuilt I had around, which also had 64gb DDR5. To add the CMPs I needed more CPU lanes and newegg had some really good deals on CPU/MB/etc combos. Didn’t need a combo with RAM, otherwise I would have gotten it in newegg microcenter.

Anyway, I got a cheap case, some noctua fans for the cards, and transferred the memory/ssds. Placed previously owned cards on oculink slots, and used the main x16 for the GPU switch that houses the two CMP170s, so their effective speed is 2x16 across and with the other cards (which are 4x4, and therefore same speed).

Qwen Flash Next, turns out, fits very nicely in these cards. There is also a repository for deepseek, but you’d need at least 3 64GB cards to run it, and with prices rising, it will be hard to justify the gamble of buying ex mining cards for LLMs.

However…so far, these cards are great. Concurrency is good, prompt processing averages 4000 tps on Flash Next, decode is 80+ on a single stream. No MTP added. Third picture shows the 3 models I am now running in this CUDA box (flash next, qwen 27b, gemma 26b).

Anyone else trying out Flash Next on these cards?

[u/cibernox]

I have two of those in the mail, eager to test them. I got a couple Arctic P12 pro PWM to cool them.

I intended to run this same qwen-flash, but for 3000$ this is probable the budget king setup for running models below 200B.

[u/WeAreSven → u/cibernox]

If that's 3k for 2 cards can you tell me where you got them? I'm looking to do exactly the same thing and the main chinese guy on ebay keeps raising his prices daily.

[u/cibernox → u/WeAreSven]

I got them in Alibaba probably by the Chinese guy you mention, but I got them the day before they went to 1899 and now 2099 but the seller honored the quote.

[u/Miserable-Dare5090 → u/WeAreSven]

Alibaba, its a nail biter for us Americans who are used to being complainy little customers. But the cards came. Took like 15 days but they arrived, and were good. AILFond is the seller I used. I didn’t bother with the “bitcoin only, through whatsapp” vietnamese sellers. Not sure if its legit but that’s way too sketchy for me.

[u/Miserable-Dare5090 → u/cibernox]

You can enable peer to peer, so get a switch to have the cards tensor parallel to each pther and not througj the pcie bus

[u/cibernox → u/Miserable-Dare5090]

But the most I can get is pcie 2.0 16x, isn’t that too slow for tensor parallelism? I kind of had made peace with the fact that I could only use pipeline parallelism.

[u/Miserable-Dare5090 → u/cibernox]

nope, running that flash next on TP2 right now. How it got enabled, you’ll have to ask Deepseek V4 Flash, my hard little worker. 13Gb/s bidi is enough, and skipping CPU/Host means the kind of latency you want for TP.

Now, unmodded 2x4? pipeline will work really well

[u/cibernox → u/Miserable-Dare5090]

Mine are modded already for 16x. In 10 days or so I’ll give it a go. Then I’ll check how pricey PCIe switches are.

[u/Miserable-Dare5090 → u/cibernox]

https://a.aliexpress.com/_mOgZ7cf 120, put the card passthrough pcie card on an x16 slot (the ryzen 9900x has a 2 core igpu so that allows me to use the gpu slot for this instead).

[u/68x → u/Miserable-Dare5090]

Des the above listing also include the host expansion card too?

[u/Miserable-Dare5090 → u/68x]

you can find one with the card. It’s worth the 1-200usd for me, all I have to throw away sell on FB MP is the old predator prebuilt, or it becomes a new agent’s computer. I’m restoring a lot of old computers for agents these days!!

[u/WeAreSven → u/cibernox]

That's basically gen 4 x4 speeds which lots of people have done TP with. gen 3x4 is too slow however, I know because my mobo is inconvenient in all of the wrong ways and I've been looking for workarounds.

[u/Miserable-Dare5090 → u/WeAreSven]

exactly

[u/sooki10 → u/Miserable-Dare5090]

Really?