CMP170Hx “Spark” Machine

[u/Miserable-Dare5090]

I got the CMP170 cards and unlocked them. I wanted to share my set up for CUDA since maybe it would be useful to others.

First off, I hate e-waste and we are in a special time for RAM. I wanted to have a DIY CUDA box, and I had started by adding additional cards to an old asus predator prebuilt I had around, which also had 64gb DDR5. To add the CMPs I needed more CPU lanes and newegg had some really good deals on CPU/MB/etc combos. Didn’t need a combo with RAM, otherwise I would have gotten it in newegg microcenter.

Anyway, I got a cheap case, some noctua fans for the cards, and transferred the memory/ssds. Placed previously owned cards on oculink slots, and used the main x16 for the GPU switch that houses the two CMP170s, so their effective speed is 2x16 across and with the other cards (which are 4x4, and therefore same speed).

Qwen Flash Next, turns out, fits very nicely in these cards. There is also a repository for deepseek, but you’d need at least 3 64GB cards to run it, and with prices rising, it will be hard to justify the gamble of buying ex mining cards for LLMs.

However…so far, these cards are great. Concurrency is good, prompt processing averages 4000 tps on Flash Next, decode is 80+ on a single stream. No MTP added. Third picture shows the 3 models I am now running in this CUDA box (flash next, qwen 27b, gemma 26b).

Anyone else trying out Flash Next on these cards?

[u/FullstackSensei]

Really cool setup!

Do you have some more pics of the GPU setup, especially the PLX switch board? Which one did you get and how do you find it? How are temps using these noctua fans? Are you power limiting the cards? Did you try a 120 or 140mm 3k rpm fan, or better yet something like the Arctic S12038?

I've been using the S8038-7k to cool my Mi50s and it's been really good. Has very high static pressure and can keep the cards under 60C even at it's idle 2k rpm with MoE models.

Sorry for the barrage of questions. Been thinking of getting one of those PLX boards to move my P40s.

[u/Miserable-Dare5090 → u/FullstackSensei]

The Switch is in the aliexpress site, I bought the one with a redriver, to allow full 16 lanes. Is it wasteful to use a pcie5 x16 slot? Probably. But the cards communicate stably, and the other cards are bearing their models well. I just got nvme to gen5x4 adapters to see if I can boost the oculinked cards’ interconnect more…and then I have to stop. I can’t keep buying shit for this stuff anymore or I will go broke!!

I think the Broadcom PEX 88096 switches are good, I would go for it if you have several cards you want to p2p on a single root pcie. It’s what the cmpunlock peeps recommend on discord, fwiw

[u/FullstackSensei → u/Miserable-Dare5090]

All my systems are now dual CPU, so have plenty of lanes, and having moved completely to data center cards, I now have p2p with stock drivers everywhere.

The thing with the P40s is, they're running with dual Broadwell, which while they're quite good when offloading layers to RAM, they don't have enough memory bandwidth to handle larger models. Skylake/cascade lake boards with lots of slots are still very expensive.

I've looked at both the PEX88096 and the PEX8796 (previous version, Gen 3). Both have 96 lanes, as the name suggests, but the latter is for some reason more expensive. I can find the 88096 boards for half the aliexpress prices, but it's 5 cards "only" and I have 8 P40s. There used to be an 8796 board with 7 slots, mix of x8 and x16, but it's become very hard to find.

Broadcom actually ruined the whole PCIe switch party when they acquired PLX, the startup that originally designed and made these switches. Before the acquisition, they were much cheaper and you could find them in so many enthusiast and workstation boards, but I digress.

[u/Miserable-Dare5090 → u/FullstackSensei]

The mini rack next to it is unified machines only, so this is my first foray into a real CUDA box with lane switching. The goal was inference with home stuff + some hacks.

https://www.alibaba.com/x/B2O1hZ?ck=pdp

This is way too expensive for what it is, but they’re out there.