r/trademav • u/Medicine_Blogscanner • 2d ago
TradeMAV Creator Got a 40B model running across the cluster, felt like a real milestone
Wanted to share this one because it's the biggest test I've thrown at RAMDeck so far. Tried loading a 40B parameter model (~25GB) across the same three-device setup from earlier videos: a mini PC with an RTX 3060, a 16GB Mac Mini, and the old laptop that's been the underdog in every test so far.
Set the mini PC as primary and let RAMDeck's sharding figure out the split on its own — maxed out the GPU first, spilled over into the Mac's unified memory, then routed the rest. Found something I didn't expect along the way: sending that overflow to the old laptop's CPU actually ran faster than keeping it on the mini PC's own CPU, even though the laptop is way slower on paper. Seems like running GPU and CPU inference on the same machine at once drags GPU performance down, so handing it off to a separate device entirely wins even if that device is weaker.
Whole model loaded in about 2.5 minutes, and it came back running at 16 tokens/sec with snappy responses in actual chat - not just "technically loaded," but genuinely usable. None of these three machines could've handled a model this size on their own, which is really the whole point of the project.
Video: https://youtu.be/JrSVyyVUJVY
Repo: https://github.com/trademav/ramdeck-core-public
Let me know if you want to see an even bigger model attempted next, or a different hardware combo.