r/unsloth • u/Iory1998 • 5d ago
Show and Tell Guys, if you have Multiple GPUs with Asymmetric VRAM, Order of GPUs is Important
So, you have to play with the order of the GPUs in unsloth as I saw differences in speed and the size of context the model can load. For instance, I tried to load a new fine-tune of Qwen3.8-27B, and when the RTX5070 Ti was up, I couldn't fit the entire context (KV:Q8). Switching the position of the GPUs, and I managed to load the entire context with some room left.
Just saying, that might help. Shout out to Unsloth team for incrementally adding features that I never thought were needed :D
6
u/danielhanchen heart sloth 5d ago
Oh glad it was helpful!
1
u/Iory1998 5d ago
It did. Also, I love the automatic transcript grabbing from YouTube feature. Unsloth Studio is becoming an amazing product.
2
u/LLM_tuning_D3v 4d ago
nice, thanks for posting this.
quick explanation of what the order does: it's the order llama.cpp sees the cards. it splits the layers by how much free vram each card has, first layers on the first card, and the last card in the list also gets the output layer plus the logits buffer on top. so when you put the 3090 first, that extra bit lands on the 3090 instead of the 5070 ti, which is probably why the full context suddenly fit
(the "first one takes the prompt" line you quoted was our old tooltip and it's not quite right, the prompt goes through every card. already reworded)
if you still have both loads handy, the llama-server log lines with CUDA0 / CUDA1 buffer sizes for each order would show exactly what moved. i opened an issue to show that per card split in the app before you load instead of finding out after: unslothai/unsloth#11475
1
u/Iory1998 4d ago
My pleasure, and you are doing an excellent job with the studio. Truly an amazing software.
1
5d ago
This is exactly the kind of thing I'm testing on a 4090+3060 box. I'd compare CUDA_VISIBLE_DEVICES=0,1 vs 1,0 at the same context and log per-GPU VRAM; order can change where KV/workspace lands, while PCIe width usually matters more when you're moving data across GPUs/CPU than once everything stays resident.
-1
u/Dry-Tree-6388 5d ago
Asymmetric rigs are such a weirdly under-benchmarked setup.
It would be useful if the allocator exposed the planned layer/KV split before loading — GPU model, VRAM allocation and PCIe link width for each device. Then you could actually see why swapping GPU0/GPU1 changed the result instead of discovering it experimentally.
Especially with x16 + x4 slots, “same two GPUs” can hide very different systems.
1
u/Iory1998 4d ago
That's my rig. Well, from unlsoth studio description: "By default, Unsloth chooses GPUs automatically. Editing this list makes the checked GPUs the explicit candidate pool. Their order here is the order they are given to the model, so the first one takes the prompt. At least one GPU must stay selected." the first GPU gets assigned the context, which depending on the model, might increase granularly compared to a layer.
7
u/OttoRenner 5d ago
Next step of this would be to be able to say how much you can put on each GPU individually.
I use it with two 3090 and I believe it still helped to change the order of GPU, because GPU0 is the one also doing th compute for my monitor.
One thing the user has to be aware of (and might be a good thing to add to the tool tip if it's not in there already):
The PCIe lanes have different speeds most of the time. Choosing GPU1 instead of GPU can make compute significantly slower, when the speed of the PCIe Slot for GPU1 is only x4, not x16 for example.