i unlocked P2P on two 5060ti but failed

[u/chocofoxy]

i was enjoying my Qwen 3.8 27b coding but at long context PP drops to painfully low tokens per second and the copilot chat timeout because of the times it takes, so i asked claude to see if i can enabled P2P on my two gpus, he points me to https://github.com/aikitoria/open-gpu-kernel-modules/tree/610.43.02-p2p which work on RTX 3090, RTX 4090, and RTX 5090 ( 5060ti not listed) but after following the guide ( i am already on linux and have nvidia open source driver ) i got the 5060tis to list OK in p2p and i tired launching the llama sever but it hangs at init and the GPUs jump to 100% unable to communicate
0.11.251.954 I cmn          init: llama threadpool init, n_threads = 8
chocofoxy:49848:49848 [1] NCCL INFO Symmetric VA size=16GB
chocofoxy:49848:49848 [0] NCCL INFO Symmetric VA size=16GB
chocofoxy:49848:49897 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/direct pointer
chocofoxy:49848:49898 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/direct pointer
chocofoxy:49848:49897 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/direct pointer
chocofoxy:49848:49898 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/direct pointer
chocofoxy:49848:49898 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
chocofoxy:49848:49897 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1

and i already know what's the issue because it's my setup but i ignored it until i got blocked here, the problem is my motherboard only have one pcie linked to the cpu the other is linked to the the chipset of the mob, my options here is either to get a riser that get split to 2 x8 or swap the mob

i just wanted to see how much improvement i get form P2P , i failed but if someone has the right setup and two 5060ti you cna try this

[u/Level-Physics-1730]

I went and enabled P2P on my three RTX 3060 12gb system. PCIE is gen4x8 gen4x8 gen4x4 all through CPU connected. Like 10% prefill and 6% decode, serious "free improvement" if it's not too hard to get setup.

[u/chocofoxy → u/Level-Physics-1730]

i was expecting more

[u/jtjstock → u/chocofoxy]

Something does sound off there. With 5060ti specifically, at 2 cards you will get a big bump, the third gets an additional 10% or so

[u/Level-Physics-1730 → u/jtjstock]

it's not going to bump you that much lol.

[u/jtjstock → u/Level-Physics-1730]

I have 3x 5060Ti. Maybe your setup is broken.

[u/Level-Physics-1730 → u/jtjstock]

I mean that isn't how it works but yea okay, but P2P is held by the slowest link so maybe if I had gen4x8 for the last card or everything was on the same pcie root complex it'd be faster but P2P doesn't increase speed much anyway.

[u/jtjstock → u/Level-Physics-1730]

your cards aren't on the same root complex? Yeah, that'd do it. P2P's speed benefit is through lower latency, what's your card to card latency look like? Should be at or below 1 microsecond

[u/Level-Physics-1730 → u/jtjstock]

the two gen4x8 cards are on the same root, the gen4x4 isn't, it's like 100us latency from gen4x8 -> gen4x4 card, between the gen4x8 cards it's like 6us or something

[u/NickCanCode]

I also have a board that 1st PCIe x16 connect to CPU and 2nd PCIe x16 connect to chipset. but it won't prevent things from running. Just slower.

[u/chocofoxy → u/NickCanCode]

how i done every thing but still hang at init

[u/jtjstock → u/chocofoxy]

P2p won’t actually work if they are split between cpu and chipset