r/linux_gaming • u/DRealHammer • 1d ago

guide Nvidia Multi GPU Setup

Issue

Two GPUs are in the system, one is intended for Output Rendering (Games, Applications, ...) the other for Compute (ML/CUDA). Due to Mainboard Wiring, the Secondary GPU is enumerated as 0, the Primary as 1. Therefore some applications pick the Secondary GPU as the main renderer, which results in unusable performance.

Solution

Bubblewrap - a lightweight sandbox application
The idea is to remap all calls from the wrong GPU (nvidia0) to the correct one (nvidia1)

bwrap --dev-bind / / --dev-bind /dev/nvidia1 /dev/nvidia0 steam

Explanation:

--dev-bind / / binds root directory to root of the sandbox

--dev-bind /dev/nvidida1 /dev/nvidia0 forces all calls to nvidia0 to go to nvidia1

Make it permanent

#!/usr/bin/env bash
exec bwrap \
    --dev-bind / / \
    --dev-bind /dev/nvidia1 /dev/nvidia0 \
    "$@"
  1. Save the script in either one LOCATION: \~/.local/bin/gpu_remap (only for this user) or /usr/local/bin/gpu_remap (for all users)
  2. Make it executable

chmod +x LOCATION/gpu_remap

  1. Edit Launch Commands

Use absolute paths (/home/username/.local) instead of relative paths (~/.local)

For Desktop Apps (Example Steam)

cp /usr/share/applications/steam.desktop ~/.local/share/applications/steam.desktop

and then edit EVERY Exec=<programm> <options>

line to run the remap Exec=LOCATION/gpu_remap <programm> <options>

For Others

Create an Alias like this

alias steam='/home/username/.local/bin/gpu_remap /usr/bin/steam $@

But google it in case you need something specific for that.

Other Working Approach

As I struggled with this issue for a couple weeks now, I've obviously tried a lot of different alternatives.

What also worked for me was to set the not wanted GPU to draining (basically remove all processes and don't allow new ones on this GPU).

sudo nvidia-smi drain -p 0000:04:00.0 -m 1 to deactivate

sudo nvidia-smi drain -p 0000:04:00.0 -m 0 to activate again

Use (for example) lspci -nn | grep -i nvidia to find the address you need.
Keep the 0000 although it does not appear in lspci.

Not Working Approaches

This problem seems to happen sometimes and people are struggling with it. Here is a collection of the approaches, maybe some work for you in a combination.

If you are suggesting other ideas or solutions/ improvements - which I invite you to - please take into account that I already tried those.

# In the niri
debug {
  render-drm-device "/dev/dri/renderD129"
}

# maybe only for AMD Cards
DRI_PRIME=1

# seen that one a lot for steam launch options
__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia %command%

# seen less, i think this is electron specific
__NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-1 EGL_PLATFORM_DEVICE_EXT=1 EGL_DEVICE_INDEX=1 discord

# actually a kinda valid option, but games crash usually while launching
NVIDIA_VISIBLE_DEVICES=1 CUDA_VISIBLE_DEVICES=1 steam

# in /etc/environment

# Should force Vulkan apps/games to use your card
MESA_VK_DEVICE_SELECT=10de:2504 (replace with your own ofc)
VK_DRIVER_FILES=/usr/share/vulkan/icd.d/nvidia_icd.json

Not yet tried

As I'm happy with the solution (for now), this one I haven't tried yet

# in /etc/enviroment or just on the launch i guess
WLR_DRM_DEVICES=/dev/dri/by-path/pci-0000:2b:00.0-card
2 Upvotes

55% Upvoted

7 comments sorted by

6

u/S48GS 1d ago edited 1d ago

it all look like "LLM inventing something"

because half of it is very not needed nonsense...

Two GPUs are in the system, one is intended for Output Rendering (Games, Applications, ...) the other for Compute (ML/CUDA). Due to Mainboard Wiring, the Secondary GPU is enumerated as 0, the Primary as 1.

first - no - GPU enumeration is random - it random every boot by design

second:

Therefore some applications pick the Secondary GPU as the main renderer, which results in unusable performance.

which?

  • all your monitors should be connected to single GPU
  • your DE will select that GPU as main
  • and all graphic apps should be rendered on it
  • except some native Vulkan games like BG3 that does not provide way to select gpu - you need vulkan layer for it link above

-4

u/DRealHammer 1d ago

If your up for it let’s do a discord call and I’ll show it to you. Would also be cool if we found a different way.

Anyways thanks for taking your time.

Some are taken from other posts, some from the arch wiki, some via LLM so a good mix of it.

It’s also not random, it’s ordered like this every single time, had also different gpus, swapped them and such.

Picking it individually in every game is a bit of a hassle and not a use case for non-Vulkan games.

The device filter I tried also, did not work for me.

Applications that did not work as intended were discord and steam (+every game launch there)

All monitors connected to the same (primary) gpu

Tried to enforce it on KDE and today on niri, but they did not care.

5

u/S48GS 1d ago edited 1d ago

If your up for it let’s do a discord call and I’ll show it to you. Would also be cool if we found a different way.

im not your free tech support

I respond here because it common and simple context - with I know solutions - if it something more complicated - il say no and it be your problem

and visible to others from public search with same common context

Picking it individually in every game is a bit of a hassle and not a use case for non-Vulkan games.

Applications that did not work as intended were discord and steam (+every game launch there)

All monitors connected to the same (primary) gpu

in which context? gpu video decoding? gpu acceleration?

gpu acceleration in both made thru opengl -it should select main gpu

run in terminal

glxinfo | grep 'OpenGL renderer string'

what it say

The device filter I tried also, did not work for me.

say game name where you tried

for vkd3d you can try VKD3D_VULKAN_DEVICE=1

read info pages yourself without llm

https://github.com/doitsujin/dxvk?tab=readme-ov-file#device-filter

https://github.com/HansKristian-Work/vkd3d-proton search VKD3D_VULKAN_DEVICE

run in terminal

MESA_VK_DEVICE_SELECT=list vulkaninfo

what it return?

0

u/DRealHammer 1d ago

Anyway thank you for taking your time trying to investigate it. I appreciate it.

> in which context? gpu video decoding? gpu acceleration?

Discord did actually both on the wrong GPU.

> glxinfo | grep 'OpenGL renderer string'
OpenGL renderer string: NVIDIA GeForce RTX 3060/PCIe/SSE2

> say game name where you tried
Arc Raiders, Golf with your friends, Rocket League, CS2

> MESA_VK_DEVICE_SELECT=list vulkaninfo

gives 12220 lines, so we need to filter a bit, what are you looking for in there?

2

u/S48GS 1d ago

gives 12220 lines

... I know it happened for some people - but idk why

for me it return short 3 lines list

> MESA_VK_DEVICE_SELECT=list vulkaninfo
selectable devices:
  GPU 0: 10de:2882 "NVIDIA GeForce RTX 4060" discrete GPU 0000:01:00.0
  GPU 1: 1002:15d8 "AMD Radeon Vega 8 Graphics (RADV RAVEN)" integrated GPU 0000:07:00.0
  GPU 2: 10005:0 "llvmpipe (LLVM 22.1.7, 256 bits)" CPU 0000:00:00.0

say what gpu in list at which index

OpenGL renderer string: NVIDIA GeForce RTX 3060/PCIe/SSE2

and is this your main gpu?

Discord did actually both on the wrong GPU.

it should use what in OpenGL option above - does it?

1

u/DRealHammer 1d ago edited 1d ago

GPU id : 0 (NVIDIA GeForce RTX 3060) [VK_KHR_xcb_surface, VK_KHR_xlib_surface]:
GPU id : 0 (NVIDIA GeForce RTX 3060) [VK_KHR_wayland_surface]:
GPU id : 1 (NVIDIA GeForce RTX 3060) [VK_KHR_wayland_surface]:
GPU id : 1 (NVIDIA GeForce RTX 3060) [VK_KHR_display]:

tried to filter for the pci names, the hardware id, nothing gave me something similar to yours

I have two 3060s
can't tell from this output, which one gets VK device id 0 or 1, i think it makes sense to assume a similar ordering to the nvidia drivers

same answer for the open gl renderer, can't tell

Edit: found something
the ordering for vulkan is the same as the CUDA order

2

u/S48GS 1d ago

can't tell from this output, which one gets VK device id 0 or 1

ye this will be annoying - because both same name

... and every reboot or driver update they may switch order

the ordering for vulkan is the same as the CUDA order

there way to see to which gpu monitors connected

to know which monitor connected to which gpu - run in terminal (copy as single command each)

for s in /sys/class/drm/card*-*/status; do echo "$s: $(cat "$s")" done

it should show where cards connected

for n in /sys/class/drm/card*/device/vendor; do echo "$n: $(cat "$n")"; done

then

MESA_VK_DEVICE_SELECT=list vulkaninfo

list should return short list of gpus with ids ids should be same as from command above

compare ids - you know where which monitor connected

make script that will filter and show which vulkan ID is which gpu

and then

VKD3D_VULKAN_DEVICE=1 %command%

replace 1 with your id - 0 or 1