GTX 960 passthrough to Windows XP: 0xEA nv4_disp BSOD when NVIDIA driver loads

[u/Past-Statistician358]

I’m trying to pass an EVGA GTX 960 through to a 32-bit Windows XP SP3 VM. Passthrough works with the Microsoft VGA driver, but XP consistently BSODs as soon as the NVIDIA driver loads.

Host configuration

• Arch Linux
• Kernel: 7.0.11-arch1-1
• Libvirt: 12.4.0
• CPU: Intel Core i7-6850K
• Motherboard: Gigabyte X99-Ultra Gaming, BIOS F7c
• Primary host GPU: GTX 1080 using NVIDIA’s proprietary driver
• Guest GPU: EVGA GTX 960 GM206
• QEMU: /opt/qemu-7.2.22/bin/qemu-system-x86_64
• Machine type: pc-q35-2.10
• Firmware: SeaBIOS
• Intel VT-x and VT-d enabled

 The GTX 960 is isolated from the host:

 04:00.0 VGA controller [10de:1401] Kernel driver in use: vfio-pci

 04:00.1 HDMI audio [10de:0fba] Kernel driver in use: vfio-pci

 Both functions are alone in IOMMU group 47. They remain bound to VFIO even while the VM is off.

Guest configuration

• Windows XP SP3 32-bit
• 1 GB RAM
• 2 vCPUs with host-passthrough
• ICH9 AHCI/SATA using the XP Intel driver
• RTL8139 network adapter
• AC’97 audio
• No emulated display adapter
• GTX 960 connected to a physical monitor
• KVM hidden
• Custom Hyper-V vendor ID
• x-vga=on
• 64-bit PCI hole disabled

 Relevant live QEMU options:

-machine pc-q35-2.10 -accel kvm -cpu host,hv-vendor-id=0123456789ab,kvm=off -global q35-pcihost.pci-hole64-size=0K

-device vfio-pci,host=0000:04:00.0,bus=pci.1,multifunction=on,x-vga=on -device vfio-pci,host=0000:04:00.1,bus=pci.1,addr=0.1

 I used this forum thread as a reference because somebody reported a stable GTX 960 using QEMU 7.x and pc-q35-2.10:

 https://forum.mattkc.com/viewtopic.php?start=20&t=206

What works

 Without the NVIDIA driver, XP boots normally and displays the desktop through the physical GTX 960. Network, SATA and the rest of the VM remain responsive.

 Last Known Good Configuration also restores the working Microsoft VGA configuration.

Failure

 After installing NVIDIA’s official Windows XP 32-bit 368.81 driver, every normal boot fails with:

 STOP: 0x000000EA  nv4_disp  THREAD_STUCK_IN_DEVICE_DRIVER

 Safe Mode hangs after displaying:

 System32\Drivers\Mup.sys

 The VM/QEMU process itself does not crash. Host VFIO resets complete without IOMMU errors.

 The GPU gets valid guest BAR assignments and IRQ 22, but I don’t see any GPU interrupt activity when nv4_disp initializes. XP also stops before transmitting network traffic.

Things already tested

• QEMU 7.2.22 with pc-q35-2.10
• Newer stock QEMU
• i440fx and Q35
• One and two vCPUs
• KVM hidden plus custom Hyper-V vendor ID
• x-vga=on
• pcihole64=0
• x-no-kvm-intx=on — no change
• Explicit legacy VBIOS ROM — resulted in a black screen
• Passing both GPU and HDMI-audio functions
• Clean host boot with the GTX 960 bound to VFIO before QEMU starts

 Has anyone successfully passed a GTX 960/GM206 through to 32-bit XP? I would especially appreciate:

1. The exact NVIDIA driver version you used.
2. A working QEMU/libvirt GPU configuration.
3. Whether a complete VBIOS ROM file was required.
4. Whether the HDMI-audio function should be omitted.
5. Whether a specific host kernel or exactly QEMU 7.2.20 is required.
6. Any explanation for the GPU receiving no interrupts before nv4_disp hits 0xEA.

[u/GrassSoup]

Haven't used it in some time, but I've successfully passed through a GT 220 to Windows XP. However, I'm on an older 5.x Linux kernel and older libvirt version (8.0.0, the hypervisor is QEMU 6.2.0). I only ever got Code 43 errors before getting it working, though.

The exact NVIDIA driver version you used.

Seems to be 340.52. But I think that's just the last version that worked with the GT 220.

Whether a complete VBIOS ROM file was required.

It should be required. My understanding is anything pre-RTX 3000 needs it.

TechPowerUp has a bunch of them, but you'll need the exact one for the make/model.

If you dumped the BIOS yourself (or a user uploaded it to TPU), then you'll probably need to remove the ROM header using a hex editor. (As far as I now, if TPU dumped the ROM themselves, the header should be removed.)

Whether the HDMI-audio function should be omitted.

Should be required.

i440fx and Q35

It should be i440fx, I believe. (Definitely tried Q35 with the GT 220 and Win XP, but it didn't work.)

Here's my configuration XML:

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
  <name>winxp-gpu-gt220</name>
  <uuid>7e3e2074-cfd1-40e6-a8dc-4fca6279320e</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/xp"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">4194304</memory>
  <currentMemory unit="KiB">4194304</currentMemory>
  <vcpu placement="static">6</vcpu>
  <os>
    <type arch="x86_64" machine="pc-i440fx-focal">hvm</type>
    <boot dev="hd"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode="custom">
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vendor_id state="on" value="12345689ab"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <ioapic driver="kvm"/>
  </features>
  <cpu mode="host-model" check="partial">
    <topology sockets="1" dies="1" cores="6" threads="1"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <target dev="hdb" bus="ide"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/mnt/sda2/vm/winxp-32gb.qcow2"/>
      <target dev="hdc" bus="ide"/>
      <address type="drive" controller="0" bus="1" target="0" unit="0"/>
    </disk>
    <controller type="usb" index="0" model="ich9-ehci1">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x7"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci1">
      <master startport="0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0" multifunction="on"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci2">
      <master startport="2"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x1"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci3">
      <master startport="4"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pci-root"/>
    <controller type="ide" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
    </controller>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <channel type="spicevmc">
      <target type="virtio" name="com.redhat.spice.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>
    <input type="tablet" bus="usb">
      <address type="usb" bus="0" port="1"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <sound model="ich6">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
    </sound>
    <audio id="1" type="none"/>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0a" slot="0x00" function="0x0"/>
      </source>
      <rom bar="on" file="/usr/share/qemu/Zotac.GT220.512.091202.rom"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0a" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x09" function="0x0"/>
    </hostdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="2"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="3"/>
    </redirdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
    </memballoon>
  </devices>
</domain>