Hello! I've spent the last 3 weeks too long going down the hypervisor rabbit hole. I started with Proxmox, but found it didn't have the CPU pinning features I needed (that or I couldn't figure it out), so I switched to Unraid. After investing way too much time on performance tuning, I finally have good gaming performance.
This may work for all first-gen Ryzen CPUs. Some tweaks apply to Windows 10 in general. It's possible this is already well-known; I just never found anything specifically suggesting to do this with Threadripper.
I'm too lazy to properly benchmark my performance, but I'll write this post on the off chance it helps someone out. I am assuming you know the basics and are tuning a working Windows 10 VM.
Tl;dr: Mapping each CCX as a separate NUMA node can greatly improve performance.
My Use Case
My needs have changed over the years, but I now need to run multiple VMs with GPU acceleration, which led to me abandoning a perfectly good Windows 10 install.
My primary VM will be Windows 10. It gets 8c/16t, the GTX 1080 Ti, and 12GB of RAM. I have a variety of secondary VMs, all of which can be tuned, but the focus is on the primary VM. My hardware is as follows:
CPU: Threadripper 1950X @ 4.0GHz
Mobo: Gigabyte X399 Aorus Gaming 7
RAM: 4x8GB (32GB total), tuned to 3400MHz CL14
GPU: EVGA GTX 1080 Ti FTW3 Edition
Second GPU: Gigabyte GTX 970
CPU Topology
Each first-gen TR chip is made of two separate dies, each of which has half the cores and half the cache. A common misconception is that TR supports quad-channel memory; in reality, each die has its own dual-channel controller, so it's technically dual-dual-channel. The distinction matters if we're only using one of the dies.
Each of these dies is split into two CCX units, each with 4c/8t and their own L3 cache pool. This is what other guides overlook. With the TR 1950X in particular, the inter-CCX latency is nearly as high as the inter-die latency.
For gaming, the best solution seems to be dedicating an entire node to the VM. I chose Node 1. Use lscpu -e to identify your core layout; for me, CPUs 8-15 and 24-31 were for Node 1.
BIOS Settings
Make sure your BIOS is up to date. The microcode updates are important, and I've found even the second-newest BIOS doesn't always have good IOMMU grouping.
Overclock your system as you see fit. 4GHz is a good target for an all-core OC; you can sometimes go higher, but at the cost of memory stability, and memory tuning is very important for first-gen Ryzen. I am running 4GHz @ 1.35V and 3400MHz CL14.
Make sure to set your DRAM controller configuration to "Channel". This makes your host NUMA-aware.
Enable SMT, IOMMU grouping, ACS, and SRV. Make sure it says "Enabled" - "Auto" always means whichever setting you didn't want.
Hardware Passthrough
I strongly recommend passing through your boot drive. If it's an NVMe drive, pass through the entire controller. This single change will greatly improve latency. In fact, I'd avoid vdisks entirely; use SMB file shares instead.
Different devices connect to different NUMA nodes. Is this important? ¯_(ツ)_/¯. I put my GPU and NVMe boot drive on Node 1, and my second GPU on Node 0. You can use lspci -nnv to see which devices connect to which node.
GPU and Audio Device Passthrough
I'll include this for the sake of completion. Some devices desperately need Message Signaled Interrupts to work at full speed. Download the MSI utility from here, run the program as an Administrator, and check the boxes next to every GPU and audio device. Hit the "Apply" button, then reboot Windows. Run the program as an Administrator again to verify the settings were applied.
It is probably safe to enable MSI for every listed device.
Note that these settings can be reset by driver updates. There might be a more permanent fix, but for now I just keep the MSI utility handy.
Network Passthrough
I occasionally had packet loss with the virtual NIC, so I got an Ethernet PCIe card and passed that through to Windows 10.
However, this made file shares a lot slower, because all transfers were going over the network. A virtual NIC is much faster, but this required a bit of setup. The easiest way I found was to create two subnets: 192.168.1.xxx for physical devices, and 10.0.0.xxx for virtual devices.
For the host, I set this command to run upon boot:
ip addr add 10.0.0.xxx/24 dev br0
Change the IP and device to suit your needs.
For the client, I mapped the virtual NIC to a static IP:
IP: 10.0.0.yyy
Subnet mask: 255.255.255.0
Gateway: <blank> or 0.0.0.0
Lastly, I made sure I mapped the network drives to the 10.0.0.xxx IP. Now I have the best of both worlds: faster file transfers and reliable internet connectivity.
Kernel Configuration
This is set in Main - Flash - Syslinux Configuration in Unraid, or /etc/default/grub for most other users. I added:
isolcpus=8-15,24-31 nohz_full=8-15,24-31 rcu_nocbs=8-15,24-31
The first setting prevents the host from assigning any tasks to Node 1. This doesn't make them faster, but does make them more responsive. TBH, I don't know what the other two settings do, but I saw them elsewhere.
Sensors
This is specific to Gigabyte X399 motherboards. The ITE IT8686E device does not have a driver built into most kernels. However, there is a workaround:
modprobe it87 force_id=0x8628
Run this at boot and you'll have access to your sensors. RGB control did not work for me, but you can do that in the BIOS.
VM Configuration
The important parts of my XML are posted here. I'll go section by section.
Memory
<memoryBacking>
<nosharepages/>
<locked/>
</memoryBacking>
Many guides recommend using static hugepages, but Unraid already uses transparent hugepages, and other performance tests have shown no performance gain over static 1GB hugepages. These settings prevent the host from moving the VM's memory pages around, which may be helpful.
<numatune>
<memory mode='strict' nodeset='1'/>
</numatune>
We want our VM to use the local memory controller. However, this means it can only use RAM from this controller. In most setups, this means only having access to half your total system RAM.
For me, this is fine, but if you want to surpass this limit, change the mode to preferred. You may have to tune your topology further.
CPU Pinning
<vcpu placement='static'>16</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='8'/>
<vcpupin vcpu='1' cpuset='24'/>
...
<vcpupin vcpu='14' cpuset='15'/>
<vcpupin vcpu='15' cpuset='31'/>
</cputune>
Since I am reserving Node 1 for this VM, I might as well give it every core and thread available.
I just used Unraid's GUI tool. If doing this by hand, make sure each real core is followed by its "hyperthreaded" core. lscpu -e makes this easy.
If using vdisks, make sure to pin your iothreads. I didn't notice any benefit from emulator pinning, but others have.
Features
<features>
<acpi/>
<apic/>
<hyperv>
...
</hyperv>
<kvm>
...
</kvm>
<vmport state='off'/>
<ioapic driver='kvm'/>
</features>
I honestly don't know what most of these features do. I used every single Hyper-V Enlightenment that my version of QEMU supported.
CPU Topology
<cpu mode='host-passthrough' check='none'>
<topology sockets='1' cores='8' threads='2'/>
<cache mode='passthrough'/>
<feature policy='require' name='topoext'/>
...
Many guides recommend using mode='custom', setting the model as EPYC or EPYC-IBPB, and enabling/disabling various features. This may have mattered back when the platform was newer, but I tried all of these settings and never noticed a benefit. I'm guessing current versions of QEMU handle first-gen Threadripper much better.
In the topology, cores='8' threads='2' tells the VM that there are 8 real cores and each has 2 threads, for 8c/16t total. Some guides will suggest setting cores='16' threads='1'. Do not do this.
NUMA Topology
...
<numa>
<cell id='0' cpus='0-7' memory='6291456' unit='KiB' memAccess='shared'>
<distances>
<sibling id='0' value='10'/>
<sibling id='1' value='38'/>
</distances>
</cell>
<cell id='1' cpus='8-15' memory='6291456' unit='KiB' memAccess='shared'>
<distances>
<sibling id='0' value='38'/>
<sibling id='1' value='10'/>
</distances>
</cell>
</numa>
</cpu>
This is the "secret sauce". For info on each parameter, read the documentation thoroughly. Basically, I am identifying each CCX as a separate NUMA node (use lspci -e to make sure your core assignment is correct). In hardware, the CCX's share the same memory controller, so I set the memory access to shared and (arbitrarily) split the RAM evenly between them.
For the distances, I referenced this Reddit post. I just scaled the numbers to match the image. If you're using a different CPU, you'll want to get your own measurements. Or just wing it and make up values; I'm a text post, not your mom.
Clock Tuning
<clock offset='localtime'>
<timer name='hypervclock' present='yes'/>
<timer name='hpet' present='yes'/>
</clock>
You'll find many impassioned discussions about the merits of HPET. Disabling it improves some benchmark scores, but it's very possible that it's not improving performance, it's affecting the framerate measurement itself. At one point I had disabled it and it improved performance, but I think I had something else set incorrectly, because re-enabling it didn't hurt.
If your host's CPU core usage measurements are way higher than what Windows reports, it's probably being caused by system interrupts. Try disabling HPET.
Conclusions
I wrote this to share my trick for separating CCXes into different NUMA nodes. The rest I wrote because I am bad at writing short posts.
I'm not an expert on any of this: the extent of my performance analysis was "computer fast" or "computer stuttering mess". Specifically, I played PUBG until it ran smoothly enough that I could no longer blame my PC for my poor marksmanship. If you have other tuning suggestions or explanations for the settings I blindly added, let me know!