Threadripper - Graphics Card Passthrough

Getting GPU passthrough to work with a host containing two NVIDIA GTX1080TI cards on Threadripper

Note, These are somewhat old and there may be better ways available. See level1techs.com for more information. This tutorial may work better for you. https://forum.level1techs.com/t/play-games-in-windows-on-linux-pci-passthrough-quick-guide/108981 Also here for the patched information. https://forum.level1techs.com/t/threadripper-reset-fixes/123937

HOST : System Hosting the VM’s VM : System running as a VM

System tutorial is for Fedora 27 Patching the kernel: This is the source for the patching instructions. https://forum.level1techs.com/t/threadripper-reset-fixes/123937

In short:

Install required packages

sudo dnf install fedpkg fedora-packager rpmdevtools ncurses-devel pesign
sudo dnf install rpm-build flex perl-devel perl-generators openssl-devel hmaccalc elfutils-devel

Make a directory to work on kernel source within

mkdir src
cd src

Anonymously clone the fedora kernel and go to that directory

fedpkg clone -a kernel
cd kernel

Switch to the branch f27

fedpkg switch-branch f27

Within the kernel.spec define your own kernel name change the ‘# define buildid .local’ to ‘%define buildid .threadripper’

Adding the kernel patch Download the patch and save it as ‘tr.patch’ within the ‘src/kernel’ directory Threadripper iommu patch

Within the file ‘kernel.spec’ look for the line ‘# END OF PATCH DEFINITIONS’ and add the following right above it.

Patch999: tr.patch
# END OF PATCH DEFINITIONS

Perform the compile and install of your custom kernel. Use htop to watch those threads work!!!

fedpkg local

Picking the correct card.

If you already have the nvidia drivers installed on the HOST you may use nvidia settings to grab the pci numbers

nvidia-settings

All boot devices, but it doesn’t single out any of them.

‘find /sys/devices/pci* -name boot_vga’

This link says that passthrough and secondary display like spice dont play nice together. Lets test that.

command to list pci devices:

lspci -vnn

My script located in:

/usr/sbin/vfio-pci-override-vga.sh

Change the permissions to make it executable

chmod 755 /usr/sbin/vfio-pci-override-vga.sh

#!/bin/sh
echo "vfio-pci" > /sys/devices/pci0000:00/0000:00:03.1/0000:08:00.0/driver_override
echo "vfio-pci" > /sys/devices/pci0000:00/0000:00:03.1/0000:08:00.1/driver_override
DEVS=0000:00:08.0
for DEV in $DEVS; do
   echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done
modprobe -i vfio-pci

Now we need dracut to refernece the file and address the gpu hardware id’s in file /etc/dracut.conf.d/vfio.conf add the following

add_drivers+=“vfio vfio_iommu_type1 vfio_pci vfio_virqfd”
install_items+=“/sbin/vfio-pci-override-vga.sh”

Verification: First off you will see your HOST desktop on boot up just as you always did. If not then you messed up.

If it worked then verify the secondary card worked correctly. Run the lspci command, you will see the card to be dedicated to the VM with the words ‘Kernel driver in use: vfio-pci’ under its listing.

Installing windows 8.1 Need to select UEFI during the ‘New VM’ creation process. On Step 5 of the ‘Create a new virtual machine’ dialog make sure to check the “Customize configuration before install”. After clicking finish Select the ‘Overview’ item on the left hand side. Under ‘Hypervisor Details’ ‘Firmware’ dropdown select a UEFI firmware. CLICK APPLY!!!!!! if you dont it will keep bios and this wont work. What if you dont see the firmware drop down, then you need to install it and restart libvirtd.

dnf install edk2-ovmf edk2-ovmf-ia32
systemctl restart libvirtd

Setting up the VM, If you do not do this the NVIDIA driver will disabled the video card with an error 43. Use Device Manager to view this. Instructions from https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#.22Error_43:_Driver_failed_to_load.22_on_Nvidia_GPUs_passed_to_Windows_VMs

In short USE COMMAND “virsh edit vmname” and add the hyperv vendor_id field and kvm “hidden” field to that file

To get the virtio drivers, install the driver disk. Follow the instructions for installing the repo, virtio-win package, and enabling the virtio-win-latest https://docs.fedoraproject.org/quick-docs/en-US/creating-windows-virtual-machines-using-virtio-drivers.html it should allow for access to an ISO with the virtio drivers that you may then use on the VM.