Threadripper - Graphics Card Passthrough

    reference   fedora gpu virtualization windows linux administration threadripper  Comments

Getting GPU passthrough to work with a host containing two NVIDIA GTX1080TI cards on Threadripper

Qudib Renegade has a very good instruction set for setting up GPU passthrough.

https://qubitrenegade.com/virtualization/kvm/vfio/2019/07/17/VFIO-Fedora-Notes.html

You may also want ot reference level1techs.com for more information.

https://forum.level1techs.com/t/play-games-in-windows-on-linux-pci-passthrough-quick-guide/108981

Also here for the patched information.

https://forum.level1techs.com/t/threadripper-reset-fixes/123937

Note, These are somewhat old and there may be better ways available.

HOST : System Hosting the VM's

VM : System running as a VM

System tutorial is for Fedora 27

Patching the kernel:

This is the source for the patching instructions. https://forum.level1techs.com/t/threadripper-reset-fixes/123937

In short:

Install required packages 

 
sudo dnf install fedpkg fedora-packager rpmdevtools ncurses-devel pesign
sudo dnf install rpm-build flex perl-devel perl-generators openssl-devel hmaccalc elfutils-devel

Make a directory to work on kernel source within 

mkdir src
cd src

Anonymously clone the fedora kernel and go to that directory 

 
fedpkg clone -a kernel
cd kernel

Switch to the branch f27 

fedpkg switch-branch f27

Within the kernel.spec define your own kernel name change the ‘# define buildid .local’ to ‘%define buildid .threadripper’

Adding the kernel patch

Download the patch and save it as ‘tr.patch’ within the ‘src/kernel’ directory Threadripper iommu patch

Within the file ‘kernel.spec’ look for the line ‘# END OF PATCH DEFINITIONS’ and add the following right above it. 

Patch999: tr.patch
\# END OF PATCH DEFINITIONS

Perform the compile and install of your custom kernel. Use htop to watch those threads work!!! 

fedpkg local

Picking the correct card.

If you already have the nvidia drivers installed on the HOST you may use nvidia settings to grab the pci numbers 

nvidia-settings

All boot devices, but it doesn't single out any of them. 

'find /sys/devices/pci* -name boot_vga'

This link says that passthrough and secondary display like spice dont play nice together. Lets test that.

command to list pci devices in a tree: 

lspci -tvnn

That will show you the device path to build out.

start with /sys/devices/pcixxxx:xx/… and build it out to look like the following, your devices numbers will be different

/sys/devices/pci0000:00/0000:00:03.1/0000:08:00.0/driver_override 

 \-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
             +-00.2  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
             +-01.0  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
             +-01.1-[01-07]--+-00.0  Advanced Micro Devices, Inc. [AMD] X399 Series Chipset USB 3.1 xHCI Controller [1022:43ba]
             |               +-00.1  Advanced Micro Devices, Inc. [AMD] X399 Series Chipset SATA Controller [1022:43b6]
             |               \-00.2-[02-07]--+-00.0-[03]----00.0  Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1d6a:d107]
             |                               +-04.0-[04]----00.0  Intel Corporation I211 Gigabit Network Connection [8086:1539]
             |                               +-05.0-[05]----00.0  Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] [8086:24fb]
             |                               +-06.0-[06]----00.0  Intel Corporation I211 Gigabit Network Connection [8086:1539]
             |                               \-07.0-[07]----00.0  Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014]
             +-02.0  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
             +-03.0  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
             +-03.1-[08]--+-00.0  NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06]
             |            \-00.1  NVIDIA Corporation GP102 HDMI Audio Controller [10de:10ef]
Will build to: 
/sys/devices/pci0000:00/0000:00:03.1/0000:08:00.0/driver_override
/sys/devices/pci0000:00/0000:00:03.1/0000:08:00.1/driver_override

Then the Renesas usb device I pass through as well:

/sys/devices/pci0000:00/0000:00:01.1/0000:01:00.2/0000:02:07.0/0000:07:00.0/driver_override
Turns out that the Renesas did not work because it was not within it own iommu group. You may need to switch it to a differnet slot. Here is a thread about that.

My script located in: 

/usr/sbin/vfio-pci-override-vga.sh

Change the permissions to make it executable 

chmod 755 /usr/sbin/vfio-pci-override-vga.sh

#!/bin/sh
#echo "vfio-pci" > /sys/devices/pci0000:00/0000:00:03.1/0000:08:00.0/driver_override
#echo "vfio-pci" > /sys/devices/pci0000:00/0000:00:03.1/0000:08:00.1/driver_override
#echo "vfio-pci" > /sys/devices/pci0000:00/0000:00:01.1/0000:01:00.2/0000:02:07.0/0000:07:00.0/driver_override

#  Or specify the id's by the device group
DEVS="0000:00:08.0 0000:00:08.1 0000:00:07.0"
for DEV in $DEVS; do
   echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

# regardless run modprobe to load the vfio-pci kernel driver.
modprobe -i vfio-pci

Now we need dracut to refernece the file and address the gpu hardware id's in file /etc/dracut.conf.d/vfio.conf add the following

add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"
install_items+="/sbin/vfio-pci-override-vga.sh"
# run dracut
dracut -f

You will need to send this information to the kernel as well during boot.

# append the following to your kernel GRUB_CMDLINE_LINUX
# Replace the pci.ids with the ones from your devices, as shown from the lspci output above at the end of each line for that device.
d.driver.blacklist=nouveau rd.driver.pre=vfio-pci iommu=1 amd_iommu=on vfio-pci.ids=10de:1b06,10de:10ef,1912:0014

Verification:

First off you will see your HOST desktop on boot up just as you always did. If not then you messed up.

If it worked then verify the secondary card worked correctly. Run the lspci command, you will see the card to be dedicated to the VM with the words ‘Kernel driver in use: vfio-pci’ under its listing.

Installing windows 8.1

Need to select UEFI during the ‘New VM’ creation process. On Step 5 of the ‘Create a new virtual machine’ dialog make sure to check the “Customize configuration before install”. After clicking finish Select the ‘Overview’ item on the left hand side. Under ‘Hypervisor Details’ ‘Firmware’ dropdown select a UEFI firmware. CLICK APPLY!!!!!! if you dont it will keep bios and this wont work.

What if you dont see the firmware drop down, then you need to install it and restart libvirtd. 

dnf install edk2-ovmf edk2-ovmf-ia32
systemctl restart libvirtd

Setting up the VM, If you do not do this the NVIDIA driver will disabled the video card with an error 43. Use Device Manager to view this. Instructions from https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#.22Error_43:_Driver_failed_to_load.22_on_Nvidia_GPUs_passed_to_Windows_VMs

In short USE COMMAND “virsh edit vmname” and add the hyperv vendor_id field and kvm “hidden” field to that file 

<features>
	<hyperv>
		...
		<vendor_id state='on' value='123456789ab'/>
		...
	</hyperv>
	...
	<kvm>
	<hidden state='on'/>
	</kvm>
</features>

To get the virtio drivers, install the driver disk. Follow the instructions for installing the repo, virtio-win package, and enabling the virtio-win-latest https://docs.fedoraproject.org/quick-docs/en-US/creating-windows-virtual-machines-using-virtio-drivers.html

it should allow for access to an ISO with the virtio drivers that you may then use on the VM.