Skip to main content

Passthrough the GPU!

Explore IOMMU Grouping

Let passthrough our second GPU now, run the following command in your terminal:

#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

This script provides a detailed view of IOMMU groups and their associated devices, helping you understand the connections within your system. This information is vital for configuring device passthrough in virtualization.

1.2 Identify IOMMU GPU IDs​

The output from the above command will list your GPUs. Your second GPU, the one designated for passthrough, will be included in this list. Note its PCI (Peripheral Component Interconnect) numbers, also known as PCI IDs. These IDs are critical for the upcoming steps.

Here’s an example of what you might see:

IOMMU Group 22:
	09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] [1002:73ff] (rev c7)
IOMMU Group 23:
	09:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]

Example IDs​

Suppose your investigation reveals the following GPU IDs:

  • 1002:73ff (GPU)
  • 1002:ab28 (GPU Audio)

Keep these IDs handy, as they are crucial for the next steps.

1.3 Embed IDs in Your Bootloader Config​

Configuring GPU passthrough involves specifying which GPU to allocate to the virtual machine by using its PCI IDs. These IDs pinpoint the GPU within the system, enabling direct hardware access.

Add the PCI IDs to your bootloader configuration. If you're using GRUB or another bootloader, include the following in the configuration line where you enabled virtualization:

vfio-pci.ids=1002:73ff,1002:ab28

Systemd-boot example​

Your updated configuration with systemd-boot should look like this:

title Arch Linux
linux /vmlinuz-linux
initrd /initramfs-linux.img
options root=<root> quiet rw intel_iommu=on iommu=pt vfio-pci.ids=1002:73ff,1002:ab28

Tips: for user systemd-boot, you do not need to regenerate the config.

GRUB example​

Your updated configuration with GRUB should look like this:

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt vfio-pci.ids=1002:73ff,1002:ab28 ..."

For the GRUB user, you will need to regenerate the config file:

sudo grub-mkconfig -o /boot/grub/grub.cfg

2. Create a vfio.conf File​

Next, create a new configuration file for vfio:

sudo vim /etc/modprobe.d/vfio.conf

Add the following content:

options vfio-pci ids=1002:73ff,1002:ab28

3. Update mkinitcpio.conf​

Ensure that the necessary vfio modules are included in your initial RAM disk. Edit the /etc/mkinitcpio.conf file and update the MODULES() line:

MODULES=(vfio_pci vfio vfio_iommu_type1)

3.2 Regenerate the Image​

With the configuration updated, rebuild the initial RAM disk and reboot your system:

sudo mkinitcpio -P # Regenerate all image
sudo mkinitcpio -p linux # Regenerate image only linux

4. Verify GPU Passthrough Status​

Now reboot your system and check the status of your GPU. To check if your GPU is correctly configured for passthrough, run

lspci -k

Look at your PCI setup. The Kernel driver in use entry should show vfio-pci. For example:

09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] (rev c7)
	Subsystem: Tul Corporation / PowerColor Navi 23 [Radeon RX 6600/6600 XT/6600M]
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu
09:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

If you see vfio-pci, your GPU and GPU Audio are successfully set up for passthrough. If not, review your steps and try again. Success is within reach!

Next Step​

In the next steps, we’ll cover installing the necessary packages, setting up libvirt permissions, configuring group memberships, adjusting QEMU permissions, and managing the libvirt service. Start by installing the required packages for virtualization. Then, configure permissions for libvirt to ensure proper management of virtualization resources. You’ll also need to add your user to the relevant groups and set the correct permissions for QEMU. Finally, start the libvirt service to ensure it runs correctly.