QATlib VFIO no-IOMMU Mode Support

Overview

QATlib now supports VFIO no-IOMMU mode, allowing customers to use VFIO without viommu in virtual machines (VMs). In this mode, QATlib passes the guest physical address directly to the host IOMMU, instead of using an IOVA. This feature is automatically detected by QATlib when it identifies QAT devices on VMs, switching to no-IOMMU mode as needed.

Prerequisites

  • Ensure hugepages are enabled on both the host and guest VM to pin host physical memory and maintain contiguous guest physical memory for host IOMMU operations.

  • For Kernel versions 6.12 to 6.15, enabling SR-IOV when IOMMU is disabled requires an additional patch to the QAT kernel driver code. The patch can be found here. The QAT kernel driver will need to be recompiled to support this usage.

Process to Enable VFIO no-IOMMU Mode for QAT Applications

Follow the sections below to enable VFIO no-IOMMU mode for QAT applications.

Setup Hugepages on the Host

Hugepages must be enabled for optimal performance. Hugepage support is required to reserve large page sizes (2MB or 1GB per page) to reduce TLB (Translation Lookaside Buffers) misses and improve cache efficiency.

For 2MB pages, you can activate hugepages while booting or at any time after the system is booted. Define hugepages setting in /etc/default/grub to activate it while booting, or overwrite the number of 2MB hugepages as follows:

echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

In this case, 1024 pages of 2MB (totally 2048 MB) are reserved.

  1. Create a directory for hugepages:

    mkdir -p /mnt/hugepages

  2. Mount the hugepages filesystem:

    mount -t hugetlbfs nodev /mnt/hugepages

  3. Reserve hugepages:

    echo 2048 > /proc/sys/vm/nr_hugepages

Configure QEMU for Hugepages

Add hugepage options in the QEMU command to allocate memory for the VM:

qemu-system-x86_64 \
      -m 4G \
      -object memory-backend-file,id=mem0,mem-path=/mnt/hugepages,size=4G,share=on,prealloc=on \
      -machine memory-backend=mem0 \

Boot VM Without viommu

Remove intel_iommu=on from the boot command line to disable viommu.

Enable VFIO no-IOMMU Mode in VM

If there is no IOMMU available on the system, VFIO can still be used by enabling the unsafe no-IOMMU mode. This mode allows VFIO to function without IOMMU protection, which is inherently unsafe but necessary in certain situations.

Once inside the VM, enable VFIO no-IOMMU mode:

echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

Important

Since no-IOMMU mode forgoes IOMMU protection, it is inherently unsafe. Ensure that you understand the risks before enabling this mode.

Load VFIO-PCI Driver

Load the VFIO-PCI driver (or qat_vfio_pci):

modprobe vfio-pci

Verify VFIO Groups

Check VFIO groups to ensure they are in no-IOMMU mode:

ls /dev/vfio

You should see VFIO groups as noiommu-<group id>.

Setup Hugepages on the Guest VM

Make the memory available for using hugepages. The mount point for hugepages can be made permanent across reboots by configuring /etc/fstab.

  1. Create a directory for hugepages:

    mkdir -p /mnt/hugepages

  2. Mount the hugepages filesystem:

    mount -t hugetlbfs nodev /mnt/hugepages

  3. Reserve hugepages:

    echo 2048 > /proc/sys/vm/nr_hugepages

Run Sample Applications

With VFIO no-IOMMU support enabled in QATlib, you can now run sample applications to test the setup.