QATlib VFIO no-IOMMU Mode Support

Overview

QATlib now supports VFIO no-IOMMU mode, allowing customers to use VFIO without viommu in virtual machines (VMs). In this mode, QATlib passes the guest physical address directly to the host IOMMU, instead of using an IOVA. This feature is automatically detected by QATlib when it identifies QAT devices on VMs, switching to no-IOMMU mode as needed.

Prerequisites

Ensure hugepages are enabled on both the host and guest VM to pin host physical memory and maintain contiguous guest physical memory for host IOMMU operations.
For Kernel versions 6.12 to 6.15, enabling SR-IOV when IOMMU is disabled requires an additional patch to the QAT kernel driver code. The patch can be found here. The QAT kernel driver will need to be recompiled to support this usage.

Process to Enable VFIO no-IOMMU Mode for QAT Applications

Follow the sections below to enable VFIO no-IOMMU mode for QAT applications.

Setup Hugepages on the Host

Hugepages must be enabled for optimal performance. Hugepage support is required to reserve large page sizes (2MB or 1GB per page) to reduce TLB (Translation Lookaside Buffers) misses and improve cache efficiency.

For 2MB pages, you can activate hugepages while booting or at any time after the system is booted. Define hugepages setting in /etc/default/grub to activate it while booting, or overwrite the number of 2MB hugepages as follows:

echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

In this case, 1024 pages of 2MB (totally 2048 MB) are reserved.

Create a directory for hugepages:
mkdir -p /mnt/hugepages
Mount the hugepages filesystem:
mount -t hugetlbfs nodev /mnt/hugepages
Reserve hugepages:
echo 2048 > /proc/sys/vm/nr_hugepages

Configure QEMU for Hugepages

Add hugepage options in the QEMU command to allocate memory for the VM:

qemu-system-x86_64 \
      -m 4G \
      -object memory-backend-file,id=mem0,mem-path=/mnt/hugepages,size=4G,share=on,prealloc=on \
      -machine memory-backend=mem0 \

Boot VM Without viommu

Remove intel_iommu=on from the boot command line to disable viommu.

Enable VFIO no-IOMMU Mode in VM

If there is no IOMMU available on the system, VFIO can still be used by enabling the unsafe no-IOMMU mode. This mode allows VFIO to function without IOMMU protection, which is inherently unsafe but necessary in certain situations.

Once inside the VM, enable VFIO no-IOMMU mode:

echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

Important

Since no-IOMMU mode forgoes IOMMU protection, it is inherently unsafe. Ensure that you understand the risks before enabling this mode.

Load VFIO-PCI Driver

Load the VFIO-PCI driver (or qat_vfio_pci):

modprobe vfio-pci

Verify VFIO Groups

Check VFIO groups to ensure they are in no-IOMMU mode:

ls /dev/vfio

You should see VFIO groups as noiommu-<group id>.

Setup Hugepages on the Guest VM

Make the memory available for using hugepages. The mount point for hugepages can be made permanent across reboots by configuring /etc/fstab.

Create a directory for hugepages:
mkdir -p /mnt/hugepages
Mount the hugepages filesystem:
mount -t hugetlbfs nodev /mnt/hugepages
Reserve hugepages:
echo 2048 > /proc/sys/vm/nr_hugepages

Configure Permissions for Non-Root Users

By default, QAT applications in no-IOMMU mode require root privileges to access VFIO devices and hugepages. If you need to run QAT applications as non-root users, additional configuration is required.

Note

These permission changes are required because no-IOMMU mode needs elevated privileges to access VFIO devices, hugepages, and system resources that are normally restricted to root users.

Set Device Permissions

Grant the qat group access to VFIO devices and hugepages:

# Grant qat group access to VFIO devices
sudo chgrp qat /dev/vfio/noiommu-*
sudo chmod 660 /dev/vfio/noiommu-*

# Grant qat group access to hugepages directory
sudo chgrp qat /dev/hugepages
sudo chmod 770 /dev/hugepages

Set Application Capabilities

For each QAT application binary that non-root users will execute, set the required capabilities:

sudo setcap cap_dac_override,cap_sys_rawio,cap_sys_admin+ep /path/to/your/qat_application

For example, to enable cpa_sample_code:

sudo setcap cap_dac_override,cap_sys_rawio,cap_sys_admin+ep /usr/local/bin/cpa_sample_code

Add Users to QAT Group

Add the desired users to the qat group:
```
sudo usermod -a -G qat <username>
```
The user will need to log out and log back in for the group membership to take effect.

Warning

Setting capabilities on executables and modifying device permissions reduces system security. Only apply these changes if non-root access to QAT applications is required, and ensure you understand the security implications.

Run Sample Applications

With VFIO no-IOMMU support enabled in QATlib, you can now run sample applications to test the setup.

For root users, applications can be run directly:

sudo /usr/local/bin/cpa_sample_code

For non-root users (after completing the permission configuration above):

/usr/local/bin/cpa_sample_code