Configuration and Tuning

There are parameters which can be tweaked to optimize for an application’s workload which might work in two modes:

• Managed Mode (when the qat service is enabled) .

• Standalone Mode (without the qat service).

Managed Mode

In Managed Mode, qatmgr manages the allocation of the Virtual Function (VF) resources to each process on process startup.

The following parameters can be set in /etc/sysconfig/qat.

Note

End of line comments are not supported in this file.

Policy

Indicates how many VFs will be assigned to each process. The maximum value allowed for POLICY is 512. In practice this value must not exceed number of QAT PFs * 16.

Policy Settings

Value	Description
Unset/0	One VF from each PF will be assigned to each process.
N	N VFs will be assigned to each process. Maximum value is: # of PFs * 16

Services Enabled

Tells the kernel driver which services to enable on each Physical Function (PF). All VFs on a PF enable the same services. There is a maximum of two services that can be enabled on each PF.

Important

Newer QAT Kernel modules include support for the following configuration options.

Refer to QAT Kernel Driver Releases & Features section of the In-Tree Release Notes for details.

Important

Newer QAT Kernel modules include support for the following configuration options.

Refer to QAT Kernel Driver Releases & Features section of the In-Tree Release Notes for details.

Services Enabled Settings

Value	Description
Unset	even-numbered PFs have sym;asym, odd-numbered PFs have dc. For example: PF0, PF2 … support crypto and PF1, PF3 … support data compression.
sym;asym	All PFs, and so all VFs, have sym;asym only.
asym;sym	Identical to sym;asym.
dc	The device is configured for running compression services only.
sym	The device is configured for running symmetric crypto services only.
asym	The device is configured for running asymmetric crypto services only.
asym;dc	The device is configured for running asymmetric crypto services and compression services.
dc;asym	Identical to asym;dc above.
sym;dc	The device is configured for running symmetric crypto services and compression services.
dc;sym	Identical to sym;dc above.
dcc	The device is configured for running chaining operations (hash then compress) only. Note: Throughput is lower than dc. Only use setting when chaining is needed.

Performance Considerations

Important

The following is applicable to QAT2.0 devices only.

In order to maximize QAT throughput performance for a given service type, only one Services type is enabled per physical function (PF).

By design, two Acceleration Engine clusters are available, each containing four Acceleration Engines. Each of these two clusters are limited to using a single service. Therefore, the possible split options are 4|4 or 8 for a given service type.

Configuration examples:

• While using VFs on a system with a PF configured for SYM and DC, 4 acceleration engines will be dedicated to SYM and 4 acceleration engines will be dedicated to DC, so only 2 resource providers per child VF can be used for SYM. Here, we should expect some performance degradation for SYM (even if DC is not used).

• While using VFs on a system with a PF configured for SYM, all 8 acceleration engines will be dedicated to SYM, so all 4 resource providers of a child VF can be used for SYM only. Here, we will see the best SYM performance.

Note

Packet size will also modulate the impact of the above configuration settings.

Example Configurations

The following examples are based on a one-socket Intel^® QAT 4xxx platform with 4 PFs, each PF having 16 VFs.

• Each VF enabled for dc has four instances,

• Each VF enabled for sym;asym has two sym instances and two asym instances.

Tip

An instance is an abstraction used on the APIs to identify a unique path to the hardware. In a multi-threaded process, typically at least one instance is needed per thread.

Flexibility

With the default configuration, neither the Policy or the Services Enabled parameter is set.

Each process will allocate one VF, from each PF, providing a total of four VFs.

This enables two sym;asym VFs, which provides four sym instances and four asym instances and two dc VFs, which provides eight dc instances.

The maximum number of processes is 16.

With N sockets, the maximum number of processes is still 16, each will have instances as above * N.

Scalability and Flexibility

To support a process that requires access to both sym;asym and dc services, the Policy parameter is set. The configuration file /etc/sysconfig/qat will look like:

POLICY=2

Each process will allocate two VFs, one with sym;asym, one with dc.

This provides two sym instances, two asym instance and four dc instances.

There is a maximum of 32 processes on one-socket system (16 VFs x 4 PFs / POLICY).

A system with N sockets will have 32 x N processes in total.

Scalability and Flexibility for Symmetric or Asymmetric Plus Compression

To support a process that requires access to either sym;dc or asym;dc services, the Policy parameter is set. The configuration file /etc/sysconfig/qat will look like either:

POLICY=1
ServicesEnabled=asym;dc

Or:

POLICY=1
ServicesEnabled=sym;dc

Each process will allocate one VFs.

This provides two sym instances or two asym instance and two dc instances.

There is a maximum of 64 processes on one-socket system (16 VFs x 4 PFs / POLICY).

A system with N sockets will have 64 x N processes in total.

Crypto-Only Scalability

To enable the sym;asym service, both the Policy and the Services Enabled parameters are set. The configuration file /etc/sysconfig/qat will look like:

POLICY=1
ServicesEnabled=sym;asym

Each process will allocate one VF, so will have two sym instances and two asym instances.

There is a maximum of 64 processes on one-socket system.

A system with N sockets will have 64 x N processes in total.

Note

This configuration also provides optimized throughput for asym.

Crypto-Only Optimized Throughput

To enable the sym;asym service, both the Policy and the Services Enabled parameters are set. The configuration file /etc/sysconfig/qat will look like:

POLICY=0
ServicesEnabled=sym;asym

Each process will allocate one VF, so will have two sym instances and two asym instances.

There is a maximum of 16 processes on one-socket system.

A system with N sockets will have 16 x N processes in total.

Symmetric-Only Optimized Throughput

To enable the sym service, both the Policy and the Services Enabled parameters are set. The configuration file /etc/sysconfig/qat will look like:

POLICY=0
ServicesEnabled=sym

Each process will allocate one VF, so will have four sym instances.

There is a maximum of 16 processes on one-socket system.

A system with N sockets will have 16 x N processes in total.

Compression-Only Scalability

To enable the dc service both the Policy and the Services Enabled parameter are set. The configuration file /etc/sysconfig/qat will look like:

POLICY=1
ServicesEnabled=dc

All VFs have the dc service enabled.

Each process will allocate one VF, so will have four dc instances.

There is a maximum of 64 processes on one-socket system.

A system with N sockets will have 64 x N processes in total.

Compression-Only Optimized Throughput

To enable the sym service, both the Policy and the Services Enabled parameters are set. The configuration file /etc/sysconfig/qat will look like:

POLICY=0
ServicesEnabled=dc

Each process will allocate one VF, so will have four dc instances.

There is a maximum of 16 processes on one-socket system.

A system with N sockets will have 16 x N processes in total.

Compression Chaining Scalability

To enable the dcc service both the Policy and the Services Enabled parameter are set. The configuration file /etc/sysconfig/qat will look like:

POLICY=1
ServicesEnabled=dcc

All VFs have the dc with sym capabilities but only for chaining operations.

Each process will allocate one VF, so will have four dc instances.

There is a maximum of 64 processes on one-socket system.

A system with N sockets will have 64 x N processes in total.

qat script

This qat script can be used to control qatlib stack when running in Managed mode.

The following command line options are supported:

qat -h

optional arguments:
-h, --help                  show this help message and exit
--config, -c                Configure QAT driver (Note: Specify mode/policy or default option will be used)
--reload, -r                Reload QAT driver without any configuration change
--status, -s                Print current QAT configuration for each VF
--mode MODE, -m MODE        Valid options are 1 to 7: 1->sym;asym | 2->dc | 3->sym | 4->asym | 5->asym;dc | 6->sym;dc | 7->dcc
--policy POLICY, -p POLICY  Specify QAT config POLICY value to set
--version, -v               Print version

Supported Operations

• Display Current Configuration

  We can display current configuration with the following command:

  qat --status
  

  Output will look like:

  +-----+---------------+------+--------------+--------------+----------+
  |  #  |   VFIO GROUP  | NODE |    PF BDF    |    VF BDF    | SERVICES |
  +-----+---------------+------+--------------+--------------+----------+
  |  1  | /dev/vfio/419 |  0   | 0000:6b:00.0 | 0000:6b:00.1 | sym;asym |
  |  2  | /dev/vfio/420 |  0   | 0000:6b:00.0 | 0000:6b:00.2 | sym;asym |
  |  3  | /dev/vfio/421 |  0   | 0000:6b:00.0 | 0000:6b:00.3 | sym;asym |
  |  4  | /dev/vfio/422 |  0   | 0000:6b:00.0 | 0000:6b:00.4 | sym;asym |
  |  5  | /dev/vfio/423 |  0   | 0000:6b:00.0 | 0000:6b:00.5 | sym;asym |
  |  6  | /dev/vfio/424 |  0   | 0000:6b:00.0 | 0000:6b:00.6 | sym;asym |
  |  7  | /dev/vfio/425 |  0   | 0000:6b:00.0 | 0000:6b:00.7 | sym;asym |
  |  8  | /dev/vfio/426 |  0   | 0000:6b:00.0 | 0000:6b:01.0 | sym;asym |
  |  9  | /dev/vfio/427 |  0   | 0000:6b:00.0 | 0000:6b:01.1 | sym;asym |
  |  10 | /dev/vfio/428 |  0   | 0000:6b:00.0 | 0000:6b:01.2 | sym;asym |
  |  11 | /dev/vfio/429 |  0   | 0000:6b:00.0 | 0000:6b:01.3 | sym;asym |
  |  12 | /dev/vfio/430 |  0   | 0000:6b:00.0 | 0000:6b:01.4 | sym;asym |
  |  13 | /dev/vfio/431 |  0   | 0000:6b:00.0 | 0000:6b:01.5 | sym;asym |
  |  14 | /dev/vfio/432 |  0   | 0000:6b:00.0 | 0000:6b:01.6 | sym;asym |
  |  15 | /dev/vfio/433 |  0   | 0000:6b:00.0 | 0000:6b:01.7 | sym;asym |
  |  16 | /dev/vfio/434 |  0   | 0000:6b:00.0 | 0000:6b:02.0 | sym;asym |
  |  17 | /dev/vfio/435 |  0   | 0000:70:00.0 | 0000:70:00.1 |    dc    |
  |  18 | /dev/vfio/436 |  0   | 0000:70:00.0 | 0000:70:00.2 |    dc    |
  |  19 | /dev/vfio/437 |  0   | 0000:70:00.0 | 0000:70:00.3 |    dc    |
  |  20 | /dev/vfio/438 |  0   | 0000:70:00.0 | 0000:70:00.4 |    dc    |
  

  Information includes:

  • VFIO Group: This information is required when passing QAT resources to container

  • NODE: Which processor node the QAT resource comes from

  • PF BDF: Bus/device/function of QAT PF

  • VF BDF: Bus/device/function of QAT VF

  • SERVICES: Current services that are enabled on device

• Update Configuration

  We can update configuration using command similar to the following:

  qat --config --policy 1 --mode 5
  

  In the example above we update the policy setting to 1 and mode to asym;dc.

  Note

  If --policy or --mode is not specified, the default values are used.

  This script will update the /etc/sysconfig/qat file and restart QAT kernel modules.

  Note

  Older QAT kernel modules and user space library may not support all mode settings included here.

  This script includes logic to detect these issues and provide guidance.

Requirements for running script

1. Install Dependencies

   For RPM-based distros:

   sudo dnf install -y python3
   sudo dnf install -y python3-venv
   sudo dnf install -y python3-pip
   

   Ensure /usr/bin/python exists and points to the installed python3 binary.

   #!/bin/bash
   
   # Check if Python 3 is installed
   if command -v python3 &>/dev/null; then
       # Get the path to the Python 3 executable
       PYTHON3_PATH=$(which python3)
   
       # Check if /usr/bin/python already exists
       if [ -L /usr/bin/python ] || [ -e /usr/bin/python ]; then
           echo "A file or symbolic link already exists at /usr/bin/python."
           echo "Please remove or rename it before running this script."
           exit 1
       else
           # Create a symbolic link for Python 3
           sudo ln -s "$PYTHON3_PATH" /usr/bin/python
           echo "Symbolic link created for Python 3 at /usr/bin/python."
       fi
   else
       echo "Python 3 is not installed or not found in the system PATH."
       exit 1
   fi
   

   For Ubuntu and other Debian-based distros, install these dependencies:

   sudo apt-get install -y python3
   sudo apt-get install -y python3-venv
   sudo apt-get install -y python-is-python3
   sudo apt-get install -y python3-pip
   sudo apt-get install -y python3-packaging
   

2. Setup Virtual Environment Install Python packages

   python -m venv myenv
   source ~/myenv/bin/activate
   pip install prettytable
   deactivate
   

3. Copy script to system directory

   This script is available here.

   After copying to local directory, this can be moved to system directory.

   mv ./qat /usr/bin
   sudo chmod 755 /usr/bin/qat
   

Standalone Mode

In Standalone Mode, there is no qat service running.

This is the preferred way to run on containers where a subset of VFs are explicitly passed through and no qatmgr managed resource manager is needed.

Each standalone process assumes VF devices which it can’t open are in use by another process and looks for unopened VFs.

In this mode, the /quickassist/utilities/service/qat_init.sh script can be called on the host to bind VFs to vfio-pci and to enable services based on the Services Enabled parameter.

In this mode the Policy parameter should not be set in /etc/sysconfig/qat. If it is set, it will be ignored.

Instead an environment variable QAT_POLICY can be set.

Please note that the QAT_POLICY=0 is not a valid setting in standalone mode. The valid setting range starts from 1 and goes up to max PFs * 16.

Running Without qat_init.sh

In some scenarios, it is not possible to use qat_init.sh. This section provides details on the steps that need to be performed in this usage.

1. Add vfio-pci.ids=8086:4941 to the kernel command line.

   Note

   Depending on your device, it might be 4941, 4943, 4945 or 4947. This is the device ID of the VFs.

   If your PF is 4940, the VF device ID is 4941, if the PF is 4942, the VF is 4943.

   See Supported Devices for additional details.

2. Obtain the Domain Bus Device Function (BDF) of the QAT PF with lspci

   This will be used when accessing sysfs.

   echo `(lspci -nd 8086:4940 && lspci -nd 8086:4942 && lspci -nd 8086:4944 && lspci -nd 8086:4946)`
   

3. Load the qat driver if not loaded

   sudo modprobe qat_4xxx
   

   Note

   Refer to Supported Devices for appropriate kernel module for your device.

4. Load vfio-pci driver if not loaded

   sudo modprobe vfio-pci
   

5. Configure devices for required service.

   Note

   This needs to be repeated for each device in the system.

   In the example below, we are enabling compression.

   echo down > /sys/bus/pci/devices/<BDF of PF>/qat/state
   echo dc > /sys/bus/pci/devices/<BDF of PF>/qat/cfg_services
   echo up > /sys/bus/pci/devices/<BDF of PF>/qat/state
   

6. Enable VFs

   Note

   This needs to be repeated for each device in the system.

   echo 16 > /sys/bus/pci/devices/<BDF of PF>/sriov_numvfs
   

7. Set permissions/groups to vfio groups

   chown root:qat /dev/vfio/[0-9]*
   chmod g+rw /dev/vfio/[0-9]*
   

Display Current Configuration

Note

This functionality is also available in the qat script described in this section.

• Standalone Mode (without the qat service).

The script below can be used to view the PF/VF service configuration.

#!/bin/bash

printf "%-13s | %-5s | %-12s | %-12s | %-10s\n"  "VFIO GROUP" "NODE"   \
"PF BDF" "VF BDF" "SERVICES"
echo "--------------------------------------------------------------"

for vfio_group in /dev/vfio/*; do
    if [ $vfio_group = "/dev/vfio/vfio" ]; then
        continue
    fi
    if [ $vfio_group = "/dev/vfio/devices" ]; then
        continue
    fi

    group=${vfio_group##*/}
    # assume one bdf per iommu group
    bdf=$(ls /sys/kernel/iommu_groups/$group/devices/)
    vendor=$(cat /sys/kernel/iommu_groups/$group/devices/$bdf/vendor)
    node=$(cat /sys/kernel/iommu_groups/$group/devices/$bdf/numa_node)
    did=$(cat /sys/kernel/iommu_groups/$group/devices/$bdf/device)

    if [ "$vendor" != "0x8086" ]; then
        continue
    fi

    if [ "$did" != "0x4941" ] && [ "$did" != "0x4943" ] && [ "$did" != "0x4945" ] && [ "$did" != "0x4947" ]; then
        continue
    fi

    regex='([a-z0-9]+):([a-z0-9]+):.*'
    [[ $bdf =~ $regex ]]
    pf_domain=${BASH_REMATCH[1]}
    pf_bus=${BASH_REMATCH[2]}
    pf_bdf="$pf_domain:$pf_bus:00.0"

    printf "%-15s %-7s %-14s %-14s %-10s\n"  "$vfio_group"  "$node"    \
    "$pf_bdf"   "$bdf"                                                 \
    "$(cat /sys/bus/pci/devices/$pf_bdf/qat/cfg_services)"

done

Custom Configuration

The configuration of services can also be set per PF.

Instructions are provided in the following link: https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-driver-qat.

Warning

• If custom configuration is set per PF following these instructions, then neither ServicesEnabled nor POLICY should be set in /etc/sysconfig/qat.

• It is not deterministic which instances a process receives in this configuration.