Configuration and Tuning

There are parameters which can be tweaked to optimize for an application’s workload which might work in two modes:

Managed Mode (when the qat service is enabled) .
Standalone Mode (without the qat service).

Managed Mode

In Managed Mode, qatmgr manages the allocation of the Virtual Function (VF) resources to each process on process startup.

The following parameters can be set in /etc/sysconfig/qat.

Policy

Indicates how many VFs will be assigned to each process. The maximum value allowed for POLICY is 512. In practice this value must not exceed number of QAT PFs * 16.

Policy Settings
Value	Description
`Unset/0`	One VF from each PF will be assigned to each process.
`N`	N VFs will be assigned to each process. Maximum value is: *# of PFs 16**

Services Enabled

Tells the kernel driver which services to enable on each Physical Function (PF). All VFs on a PF enable the same services. There is a maximum of two services that can be enabled on each PF.

Important

Newer QAT Kernel modules include support for the following configuration options.

Refer to QAT Kernel Driver Releases & Features section of the In-Tree Release Notes for details.

Important

Newer QAT Kernel modules include support for the following configuration options.

Refer to QAT Kernel Driver Releases & Features section of the In-Tree Release Notes for details.

Services Enabled Settings
Value	Description
`Unset`	even-numbered PFs have `sym;asym`, odd-numbered PFs have `dc`. For example: PF0, PF2 … support crypto and PF1, PF3 … support data compression.
`sym;asym`	All PFs, and so all VFs, have `sym;asym` only.
`asym;sym`	Identical to `sym;asym`.
`dc`	The device is configured for running compression services only.
`sym`	The device is configured for running symmetric crypto services only.
`asym`	The device is configured for running asymmetric crypto services only.
`asym;dc`	The device is configured for running asymmetric crypto services and compression services.
`dc;asym`	Identical to `asym;dc` above.
`sym;dc`	The device is configured for running symmetric crypto services and compression services.
`dc;sym`	Identical to `sym;dc` above.
`dcc`	The device is configured for running chaining operations (hash then compress) only. Note: Throughput is lower than dc. Only use setting when chaining is needed.

Performance Considerations

Important

The following is applicable to QAT2.0 devices only.

In order to maximize QAT throughput performance for a given service type, only one Services type is enabled per physical function (PF).

By design, two Acceleration Engine clusters are available, each containing four Acceleration Engines. Each of these two clusters are limited to using a single service. Therefore, the possible split options are 4|4 or 8 for a given service type.

Configuration examples:

While using VFs on a system with a PF configured for SYM and DC, 4 acceleration engines will be dedicated to SYM and 4 acceleration engines will be dedicated to DC, so only 2 resource providers per child VF can be used for SYM. Here, we should expect some performance degradation for SYM (even if DC is not used).
While using VFs on a system with a PF configured for SYM, all 8 acceleration engines will be dedicated to SYM, so all 4 resource providers of a child VF can be used for SYM only. Here, we will see the best SYM performance.

Note

Packet size will also modulate the impact of the above configuration settings.

Example Configurations

The following examples are based on a one-socket Intel^® QAT 4xxx platform with 4 PFs, each PF having 16 VFs.

Each VF enabled for dc has four instances,
Each VF enabled for sym;asym has two sym instances and two asym instances.

Tip

An instance is an abstraction used on the APIs to identify a unique path to the hardware. In a multi-threaded process, typically at least one instance is needed per thread.

Flexibility

With the default configuration, neither the Policy or the Services Enabled parameter is set.

Each process will allocate one VF, from each PF, providing a total of four VFs.

This enables two sym;asym VFs, which provides four sym instances and four asym instances and two dc VFs, which provides eight dc instances.

The maximum number of processes is 16.

With N sockets, the maximum number of processes is still 16, each will have instances as above * N.

Scalability and Flexibility

To support a process that requires access to both sym;asym and dc services, the Policy parameter is set. The configuration file /etc/sysconfig/qat will look like:

POLICY=2

Each process will allocate two VFs, one with sym;asym, one with dc.

This provides two sym instances, two asym instance and four dc instances.

There is a maximum of 32 processes on one-socket system (16 VFs x 4 PFs / POLICY).

A system with N sockets will have 32 x N processes in total.

Scalability and Flexibility for Symmetric or Asymmetric Plus Compression

To support a process that requires access to either sym;dc or asym;dc services, the Policy parameter is set. The configuration file /etc/sysconfig/qat will look like either:

POLICY=1
ServicesEnabled=asym;dc

Or:

POLICY=1
ServicesEnabled=sym;dc

Each process will allocate one VFs.

This provides two sym instances or two asym instance and two dc instances.

There is a maximum of 64 processes on one-socket system (16 VFs x 4 PFs / POLICY).

A system with N sockets will have 64 x N processes in total.

Crypto-Only Scalability

To enable the sym;asym service, both the Policy and the Services Enabled parameters are set. The configuration file /etc/sysconfig/qat will look like:

POLICY=1
ServicesEnabled=sym;asym

Each process will allocate one VF, so will have two sym instances and two asym instances.

There is a maximum of 64 processes on one-socket system.

A system with N sockets will have 64 x N processes in total.

Note

This configuration also provides optimized throughput for asym.

Crypto-Only Optimized Throughput

To enable the sym;asym service, both the Policy and the Services Enabled parameters are set. The configuration file /etc/sysconfig/qat will look like:

POLICY=0
ServicesEnabled=sym;asym

Each process will allocate one VF, so will have two sym instances and two asym instances.

There is a maximum of 16 processes on one-socket system.

A system with N sockets will have 16 x N processes in total.

Symmetric-Only Optimized Throughput

To enable the sym service, both the Policy and the Services Enabled parameters are set. The configuration file /etc/sysconfig/qat will look like:

POLICY=0
ServicesEnabled=sym

Each process will allocate one VF, so will have four sym instances.

There is a maximum of 16 processes on one-socket system.

A system with N sockets will have 16 x N processes in total.

Compression-Only Scalability

To enable the dc service both the Policy and the Services Enabled parameter are set. The configuration file /etc/sysconfig/qat will look like:

POLICY=1
ServicesEnabled=dc

All VFs have the dc service enabled.

Each process will allocate one VF, so will have four dc instances.

There is a maximum of 64 processes on one-socket system.

A system with N sockets will have 64 x N processes in total.

Compression-Only Optimized Throughput

To enable the sym service, both the Policy and the Services Enabled parameters are set. The configuration file /etc/sysconfig/qat will look like:

POLICY=0
ServicesEnabled=dc

Each process will allocate one VF, so will have four dc instances.

There is a maximum of 16 processes on one-socket system.

A system with N sockets will have 16 x N processes in total.

Compression Chaining Scalability

To enable the dcc service both the Policy and the Services Enabled parameter are set. The configuration file /etc/sysconfig/qat will look like:

POLICY=1
ServicesEnabled=dcc

All VFs have the dc with sym capabilities but only for chaining operations.

Each process will allocate one VF, so will have four dc instances.

There is a maximum of 64 processes on one-socket system.

A system with N sockets will have 64 x N processes in total.

qat script

This qat script can be used to control qatlib stack when running in Managed mode.

The following command line options are supported:

qat -h

optional arguments:
-h, --help                  show this help message and exit
--config, -c                Configure QAT driver (Note: Specify mode/policy or default option will be used)
--reload, -r                Reload QAT driver without any configuration change
--status, -s                Print current QAT configuration for each VF
--mode MODE, -m MODE        Valid options are 1 to 7: 1->sym;asym | 2->dc | 3->sym | 4->asym | 5->asym;dc | 6->sym;dc | 7->dcc
--policy POLICY, -p POLICY  Specify QAT config POLICY value to set
--version, -v               Print version

Supported Operations

Display Current Configuration

We can display current configuration with the following command:

qat --status

Output will look like:

+-----+---------------+------+--------------+--------------+----------+
|  #  |   VFIO GROUP  | NODE |    PF BDF    |    VF BDF    | SERVICES |
+-----+---------------+------+--------------+--------------+----------+
|  1  | /dev/vfio/419 |  0   | 0000:6b:00.0 | 0000:6b:00.1 | sym;asym |
|  2  | /dev/vfio/420 |  0   | 0000:6b:00.0 | 0000:6b:00.2 | sym;asym |
|  3  | /dev/vfio/421 |  0   | 0000:6b:00.0 | 0000:6b:00.3 | sym;asym |
|  4  | /dev/vfio/422 |  0   | 0000:6b:00.0 | 0000:6b:00.4 | sym;asym |
|  5  | /dev/vfio/423 |  0   | 0000:6b:00.0 | 0000:6b:00.5 | sym;asym |
|  6  | /dev/vfio/424 |  0   | 0000:6b:00.0 | 0000:6b:00.6 | sym;asym |
|  7  | /dev/vfio/425 |  0   | 0000:6b:00.0 | 0000:6b:00.7 | sym;asym |
|  8  | /dev/vfio/426 |  0   | 0000:6b:00.0 | 0000:6b:01.0 | sym;asym |
|  9  | /dev/vfio/427 |  0   | 0000:6b:00.0 | 0000:6b:01.1 | sym;asym |
|  10 | /dev/vfio/428 |  0   | 0000:6b:00.0 | 0000:6b:01.2 | sym;asym |
|  11 | /dev/vfio/429 |  0   | 0000:6b:00.0 | 0000:6b:01.3 | sym;asym |
|  12 | /dev/vfio/430 |  0   | 0000:6b:00.0 | 0000:6b:01.4 | sym;asym |
|  13 | /dev/vfio/431 |  0   | 0000:6b:00.0 | 0000:6b:01.5 | sym;asym |
|  14 | /dev/vfio/432 |  0   | 0000:6b:00.0 | 0000:6b:01.6 | sym;asym |
|  15 | /dev/vfio/433 |  0   | 0000:6b:00.0 | 0000:6b:01.7 | sym;asym |
|  16 | /dev/vfio/434 |  0   | 0000:6b:00.0 | 0000:6b:02.0 | sym;asym |
|  17 | /dev/vfio/435 |  0   | 0000:70:00.0 | 0000:70:00.1 |    dc    |
|  18 | /dev/vfio/436 |  0   | 0000:70:00.0 | 0000:70:00.2 |    dc    |
|  19 | /dev/vfio/437 |  0   | 0000:70:00.0 | 0000:70:00.3 |    dc    |
|  20 | /dev/vfio/438 |  0   | 0000:70:00.0 | 0000:70:00.4 |    dc    |

Information includes:

VFIO Group: This information is required when passing QAT resources to container
NODE: Which processor node the QAT resource comes from
PF BDF: Bus/device/function of QAT PF
VF BDF: Bus/device/function of QAT VF
SERVICES: Current services that are enabled on device

Update Configuration
We can update configuration using command similar to the following:
qat --config --policy 1 --mode 5
In the example above we update the policy setting to 1 and mode to asym;dc.

Note

If --policy or --mode is not specified, the default values are used.

This script will update the /etc/sysconfig/qat file and restart QAT kernel modules.

Note

Older QAT kernel modules and user space library may not support all mode settings included here.

This script includes logic to detect these issues and provide guidance.

Requirements for running script

Install Dependencies

For RPM-based distros:

sudo dnf install -y python3
sudo dnf install -y python3-venv
sudo dnf install -y python3-pip

Ensure /usr/bin/python exists and points to the installed python3 binary.

#!/bin/bash

# Check if Python 3 is installed
if command -v python3 &>/dev/null; then
    # Get the path to the Python 3 executable
    PYTHON3_PATH=$(which python3)

    # Check if /usr/bin/python already exists
    if [ -L /usr/bin/python ] || [ -e /usr/bin/python ]; then
        echo "A file or symbolic link already exists at /usr/bin/python."
        echo "Please remove or rename it before running this script."
        exit 1
    else
        # Create a symbolic link for Python 3
        sudo ln -s "$PYTHON3_PATH" /usr/bin/python
        echo "Symbolic link created for Python 3 at /usr/bin/python."
    fi
else
    echo "Python 3 is not installed or not found in the system PATH."
    exit 1
fi

For Ubuntu and other Debian-based distros, install these dependencies:

sudo apt-get install -y python3
sudo apt-get install -y python3-venv
sudo apt-get install -y python-is-python3
sudo apt-get install -y python3-pip
sudo apt-get install -y python3-packaging

Setup Virtual Environment Install Python packages

python -m venv myenv
source ~/myenv/bin/activate
pip install prettytable
deactivate

Copy script to system directory
This script is available here.

After copying to local directory, this can be moved to system directory.
mv ./qat /usr/bin sudo chmod 755 /usr/bin/qat

Standalone Mode

In Standalone Mode, there is no qat service running.

This is the preferred way to run on containers where a subset of VFs are explicitly passed through and no qatmgr managed resource manager is needed.

Each standalone process assumes VF devices which it can’t open are in use by another process and looks for unopened VFs.

In this mode, the /quickassist/utilities/service/qat_init.sh script can be called on the host to bind VFs to vfio-pci and to enable services based on the Services Enabled parameter.

In this mode the Policy parameter should not be set in /etc/sysconfig/qat. If it is set, it will be ignored.

Instead an environment variable QAT_POLICY can be set.

Valid values are the same as for POLICY, except that for QAT_POLICY=0 a maximum of 6 VFs will be allocated.

Running Without qat_init.sh

In some scenarios, it is not possible to use qat_init.sh. This section provides details on the steps that need to be performed in this usage.

Add vfio-pci.ids=8086:4941 to the kernel command line.

Note

Depending on your device, it might be 4941, 4943 or 4947. This is the device ID of the VFs.

If your PF is 4940, the VF device ID is 4941, if the PF is 4942, the VF is 4943.

See Supported Devices for additional details.
Obtain the Domain Bus Device Function (BDF) of the QAT PF with lspci
This will be used when accessing sysfs.
echo `(lspci -nd 8086:4940 && lspci -nd 8086:4942 && lspci -nd 8086:4946)`
Load the qat driver if not loaded
modprobe qat_4xxx
Load vfio-pci driver if not loaded
modprobe vfio-pci

Configure devices for required service.

Note

This needs to be repeated for each device in the system.

In the example below, we are enabling compression.
echo down > /sys/bus/pci/devices/<BDF of PF>/qat/state
echo dc > /sys/bus/pci/devices/<BDF of PF>/qat/cfg_services
echo up > /sys/bus/pci/devices/<BDF of PF>/qat/state

Enable VFs
Note

This needs to be repeated for each device in the system.
echo 16 > /sys/bus/pci/devices/<BDF of PF>/sriov_numvfs

Set permissions/groups to vfio groups

chown root:qat /dev/vfio/[0-9]*
chmod g+rw /dev/vfio/[0-9]*

Display Current Configuration

Note

This functionality is also available in the qat script described in this section.

Standalone Mode (without the qat service).

The script below can be used to view the PF/VF service configuration.

#!/bin/bash

printf "%-13s | %-5s | %-12s | %-12s | %-10s\n"  "VFIO GROUP" "NODE"   \
"PF BDF" "VF BDF" "SERVICES"
echo "--------------------------------------------------------------"

for vfio_group in /dev/vfio/*; do
    if [ $vfio_group = "/dev/vfio/vfio" ]; then
        continue
    fi
    if [ $vfio_group = "/dev/vfio/devices" ]; then
        continue
    fi

    group=${vfio_group##*/}
    # assume one bdf per iommu group
    bdf=$(ls /sys/kernel/iommu_groups/$group/devices/)
    vendor=$(cat /sys/kernel/iommu_groups/$group/devices/$bdf/vendor)
    node=$(cat /sys/kernel/iommu_groups/$group/devices/$bdf/numa_node)
    did=$(cat /sys/kernel/iommu_groups/$group/devices/$bdf/device)

    if [ "$vendor" != "0x8086" ]; then
        continue
    fi

    if [ "$did" != "0x4941" ] && [ "$did" != "0x4943" ] && [ "$did" != "0x4947" ]; then
        continue
    fi

    regex='([a-z0-9]+):([a-z0-9]+):.*'
    [[ $bdf =~ $regex ]]
    pf_domain=${BASH_REMATCH[1]}
    pf_bus=${BASH_REMATCH[2]}
    pf_bdf="$pf_domain:$pf_bus:00.0"

    printf "%-15s %-7s %-14s %-14s %-10s\n"  "$vfio_group"  "$node"    \
    "$pf_bdf"   "$bdf"                                                 \
    "$(cat /sys/bus/pci/devices/$pf_bdf/qat/cfg_services)"

done

Custom Configuration

The configuration of services can also be set per PF.

Instructions are provided in the following link: https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-driver-qat.

Warning

If custom configuration is set per PF following these instructions, then neither ServicesEnabled nor POLICY should be set in /etc/sysconfig/qat.
It is not deterministic which instances a process receives in this configuration.