Run Containers using Intel® QAT Acceleration

This section includes details on how to launch a container image with QAT support.

Topics include:

  • Command Line Parameters

  • Assigning QAT Resources

  • Tuning Parameters

Command Line Parameters

Here are list of required and optional command line arguments when launching containers with QAT support.

memlock (REQUIRED)

memlock must be enabled within the container. This can be done with the following parameter:

--cap-add=IPC_LOCK

Refer to *docker* docs for additional information.

Security Configuration (OPTIONAL)

Security options for the container is set using the --security-opt parameter.

In examples below this parameter is set to:

--security-opt seccomp=unconfined
--security-opt apparmor=unconfined

Including these parameters turns off seccomp confinement for the container and disables apparmor within the container.

Refer to *docker* docs for additional information.

CPU Pinning (OPTIONAL)

This allows locking of CPUs the container will run. The can be done with the following parameter:

--cpuset-cpus 10-20

This would result in the the container images running on cores 10 through 20.

Important

For optimal performance ensure the cores that the container is running on are on the same NUMA node as the QAT devices

Refer to *docker* docs for additional information.

Memory Node Pinning

This parameter allows controlling which memory nodes to allow execution. This can be done as follows:

--cpuset-mems="0"

This would mean processes in the container can only use memory from memory node 0.

Refer to *docker* docs for additional information.

Providing QAT Resources to Container

This section describes how QAT resources are provided to the container images.

The the examples below, the following command line parameter is used to provide QAT resources to the container image.

$(for i in `ls /dev/vfio/*`; do echo --device $i; done)

This results in all QAT resources being allocated to the container image. In this case only one container image that uses QAT resources can run at a time as this container is consuming all available QAT resources.

In order to support multiple containers that utilize QAT we need to provide a subset of VFIO IDs instead of IDs. This raises the question of which VFIO IDS to provide and how many.

To answer the question of which VFIOs to provide, we can use this configuration script to obtain the information.

[sdp@fl6u41 ~]$ qatlistconfig.sh
VFIO GROUP    | NODE  | PF BDF       | VF BDF       | SERVICES
--------------------------------------------------------------
/dev/vfio/449   0       0000:6b:00.0   0000:6b:00.1   sym;asym
/dev/vfio/450   0       0000:6b:00.0   0000:6b:00.2   sym;asym
/dev/vfio/451   0       0000:6b:00.0   0000:6b:00.3   sym;asym
/dev/vfio/452   0       0000:6b:00.0   0000:6b:00.4   sym;asym
/dev/vfio/453   0       0000:6b:00.0   0000:6b:00.5   sym;asym
/dev/vfio/454   0       0000:6b:00.0   0000:6b:00.6   sym;asym
/dev/vfio/455   0       0000:6b:00.0   0000:6b:00.7   sym;asym
/dev/vfio/456   0       0000:6b:00.0   0000:6b:01.0   sym;asym
/dev/vfio/457   0       0000:6b:00.0   0000:6b:01.1   sym;asym
/dev/vfio/458   0       0000:6b:00.0   0000:6b:01.2   sym;asym
/dev/vfio/459   0       0000:6b:00.0   0000:6b:01.3   sym;asym
/dev/vfio/460   0       0000:6b:00.0   0000:6b:01.4   sym;asym
/dev/vfio/461   0       0000:6b:00.0   0000:6b:01.5   sym;asym
/dev/vfio/462   0       0000:6b:00.0   0000:6b:01.6   sym;asym
/dev/vfio/463   0       0000:6b:00.0   0000:6b:01.7   sym;asym
/dev/vfio/464   0       0000:6b:00.0   0000:6b:02.0   sym;asym
/dev/vfio/465   0       0000:70:00.0   0000:70:00.1   dc
/dev/vfio/466   0       0000:70:00.0   0000:70:00.2   dc
/dev/vfio/467   0       0000:70:00.0   0000:70:00.3   dc
/dev/vfio/468   0       0000:70:00.0   0000:70:00.4   dc
/dev/vfio/469   0       0000:70:00.0   0000:70:00.5   dc
/dev/vfio/470   0       0000:70:00.0   0000:70:00.6   dc
/dev/vfio/471   0       0000:70:00.0   0000:70:00.7   dc

In the output above we can see which VFIO IDs support dc. Next questions would include:

  • how many VFIO device IDs should be provided

  • should they come from the same QAT PF, or be spread across multiple end-points.

The answer to these questions will depend on resources available and performance targets for individual container images.

As we can see setting this up can be quite complex. This complexity can be mitigated by using orchestration software that hides the complexity of assigning QAT resources to the container image.

Important

When providing subset of available VFIO device IDs to container, it is very important to include QAT_POLICY command line parameter when launching the container.

Refer to Orchestration Software for additional information.

Tuning Parameters

Container Runtime Memory Lock Limit (REQUIRED)

This section assigns the memory lock limit for the container. How much memory is required depends on the application as well as memory requirements for the QAT library (qatlib) qatlib requires 4MB per VF of locked memory.

  1. Create a systemd directory for the docker service.

    sudo mkdir -p /etc/systemd/system/docker.service.d

  2. Navigate to the docker service directory.

    cd /etc/systemd/system/docker.service.d

  3. Create a file name memlock.conf.

    sudo vi memlock.conf

  4. Add the following variable to the memlock.conf file:

    [Service]
LimitMEMLOCK=16777216

    Note

    Value defined here depends on your applications needs plus memory requirements for qatlib.

  5. Save the file.

  6. Flush the changes.

    sudo systemctl daemon-reload

  7. Restart Docker.

    sudo systemctl restart docker