Run Containers using Intel® QAT Acceleration
This section includes details on how to launch a container image with QAT support.
Topics include:
Command Line Parameters
Assigning QAT Resources
Tuning Parameters
Command Line Parameters
Here are list of required and optional command line arguments when launching containers with QAT support.
memlock (REQUIRED)
memlock must be enabled within the container. This can be done with the following parameter:
--cap-add=IPC_LOCK
Refer to *docker* docs for additional information.
QAT_POLICY (RECOMMENDED)
It is important to specify the QAT_POLICY for the container. This affects how resources are allocated. For most configurations, setting QAT_POLICY to 1 is appropriate.
--env QAT_POLICY=1
Refer to QATlib User’s Guide for additional details on POLICY settings.
Security Configuration (OPTIONAL)
Security options for the container is set using the --security-opt
parameter.
In examples below this parameter is set to:
--security-opt seccomp=unconfined --security-opt apparmor=unconfined
Including these parameters turns off seccomp confinement for the container and disables apparmor within the container.
Refer to *docker* docs for additional information.
CPU Pinning (OPTIONAL)
This allows locking of CPUs the container will run. The can be done with the following parameter:
--cpuset-cpus 10-20
This would result in the the container images running on cores 10 through 20.
Important
For optimal performance ensure the cores that the container is running on are on the same NUMA node as the QAT devices
Refer to *docker* docs for additional information.
Memory Node Pinning
This parameter allows controlling which memory nodes to allow execution. This can be done as follows:
--cpuset-mems="0"
This would mean processes in the container can only use memory from memory node 0.
Refer to *docker* docs for additional information.
Providing QAT Resources to Container
This section describes how QAT resources are provided to the container images.
The the examples below, the following command line parameter is used to provide QAT resources to the container image.
$(for i in `ls /dev/vfio/*`; do echo --device $i; done)
This results in all QAT resources being allocated to the container image. In this case only one container image that uses QAT resources can run at a time as this container is consuming all available QAT resources.
In order to support multiple containers that utilize QAT we need to provide a subset of VFIO IDs instead of IDs. This raises the question of which VFIO IDS to provide and how many.
To answer the question of which VFIOs to provide, we can use this configuration script to obtain the information.
[sdp@fl6u41 ~]$ qatlistconfig.sh
VFIO GROUP | NODE | PF BDF | VF BDF | SERVICES
--------------------------------------------------------------
/dev/vfio/449 0 0000:6b:00.0 0000:6b:00.1 sym;asym
/dev/vfio/450 0 0000:6b:00.0 0000:6b:00.2 sym;asym
/dev/vfio/451 0 0000:6b:00.0 0000:6b:00.3 sym;asym
/dev/vfio/452 0 0000:6b:00.0 0000:6b:00.4 sym;asym
/dev/vfio/453 0 0000:6b:00.0 0000:6b:00.5 sym;asym
/dev/vfio/454 0 0000:6b:00.0 0000:6b:00.6 sym;asym
/dev/vfio/455 0 0000:6b:00.0 0000:6b:00.7 sym;asym
/dev/vfio/456 0 0000:6b:00.0 0000:6b:01.0 sym;asym
/dev/vfio/457 0 0000:6b:00.0 0000:6b:01.1 sym;asym
/dev/vfio/458 0 0000:6b:00.0 0000:6b:01.2 sym;asym
/dev/vfio/459 0 0000:6b:00.0 0000:6b:01.3 sym;asym
/dev/vfio/460 0 0000:6b:00.0 0000:6b:01.4 sym;asym
/dev/vfio/461 0 0000:6b:00.0 0000:6b:01.5 sym;asym
/dev/vfio/462 0 0000:6b:00.0 0000:6b:01.6 sym;asym
/dev/vfio/463 0 0000:6b:00.0 0000:6b:01.7 sym;asym
/dev/vfio/464 0 0000:6b:00.0 0000:6b:02.0 sym;asym
/dev/vfio/465 0 0000:70:00.0 0000:70:00.1 dc
/dev/vfio/466 0 0000:70:00.0 0000:70:00.2 dc
/dev/vfio/467 0 0000:70:00.0 0000:70:00.3 dc
/dev/vfio/468 0 0000:70:00.0 0000:70:00.4 dc
/dev/vfio/469 0 0000:70:00.0 0000:70:00.5 dc
/dev/vfio/470 0 0000:70:00.0 0000:70:00.6 dc
/dev/vfio/471 0 0000:70:00.0 0000:70:00.7 dc
In the output above we can see which VFIO IDs support dc
. Next questions would include:
how many VFIO device IDs should be provided
should they come from the same QAT PF, or be spread across multiple end-points.
The answer to these questions will depend on resources available and performance targets for individual container images.
As we can see setting this up can be quite complex. This complexity can be mitigated by using orchestration software that hides the complexity of assigning QAT resources to the container image.
Important
When providing subset of available VFIO device IDs to container, it is very important to include QAT_POLICY command line parameter when launching the container.
Refer to Orchestration Software for additional information.
Tuning Parameters
Container Runtime Memory Lock Limit (REQUIRED)
This section assigns the memory lock limit for the container. How much memory is required depends on the application as well as memory requirements for the QAT library (qatlib) qatlib requires 4MB per VF of locked memory.
Create a systemd directory for the docker service.
sudo mkdir -p /etc/systemd/system/docker.service.d
Navigate to the docker service directory.
cd /etc/systemd/system/docker.service.d
Create a file name memlock.conf.
sudo vi memlock.conf
Add the following variable to the memlock.conf file:
[Service] LimitMEMLOCK=16777216
Note
Value defined here depends on your applications needs plus memory requirements for qatlib.
Save the file.
Flush the changes.
sudo systemctl daemon-reload
Restart Docker.
sudo systemctl restart docker