Asynch Mode for NGINX*

Installation

The Asynch Mode for NGINX (asynch_nginx) package can be installed using the instructions in this section.

Note

The instructions here are to be used when using asynch_nginx with qatlib.

If using the out-of-tree QAT package, refer to the latest installation instructions available at the asynch_mode_nginx GitHub repository.

Package Dependencies

This section includes instructions on installing the required packages dependencies.

RPM-based package dependencies

Note

This section is required for RPM-based Linux distributions such as RHEL*, CentOS*, and Fedora*.

  1. Install the RPM-based package dependencies:

    sudo dnf groupinstall "Development Tools"
sudo dnf install -y gcc
sudo dnf install -y zlib
sudo dnf install -y openssl-devel
sudo dnf install -y zlib-devel
sudo dnf install -y make
sudo dnf install -y pcre
sudo dnf install -y pcre-devel

DEB-based package dependencies

Note

This section is required for DEB-based Linux distributions such as Ubuntu*.

  1. Install the DEB-based package dependencies:

    sudo apt-get update
sudo apt-get install -y zlib1g-dev
sudo apt-get install -y libssl-dev
sudo apt-get install -y libpcre3
sudo apt-get install -y libpcre3-dev

Pre-Requisites

Prior to installing async_nginx ensure the following packages have been installed:

If using compression service, ensure the following package has been installed:

Installation instructions

  1. Obtain the source package for async_nginx by running the following commands:

    cd ~
git clone https://github.com/intel/asynch_mode_nginx.git
cd asynch_mode_nginx/

  2. Configure the async_nginx application by running the below commands:

    Important

    Improve Website Performance Using Compression

    • Website performance is partially dependent on the size of the files that user browser must download.

    • Reducing the size of these files can increase the speed of the website.

    • Using QAT for compression provides lower latency, lower power utilization, and reduced CPU utilization when compared to using cores and the default gzip library.

    To configure with support for both encryption and data compression:

    export NGINX_INSTALL_DIR=/usr/local/nginx_qat
sudo mkdir -p $NGINX_INSTALL_DIR
./configure \
   --prefix=$NGINX_INSTALL_DIR \
   --with-http_ssl_module \
   --add-dynamic-module=modules/nginx_qatzip_module \
   --add-dynamic-module=modules/nginx_qat_module/ \
   --with-cc-opt="-DNGX_SECURE_MEM  -Wno-error=deprecated-declarations" \
   --with-ld-opt="-lqatzip -lz"

    To configure with support for just encryption:

    export NGINX_INSTALL_DIR=/usr/local/nginx_qat
sudo mkdir -p $NGINX_INSTALL_DIR
./configure \
   --prefix=$NGINX_INSTALL_DIR \
   --with-http_ssl_module \
   --add-dynamic-module=modules/nginx_qat_module/ \
   --with-cc-opt="-DNGX_SECURE_MEM  -Wno-error=deprecated-declarations"

  3. Build and install the async_nginx application by running the following commands:

    make -j
sudo make install

Configuration

QAT Configuration

  • At least one QAT endpoint needs to have asymmetric/symmetric (asym;sym) services enabled to utilize cryptography.

  • At least one QAT endpoint needs to have compression (dc) service enabled to utilize compression.

Important

Newer versions of the QAT Kernel modules include support for additional service configuration options, including asymmetric/compression (asym;dc). This mode is ideal if both compression and asymmetric crypto support is required.

Refer to Services Enabled section of the QATlib User’s Guide for additional details.

Refer to the following sections for details on configuring qatlib.

NGINX Configuration

This section includes details on configuring NGINX to utilize QAT. Example configuration file is also available for download.

General Section

Here is example of the general section of the NGINX configuration file:

user  www www;
worker_processes  auto;
error_log  logs/error.log;
pid        logs/nginx.pid;
worker_rlimit_nofile 500000;

load_module modules/ngx_http_qatzip_filter_module.so;
load_module modules/ngx_ssl_engine_qat_module.so;

events {
   use epoll;
   worker_connections 102400;
   accept_mutex off;
}

# Enable QAT engine in heuristic mode.
ssl_engine {
   use_engine qatengine;
   default_algorithms ALL;
   qat_engine {
      qat_offload_mode async;
      qat_notify_mode poll;
      qat_poll_mode heuristic;
      qat_sw_fallback off;
   }
}

Key parameters include:

worker_processes  auto; : This sets the number of NGINX worker processes to the number of cores available in the system/virtual machine/container.

Important

For optimal performance:

  • Ensure there are enough QAT VFs configured for each service (asym, dc) being utilized.

  • If both asym and dc are used, the suggested POLICY setting is 2.

  • If just one service is used, the suggested POLICY setting is 1.

Refer to Configuration and Tuning for details.

worker_rlimit_nofile 500000; : Sets numbers of open files that each worker process can have open at one time.

load_module modules/ngx_http_qatzip_filter_module.so; : Loads QATzip NGINX module. This is required when compression is enabled.

load_module modules/ngx_ssl_engine_qat_module.so; : Loads QAT_Engine NGINX module. This is required when encryption is enabled.

Compression Settings

Compression settings are included in the http block of the NGINX configuration file. Here is example compression configuration.

http {
 include       mime.types;
 default_type  application/octet-stream;
 gzip on;
 gzip_min_length     128;
 gzip_comp_level     1;
 gzip_types  text/css text/javascript text/xml text/plain text/x-component application/javascript application/json application/xml application/rss+xml font/truetype font/opentype application/vnd.ms-fontobject image/svg+xml;
 gzip_vary            on;
 gzip_disable        "msie6";
 gzip_http_version   1.0;

 qatzip_sw failover;
 qatzip_min_length 128;
 qatzip_comp_level 1;
 qatzip_buffers 16 8k;
 qatzip_types text/css text/javascript text/xml text/plain text/x-component application/javascript application/json application/xml application/rss+xml font/truetype font/opentype application/vnd.ms-fontobject image/svg+xml application/octet-stream image/jpeg;
 qatzip_chunk_size   64k;
 qatzip_stream_size  256k;
 qatzip_sw_threshold 256;

Key parameters include:

gzip on; : Enables compression

qatzip_min_length : Sets the minimum compression size that would be sent to QAT. Requests smaller than this threshold are compressed using gzip.

qat_comp_level : Sets the compression level that should be used. This can be set from 1 to 9. Higher compression levels allow for better compression ratios.

HTTPS Settings

We can configure NGINX to use QAT for encryption with updates to the HTTPS server section. An example of this configuration is:

server {
     listen       443 ssl asynch;
     server_name  localhost;

     ssl_protocols       TLSv1.2;

     ssl_certificate      /usr/local/nginx_qat_module/certs/server.crt;
     ssl_certificate_key  /usr/local/nginx_qat_module/certs/server.key;
     access_log off;
 }

Key items to call out here include:

listen       443 ssl asynch; : This listens on port 443 and enables asynchronous access to OpenSSL. asynch is required for optimal performance.

ssl_certificate : Specifies the certificate to be used for public key encryption.

ssl_certificate_key : Specifies private key to be used for public key encryption.

Benchmarking NGINX Server

When benchmarking NGINX server we can focus on two main areas:

  • Connections Per Second(CPS)

  • Throughput

With CPS testing, we are interested in how many connections per second can the NGINX server process. This test highlights the value QAT brings to Public Key Encryption.

Throughput tests would focus on the data being encrypted and optionally compressed. This test highlights the value qat_sw brings to the symmetric crypto operations and the value QAT brings to compression.

CPS Testing

The following scripts can be used benchmark CPS:

connection_test.sh

Script invokes openssl s_time to create connection requests to the specified NGINX server. The script has has the following parameters that can be changed to meet user specific test requirements:

  • ip_address: IP address of NGINX server

  • _time: How long to run test specified in seconds

  • clients: Number of simultaneous connections

  • port: Port number of NGINX server

After the _time expires, resulting number of connections per second is displayed to screen.

Here is an example of the output from the script:

[sdp@fl6u41 nginx]$ ./connection_test.sh localhost

Location of OpenSSL:
IP Addresses:                  localhost
Time:                          10
Clients:                       300
Port:                          443
Cipher:                        AES128-SHA
Connections per second:      12424 CPS
Finished in 12 seconds (0 seconds waiting for procs to start)

cps_test.py

This is a wrapper python script around the connection_test.sh. The test script is called in an endless loop capturing test results (Connections per Second) to log file. In the example script we are writing to a textfile_collector file for Prometheus. This allows us to monitor results via Granfana dashboard. If this is not required, simple modifications to the script could be made to output to stdout or other log file.

Throughput Testing

For throughput testing we are using the wrk application to generate client traffic requests to the specified NGINX server. These requests include the downloading of a file from the NGINX server.

The example test script is:

Prior to running the script, ensure the NGINX server includes the file to be used for testing.

The following command line arguments are available:

$ python3 wrk_test.py -h
   usage: wrk_test.py [-h] [-f FILE] [-a ADDRESS] [-t THREADS] [-n CONNECTIONS] [-d DURATION] [-R RATE] [-c COMPRESS] [-i ITERATIONS]

   optional arguments:
   -h, --help            show this help message and exit
   -f FILE, --file FILE  File to download
   -a ADDRESS, --address ADDRESS
                           address of nginx server
   -t THREADS, --threads THREADS
                           Number of threads (default 50)
   -n CONNECTIONS, --connections CONNECTIONS
                           Number of connections per thread (default 400)
   -d DURATION, --duration DURATION
                           Duration of the test in seconds (default 10)
   -R RATE, --rate RATE  Rate (default 10000)
   -c COMPRESS, --compress COMPRESS
                           Allow compressed files (default = False)
   -i ITERATIONS, --iterations ITERATIONS
                           Number of times to loop (default = 35)

For each iteration, the following telemetry values are captured from the wrk output:

  • requests

  • requests per second

  • size of data read

  • transfers per second

  • p50 latency

  • p75 latency

  • p99 latency

  • average latency

The results are logged to textfile_collector file for Prometheus. This allows us to monitor results via Granfana dashboard. If this is not required, simple modifications to the script could be made to output to stdout or other log file.

Here is example of the contents of the data collected:

nginx_request_count 3462068
nginx_requests_per_second 9893.86
nginx_data_read 62.13000000000003
nginx_transfers_per_second 181.64
nginx_p50_latency 1.98
nginx_p75_latency 3.11
nginx_p99_latency 206.08
nginx_ave_latency 8.24
nginx_connect_errors 0
nginx_timeout_errors 0