Quick Start

Contents

Quick Start#

Low-Level C API#

We walk through an example to learn the basic workflow of Intel® Query Processing Library (Intel® QPL) low-level C API. The key agent of this API is the qpl_job data structure. To work with Intel QPL low-level C API, the application will need to:

  1. Allocate buffer required for compression.

  2. Query the memory size required for the job structure.

  3. Allocate memory for the job structure according to the queried size.

  4. Initialize the job structure and fill in necessary parameters.

  5. Pass the job structure (along with the allocated memory) to Intel QPL.

  6. When the operations are finished, free the resources.

The example below compresses and decompresses data with Deflate dynamic Huffman encoding via Intel QPL low-level C API. For our purpose to understand the workflow, we only focus on the compression part here. See the comments after the code block.

 1/*******************************************************************************
 2 * Copyright (C) 2022 Intel Corporation
 3 *
 4 * SPDX-License-Identifier: MIT
 5 ******************************************************************************/
 6
 7//* [QPL_LOW_LEVEL_COMPRESSION_EXAMPLE] */
 8
 9#include <iostream>
10#include <memory>
11#include <vector>
12
13#include "qpl/qpl.h"
14
15#include "examples_utils.hpp" // for argument parsing function
16
17constexpr const uint32_t source_size = 1000;
18
19auto main(int argc, char** argv) -> int {
20    std::cout << "Intel(R) Query Processing Library version is " << qpl_get_library_version() << ".\n";
21
22    // Default to Software Path
23    qpl_path_t execution_path = qpl_path_software;
24
25    // Get compression buffer size estimate
26    const uint32_t compression_size = qpl_get_safe_deflate_compression_buffer_size(source_size);
27    if (compression_size == 0) {
28        std::cout << "Invalid source size. Source size exceeds the maximum supported size.\n";
29        return 1;
30    }
31
32    // Source and output containers
33    std::vector<uint8_t> source(source_size, 5);
34    std::vector<uint8_t> destination(compression_size, 4);
35    std::vector<uint8_t> reference(source_size, 7);
36
37    std::unique_ptr<uint8_t[]> job_buffer;
38    uint32_t                   size = 0;
39
40    // Job initialization
41    qpl_status status = qpl_get_job_size(execution_path, &size);
42    if (status != QPL_STS_OK) {
43        std::cout << "An error " << status << " acquired during job size getting.\n";
44        return 1;
45    }
46
47    job_buffer   = std::make_unique<uint8_t[]>(size);
48    qpl_job* job = reinterpret_cast<qpl_job*>(job_buffer.get());
49
50    status = qpl_init_job(execution_path, job);
51    if (status != QPL_STS_OK) {
52        std::cout << "An error " << status << " acquired during job initializing.\n";
53        return 1;
54    }
55
56    // Performing a compression operation
57    job->op            = qpl_op_compress;
58    job->level         = qpl_default_level;
59    job->next_in_ptr   = source.data();
60    job->next_out_ptr  = destination.data();
61    job->available_in  = source_size;
62    job->available_out = static_cast<uint32_t>(destination.size());
63    job->flags         = QPL_FLAG_FIRST | QPL_FLAG_LAST | QPL_FLAG_DYNAMIC_HUFFMAN | QPL_FLAG_OMIT_VERIFY;
64
65    // Compression
66    status = qpl_execute_job(job);
67    if (status != QPL_STS_OK) {
68        std::cout << "An error " << status << " acquired during compression.\n";
69        return 1;
70    }
71
72    const uint32_t compressed_size = job->total_out;
73
74    // Freeing resources
75    status = qpl_fini_job(job);
76    if (status != QPL_STS_OK) {
77        std::cout << "An error " << status << " acquired during job finalization.\n";
78        return 1;
79    }
80
81    return 0;
82}
83
84//* [QPL_LOW_LEVEL_COMPRESSION_EXAMPLE] */

The application only needs to include one header file qpl/qpl.h, which specifies the prototypes of all the functions.

At line 26, we call qpl_get_safe_deflate_compression_buffer_size() to estimate the size of the output buffer required for compression operations. The size is calculated based on the input data size and return 0 if the source size exceeds the maximum supported size. For more details, refer to Estimating the Size of the Deflate Compression Buffer page.

At line 34, we allocate the output buffer based on the estimation we obtained earlier.

At line 41, we call qpl_get_job_size() to query the required memory size based on the specified execution path.

At lines 47-48, we allocate memory according to the returned value of size. Note that the value of size is greater than the size of the job structure qpl_job. The leading portion of the allocated memory is used to store the job structure, while the remaining portion is a buffer for internal usages.

At line 50, we call qpl_init_job() to initialize the job structure and buffer, then we fill in necessary parameters at lines 57 to 63.

The job structure and the allocated buffer are passed to Intel QPL at line 66. After qpl_execute_job() completes successfully, we can retrieve the results stored in the job structure.

Finally, we call qpl_fini_job() at line 75 to free the resources.

In order to build the library and all the examples, including the one above, follow steps at Building the Library. Compiled examples then would be located in <qpl_library>/build/examples/low-level-api/.

Alternatively, in order to build compression_example.cpp individually using existing Intel QPL installation, use:

g++ -I/<install_dir>/include -o compression_example compression_example.cpp /<install_dir>/lib64/libqpl.a -ldl

Attention

Intel QPL could be also used from C applications. This would still require C++ runtime library installed on the system. You would also need to add -lstdc++ if you are using the static library libqpl.a.

On Linux, if you installed Intel QPL system wide, you can use the dynamic library to compile the examples with:

g++ -I/<install_dir>/include -o compression_example compression_example.cpp -lqpl

In order to build an example using pkg-config for the dynamic library, set the PKG_CONFIG_PATH and compile the example using qpl.pc:

g++ `pkg-config --cflags --libs qpl` -o compression_example compression_example.cpp

To run the example on the Hardware Path (see Execution Paths), use:

./compression_example hardware_path

Attention

Either sudo privileges or elevated permissions are required to initialize Intel QPL job with qpl_path_hardware.

Refer to the Accelerator Configuration section for more details about getting permissions.

Attention

With the Hardware Path, the user must either place the libaccel-config library in /usr/lib64/ or specify the location of libaccel-config in LD_LIBRARY_PATH for the dynamic loader to find it.

Attention

In the example above we do not set qpl_job.numa_id value.

If Intel QPL version is `< 1.6.0`, the library will auto-detect NUMA node of the calling process and use Intel® In-Memory Analytics Accelerator (Intel® IAA) device(s) located on the same NUMA node.

If Intel QPL version is `>= 1.6.0`, the library will use Intel IAA device(s) located on the socket of the calling thread.

Refer to Devices Selection and NUMA Support section for more details.

To run the example on the Software Path (see Execution Paths), use:

./compression_example software_path

To run the example on the Auto Path (see Execution Paths), use:

./compression_example auto_path

Attention

If Intel QPL is built with -DDYNAMIC_LOADING_LIBACCEL_CONFIG=OFF (see Available Build Options for details), replace -ldl with -laccel-config in the compilation command. The user must either place libaccel-config in /usr/lib64/ or specify the location of libaccel-config (for example, using LD_LIBRARY_PATH and LIBRARY_PATH).

Refer to Developer Guide for more information about Intel QPL low-level C API. For more examples, see Low-Level C API Examples.

Contents