The DPC++ Runtime Plugin Interface (PI) is an interface layer between the
device-agnostic part of DPC++ runtime and the device-specific runtime layers
which control execution on devices. It employs the “plugin” mechanism to bind
to the device specific runtime layers similar to what is used by libomptarget
The picture below illustrates the placement of PI within the overall DPC++
runtime stack. Dotted lines show components or paths which are not yet available
in the runtime, but are likely to be developed.
The plugin interface and the discovery process behind it allows to dynamically
plug in implementations based on OpenCL and “native” runtime for a particular
device – such as OpenCL for
FPGA devices or native runtimes for GPUs. Implementations of the PI are
“plugins” - dynamic libraries or shared objects which expose a number of entry
points implementing the PI interface. The DPC++ runtime collects those function
pointers into a PI interface dispatch table - one per plugin - and uses this
table to dispatch to the device(s) covered by the corresponding plugin.
PI is based on a subset of OpenCL 1.2 runtime specification, it follows OpenCL’s
platform, execution and memory models in all aspects except for those explicitly
mentioned in this document. Some of PI API types and functions have exact
matches in OpenCL. Whenever there is such a match, the semantics also fully
match unless the differences are explicitly specified in this document. While
PI has roots in OpenCL, it does have many differences, and the gap is likely
to grow, for example in areas of memory model and management, program
Discovery and linkage of PI implementations
Device discovery phase enumerates all available devices and their features by
querying underlying plugins found in the system. This process is performed when
all attached platforms or devices are queried in an application; for example,
during device selection.
Plugins are physically dynamic libraries or shared objects.
The process to discover plugins follows the following guidelines.
The DPC++ Runtime reads the names of the plugins from a configuration file
at a predetermined location (TBD - Add this location). These plugins are
searched at locations in env LD_LIBRARY_PATH on Linux and env PATH on Windows.
(TBD - Extend to search the plugins at a path relative to the SYCL Runtime
installation directory by using DT_RPATH on Linux. Similar functionality can be
achieved on Windows using SetDllDirectory. This will help avoiding extra setting
To avoid any issues with read-only access, an environment variable
SYCL_PI_CONFIG can be set to point to the configuration file which lists the
Plugin names. The enviroment variable if set overrides the predetermined
location’s config file. These Plugins are then be searched in LD_LIBRARY_PATH
locations. It is the developer’s responsibility to include the plugin names from
the predetermined location’s config file to enable discovery of all plugins.
(TBD - Extend to support search in DT_RPATH as above.)
In the current implementation the plugin names are hardcoded in the library.
Configuration file or env SYCL_PI_CONFIG is currently not being considered.
A trace mechanism is provided using env SYCL_PI_TRACE to log the discovery/
binding/ device enumeration process. Different levels of tracing can be achieved
with different values of SYCL_PI_TRACE.
SYCL_PI_TRACE=0x01 provides basic trace of plugins discovered and bound. It also
lists the device selector’s selected device information.
SYCL_PI_TRACE=0x02 provides trace of all PI calls made from the DPC++ runtime
with arguments and returned values.
SYCL_PI_TRACE=-1 lists all PI Traces above and more debug messages.
Plugin binary interface
Plugins should implement all the Interface APIs required for the PI Version
it supports. There is pi.def/
pi.h file listing all PI API names that
can be called by the specific version of Plugin Interface.
It exports a function - “piPluginInit” that returns the plugin details and
function pointer table containing the list of pointers to implemented Interface
Functions defined in pi.h.
In the future, this document will list the minimum set of Interface APIs
to be supported by Plugins. This will also require adding functionality to SYCL
Runtime to work with such limited functionality plugins.
(TBD - list and describe the symbols that a plugin must implement in order to
be picked up by the DPC++ runtime for offload.)
Binding a Plugin
The DPC++ Runtime loads all discovered Plugins and tries to bind them by calling
piPluginInit API for each loaded Plugin. The Plugins return the information of
supported PI version and the list of implemented PI API Function pointers.
(TBD - Use the PI API Version information and check for compatibility.
Extend to support version compatibility checks without loading the library.
Eg:Changing the plugin name to reflect the supported Plugin Interface version.)
The information of compatible plugins (with the Function Pointer Table) is
stored in the associated platforms during platform object construction.
The PI API calls are later forwarded using this information.
A plugin is said to “bind” after this process completes with no errors.
During device selection, the user can prefer selection of a device from a
specific Plugin or Backend using the env ONEAPI_DEVICE_SELECTOR. The correspondence between
a plugin and a ONEAPI_DEVICE_SELECTOR value is currently hardcoded in the runtime.
( TBD: Make this a part of configuration file).
Eg: ONEAPI_DEVICE_SELECTOR=opencl:* corresponds to OpenCL Plugin.
OpenCL plugin is a usual plugin from DPC++ runtime standpoint, but its loading
and initialization involves a nested discovery process which finds out available
OpenCL implementations. They can be installed either in the standard Khronos
ICD-compatible way (e.g. listed in files under /etc/OpenCL/vendors on
Linux) or not, and the OpenCL plugin can hook up with both.
TBD - implement and describe the nested OpenCL implementation discovery process
performed by the OpenCL plugin
Device enumeration by plugins
Devices from all bound plugins are queried and listed as and when required, eg:
during device selection in device_selector.
The trace shows the PI API calls made when using SYCL_PI_TRACE=-1.
(TBD - Add the trace to list all available devices when plugins are successfully
The plugins not chosen to be connected to should be unloaded. piInitializePlugins()
can be called to load and bound the necessary plugins. In addition, piTearDown()
can be called when plugins are not needed any more. It notifies each
plugin to start performing its own tear-down process such as global memory
deallocation. In the future, piTearDown() can include any other jobs that need to
be done before the plugin is unloaded from memory. Possibly, a
notification of the plugin unloading to lower-level plugins can be added so that
they can clean up their own memory [TBD].
After piTearDown() is called, the plugin can be safely unloaded by calling unload(),
which is going to invoke OS-specific system calls to remove the dynamic library
Each plugin should not create global variables that require non-trivial
destructor. Pointer variables with heap memory allocation is a good example
to be created at the global scope. A std::vector object is not. piTearDown
will take care of deallocation of these global variables safely.
PI API Specification
PI interface is logically divided into few subsets:
Core API which must be implemented by all plugins for DPC++ runtime to be
able to operate on the corresponding device. The core API further breaks down
Interoperability API which allows interoperability with underlying
runtimes such as OpenCL.
See pi.h header for the full list and
descriptions of PI APIs.
The Core OpenCL-based PI APIs
This subset defines functions representing core functionality,
such as device memory management, kernel creation and parameter setting,
enqueuing kernel for execution, etc. Functions in this subset fully match
semantics of the corresponding OpenCL functions, for example:
The Extension PI APIs
Those APIs don’t have OpenCL counter parts and require full specification. For
example, the function below selects the most appropriate device binary based
on runtime information and the binary’s characteristics
pi_device_binary * binaries,
pi_device_binary * selected_binary);
PI also defines few types and string tags to describe a device binary image.
Those are used to communicate to plugins information about the images where it
is needed, currently only in the above function. The main
pi_device_binary, whose detailed description can also be found
in the header. The layout of this type strictly matches the layout of the
corresponding device binary descriptor type defined in the
clang-offload-wrapper tool which wraps device binaries into a host
object for further linkage. The wrapped binaries reside inside this descriptor
in a data section.
The Interoperability PI APIs
These are APIs needed to implement DPC++ runtime interoperability with
underlying “native” device runtimes such as OpenCL.
Interoperability extension APIs have been added to get native runtime handles
from the backend-agnostic PI Objects or to create PI Objects using the native
PI Extension mechanism
TBD This section describes a mechanism for DPC++ or other runtimes to detect
availability of and obtain interfaces beyond those defined by the PI dispatch.
TBD Add API to query PI version supported by plugin at runtime.