Features

Easy-to-use Python API

Intel® Extension for PyTorch* provides simple frontend Python APIs and utilities to get performance optimizations such as operator optimization.

Check the API Documentation for API functions description and Examples for usage guidance.

Channels Last

Compared with the default NCHW memory format, using channels_last (NHWC) memory format can further accelerate convolutional neural networks. In Intel® Extension for PyTorch*, NHWC memory format has been enabled for most key CPU and GPU operators. More detailed information is available at Channels Last.

Intel® Extension for PyTorch* automatically converts a model to channels last memory format when users optimize the model with ipex.optimize(model). With this feature, users do not need to manually apply model=model.to(memory_format=torch.channels_last) anymore. However, models running on Intel® Data Center GPU Flex Series will choose oneDNN layout, so users still need to manually convert the model and data to channels last format. More detailed information is available at Auto Channels Last.

Auto Mixed Precision (AMP)

Benefiting from less memory usage and computation, low precision data types typically speed up both training and inference workloads. On GPU side, support of BFloat16 and Float16 are both available in Intel® Extension for PyTorch*. BFloat16 is the default low precision floating data type when AMP is enabled.

Detailed information of AMP for GPU are available at Auto Mixed Precision (AMP) on GPU.

Quantization

Intel® Extension for PyTorch* currently supports imperative mode and TorchScript mode for post-training static quantization on GPU. This section illustrates the quantization workflow on Intel GPUs.

Check more detailed information for INT8 Quantization.

On Intel® GPUs, Intel® Extension for PyTorch* also provides FP8 Quantization. Check more detailed information for FP8 Quantization.

Distributed Training

To meet demands of large scale model training over multiple devices, distributed training on Intel® GPUs is supported. Two alternative methodologies are available. Users can choose either to use PyTorch native distributed training module, Distributed Data Parallel (DDP), with Intel® oneAPI Collective Communications Library (oneCCL) support via Intel® oneCCL Bindings for PyTorch (formerly known as torch_ccl) or use Horovod with Intel® oneAPI Collective Communications Library (oneCCL) support (Prototype).

For more detailed information, check DDP and Horovod (Prototype).

DLPack Solution

DLPack defines a stable in-memory data structure for sharing tensors among frameworks. It enables sharing of tensor data without copying when interoparating with other libraries. Intel® Extension for PyTorch* extends DLPack support in PyTorch* for XPU device particularly.

For more detailed information, check DLPack Solution.

DPC++ Extension

Intel® Extension for PyTorch* provides C++ APIs to get SYCL queue and configure floating-point math mode.

Check the API Documentation for the details of API functions. DPC++ Extension describes how to write customized DPC++ kernels with a practical example and build it with setuptools and CMake.

Advanced Configuration

The default settings for Intel® Extension for PyTorch* are sufficient for most use cases. However, if you need to customize Intel® Extension for PyTorch*, advanced configuration is available at build time and runtime.

For more detailed information, check Advanced Configuration.

A driver environment variable ZE_FLAT_DEVICE_HIERARCHY is currently used to select the device hierarchy model with which the underlying hardware is exposed. By default, each GPU tile is used as a device. Check the Level Zero Specification Documentation for more details.

Fully Sharded Data Parallel (FSDP)

Fully Sharded Data Parallel (FSDP) is a PyTorch* module that provides industry-grade solution for large model training. FSDP is a type of data parallel training, unlike DDP, where each process/worker maintains a replica of the model, FSDP shards model parameters, optimizer states and gradients across DDP ranks to reduce the GPU memory footprint used in training. This makes the training of some large-scale models feasible.

For more detailed information, check FSDP.

torch.compile for GPU (Beta)

Intel® Extension for PyTorch* now empowers users to seamlessly harness graph compilation capabilities for optimal PyTorch model performance on Intel GPU via the flagship torch.compile API through the default “inductor” backend (TorchInductor ).

For more detailed information, check torch.compile for GPU.

Kineto Supported Profiler Tool (Prototype)

The Kineto supported profiler tool is an extension of PyTorch* profiler for profiling operators’ executing time cost on GPU devices. With this tool, you can get information in many fields of the run models or code scripts. Build Intel® Extension for PyTorch* with Kineto support as default and enable this tool using the with statement before the code segment.

For more detailed information, check Profiler Kineto.

Compute Engine (Prototype feature for debug)

Compute engine is a prototype feature which provides the capacity to choose specific backend for operators with multiple implementations.

For more detailed information, check Compute Engine.

IPEX_LOG (Prototype feature for debug)

IPEX_LOG provides the capability to log verbose information from Intel® Extension for PyTorch* . Please use IPEX_LOG to get the log information or trace the execution from Intel® Extension for PyTorch*. Please continue using PyTorch* macros such as TORCH_CHECK, TORCH_ERROR, etc. to get the log information from PyTorch*.

For more detailed information, check IPEX_LOG.