PyTorch Static Quantization

  1. Introduction

  2. Get Started
    2.1 Static Quantization with IPEX Backend
    2.1.1 Usage Sample with IPEX
    2.1.2 Specify Quantization Rules
    2.1.3 Model Examples

Introduction

Post-Training Quantization (PTQ) is a technique used to convert a pre-trained floating-point model to a quantized model. This approach does not require model retraining. Instead, it uses calibration data to determine the optimal quantization parameters. Static quantization involves calibrating both weights and activations during the quantization process. Currently, we support static PTQ with Intel Extension for PyTorch (IPEX).

Get Started

Static Quantization with IPEX Backend

Intel Extension for PyTorch (IPEX) provides optimizations specifically for Intel hardware, improving the performance of PyTorch models through efficient execution on CPUs. IPEX supports PTQ, allowing users to quantize models to lower precision to reduce model size and inference time while maintaining accuracy.

The design philosophy of the quantization interface of Intel(R) Neural Compressor is easy-of-use. It requests user to provide model, calibration function, and example inputs. Those parameters would be used to quantize and tune the model.

model is the framework model location or the framework model object.

calibration function is used to determine the appropriate quantization parameters, such as scale and zero-point, for the model’s weights and activations. This process is crucial for minimizing the loss of accuracy that can occur when converting from floating-point to lower-precision format.

IPEX leverages just-in-time (JIT) compilation techniques for optimizing the model. example inputs is used to trace the computational graph of the model, enabling various optimizations and transformations that are specific to IPEX. This tracing process captures the operations performed by the model, allowing IPEX to apply quantization optimizations effectively. example inputs should be representative of the actual data the model will process to ensure accurate calibration.

Usage Sample with IPEX

import intel_extension_for_pytorch as ipex
from neural_compressor.torch.quantization import StaticQuantConfig, convert, prepare

quant_config = StaticQuantConfig(act_sym=True, act_algo="minmax")
prepared_model = prepare(model, quant_config=quant_config, example_inputs=example_inputs)
run_fn(prepared_model)
q_model = convert(prepared_model)

[!IMPORTANT]
To use static quantization with the IPEX backend, please explicitly import IPEX at the beginning of your program.

Specify Quantization Rules

Intel(R) Neural Compressor support specify quantization rules by operator name or operator type. Users can use set_local to fallback either op_name or op_type in StaticQuantConfig to achieve the above purpose.

  1. Example of op_name_dict Here we don’t quantize the layer named fc1.

# fallback by op_name
quant_config.set_local("fc1", StaticQuantConfig(w_dtype="fp32", act_dtype="fp32"))
prepared_model = prepare(fp32_model, quant_config=quant_config, example_inputs=example_inputs)
run_fn(prepared_model)
q_model = convert(prepared_model)

  1. Example of op_type_dict Here we don’t quantize Linear layers.

# fallback by op_type
quant_config.set_local("Linear", StaticQuantConfig(w_dtype="fp32", act_dtype="fp32"))
prepared_model = prepare(model, quant_config=quant_config, example_inputs=example_inputs)
run_fn(prepared_model)
q_model = convert(prepared_model)

Model Examples

The usage sample above demonstrates how to quantize a model with the IPEX backend.