Intel® Neural Compressor
An open-source Python library supporting popular model compression techniques on all mainstream deep learning frameworks (TensorFlow, PyTorch, ONNX Runtime, and MXNet)
Intel® Neural Compressor aims to provide popular model compression techniques such as quantization, pruning (sparsity), distillation, and neural architecture search on mainstream frameworks such as TensorFlow, PyTorch, ONNX Runtime, and MXNet, as well as Intel extensions such as Intel Extension for TensorFlow and Intel Extension for PyTorch. In particular, the tool provides the key features, typical examples, and open collaborations as below:
Support a wide range of Intel hardware such as Intel Xeon Scalable processor, Intel Xeon CPU Max Series, Intel Data Center GPU Flex Series, and Intel Data Center GPU Max Series with extensive testing; support AMD CPU, ARM CPU, and NVidia GPU through ONNX Runtime with limited testing
Validate more than 10,000 models such as Bloom-176B, OPT-30B, Stable Diffusion, GPT-J, BERT-Large, and ResNet50 from popular model hubs such as Hugging Face, Torch Vision, and ONNX Model Zoo, by leveraging zero-code optimization solution Neural Coder and automatic [accuracy-driven]./design.html#workflow) quantization strategies
Collaborate with cloud marketplace such as Google Cloud Platform, Amazon Web Services, and Azure, software platforms such as Alibaba Cloud and Tencent TACO, and open AI ecosystem such as Hugging Face, PyTorch, ONNX, and Lightning AI
The performance and accuracy results of typical models are available here. You may also want to visit the Intel® Neural Compressor online document website for more details.
Install from pypi
pip install neural-compressor
More installation methods can be found at Installation Guide. Please check out our FAQ for more details.
Quantization with Python API
# Install Intel Neural Compressor and TensorFlow pip install neural-compressor pip install tensorflow # Prepare fp32 model wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_6/mobilenet_v1_1.0_224_frozen.pb
from neural_compressor.config import PostTrainingQuantConfig from neural_compressor.data import DataLoader from neural_compressor.data import Datasets dataset = Datasets('tensorflow')['dummy'](shape=(1, 224, 224, 3)) dataloader = DataLoader(framework='tensorflow', dataset=dataset) from neural_compressor.quantization import fit q_model = fit( model="./mobilenet_v1_1.0_224_frozen.pb", conf=PostTrainingQuantConfig(), calib_dataloader=dataloader, eval_dataloader=dataloader)
More quick samples can be found in Get Started Page.
|Notebook||Examples||Intel oneAPI AI Analytics Toolkit|
|Quantization||Advanced Mixed Precision||Pruning (Sparsity)||Distillation|
|Orchestration||Benchmarking||Distributed Compression||Model Export|
|Neural Coder (Zero-code Optimization)|
|Launcher||JupyterLab Extension||Visual Studio Code Extension||Supported Matrix|
|Adaptor||Strategy||Distillation for Quantization||SmoothQuant|
Post on Social Media: Adopt with Tencent TACO: Heterogeneous optimization is also key to improving AI computing power (Mar 2023)
Post on Social Media: Training and Inference for Stable Diffusion | Intel Business (Jan 2023)
Blog by Intel: Intel® AMX Enhances AI Inference Performance (Jan 2023)
Blog by TensorFlow: Optimizing TensorFlow for 4th Gen Intel Xeon Processors (Jan 2023)
NeurIPS’2022: Fast Distilbert on CPUs (Oct 2022)
NeurIPS’2022: QuaLA-MiniLM: a Quantized Length Adaptive MiniLM (Oct 2022)
View our Full Publication List.
Welcome to raise any interesting research ideas on model compression techniques and feel free to reach us (firstname.lastname@example.org). Look forward to our collaborations on Intel Neural Compressor!