Welcome to Intel® NPU Acceleration Library’s documentation!#
The Intel® NPU Acceleration Library is a Python library designed to boost the efficiency of your applications by leveraging the power of the Intel Neural Processing Unit (NPU) to perform high-speed computations on compatible hardware.
Installation#
Check that your system has an available NPU (how-to).
You can install the packet in your machine with
pip install intel-npu-acceleration-library
Run a LLaMA model on the NPU#
To run LLM models you need to install the transformers library
pip install transformers
You are now up and running! You can create a simple script like the following one to run a LLM on the NPU
from transformers import AutoTokenizer, TextStreamer
from intel_npu_acceleration_library import NPUModelForCausalLM
import torch
model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
model = NPUModelForCausalLM.from_pretrained(model_id, use_cache=True, dtype=torch.int8).eval()
tokenizer = AutoTokenizer.from_pretrained(model_id, use_default_system_prompt=True)
tokenizer.pad_token_id = tokenizer.eos_token_id
streamer = TextStreamer(tokenizer, skip_special_tokens=True)
query = input("Ask something: ")
prefix = tokenizer(query, return_tensors="pt")["input_ids"]
generation_kwargs = dict(
input_ids=prefix,
streamer=streamer,
do_sample=True,
top_k=50,
top_p=0.9,
max_new_tokens=512,
)
print("Run inference")
_ = model.generate(**generation_kwargs)
Take note that you only need to use intel_npu_acceleration_library.compile to offload the heavy computation to the NPU.
Feel free to check Usage and LLM and the examples folder for additional use-cases and examples.
Site map#
Library overview:
Developements guide:
API Reference: