vLLM on Intel Xeon CPUs

Upstream First

Intel upstreams Xeon CPU optimizations directly to vLLM. This skill encodes deployment, tuning, validation, and a short benchmarking walkthrough — always consult upstream for the latest:

When to Use

Invoke this skill when the user wants to:

Do not use for GPU vLLM, model training, deep quantization tuning, model selection (point users at the Intel Xeon AI Performance Advisor), or non-Xeon CPUs.

Prerequisites

Item Requirement
OS Linux
Python 3.10–3.13 (only if not using Docker)
CPU 4th Gen Intel Xeon or newer; must expose amx_tile, amx_bf16, amx_int8 for best BF16/INT8 performance
Tools curl, numactl, jq, g++, python3-dev, python3-venv (sudo apt-get install -y --no-install-recommends curl git jq numactl htop python3-venv python3-full g++ python3-dev). g++ and Python headers are required by PyTorch inductor to JIT-compile CPU kernels.
Docker Recent Docker with --cap-add SYS_NICE and --security-opt seccomp=unconfined permitted

Procedure 1 — Validate Hardware

Goal: confirm Xeon generation, AMX support, and NUMA topology before deploying.

  1. Inspect CPU model, sockets, cores, threads, NUMA nodes: 
    lscpu | grep -E "Model name|Socket|Core|Thread|NUMA node"
  2. Check for AMX and AVX-512 flags: 
    lscpu | grep -E "amx_(tile|bf16|int8)|avx512_bf16|avx512f|avx2"
    • All of amx_tile, amx_bf16, amx_int8 present → proceed; BF16 AMX kernels will activate.
    • AMX missing –> Warn user. vLLM will still run, but inference throughput will be substantially lower. Recommend a 4th Gen Xeon (Sapphire Rapids) or newer.
  3. Inspect NUMA topology: 
    numactl --hardware

    Record the NUMA node count N — it drives --tensor-parallel-size and KV cache sizing.

Procedure 2 — Deploy (Docker Fast Path)

Goal: serve a model via the official vllm/vllm-openai-cpu image with Xeon-tuned env vars.

  1. (Optional) Install Docker if not present (Ubuntu 24.04): 
    sudo apt-get update
sudo apt-get install -y docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker   # apply group without re-login
  2. Pin a release tag (do not use latest-x86_64 in production): 
    export VLLM_VERSION=0.20.2   # update to the latest release that meets the minimum above
docker pull vllm/vllm-openai-cpu:v${VLLM_VERSION}-x86_64
  3. Run the container with the required Xeon env vars and Docker capabilities: 
    docker run --rm \
  --name vllm-cpu \
  --security-opt seccomp=unconfined \
  --cap-add SYS_NICE \
  --shm-size=8g \
  -p 8000:8000 \
  -e HF_TOKEN="${HF_TOKEN}" \
  -e VLLM_CPU_KVCACHE_SPACE=20 \
  -e VLLM_CPU_OMP_THREADS_BIND=auto \
  -e VLLM_CPU_NUM_OF_RESERVED_CPU=1 \
  vllm/vllm-openai-cpu:v${VLLM_VERSION}-x86_64 \
  <model-id> \
  --dtype=bfloat16 \
  --max-num-batched-tokens 2048 \
  --max-num-seqs 128
    • SYS_NICE + seccomp=unconfined enable vLLM’s NUMA memory-policy calls. Without them serving still works but logs may show get_mempolicy: Operation not permitted and NUMA placement weakens.
    • VLLM_CPU_KVCACHE_SPACE is in GiB per NUMA node (start 20–40) — must fit in node-local memory.
    • VLLM_CPU_OMP_THREADS_BIND=auto binds OpenMP workers to NUMA-local cores. For manual control use ranges like 0-31|32-63.
    • VLLM_CPU_NUM_OF_RESERVED_CPU=1 keeps a core free for API serving, tokenization, networking, and OS work.
  4. Validate the OpenAI-compatible endpoint: 
    curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-id>",
    "messages": [{"role": "user", "content": "Give three CPU inference tuning tips."}],
    "max_tokens": 128
  }'

Procedure 3 — Tune Performance Knobs

Goal: improve TTFT / TPOT / throughput methodically. Change one knob per run and compare.

  1. Pick the use case:
    • Online serving → start --max-num-batched-tokens 2048, --max-num-seqs 128.
    • Offline batch → start --max-num-batched-tokens 4096, --max-num-seqs 256.
  2. Set --tensor-parallel-size:
    • Single NUMA node → leave default.
    • Multi NUMA → set to the NUMA node count N from Procedure 1.
    • --tensor-parallel-size=6 is currently unsupported on CPU; avoid it.
  3. Size VLLM_CPU_KVCACHE_SPACE (GiB per NUMA node):
    • Start at 20–40. Larger value → more concurrency / longer context, but must fit in node-local RAM.
    • If the server OOMs or pages, halve and retry.
  4. Bind OpenMP threads with VLLM_CPU_OMP_THREADS_BIND:
    • Prefer auto. Use manual ranges (0-31|32-63) only when auto mis-pins (verify with numastat -p $(pgrep -f 'vllm serve|api_server' | head -n1)).
  5. (Experimental) Low-latency small-batch serving:
    • VLLM_CPU_SGL_KERNEL=1 enables x86 small-batch kernels. Requires AMX, BF16 weights, and compatible shapes.
  6. Quantization (when functional quality is acceptable):
    • Try INT8 or AWQ to reduce weight memory and memory-bandwidth pressure. Validate quality before promoting.

Full knob reference: tuning matrix.

Procedure 4 — Benchmark (Minimal CPU Walkthrough)

Goal: measure TTFT, TPOT, and throughput against the running server with a reproducible warm-up.

  1. Confirm the server is reachable and inspect environment: 
    curl -s http://localhost:8000/v1/models | jq .
docker exec vllm-cpu vllm collect-env   # or `vllm collect-env` for native installs
SERVER_PID=$(pgrep -f 'vllm serve|api_server' | head -n 1)
numastat -p "${SERVER_PID}"             # verify NUMA locality
  2. Run vllm bench serve with warm-ups (warm-ups avoid measuring JIT/compile overhead): 
    vllm bench serve \
  --model <model-id> \
  --dataset-name random \
  --random-input-len 128 \
  --random-output-len 128 \
  --num-prompts 100 \
  --num-warmups 5 \
  --request-rate inf \
  --save-result \
  --result-dir ./bench-results \
  --percentile-metrics ttft,tpot,itl

    Execution context: prefix vllm bench serve with docker exec vllm-cpu if you deployed via Procedure 2 (Docker), or run it directly on a native/venv install. A native Failed to infer device type error means the generic CUDA wheel is installed instead of the CPU wheel — reinstall the wheel whose version ends in +cpu (e.g. 0.20.2+cpu) from https://download.pytorch.org/whl/cpu.

  3. Sweep methodically — one variable per run:
    • Vary input/output lengths: 64, 128, 256, 512.
    • Vary concurrency via --num-prompts: 10, 50, 100, 200, 500.
    • Track TTFT, TPOT, output tokens/sec, requests/sec, peak RSS, NUMA locality, and any OOM events.
    • When tuning, change only one of VLLM_CPU_KVCACHE_SPACE, VLLM_CPU_OMP_THREADS_BIND, --max-num-batched-tokens, --max-num-seqs, or --block-size per run. Compare results across runs using the saved JSON files in ./bench-results.
  4. For the full CI-grade harness (run-performance-benchmarks.sh with ON_CPU=1, SERVING_JSON, DRY_RUN, MODEL_FILTER, DTYPE_FILTER), see the upstream vLLM benchmarking docs.

Output the Agent Should Produce

After running these procedures, return to the user:

References