Skip to content

Torch Tweak Examples

This directory contains practical examples demonstrating how to use Torch Tweak to tune different types of AI models for inference performance.

Minimal

Smallest tuning demo

Tunes a tiny MLP and selects the faster of Torch Inductor and Torch Eager. This is the shortest example in the tree.

  • Location: Minimal
  • Model: a two-layer MLP defined in the script
  • Use Case: The shortest path from a PyTorch module to a tuned backend
  • Key Features:
  • One script, no dataset and no checkpoint
  • Highest-throughput backend selection
  • Intel XPU when available, otherwise CPU

ResNet

Computer Vision - Image Classification

Shows how to tune ResNet models for image classification tasks. This example demonstrates model tuning and inference tuning for convolutional neural networks.

  • Location: ResNet
  • Model: ResNet50 image classification
  • Use Case: Optimizing CNN models for computer vision tasks
  • Key Features:
  • Model tuning with Torch Tweak
  • Image classification inference
  • Performance comparison before/after tuning
  • More Info:
  • https://huggingface.co/microsoft/resnet-50

StableDiffusion

Generative AI - Text-to-Image

Demonstrates tuning of Stable Diffusion models for text-to-image generation. This example shows how to tune diffusion models for faster and more efficient image generation.

FLUX

Generative AI - Advanced Text-to-Image

Shows tuning of the FLUX text-to-image model, demonstrating advanced diffusion model tuning techniques for high-quality image generation.

  • Location: FLUX
  • Model: FLUX.2-klein-4B from Black Forest Labs
  • Use Case: Optimizing state-of-the-art text-to-image models
  • Key Features:
  • Advanced diffusion model tuning
  • High-quality image generation
  • Efficient inference pipeline tuning
  • More Info:
  • https://huggingface.co/black-forest-labs/FLUX.2-klein-4B

ParakeetCTC

Speech AI - Automatic Speech Recognition

Demonstrates tuning of ASR (Automatic Speech Recognition) models using NVIDIA's Parakeet CTC model for speech-to-text conversion.

ParakeetRNNT

Speech AI - Automatic Speech Recognition

Demonstrates tuning of ASR (Automatic Speech Recognition) models using NVIDIA's Parakeet RNNT model for speech-to-text conversion.

ESM2

Text AI - Advanced Text Embedding

Demonstrates tuning of ESM2 model for text embedding tasks.

E5Large

Text AI - Advanced Text Embedding

Demonstrates tuning of E5Large model for text embedding tasks.

LLM

Large Language Models - Text Generation

Demonstrates tuning of Large Language Models for text generation tasks. This example shows how to optimize LLMs for efficient inference with KV cache support.

  • Location: LLM
  • Model: Microsoft Phi-3-mini-4k-instruct from HuggingFace
  • Use Case: Optimizing LLMs for text generation and inference
  • Key Features:
  • LLM model tuning with Torch Tweak
  • Static and dynamic KV cache optimization
  • Prefill and decode phase optimization
  • HuggingFace integration
  • More Info:
  • https://huggingface.co/microsoft/Phi-3-mini-4k-instruct

Each example includes:

  • Complete setup instructions
  • Usage examples with CLI commands
  • Model-specific tuning parameters

To get started, navigate to any example directory and follow the README instructions for that specific model type.