tlt.models.text_classification.text_classification_model.TextClassificationModel.benchmark

abstract TextClassificationModel.benchmark(dataset, saved_model_dir=None, warmup=10, iteration=100, cores_per_instance=None, num_of_instance=None, inter_num_of_threads=None, intra_num_of_threads=None)

Use Intel Neural Compressor to benchmark the model with the dataset argument. The dataset’s validation or test subset will be used for benchmarking, if present. Otherwise, the full training dataset is used. The model to be benchmarked can also be explicitly set to a saved_model_dir containing for example a quantized saved model.

Parameters

  • dataset (ImageClassificationDataset) – Dataset to use for benchmarking

  • saved_model_dir (str) – Optional, path to the directory where the saved model is located

  • warmup (int) – The number of iterations to perform before running performance tests, default is 10

  • iteration (int) – The number of iterations to run performance tests, default is 100

  • cores_per_instance (int or None) – The number of CPU cores to use per instance, default is None

  • num_of_instance (int or None) – The number of instances to use for performance testing, default is None

  • inter_num_of_threads (int or None) – The number of threads to use for inter-thread operations, default is None

  • intra_num_of_threads (int or None) – The number of threads to use for intra-thread operations, default is None

Returns

Benchmarking results from Intel Neural Compressor

Raises

  • NotADirectoryError – if the saved_model_dir is not None or a valid directory

  • FileNotFoundError – if a model is not found in the saved_model_dir