`neural_compressor.benchmark`¶

Benchmark is used for evaluating the model performance.

Module Contents¶

Classes¶

Benchmark

Benchmark class can be used to evaluate the model performance.

Functions¶

fit(model[, config, b_dataloader, b_func])

Benchmark the model performance with the configure.

class neural_compressor.benchmark.Benchmark(conf_fname_or_obj)¶

Bases: object

Benchmark class can be used to evaluate the model performance.

With the objective setting, user can get the data of what they configured in yaml.

Parameters:: conf_fname_or_obj (string or obj) – The path to the YAML configuration file or Benchmark_Conf class containing accuracy goal, tuning objective and preferred calibration & quantization tuning space etc.

dataloader(dataset, batch_size=1, collate_fn=None, last_batch='rollover', sampler=None, batch_sampler=None, num_workers=0, pin_memory=False, shuffle=False, distributed=False)¶: Set dataloader for benchmarking.

metric(name, metric_cls, **kwargs)¶

Set the metric class and Neural Compressor will initialize this class when evaluation.

Neural Compressor has many built-in metrics, but users can set specific metrics through this api. The metric class should take the outputs of the model or postprocess (if have) as inputs. Neural Compressor built-in metrics always take (predictions, labels) as inputs for update, and user_metric.metric_cls should be a sub_class of neural_compressor.metric.metric or an user-defined metric object

Parameters:

metric_cls (cls) – Should be a sub_class of neural_compressor.metric.BaseMetric, which takes (predictions, labels) as inputs
name (str, optional) – Name for metric. Defaults to ‘user_metric’.

postprocess(name, postprocess_cls, **kwargs)¶

Set postprocess class and neural_compressor will initialize this class when evaluation.

The postprocess function should take the outputs of the model as inputs, and outputs (predictions, labels) as inputs for metric updates.

Args: name (str, optional): Name for postprocess. postprocess_cls (cls): Should be a sub_class of neural_compressor.data.transforms.postprocess.

neural_compressor.benchmark.fit(model, config=None, b_dataloader=None, b_func=None)¶