neural_compressor.metric.metric
Neural Compressor metrics.
Module Contents
Classes
A wrapper of the information needed to construct a Metric. |
|
Tensorflow metrics collection. |
|
PyTorch metrics collection. |
|
MXNet metrics collection. |
|
ONNXRT QLinear metrics collection. |
|
ONNXRT Integer metrics collection. |
|
Intel Neural Compressor Metrics. |
|
The base class of Metric. |
|
The wrapper of Metric class for PyTorch. |
|
The wrapper of Metric class for MXNet. |
|
The wrapper of Metric class for ONNXRT. |
|
F1 score of a binary classification problem. |
|
The Accuracy for the classification tasks. |
|
A dummy PyTorch Metric. |
|
A dummy Metric. |
|
Computes Mean Absolute Error (MAE) loss. |
|
Computes Root Mean Squared Error (RMSE) loss. |
|
Computes Mean Squared Error (MSE) loss. |
|
Compute Top-k Accuracy classification score for Tensorflow model. |
|
Compute Top-k Accuracy classification score. |
|
Compute mean average precision of the detection task. |
|
Computes mean average precision. |
|
Computes mean average precision using algorithm in COCO. |
|
Computes mean average precision using algorithm in VOC. |
|
Evaluate for v1.1 of the SQuAD dataset. |
|
Compute the mean IOU(Intersection over Union) score. |
|
Compute the GLUE score. |
|
Computes ROC score. |
Functions
|
Decorate for registering all Metric subclasses. |
|
Register customer metric class or a dict of built-in metric configures. |
- class neural_compressor.metric.metric.Metric(name='user_metric', metric_cls=None, **kwargs)[source]
A wrapper of the information needed to construct a Metric.
The metric class should take the outputs of the model as the metric’s inputs, neural_compressor built-in metric always take (predictions, labels) as inputs, it’s recommended to design metric_cls to take (predictions, labels) as inputs.
- Parameters:
metric_cls (cls) – Should be a instance of sub_class of neural_compressor.metric.BaseMetric or a customer’s metric, which takes (predictions, labels) as inputs.
name (str, optional) – Name for metric. Defaults to ‘user_metric’.
- class neural_compressor.metric.metric.METRICS(framework: str)[source]
Intel Neural Compressor Metrics.
- neural_compressor.metric.metric.metric_registry(metric_type: str, framework: str)[source]
Decorate for registering all Metric subclasses.
The cross-framework metric is supported by specifying the framework param as one of tensorflow, pytorch, mxnet, onnxrt.
- Parameters:
metric_type – The metric type.
framework – The framework name.
- Returns:
The function to register metric class.
- Return type:
decorator_metric
- class neural_compressor.metric.metric.BaseMetric(metric, single_output=False, hvd=None)[source]
The base class of Metric.
- class neural_compressor.metric.metric.WrapPyTorchMetric(metric, single_output=False, hvd=None)[source]
The wrapper of Metric class for PyTorch.
- class neural_compressor.metric.metric.WrapMXNetMetric(metric, single_output=False, hvd=None)[source]
The wrapper of Metric class for MXNet.
- class neural_compressor.metric.metric.WrapONNXRTMetric(metric, single_output=False, hvd=None)[source]
The wrapper of Metric class for ONNXRT.
- class neural_compressor.metric.metric.F1[source]
F1 score of a binary classification problem.
The F1 score is the harmonic mean of the precision and recall. It can be computed with the equation: F1 = 2 * (precision * recall) / (precision + recall)
- class neural_compressor.metric.metric.Accuracy[source]
The Accuracy for the classification tasks.
The accuracy score is the proportion of the total number of predictions that were correct classified.
- class neural_compressor.metric.metric.PyTorchLoss[source]
A dummy PyTorch Metric.
A dummy metric that computes the average of predictions and prints it directly.
- class neural_compressor.metric.metric.Loss[source]
A dummy Metric.
A dummy metric that computes the average of predictions and prints it directly.
- class neural_compressor.metric.metric.MAE(compare_label=True)[source]
Computes Mean Absolute Error (MAE) loss.
Mean Absolute Error (MAE) is the mean of the magnitude of difference between the predicted and actual numeric values.
- class neural_compressor.metric.metric.RMSE(compare_label=True)[source]
Computes Root Mean Squared Error (RMSE) loss.
- class neural_compressor.metric.metric.MSE(compare_label=True)[source]
Computes Mean Squared Error (MSE) loss.
Mean Squared Error(MSE) represents the average of the squares of errors. For example, the average squared difference between the estimated values and the actual values.
- class neural_compressor.metric.metric.TensorflowTopK(k=1)[source]
Compute Top-k Accuracy classification score for Tensorflow model.
This metric computes the number of times where the correct label is among the top k labels predicted.
- class neural_compressor.metric.metric.GeneralTopK(k=1)[source]
Compute Top-k Accuracy classification score.
This metric computes the number of times where the correct label is among the top k labels predicted.
- class neural_compressor.metric.metric.COCOmAPv2(anno_path=None, iou_thrs='0.5:0.05:0.95', map_points=101, map_key='DetectionBoxes_Precision/mAP', output_index_mapping={'num_detections': -1, 'boxes': 0, 'scores': 1, 'classes': 2})[source]
Compute mean average precision of the detection task.
- class neural_compressor.metric.metric.TensorflowMAP(anno_path=None, iou_thrs=0.5, map_points=0, map_key='DetectionBoxes_Precision/mAP')[source]
Computes mean average precision.
- class neural_compressor.metric.metric.TensorflowCOCOMAP(anno_path=None, iou_thrs=None, map_points=None, map_key='DetectionBoxes_Precision/mAP')[source]
Computes mean average precision using algorithm in COCO.
- class neural_compressor.metric.metric.TensorflowVOCMAP(anno_path=None, iou_thrs=None, map_points=None, map_key='DetectionBoxes_Precision/mAP')[source]
Computes mean average precision using algorithm in VOC.
- class neural_compressor.metric.metric.mIOU(num_classes=21)[source]
Compute the mean IOU(Intersection over Union) score.
- neural_compressor.metric.metric.register_customer_metric(user_metric, framework)[source]
Register customer metric class or a dict of built-in metric configures.
neural_compressor have many built-in metrics, user can pass a metric configure dict to tell neural compressor what metric will be use. You also can set multi-metrics to evaluate the performance of a specific model.
- Single metric:
{topk: 1}
- Multi-metrics:
- {topk: 1,
MSE: {compare_label: False}, weight: [0.5, 0.5], higher_is_better: [True, False]
}
For the built-in metrics, please refer to below link: https://github.com/intel/neural-compressor/blob/master/docs/source/metric.md#supported-built-in-metric-matrix.
- User also can get the built-in metrics by neural_compressor.Metric:
Metric(name=”topk”, k=1)
User also can set specific metric through this api. The metric class should take the outputs of the model or postprocess(if have) as inputs, neural_compressor built-in metric always take(predictions, labels) as inputs for update, and user_metric.metric_cls should be sub_class of neural_compressor.metric.BaseMetric.
- Parameters:
user_metric (neural_compressor.metric.Metric or a dict of built-in metric configurations) – The object of Metric or a dict of built-in metric configurations.
framework – framework, such as: tensorflow, pytorch……