neural_compressor.experimental.metric.coco_tools

Wrappers for third party pycocotools to be used within object_detection.

Note that nothing in this file is tensorflow related and thus cannot be called directly as a slim metric, for example.

TODO(jonathanhuang): wrap as a slim metric in metrics.py

Usage example: given a set of images with ids in the list image_ids and corresponding lists of numpy arrays encoding groundtruth (boxes and classes) and detections (boxes, scores and classes), where elements of each list correspond to detections/annotations of a single image, then evaluation (in multi-class mode) can be invoked as follows:

groundtruth_dict = coco_tools.ExportGroundtruthToCOCO(

image_ids, groundtruth_boxes_list, groundtruth_classes_list, max_num_classes, output_path=None)

detections_list = coco_tools.ExportDetectionsToCOCO(

image_ids, detection_boxes_list, detection_scores_list, detection_classes_list, output_path=None)

groundtruth = coco_tools.COCOWrapper(groundtruth_dict) detections = groundtruth.LoadAnnotations(detections_list) evaluator = coco_tools.COCOEvalWrapper(groundtruth, detections,

agnostic_mode=False)

metrics = evaluator.ComputeMetrics()

Module Contents

Classes

COCOWrapper

Wrapper for the pycocotools COCO class.

COCOEvalWrapper

Wrapper for the pycocotools COCOeval class.

Functions

ExportSingleImageGroundtruthToCoco(→ list)

Export groundtruth of a single image to COCO format.

ExportSingleImageDetectionBoxesToCoco(→ list)

Export detections of a single image to COCO format.

ExportSingleImageDetectionMasksToCoco(→ list)

Export detection masks of a single image to COCO format.
class neural_compressor.experimental.metric.coco_tools.COCOWrapper(dataset: Dict[str, Any], detection_type: str = 'bbox')

Bases: pycocotools.coco.COCO

Wrapper for the pycocotools COCO class.

dataset

a dictionary holding bounding box annotations in the COCO format.

detection_type

type of detections being wrapped. Can be one of [‘bbox’, ‘segmentation’]

LoadAnnotations(annotations: list) → pycocotools.coco.COCO

Load annotations dictionary into COCO datastructure.

See http://mscoco.org/dataset/#format for a description of the annotations format. As above, this function replicates the default behavior of the API but does not require writing to external storage.

Parameters:

annotations – python list holding object detection results where each detection is encoded as a dict with required keys [‘image_id’, ‘category_id’, ‘score’] and one of [‘bbox’, ‘segmentation’] based on detection_type.

Returns:

a coco.COCO datastructure holding object detection annotations results

Raises:

ValueError – if (1) annotations is not a list or annotations do not correspond to the images contained in self.

class neural_compressor.experimental.metric.coco_tools.COCOEvalWrapper(groundtruth: pycocotools.coco.COCO = None, detections: pycocotools.coco.COCO = None, agnostic_mode=False, iou_type: str = 'bbox', iou_thrs: str | float = None, map_points=None)

Bases: pycocotools.cocoeval.COCOeval

Wrapper for the pycocotools COCOeval class.

To evaluate, create two objects (groundtruth_dict and detections_list) using the conventions listed at http://mscoco.org/dataset/#format. Then call evaluation as follows:

groundtruth = coco_tools.COCOWrapper(groundtruth_dict) detections = groundtruth.LoadAnnotations(detections_list) evaluator = coco_tools.COCOEvalWrapper(groundtruth, detections,

agnostic_mode=False)

metrics = evaluator.ComputeMetrics()

GetCategory(category_id: int) → dict

Fetch dictionary holding category information given category id.

Parameters:

category_id – integer id

Returns:

dictionary holding ‘id’, ‘name’.

GetAgnosticMode() → bool

Return whether COCO Eval is configured to evaluate in agnostic mode.

GetCategoryIdList() → List[int]

Return the list of IDs of all valid categories.

accumulate(p: pycocotools.cocoeval.Params = None)

Accumulate evaluation results per image and store it to self.eval.

Parameters:

p – input params for evaluation

ComputeMetrics(include_metrics_per_category: bool = False, all_metrics_per_category: bool = False)

Compute detection metrics.

Parameters:

  • include_metrics_per_category – Whether include metrics per category.

  • all_metrics_per_category – Whether include all the summery metrics for each category in per_category_ap. Be careful with setting it to true if you have more than handful of categories, because it will pollute your mldash.

Returns:

A tuple of (summary_metrics, per_category_ap), in which

  1. summary_metrics is a dictionary holding:

’Precision/mAP’: mean average precision over classes averaged over IOU

thresholds ranging from .5 to .95 with .05 increments;

’Precision/mAP@.50IOU’: mean average precision at 50% IOU; ‘Precision/mAP@.75IOU’: mean average precision at 75% IOU; ‘Precision/mAP (small)’: mean average precision for small objects

(area < 32^2 pixels);

’Precision/mAP (medium)’: mean average precision for medium sized

objects (32^2 pixels < area < 96^2 pixels);

’Precision/mAP (large)’: mean average precision for large objects

(96^2 pixels < area < 10000^2 pixels);

’Recall/AR@1’: average recall with 1 detection; ‘Recall/AR@10’: average recall with 10 detections; ‘Recall/AR@100’: average recall with 100 detections; ‘Recall/AR@100 (small)’: average recall for small objects with 100

detections;

’Recall/AR@100 (medium)’: average recall for medium objects with 100

detections;

’Recall/AR@100 (large)’: average recall for large objects with 100

detections;

and (2) per_category_ap is a dictionary holding category specific results with

keys of the form: ‘Precision mAP ByCategory/category’ (without the supercategory part if no supercategories exist).

For backward compatibility ‘PerformanceByCategory’ is included in the

output regardless of all_metrics_per_category. If evaluating class-agnostic mode, per_category_ap is an empty dictionary.

Raises:

ValueError – If category_stats does not exist.

neural_compressor.experimental.metric.coco_tools.ExportSingleImageGroundtruthToCoco(image_id: int | str, next_annotation_id: int, category_id_set: Set[str], groundtruth_boxes: numpy.array, groundtruth_classes: numpy.array, groundtruth_masks: numpy.array | None = None, groundtruth_is_crowd: numpy.array | None = None) → list

Export groundtruth of a single image to COCO format.

This function converts groundtruth detection annotations represented as numpy arrays to dictionaries that can be ingested by the COCO evaluation API. Note that the image_ids provided here must match the ones given to ExportSingleImageDetectionsToCoco. We assume that boxes and classes are in correspondence - that is: groundtruth_boxes[i, :], and groundtruth_classes[i] are associated with the same groundtruth annotation.

In the exported result, “area” fields are always set to the area of the groundtruth bounding box.

Parameters:

  • image_id – a unique image identifier either of type integer or string.

  • next_annotation_id – integer specifying the first id to use for the groundtruth annotations. All annotations are assigned a continuous integer id starting from this value.

  • category_id_set – A set of valid class ids. Groundtruth with classes not in category_id_set are dropped.

  • groundtruth_boxes – numpy array (float32) with shape [num_gt_boxes, 4]

  • groundtruth_classes – numpy array (int) with shape [num_gt_boxes]

  • groundtruth_masks – optional uint8 numpy array of shape [num_detections, image_height, image_width] containing detection_masks.

  • groundtruth_is_crowd – optional numpy array (int) with shape [num_gt_boxes] indicating whether groundtruth boxes are crowd.

Returns:

A list of groundtruth annotations for a single image in the COCO format.

Raises:

ValueError – if (1) groundtruth_boxes and groundtruth_classes do not have the right lengths or (2) if each of the elements inside these lists do not have the correct shapes or (3) if image_ids are not integers

neural_compressor.experimental.metric.coco_tools.ExportSingleImageDetectionBoxesToCoco(image_id: int | str, category_id_set: Set[int], detection_boxes: numpy.array, detection_scores: numpy.array, detection_classes: numpy.array) → list

Export detections of a single image to COCO format.

This function converts detections represented as numpy arrays to dictionaries that can be ingested by the COCO evaluation API. Note that the image_ids provided here must match the ones given to the ExporSingleImageDetectionBoxesToCoco. We assume that boxes, and classes are in correspondence - that is: boxes[i, :], and classes[i] are associated with the same groundtruth annotation.

Parameters:

  • image_id – unique image identifier either of type integer or string.

  • category_id_set – A set of valid class ids. Detections with classes not in category_id_set are dropped.

  • detection_boxes – float numpy array of shape [num_detections, 4] containing detection boxes.

  • detection_scores – float numpy array of shape [num_detections] containing scored for the detection boxes.

  • detection_classes – integer numpy array of shape [num_detections] containing the classes for detection boxes.

Returns:

A list of detection annotations for a single image in the COCO format.

Raises:

  • ValueError – if (1) detection_boxes, detection_scores and detection_classes

  • do not have the right lengths or (2) if each of the elements inside these –

  • lists do not have the correct shapes or (3) if image_ids are not integers. –

neural_compressor.experimental.metric.coco_tools.ExportSingleImageDetectionMasksToCoco(image_id: str | int, category_id_set: Set[int], detection_masks: numpy.array, detection_scores: numpy.array, detection_classes: numpy.array) → list

Export detection masks of a single image to COCO format.

This function converts detections represented as numpy arrays to dictionaries that can be ingested by the COCO evaluation API. We assume that detection_masks, detection_scores, and detection_classes are in correspondence - that is: detection_masks[i, :], detection_classes[i] and detection_scores[i]

are associated with the same annotation.

Parameters:

  • image_id – unique image identifier either of type integer or string.

  • category_id_set – A set of valid class ids. Detections with classes not in

  • dropped. (category_id_set are) –

  • detection_masks – uint8 numpy array of shape [num_detections, image_height,

  • detection_masks. (image_width] containing) –

  • detection_scores – float numpy array of shape [num_detections] containing

  • masks. (the classes for detection) –

  • detection_classes – integer numpy array of shape [num_detections] containing

  • masks. –

Returns:

A list of detection mask annotations for a single image in the COCO format.

Raises:

  • ValueError – if (1) detection_masks, detection_scores and detection_classes

  • do not have the right lengths or (2) if each of the elements inside these –

  • lists do not have the correct shapes or (3) if image_ids are not integers. –