neural_compressor.experimental.data.transforms.transform

Neural Compressor built-in Transforms on multiple framework backends.

Module Contents

Classes

Transforms	INC supports built-in preprocessing, postprocessing and general methods on different framework backends.
TensorflowTransforms	Tensorflow Transforms subclass.
MXNetTransforms	Mxnet Transforms subclass.
PyTorchTransforms	Pytorch Transforms subclass.
ONNXRTQLTransforms	Onnxrt_qlinearops Transforms subclass.
ONNXRTITTransforms	Onnxrt_integerops Transforms subclass.
TRANSFORMS	Transforms collection class.
BaseTransform	The base class for transform.
TensorflowWrapFunction	Tensorflow wrapper function class.
TensorflowTransform	Tensorflow transform class, the subclass of BaseTransform.
PytorchMxnetWrapFunction	Pytorch and MXNet wrapper function class.
PytorchMxnetTransform	Pytorch and Mxnet transform class, the subclass of BaseTransform.
ComposeTransform	Composes several transforms together.
CropToBoundingBox	Crops an image to a specified bounding box.
MXNetCropToBoundingBox	Crops an image to a specified bounding box.
ONNXRTCropToBoundingBox	Crops an image to a specified bounding box.
TensorflowCropToBoundingBox	Crops an image to a specified bounding box.
ResizeWithRatio	Resize image with aspect ratio and pad it to max shape(optional).
TensorflowResizeWithRatio	Resize image with aspect ratio and pad it to max shape(optional).
Transpose	Transpose image according to perm.
TensorflowTranspose	Transpose image according to perm.
MXNetTranspose	Transpose image according to perm.
PyTorchTranspose	Transpose image according to perm.
RandomVerticalFlip	Vertically flip the given image randomly.
TensorflowRandomVerticalFlip	Vertically flip the given image randomly.
RandomHorizontalFlip	Horizontally flip the given image randomly.
TensorflowRandomHorizontalFlip	Horizontally flip the given image randomly.
ToArray	Convert PIL Image or NDArray to numpy array.
CastTFTransform	Convert image to given dtype.
CastONNXTransform	Convert image to given dtype.
CastPyTorchTransform	Convert image to given dtype.
CenterCropTFTransform	Crops the given image at the center to the given size.
PaddedCenterCropTransform	Crops the given image at the center to the given size with padding.
ResizeTFTransform	Resize the input image to the given size.
ResizePytorchTransform	Resize the input image to the given size.
RandomCropTFTransform	Crop the image at a random location to the given size.
RandomResizedCropPytorchTransform	Crop the given image to random size and aspect ratio.
RandomResizedCropMXNetTransform	Crop the given image to random size and aspect ratio.
RandomResizedCropTFTransform	Crop the given image to random size and aspect ratio.
NormalizeTFTransform	Normalize a image with mean and standard deviation.
RescaleKerasPretrainTransform	Scale the values of image to [0,1].
RescaleTFTransform	Scale the values of image to [0,1].
RescaleTransform	Scale the values of image to [0,1].
AlignImageChannelTransform	Align image channel, now just support [H,W]->[H,W,dim], [H,W,4]->[H,W,3] and [H,W,3]->[H,W].
PyTorchAlignImageChannel	Align image channel, now just support [H,W,4]->[H,W,3] and [H,W,3]->[H,W].
ToNDArrayTransform	Convert np.array to NDArray.
ResizeMXNetTransform	Resize the input image to the given size.
ResizeTransform	Resize the input image to the given size.
CropResizeTFTransform	Crop the input image with given location and resize it.
PyTorchCropResizeTransform	Crop the input image with given location and resize it.
MXNetCropResizeTransform	Crop the input image with given location and resize it.
CropResizeTransform	Crop the input image with given location and resize it.
CenterCropTransform	Crops the given image at the center to the given size.
MXNetNormalizeTransform	Normalize a image with mean and standard deviation.
PyTorchNormalizeTransform	Normalize a image with mean and standard deviation.
NormalizeTransform	Normalize a image with mean and standard deviation.
RandomCropTransform	Crop the image at a random location to the given size.
RandomResizedCropTransform	Crop the given image to random size and aspect ratio.
SquadExample	A single training/test example for simple sequence classification.
InputFeatures	A single set of features of data.
CollectTransform	Postprocess the predictions, collect data.
TFSquadV1PostTransform	Postprocess the predictions of bert on SQuAD.
TFModelZooCollectTransform	Postprocess the predictions of model zoo, collect data.
TFSquadV1ModelZooPostTransform	Postprocess the predictions of bert on SQuADV1.1.
ParseDecodeVocTransform	Parse features in Example proto.

Functions

transform_registry(transform_type, process, framework)	Class decorator used to register all transform subclasses.
get_torchvision_map(interpolation)	Get torchvision interpolation map.
get_final_text(pred_text, orig_text, do_lower_case)	Project the tokenized prediction back to the original text.
read_squad_examples(input_file)	Read a SQuAD json file into a list of SquadExample.
convert_examples_to_features(examples, tokenizer, ...)	Load a data file into a list of `InputBatch`s.

class neural_compressor.experimental.data.transforms.transform.Transforms(process, concat_general=True)

INC supports built-in preprocessing, postprocessing and general methods on different framework backends.

Transforms base class provides the abstract methods. Users can also register their own Transforms classes by inheriting this base class.

class neural_compressor.experimental.data.transforms.transform.TensorflowTransforms(process, concat_general=True)

Tensorflow Transforms subclass.

class neural_compressor.experimental.data.transforms.transform.MXNetTransforms(process, concat_general=True)

Mxnet Transforms subclass.

class neural_compressor.experimental.data.transforms.transform.PyTorchTransforms(process, concat_general=True)

Pytorch Transforms subclass.

class neural_compressor.experimental.data.transforms.transform.ONNXRTQLTransforms(process, concat_general=True)

Onnxrt_qlinearops Transforms subclass.

class neural_compressor.experimental.data.transforms.transform.ONNXRTITTransforms(process, concat_general=True)

Onnxrt_integerops Transforms subclass.

class neural_compressor.experimental.data.transforms.transform.TRANSFORMS(framework, process)

Transforms collection class.

Provide register method to register new Transforms and provide __getitem__ method to get Transforms according to Transforms type.

neural_compressor.experimental.data.transforms.transform.transform_registry(transform_type, process, framework)

Class decorator used to register all transform subclasses.

Parameters:

• transform_type (str) – Transform registration name

• process (str) – support 3 process including ‘preprocess’, ‘postprocess’, ‘general’

• framework (str) – support 4 framework including ‘tensorflow’, ‘pytorch’, ‘mxnet’, ‘onnxrt’

• cls (class) – The class of register.

Returns:

The class of register.

Return type:

cls

class neural_compressor.experimental.data.transforms.transform.BaseTransform

The base class for transform.

class neural_compressor.experimental.data.transforms.transform.TensorflowWrapFunction(transform_func)

Tensorflow wrapper function class.

class neural_compressor.experimental.data.transforms.transform.TensorflowTransform(transform_func, **kwargs)

Tensorflow transform class, the subclass of BaseTransform.

class neural_compressor.experimental.data.transforms.transform.PytorchMxnetWrapFunction(transform_func)

Pytorch and MXNet wrapper function class.

class neural_compressor.experimental.data.transforms.transform.PytorchMxnetTransform(transform_func)

Pytorch and Mxnet transform class, the subclass of BaseTransform.

neural_compressor.experimental.data.transforms.transform.get_torchvision_map(interpolation)

Get torchvision interpolation map.

class neural_compressor.experimental.data.transforms.transform.ComposeTransform(transform_list)

Composes several transforms together.

Parameters:

transform_list (list of Transform objects) – list of transforms to compose

Returns:

tuple of processed image and label

Return type:

sample (tuple)

class neural_compressor.experimental.data.transforms.transform.CropToBoundingBox(offset_height, offset_width, target_height, target_width)

Crops an image to a specified bounding box.

Parameters:

• offset_height (int) – Vertical coordinate of the top-left corner of the result in the input

• offset_width (int) – Horizontal coordinate of the top-left corner of the result in the input

• target_height (int) – Height of the result

• target_width (int) – Width of the result

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.MXNetCropToBoundingBox(offset_height, offset_width, target_height, target_width)

Crops an image to a specified bounding box.

Parameters:

• offset_height (int) – Vertical coordinate of the top-left corner of the result in the input

• offset_width (int) – Horizontal coordinate of the top-left corner of the result in the input

• target_height (int) – Height of the result

• target_width (int) – Width of the result

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.ONNXRTCropToBoundingBox(offset_height, offset_width, target_height, target_width)

Crops an image to a specified bounding box.

Parameters:

• offset_height (int) – Vertical coordinate of the top-left corner of the result in the input

• offset_width (int) – Horizontal coordinate of the top-left corner of the result in the input

• target_height (int) – Height of the result

• target_width (int) – Width of the result

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.TensorflowCropToBoundingBox(offset_height, offset_width, target_height, target_width)

Crops an image to a specified bounding box.

Parameters:

• offset_height (int) – Vertical coordinate of the top-left corner of the result in the input

• offset_width (int) – Horizontal coordinate of the top-left corner of the result in the input

• target_height (int) – Height of the result

• target_width (int) – Width of the result

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.ResizeWithRatio(min_dim=800, max_dim=1365, padding=False, constant_value=0)

Resize image with aspect ratio and pad it to max shape(optional).

If the image is padded, the label will be processed at the same time. The input image should be np.array.

Parameters:

• min_dim (int, default=800) – Resizes the image such that its smaller dimension == min_dim

• max_dim (int, default=1365) – Ensures that the image longest side doesn’t exceed this value

• padding (bool, default=False) – If true, pads image with zeros so its size is max_dim x max_dim

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.TensorflowResizeWithRatio(min_dim=800, max_dim=1365, padding=False, constant_value=0)

Resize image with aspect ratio and pad it to max shape(optional).

If the image is padded, the label will be processed at the same time. The input image should be np.array or tf.Tensor.

Parameters:

• min_dim (int, default=800) – Resizes the image such that its smaller dimension == min_dim

• max_dim (int, default=1365) – Ensures that the image longest side doesn’t exceed this value

• padding (bool, default=False) – If true, pads image with zeros so its size is max_dim x max_dim

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.Transpose(perm)

Transpose image according to perm.

Parameters:

perm (list) – A permutation of the dimensions of input image

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.TensorflowTranspose(perm)

Transpose image according to perm.

Parameters:

perm (list) – A permutation of the dimensions of input image

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.MXNetTranspose(perm)

Transpose image according to perm.

Parameters:

perm (list) – A permutation of the dimensions of input image

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.PyTorchTranspose(perm)

Transpose image according to perm.

Parameters:

perm (list) – A permutation of the dimensions of input image

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.RandomVerticalFlip

Vertically flip the given image randomly.

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.TensorflowRandomVerticalFlip

Vertically flip the given image randomly.

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.RandomHorizontalFlip

Horizontally flip the given image randomly.

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.TensorflowRandomHorizontalFlip

Horizontally flip the given image randomly.

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.ToArray

Convert PIL Image or NDArray to numpy array.

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.CastTFTransform(dtype='float32')

Convert image to given dtype.

Parameters:

dtype (str, default='float32') – A dtype to convert image to

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.CastONNXTransform(dtype='float32')

Convert image to given dtype.

Parameters:

dtype (str, default='float32') – A dtype to convert image to

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.CastPyTorchTransform(dtype='float32')

Convert image to given dtype.

Parameters:

dtype (str, default='float32') – A dtype to convert image to

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.CenterCropTFTransform(size)

Crops the given image at the center to the given size.

Parameters:

size (list or int) – Size of the result

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.PaddedCenterCropTransform(size, crop_padding=0)

Crops the given image at the center to the given size with padding.

Parameters:

• size (list or int) – Size of the result

• crop_padding (int) – crop padding number

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.ResizeTFTransform(size, interpolation='bilinear')

Resize the input image to the given size.

Parameters:

• size (list or int) – Size of the result

• interpolation (str, default='bilinear') – Desired interpolation type, support ‘bilinear’, ‘nearest’, ‘bicubic’

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.ResizePytorchTransform(size, interpolation='bilinear')

Resize the input image to the given size.

Parameters:

• size (list or int) – Size of the result

• interpolation (str, default='bilinear') – Desired interpolation type, support ‘bilinear’, ‘nearest’, ‘bicubic’

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.RandomCropTFTransform(size)

Crop the image at a random location to the given size.

Parameters:

size (list or tuple or int) – Size of the result

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.RandomResizedCropPytorchTransform(size, scale=(0.08, 1.0), ratio=(3.0 / 4.0, 4.0 / 3.0), interpolation='bilinear')

Crop the given image to random size and aspect ratio.

Parameters:

• size (list or int) – Size of the result

• scale (tuple or list, default=(0.08, 1.0)) – range of size of the origin size cropped

• ratio (tuple or list, default=(3. / 4., 4. / 3.)) – range of aspect ratio of the origin aspect ratio cropped

• interpolation (str, default='bilinear') – Desired interpolation type, support ‘bilinear’, ‘nearest’, ‘bicubic’

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.RandomResizedCropMXNetTransform(size, scale=(0.08, 1.0), ratio=(3.0 / 4.0, 4.0 / 3.0), interpolation='bilinear')

Crop the given image to random size and aspect ratio.

Parameters:

• size (list or int) – Size of the result

• scale (tuple or list, default=(0.08, 1.0)) – range of size of the origin size cropped

• ratio (tuple or list, default=(3. / 4., 4. / 3.)) – range of aspect ratio of the origin aspect ratio cropped

• interpolation (str, default='bilinear') – Desired interpolation type, support ‘bilinear’, ‘nearest’, ‘bicubic’

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.RandomResizedCropTFTransform(size, scale=(0.08, 1.0), ratio=(3.0 / 4.0, 4.0 / 3.0), interpolation='bilinear')

Crop the given image to random size and aspect ratio.

Parameters:

• size (list or int) – Size of the result

• scale (tuple or list, default=(0.08, 1.0)) – range of size of the origin size cropped

• ratio (tuple or list, default=(3. / 4., 4. / 3.)) – range of aspect ratio of the origin aspect ratio cropped

• interpolation (str, default='bilinear') – Desired interpolation type, support ‘bilinear’, ‘nearest’

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.NormalizeTFTransform(mean=[0.0], std=[1.0], rescale=None)

Normalize a image with mean and standard deviation.

Parameters:

• mean (list, default=[0.0]) – means for each channel, if len(mean)=1, mean will be broadcasted to each channel, otherwise its length should be same with the length of image shape

• std (list, default=[1.0]) – stds for each channel, if len(std)=1, std will be broadcasted to each channel, otherwise its length should be same with the length of image shape

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.RescaleKerasPretrainTransform(rescale=None)

Scale the values of image to [0,1].

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.RescaleTFTransform

Scale the values of image to [0,1].

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.RescaleTransform

Scale the values of image to [0,1].

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.AlignImageChannelTransform(dim=3)

Align image channel, now just support [H,W]->[H,W,dim], [H,W,4]->[H,W,3] and [H,W,3]->[H,W].

Input image must be np.ndarray.

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.PyTorchAlignImageChannel(dim=3)

Align image channel, now just support [H,W,4]->[H,W,3] and [H,W,3]->[H,W].

Input image must be PIL Image.

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.ToNDArrayTransform

Convert np.array to NDArray.

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.ResizeMXNetTransform(size, interpolation='bilinear')

Resize the input image to the given size.

Parameters:

• size (list or int) – Size of the result

• interpolation (str, default='bilinear') – Desired interpolation type, support ‘bilinear’, ‘nearest’, ‘bicubic’

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.ResizeTransform(size, interpolation='bilinear')

Resize the input image to the given size.

Parameters:

• size (list or int) – Size of the result

• interpolation (str, default='bilinear') – Desired interpolation type,

• 'bilinear' (support) –

• 'nearest' –

• 'bicubic'. –

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.CropResizeTFTransform(x, y, width, height, size, interpolation='bilinear')

Crop the input image with given location and resize it.

Parameters:

• x (int) – Left boundary of the cropping area

• y (int) – Top boundary of the cropping area

• width (int) – Width of the cropping area

• height (int) – Height of the cropping area

• size (list or int) – resize to new size after cropping

• interpolation (str, default='bilinear') – Desired interpolation type, support ‘bilinear’, ‘nearest’, ‘bicubic’

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.PyTorchCropResizeTransform(x, y, width, height, size, interpolation='bilinear')

Crop the input image with given location and resize it.

Parameters:

• x (int) – Left boundary of the cropping area

• y (int) – Top boundary of the cropping area

• width (int) – Width of the cropping area

• height (int) – Height of the cropping area

• size (list or int) – resize to new size after cropping

• interpolation (str, default='bilinear') – Desired interpolation type, support ‘bilinear’, ‘nearest’, ‘bicubic’

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.MXNetCropResizeTransform(x, y, width, height, size, interpolation='bilinear')

Crop the input image with given location and resize it.

Parameters:

• x (int) – Left boundary of the cropping area

• y (int) – Top boundary of the cropping area

• width (int) – Width of the cropping area

• height (int) – Height of the cropping area

• size (list or int) – resize to new size after cropping

• interpolation (str, default='bilinear') – Desired interpolation type, support ‘bilinear’, ‘nearest’, ‘bicubic’

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.CropResizeTransform(x, y, width, height, size, interpolation='bilinear')

Crop the input image with given location and resize it.

Parameters:

• x (int) – Left boundary of the cropping area

• y (int) – Top boundary of the cropping area

• width (int) – Width of the cropping area

• height (int) – Height of the cropping area

• size (list or int) – resize to new size after cropping

• interpolation (str, default='bilinear') – Desired interpolation type, support ‘bilinear’, ‘nearest’, ‘bicubic’

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.CenterCropTransform(size)

Crops the given image at the center to the given size.

Parameters:

size (list or int) – Size of the result

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.MXNetNormalizeTransform(mean=[0.0], std=[1.0])

Normalize a image with mean and standard deviation.

Parameters:

• mean (list, default=[0.0]) – means for each channel, if len(mean)=1, mean will be broadcasted to each channel, otherwise its length should be same with the length of image shape

• std (list, default=[1.0]) – stds for each channel, if len(std)=1, std will be broadcasted to each channel, otherwise its length should be same with the length of image shape

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.PyTorchNormalizeTransform(mean=[0.0], std=[1.0])

Normalize a image with mean and standard deviation.

Parameters:

• mean (list, default=[0.0]) – means for each channel, if len(mean)=1, mean will be broadcasted to each channel, otherwise its length should be same with the length of image shape

• std (list, default=[1.0]) – stds for each channel, if len(std)=1, std will be broadcasted to each channel, otherwise its length should be same with the length of image shape

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.NormalizeTransform(mean=[0.0], std=[1.0])

Normalize a image with mean and standard deviation.

Parameters:

• mean (list, default=[0.0]) – means for each channel, if len(mean)=1, mean will be broadcasted to each channel, otherwise its length should be same with the length of image shape

• std (list, default=[1.0]) – stds for each channel, if len(std)=1, std will be broadcasted to each channel, otherwise its length should be same with the length of image shape

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.RandomCropTransform(size)

Crop the image at a random location to the given size.

Parameters:

size (list or tuple or int) – Size of the result

Returns:

tuple of processed image and label

class neural_compressor.experimental.data.transforms.transform.RandomResizedCropTransform(size, scale=(0.08, 1.0), ratio=(3.0 / 4.0, 4.0 / 3.0), interpolation='bilinear')

Crop the given image to random size and aspect ratio.

Parameters:

• size (list or int) – Size of the result

• scale (tuple or list, default=(0.08, 1.0)) – range of size of the origin size cropped

• ratio (tuple or list, default=(3. / 4., 4. / 3.)) – range of aspect ratio of the origin aspect ratio cropped

• interpolation (str, default='bilinear') – Desired interpolation type, support ‘bilinear’, ‘nearest’

Returns:

tuple of processed image and label

neural_compressor.experimental.data.transforms.transform.get_final_text(pred_text, orig_text, do_lower_case)

Project the tokenized prediction back to the original text.

class neural_compressor.experimental.data.transforms.transform.SquadExample(qas_id, question_text, doc_tokens, orig_answer_text=None, start_position=None, end_position=None, is_impossible=False)

A single training/test example for simple sequence classification.

For examples without an answer, the start and end position are -1.

class neural_compressor.experimental.data.transforms.transform.InputFeatures(unique_id, example_index, doc_span_index, tokens, token_to_orig_map, token_is_max_context, input_ids, input_mask, segment_ids, start_position=None, end_position=None, is_impossible=None)

A single set of features of data.

neural_compressor.experimental.data.transforms.transform.read_squad_examples(input_file)

Read a SQuAD json file into a list of SquadExample.

neural_compressor.experimental.data.transforms.transform.convert_examples_to_features(examples, tokenizer, max_seq_length, doc_stride, max_query_length, output_fn)

Load a data file into a list of `InputBatch`s.

class neural_compressor.experimental.data.transforms.transform.CollectTransform(length=10833)

Postprocess the predictions, collect data.

class neural_compressor.experimental.data.transforms.transform.TFSquadV1PostTransform(label_file, vocab_file, n_best_size=20, max_seq_length=384, max_query_length=64, max_answer_length=30, do_lower_case=True, doc_stride=128)

Postprocess the predictions of bert on SQuAD.

Parameters:

• label_file (str) – path of label file

• vocab_file (str) – path of vocabulary file

• n_best_size (int, default=20) – The total number of n-best predictions to generate in nbest_predictions.json

• max_seq_length (int, default=384) – The maximum total input sequence length after WordPiece tokenization. Sequences longer than this will be truncated, shorter than this will be padded

• max_query_length (int, default=64) – The maximum number of tokens for the question. Questions longer than this will be truncated to this length

• max_answer_length (int, default=30) – The maximum length of an answer that can be generated. This is needed because the start and end predictions are not conditioned on one another

• do_lower_case (bool, default=True) – Whether to lower case the input text. Should be True for uncased models and False for cased models

• doc_stride (int, default=128) – When splitting up a long document into chunks, how much stride to take between chunks

Returns:

tuple of processed prediction and label

class neural_compressor.experimental.data.transforms.transform.TFModelZooCollectTransform(length=10833)

Postprocess the predictions of model zoo, collect data.

class neural_compressor.experimental.data.transforms.transform.TFSquadV1ModelZooPostTransform(label_file, vocab_file, n_best_size=20, max_seq_length=384, max_query_length=64, max_answer_length=30, do_lower_case=True, doc_stride=128)

Postprocess the predictions of bert on SQuADV1.1.

See class TFSquadV1PostTransform for more details

class neural_compressor.experimental.data.transforms.transform.ParseDecodeVocTransform

Parse features in Example proto.

Returns:

tuple of parsed image and labels