neural_compressor.tensorflow.utils.data

BaseDataloder of all dataloaders.

Classes

BaseDataLoader	Base class for all DataLoaders.
DummyDataset	Dataset used for dummy data generation.
DummyDatasetV2	Dataset used for dummy_v2 data generation.

Module Contents

class neural_compressor.tensorflow.utils.data.BaseDataLoader(dataset, batch_size=1, last_batch='rollover', collate_fn=None, sampler=None, batch_sampler=None, num_workers=0, pin_memory=False, shuffle=False, distributed=False)

Base class for all DataLoaders.

_generate_dataloader is needed to create a dataloader object from the general params like batch_size and sampler. The dynamic batching is just to generate a new dataloader by setting batch_size and last_batch.

class neural_compressor.tensorflow.utils.data.DummyDataset(shape, low=-128.0, high=127.0, dtype='float32', label=True, transform=None, filter=None)

Dataset used for dummy data generation.

This Dataset is to construct a dataset from a specific shape. The value range is calculated from: low * stand_normal(0, 1) + high. (TODO) construct dummy data from real dataset or iteration of data.

class neural_compressor.tensorflow.utils.data.DummyDatasetV2(input_shape, label_shape=None, low=-128.0, high=127.0, dtype='float32', transform=None, filter=None)

Dataset used for dummy_v2 data generation.

This Dataset is to construct a dataset from a input shape and label shape. The value range is calculated from: low * stand_normal(0, 1) + high.