Dataset
=======

Users can use Neural Compressor built-in dataset objects as well as register their own datasets.

## Built-in dataset support list

Neural Compressor supports built-in dataloaders on popular industry datasets. Refer to this [HelloWorld example](/examples/helloworld/tf_example6) to learn how to configure a built-in dataloader.

#### TensorFlow

| Dataset | Parameters | Comments | Usage |
| :------ | :------ | :------ | :------ |
| MNIST(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train** (bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/MNIST/, otherwise user should put mnist.npz under root/MNIST/ manually. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; MNIST: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['MNIST'] (root=root, train=False, transform=transform, filter=None, download=True) |
| FashionMNIST(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train**(bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/FashionMNIST/, otherwise user should put train-labels-idx1-ubyte.gz, train-images-idx3-ubyte.gz, t10k-labels-idx1-ubyte.gz and t10k-images-idx3-ubyte.gz under root/FashionMNIST/ manually.| **In yaml file:** <br> dataset: <br> &ensp;&ensp; FashionMNIST: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['FashionMNIST'] (root=root, train=False, transform=transform, filter=None, download=True) |
| CIFAR10(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train** (bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/ and extract it automatically, otherwise user can download file from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz manually to root/ and extract it. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; CIFAR10: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['CIFAR10'] (root=root, train=False, transform=transform, filter=None, download=True) |
| CIFAR100(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train** (bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/ and extract it automatically, otherwise user can download file from https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz manually to root/ and extract it. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; CIFAR100: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['CIFAR100'] (root=root, train=False, transform=transform, filter=None, download=True) |
| ImageRecord(root, transform, filter) | **root** (str): Root directory of dataset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | Please arrange data in this way: <br> root/validation-000-of-100 <br> root/validation-001-of-100 <br> ... <br> root/validation-099-of-100 <br> The file name needs to follow this pattern: '* - * -of- *' | **In yaml file:** <br> dataset: <br> &ensp;&ensp; ImageRecord: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['ImageRecord'] (root=root, transform=transform, filter=None) <br> |
| ImageFolder(root, transform, filter) | **root** (str): Root directory of dataset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | Please arrange data in this way: <br> root/class_1/xxx.png <br> root/class_1/xxy.png <br> root/class_1/xxz.png <br> ... <br> root/class_n/123.png <br> root/class_n/nsdf3.png <br> root/class_n/asd932_.png <br> Please put images of different categories into different folders. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; ImageFolder: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['ImageFolder'] (root=root,transform=transform, filter=None) |
| ImagenetRaw(data_path, image_list, transform, filter) | **data_path** (str): Root directory of dataset <br> **image_list** (str): data file, record image_names and their labels <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | Please arrange data in this way: <br> data_path/img1.jpg <br> data_path/img2.jpg <br> ... <br> data_path/imgx.jpg <br> dataset will read name and label of each image from image_list file, if user set image_list to None, it will read from data_path/val_map.txt automatically. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; ImagenetRaw: <br> &ensp;&ensp;&ensp;&ensp; data_path: /path/to/image <br> &ensp;&ensp;&ensp;&ensp; image_list: /path/to/label <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['ImagenetRaw'] (data_path, image_list, transform=transform, filter=None) |
| COCORecord(root, num_cores, transform, filter) | **root** (str): Root directory of dataset <br> **num_cores** (int, default=28):The number of input Datasets to interleave from in parallel <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | Root is a full path to tfrecord file, which contains the file name. <br> **Please use Resize transform when batch_size > 1** | **In yaml file:** <br> dataset: <br> &ensp;&ensp; COCORecord: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/tfrecord <br> &ensp;&ensp;&ensp;&ensp; num_cores: 28 <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['COCORecord'] (root, num_cores=28, transform=transform, filter=None) |
| COCORaw(root, img_dir, anno_dir, transform, filter) | **root** (str): Root directory of dataset <br> **img_dir** (str, default='val2017'): image file directory <br> **anno_dir** (str, default='annotations/instances_val2017.json'): annotation file directory <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | Please arrange data in this way: <br> /root/img_dir/1.jpg <br> /root/img_dir/2.jpg <br> ... <br> /root/img_dir/n.jpg <br> /root/anno_dir <br> **Please use Resize transform when batch_size > 1** | **In yaml file:** <br> dataset: <br> &ensp;&ensp; COCORaw: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; img_dir: /path/to/image <br> &ensp;&ensp;&ensp;&ensp; anno_dir: /path/to/annotation <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['COCORaw'] (root, img_dir, anno_dir, transform=transform, filter=None) <br> If anno_dir is not set, the dataset will use default label map |
| COCONpy(root, npy_dir, anno_dir) | **root** (str): Root directory of dataset <br> **npy_dir** (str, default='val2017'): npy file directory <br> **anno_dir** (str, default='annotations/instances_val2017.json'): annotation file directory | Please arrange data in this way: <br> /root/npy_dir/1.jpg.npy <br> /root/npy_dir/2.jpg.npy <br> ... <br> /root/npy_dir/n.jpg.npy <br> /root/anno_dir <br> **Please use Resize transform when batch_size > 1** | **In yaml file:** <br> dataset: <br> &ensp;&ensp; COCORaw: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; npy_dir: /path/to/npy <br> &ensp;&ensp;&ensp;&ensp; anno_dir: /path/to/annotation <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['COCONpy'] (root, npy_dir, anno_dir) <br> If anno_dir is not set, the dataset will use default label map |
| dummy(shape, low, high, dtype, label, transform, filter) | **shape** (list or tuple):shape of total samples, the first dimension should be the sample count of the dataset. support create multi shape tensors, use list of tuples for each tuple in the list, will create a such size tensor. <br> **low** (list or float, default=-128.):low out the tensor value range from[0, 1] to [0, low] or [low, 0] if low < 0, if float, will implement all tensors with same low value. <br> **high** (list or float, default=127.):high the tensor value by add all tensor element value high. If list, length of list should be same with shape list <br> **dtype** (list or str, default='float32'):support multi tensor dtype setting. If list, length of list should be same with shape list, if str, all tensors will use same dtype. dtype support 'float32', 'float16', 'uint8', 'int8', 'int32', 'int64', 'bool' <br> **label** (bool, default=True):whether to return 0 as label <br> **transform** (transform object, default=None): dummy dataset does not need transform. If transform is not None, it will ignore it. <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | This dataset is to construct a dataset from a specific shape, the value range is calculated from: low * stand_normal(0, 1) + high. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; dummy: <br> &ensp;&ensp;&ensp;&ensp; shape: [3, 224, 224, 3] <br> &ensp;&ensp;&ensp;&ensp; low: 0.0 <br> &ensp;&ensp;&ensp;&ensp; high: 127.0 <br> &ensp;&ensp;&ensp;&ensp; dtype: float32 <br> &ensp;&ensp;&ensp;&ensp; label: True <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['dummy'] (shape, low, high, dtype, label, transform=None, filter=None) |
| dummy_v2(input_shape, label_shape, low, high, dtype, transform, filter) | **input_shape** (list or tuple):create single or multi input tensors list represent the sample shape of the dataset, eg and image size should be represented as (224, 224, 3), tuple contains multiple list and represent multi input tensors. <br> **label_shape** (list or tuple):create single or multi label tensors list represent the sample shape of the label, eg and label size should be represented as (1,), tuple contains multiple list and represent multi label tensors. In yaml usage, it offers (1,) as the default value. <br> **low** (list or float, default=-128.):low out the tensor value range from[0, 1] to [0, low] or [low, 0] if low < 0, if float, will implement all tensors with same low value. <br> **high** (list or float, default=127.):high the tensor value by add all tensor element value high. If list, length of list should be same with shape list <br> **dtype** (list or str, default='float32'):support multi tensor dtype setting. If list, length of list should be same with shape list, if str, all tensors will use same dtype. dtype support 'float32', 'float16', 'uint8', 'int8', 'int32', 'int64', 'bool' <br>  **transform** (transform object, default=None): dummy dataset does not need transform. If transform is not None, it will ignore it. <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | This dataset is to construct a dataset from a specific shape, the value range is calculated from: low * stand_normal(0, 1) + high. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; dummy_v2: <br> &ensp;&ensp;&ensp;&ensp; input_shape: [224, 224, 3] <br> &ensp;&ensp;&ensp;&ensp; label_shape: [1] <br> &ensp;&ensp;&ensp;&ensp; low: 0.0 <br> &ensp;&ensp;&ensp;&ensp; high: 127.0 <br> &ensp;&ensp;&ensp;&ensp; dtype: float32 <br> <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['dummy_v2'] (input_shape, low, high, dtype, transform=None, filter=None) |
| style_transfer(content_folder, style_folder, crop_ratio, resize_shape, image_format, transform, filter) | **content_folder** (str):Root directory of content images <br> **style_folder** (str):Root directory of style images <br> **crop_ratio** (float, default=0.1):cropped ratio to each side <br> **resize_shape** (tuple, default=(256, 256)):target size of image <br> **image_format** (str, default='jpg'): target image format <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | Dataset used for style transfer task. This Dataset is to construct a dataset from two specific image holders representing content image folder and style image folder. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; style_transfer: <br> &ensp;&ensp;&ensp;&ensp; content_folder: /path/to/content_folder <br> &ensp;&ensp;&ensp;&ensp; style_folder: /path/to/style_folder <br> &ensp;&ensp;&ensp;&ensp; crop_ratio: 0.1 <br> &ensp;&ensp;&ensp;&ensp; resize_shape: [256, 256] <br> &ensp;&ensp;&ensp;&ensp; image_format: 'jpg' <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['style_transfer'] (content_folder, style_folder, crop_ratio, resize_shape, image_format, transform=transform, filter=None) |
| TFRecordDataset(root, transform, filter) | **root** (str): filename of dataset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions |Root is a full path to tfrecord file, which contains the file name. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; TFRecordDataset: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/tfrecord <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['TFRecordDataset'] (root, transform=transform) |
| bert(root, label_file, task, transform, filter) | **root** (str): path of dataset <br> **label_file** (str): path of label file <br> **task** (str, default='squad'): task type of model <br> **model_type** (str, default='bert'): model type, support 'bert'.<br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | This dataset supports tfrecord data, please refer to [Guide](../examples/tensorflow/nlp/bert_large_squad/quantization/ptq/README.md) to create tfrecord file first. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; bert: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; label_file: /path/to/label_file  <br> &ensp;&ensp;&ensp;&ensp; task: squad <br> &ensp;&ensp;&ensp;&ensp; model_type: bert <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['bert'] (root, label_file, transform=transform) | 


#### PyTorch

| Dataset | Parameters | Comments | Usage |
| :------ | :------ | :------ | :------ |
| MNIST(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train** (bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/MNIST/, otherwise user should put mnist.npz under root/MNIST/ manually. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; MNIST: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['MNIST'] (root=root, train=False, transform=transform, filter=None, download=True) |
| FashionMNIST(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train**(bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/FashionMNIST/, otherwise user should put train-labels-idx1-ubyte.gz, train-images-idx3-ubyte.gz, t10k-labels-idx1-ubyte.gz and t10k-images-idx3-ubyte.gz under root/FashionMNIST/ manually.| **In yaml file:** <br> dataset: <br> &ensp;&ensp; FashionMNIST: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['FashionMNIST'] (root=root, train=False, transform=transform, filter=None, download=True) |
| CIFAR10(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train** (bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/ and extract it automatically, otherwise user can download file from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz manually to root/ and extract it. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; CIFAR10: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['CIFAR10'] (root=root, train=False, transform=transform, filter=None, download=True) |
| CIFAR100(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train** (bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/ and extract it automatically, otherwise user can download file from https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz manually to root/ and extract it. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; CIFAR100: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['CIFAR100'] (root=root, train=False, transform=transform, filter=None, download=True) |
| ImageFolder(root, transform, filter) | **root** (str): Root directory of dataset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | Please arrange data in this way: <br> root/class_1/xxx.png <br> root/class_1/xxy.png <br> root/class_1/xxz.png <br> ... <br> root/class_n/123.png <br> root/class_n/nsdf3.png <br> root/class_n/asd932_.png <br> Please put images of different categories into different folders. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; ImageFolder: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['ImageFolder'] (root=root,transform=transform, filter=None) |
| ImagenetRaw(data_path, image_list, transform, filter) | **data_path** (str): Root directory of dataset <br> **image_list** (str): data file, record image_names and their labels <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | Please arrange data in this way: <br> data_path/img1.jpg <br> data_path/img2.jpg <br> ... <br> data_path/imgx.jpg <br> dataset will read name and label of each image from image_list file, if user set image_list to None, it will read from data_path/val_map.txt automatically. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; ImagenetRaw: <br> &ensp;&ensp;&ensp;&ensp; data_path: /path/to/image <br> &ensp;&ensp;&ensp;&ensp; image_list: /path/to/label <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['ImagenetRaw'] (data_path, image_list, transform=transform, filter=None) |
| COCORaw(root, img_dir, anno_dir, transform, filter) | **root** (str): Root directory of dataset <br> **img_dir** (str, default='val2017'): image file directory <br> **anno_dir** (str, default='annotations/instances_val2017.json'): annotation file directory <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | Please arrange data in this way: <br> /root/img_dir/1.jpg <br> /root/img_dir/2.jpg <br> ... <br> /root/img_dir/n.jpg <br> /root/anno_dir <br> **Please use Resize transform when batch_size>1**| **In yaml file:** <br> dataset: <br> &ensp;&ensp; COCORaw: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; img_dir: /path/to/image <br> &ensp;&ensp;&ensp;&ensp; anno_dir: /path/to/annotation <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['COCORaw'] (root, img_dir, anno_dir, transform=transform, filter=None) <br> If anno_dir is not set, the dataset will use default label map |
| dummy(shape, low, high, dtype, label, transform, filter) | **shape** (list or tuple):shape of total samples, the first dimension should be the sample count of the dataset. support create multi shape tensors, use list of tuples for each tuple in the list, will create a such size tensor. <br> **low** (list or float, default=-128.):low out the tensor value range from[0, 1] to [0, low] or [low, 0] if low < 0, if float, will implement all tensors with same low value. <br> **high** (list or float, default=127.):high the tensor value by add all tensor element value high. If list, length of list should be same with shape list <br> **dtype** (list or str, default='float32'):support multi tensor dtype setting. If list, length of list should be same with shape list, if str, all tensors will use same dtype. dtype support 'float32', 'float16', 'uint8', 'int8', 'int32', 'int64', 'bool' <br> **label** (bool, default=True):whether to return 0 as label <br> **transform** (transform object, default=None): dummy dataset does not need transform. If transform is not None, it will ignore it. <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | This dataset is to construct a dataset from a specific shape, the value range is calculated from: low * stand_normal(0, 1) + high. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; dummy: <br> &ensp;&ensp;&ensp;&ensp; shape: [3, 224, 224, 3] <br> &ensp;&ensp;&ensp;&ensp; low: 0.0 <br> &ensp;&ensp;&ensp;&ensp; high: 127.0 <br> &ensp;&ensp;&ensp;&ensp; dtype: float32 <br> &ensp;&ensp;&ensp;&ensp; label: True <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['dummy'] (shape, low, high, dtype, label, transform=None, filter=None) |
| dummy_v2(input_shape, label_shape, low, high, dtype, transform, filter) | **input_shape** (list or tuple):create single or multi input tensors list represent the sample shape of the dataset, eg and image size should be represented as (224, 224, 3), tuple contains multiple list and represent multi input tensors. <br> **label_shape** (list or tuple):create single or multi label tensors list represent the sample shape of the label, eg and label size should be represented as (1,), tuple contains multiple list and represent multi label tensors. In yaml usage, it offers (1,) as the default value. <br> **low** (list or float, default=-128.):low out the tensor value range from[0, 1] to [0, low] or [low, 0] if low < 0, if float, will implement all tensors with same low value. <br> **high** (list or float, default=127.):high the tensor value by add all tensor element value high. If list, length of list should be same with shape list <br> **dtype** (list or str, default='float32'):support multi tensor dtype setting. If list, length of list should be same with shape list, if str, all tensors will use same dtype. dtype support 'float32', 'float16', 'uint8', 'int8', 'int32', 'int64', 'bool' <br>  **transform** (transform object, default=None): dummy dataset does not need transform. If transform is not None, it will ignore it. <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | This dataset is to construct a dataset from a specific shape, the value range is calculated from: low * stand_normal(0, 1) + high. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; dummy_v2: <br> &ensp;&ensp;&ensp;&ensp; input_shape: [224, 224, 3] <br> &ensp;&ensp;&ensp;&ensp; label_shape: [1] <br> &ensp;&ensp;&ensp;&ensp; low: 0.0 <br> &ensp;&ensp;&ensp;&ensp; high: 127.0 <br> &ensp;&ensp;&ensp;&ensp; dtype: float32 <br> <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['dummy_v2'] (input_shape, low, high, dtype, transform=None, filter=None) |
| bert(dataset, task, model_type, transform, filter) | **dataset** (list): list of data <br> **task** (str): the task of the model, support "classifier", "squad" <br> **model_type** (str, default='bert'): model type, support 'distilbert', 'bert', 'xlnet', 'xlm' <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | This Dataset is to construct from the Bert TensorDataset and not a full implementation from yaml config. The original repo link is: https://github.com/huggingface/transformers. When you want use this Dataset, you should add it before you initialize your DataLoader. | **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['bert'] (dataset, task, model_type, transform=transform, filter=None) <br> Now not support yaml implementation |

#### MXNet

| Dataset | Parameters | Comments | Usage |
| :------ | :------ | :------ | :------ |
| MNIST(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train** (bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/MNIST/, otherwise user should put mnist.npz under root/MNIST/ manually. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; MNIST: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['MNIST'] (root=root, train=False, transform=transform, filter=None, download=True) |
| FashionMNIST(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train**(bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/FashionMNIST/, otherwise user should put train-labels-idx1-ubyte.gz, train-images-idx3-ubyte.gz, t10k-labels-idx1-ubyte.gz and t10k-images-idx3-ubyte.gz under root/FashionMNIST/ manually.| **In yaml file:** <br> dataset: <br> &ensp;&ensp; FashionMNIST: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['FashionMNIST'] (root=root, train=False, transform=transform, filter=None, download=True) |
| CIFAR10(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train** (bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/ and extract it automatically, otherwise user can download file from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz manually to root/ and extract it. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; CIFAR10: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['CIFAR10'] (root=root, train=False, transform=transform, filter=None, download=True) |
| CIFAR100(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train** (bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/ and extract it automatically, otherwise user can download file from https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz manually to root/ and extract it. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; CIFAR100: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['CIFAR100'] (root=root, train=False, transform=transform, filter=None, download=True) |
| ImageFolder(root, transform, filter) | **root** (str): Root directory of dataset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | Please arrange data in this way: <br> root/class_1/xxx.png <br> root/class_1/xxy.png <br> root/class_1/xxz.png <br> ... <br> root/class_n/123.png <br> root/class_n/nsdf3.png <br> root/class_n/asd932_.png <br> Please put images of different categories into different folders. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; ImageFolder: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['ImageFolder'] (root=root,transform=transform, filter=None) |
| ImagenetRaw(data_path, image_list, transform, filter) | **data_path** (str): Root directory of dataset <br> **image_list** (str): data file, record image_names and their labels <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | Please arrange data in this way: <br> data_path/img1.jpg <br> data_path/img2.jpg <br> ... <br> data_path/imgx.jpg <br> dataset will read name and label of each image from image_list file, if user set image_list to None, it will read from data_path/val_map.txt automatically. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; ImagenetRaw: <br> &ensp;&ensp;&ensp;&ensp; data_path: /path/to/image <br> &ensp;&ensp;&ensp;&ensp; image_list: /path/to/label <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['ImagenetRaw'] (data_path, image_list, transform=transform, filter=None) |
| COCORaw(root, img_dir, anno_dir, transform, filter) | **root** (str): Root directory of dataset <br> **img_dir** (str, default='val2017'): image file directory <br> **anno_dir** (str, default='annotations/instances_val2017.json'): annotation file directory <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | Please arrange data in this way: <br> /root/img_dir/1.jpg <br> /root/img_dir/2.jpg <br> ... <br> /root/img_dir/n.jpg <br> /root/anno_dir <br> **Please use Resize transform when batch_size > 1**| **In yaml file:** <br> dataset: <br> &ensp;&ensp; COCORaw: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; img_dir: /path/to/image <br> &ensp;&ensp;&ensp;&ensp; anno_dir: /path/to/annotation <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['COCORaw'] (root, img_dir, anno_dir, transform=transform, filter=None) <br> If anno_dir is not set, the dataset will use default label map |
| dummy(shape, low, high, dtype, label, transform, filter) | **shape** (list or tuple):shape of total samples, the first dimension should be the sample count of the dataset. support create multi shape tensors, use list of tuples for each tuple in the list, will create a such size tensor. <br> **low** (list or float, default=-128.):low out the tensor value range from[0, 1] to [0, low] or [low, 0] if low < 0, if float, will implement all tensors with same low value. <br> **high** (list or float, default=127.):high the tensor value by add all tensor element value high. If list, length of list should be same with shape list <br> **dtype** (list or str, default='float32'):support multi tensor dtype setting. If list, length of list should be same with shape list, if str, all tensors will use same dtype. dtype support 'float32', 'float16', 'uint8', 'int8', 'int32', 'int64', 'bool' <br> **label** (bool, default=True):whether to return 0 as label <br> **transform** (transform object, default=None): dummy dataset does not need transform. If transform is not None, it will ignore it. <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | This dataset is to construct a dataset from a specific shape, the value range is calculated from: low * stand_normal(0, 1) + high. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; dummy: <br> &ensp;&ensp;&ensp;&ensp; shape: [3, 224, 224, 3] <br> &ensp;&ensp;&ensp;&ensp; low: 0.0 <br> &ensp;&ensp;&ensp;&ensp; high: 127.0 <br> &ensp;&ensp;&ensp;&ensp; dtype: float32 <br> &ensp;&ensp;&ensp;&ensp; label: True <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['dummy'] (shape, low, high, dtype, label, transform=None, filter=None) |
| dummy_v2(input_shape, label_shape, low, high, dtype, transform, filter) | **input_shape** (list or tuple):create single or multi input tensors list represent the sample shape of the dataset, eg and image size should be represented as (224, 224, 3), tuple contains multiple list and represent multi input tensors. <br> **label_shape** (list or tuple):create single or multi label tensors list represent the sample shape of the label, eg and label size should be represented as (1,), tuple contains multiple list and represent multi label tensors. In yaml usage, it offers (1,) as the default value. <br> **low** (list or float, default=-128.):low out the tensor value range from[0, 1] to [0, low] or [low, 0] if low < 0, if float, will implement all tensors with same low value. <br> **high** (list or float, default=127.):high the tensor value by add all tensor element value high. If list, length of list should be same with shape list <br> **dtype** (list or str, default='float32'):support multi tensor dtype setting. If list, length of list should be same with shape list, if str, all tensors will use same dtype. dtype support 'float32', 'float16', 'uint8', 'int8', 'int32', 'int64', 'bool' <br>  **transform** (transform object, default=None): dummy dataset does not need transform. If transform is not None, it will ignore it. <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | This dataset is to construct a dataset from a specific shape, the value range is calculated from: low * stand_normal(0, 1) + high. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; dummy_v2: <br> &ensp;&ensp;&ensp;&ensp; input_shape: [224, 224, 3] <br> &ensp;&ensp;&ensp;&ensp; label_shape: [1] <br> &ensp;&ensp;&ensp;&ensp; low: 0.0 <br> &ensp;&ensp;&ensp;&ensp; high: 127.0 <br> &ensp;&ensp;&ensp;&ensp; dtype: float32 <br> <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['dummy_v2'] (input_shape, low, high, dtype, transform=None, filter=None) |

#### ONNXRT

| Dataset | Parameters | Comments | Usage |
| :------ | :------ | :------ | :------ |
| MNIST(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train** (bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/MNIST/, otherwise user should put mnist.npz under root/MNIST/ manually. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; MNIST: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['MNIST'] (root=root, train=False, transform=transform, filter=None, download=True) |
| FashionMNIST(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train**(bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/FashionMNIST/, otherwise user should put train-labels-idx1-ubyte.gz, train-images-idx3-ubyte.gz, t10k-labels-idx1-ubyte.gz and t10k-images-idx3-ubyte.gz under root/FashionMNIST/ manually.| **In yaml file:** <br> dataset: <br> &ensp;&ensp; FashionMNIST: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['FashionMNIST'] (root=root, train=False, transform=transform, filter=None, download=True) |
| CIFAR10(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train** (bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/ and extract it automatically, otherwise user can download file from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz manually to root/ and extract it. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; CIFAR10: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['CIFAR10'] (root=root, train=False, transform=transform, filter=None, download=True) |
| CIFAR100(root, train, transform, filter, download) | **root** (str): Root directory of dataset <br> **train** (bool, default=False): If True, creates dataset from train subset, otherwise from validation subset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions <br> **download** (bool, default=True): If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. | If download is True, it will download dataset to root/ and extract it automatically, otherwise user can download file from https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz manually to root/ and extract it. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; CIFAR100: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; train: False <br> &ensp;&ensp;&ensp;&ensp; download: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['CIFAR100'] (root=root, train=False, transform=transform, filter=None, download=True) |
| ImageFolder(root, transform, filter) | **root** (str): Root directory of dataset <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | Please arrange data in this way: <br> root/class_1/xxx.png <br> root/class_1/xxy.png <br> root/class_1/xxz.png <br> ... <br> root/class_n/123.png <br> root/class_n/nsdf3.png <br> root/class_n/asd932_.png <br> Please put images of different categories into different folders. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; ImageFolder: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['ImageFolder'] (root=root,transform=transform, filter=None) |
| ImagenetRaw(data_path, image_list, transform, filter) | **data_path** (str): Root directory of dataset <br> **image_list** (str): data file, record image_names and their labels <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | Please arrange data in this way: <br> data_path/img1.jpg <br> data_path/img2.jpg <br> ... <br> data_path/imgx.jpg <br> dataset will read name and label of each image from image_list file, if user set image_list to None, it will read from data_path/val_map.txt automatically. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; ImagenetRaw: <br> &ensp;&ensp;&ensp;&ensp; data_path: /path/to/image <br> &ensp;&ensp;&ensp;&ensp; image_list: /path/to/label <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['ImagenetRaw'] (data_path, image_list, transform=transform, filter=None) |
| COCORaw(root, img_dir, anno_dir, transform, filter) | **root** (str): Root directory of dataset <br> **img_dir** (str, default='val2017'): image file directory <br> **anno_dir** (str, default='annotations/instances_val2017.json'): annotation file directory <br> **transform** (transform object, default=None):  transform to process input data <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | Please arrange data in this way: <br> /root/img_dir/1.jpg <br> /root/img_dir/2.jpg <br> ... <br> /root/img_dir/n.jpg <br> /root/anno_dir <br> ***Please use Resize transform when batch_size > 1**| **In yaml file:** <br> dataset: <br> &ensp;&ensp; COCORaw: <br> &ensp;&ensp;&ensp;&ensp; root: /path/to/root <br> &ensp;&ensp;&ensp;&ensp; img_dir: /path/to/image <br> &ensp;&ensp;&ensp;&ensp; anno_dir: /path/to/annotation <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['COCORaw'] (root, img_dir, anno_dir, transform=transform, filter=None) <br> If anno_dir is not set, the dataset will use default label map |
| dummy(shape, low, high, dtype, label, transform, filter) | **shape** (list or tuple):shape of total samples, the first dimension should be the sample count of the dataset. support create multi shape tensors, use list of tuples for each tuple in the list, will create a such size tensor. <br> **low** (list or float, default=-128.):low out the tensor value range from[0, 1] to [0, low] or [low, 0] if low < 0, if float, will implement all tensors with same low value. <br> **high** (list or float, default=127.):high the tensor value by add all tensor element value high. If list, length of list should be same with shape list <br> **dtype** (list or str, default='float32'):support multi tensor dtype setting. If list, length of list should be same with shape list, if str, all tensors will use same dtype. dtype support 'float32', 'float16', 'uint8', 'int8', 'int32', 'int64', 'bool' <br> **label** (bool, default=True):whether to return 0 as label <br> **transform** (transform object, default=None): dummy dataset does not need transform. If transform is not None, it will ignore it. <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | This dataset is to construct a dataset from a specific shape, the value range is calculated from: low * stand_normal(0, 1) + high. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; dummy: <br> &ensp;&ensp;&ensp;&ensp; shape: [3, 224, 224, 3] <br> &ensp;&ensp;&ensp;&ensp; low: 0.0 <br> &ensp;&ensp;&ensp;&ensp; high: 127.0 <br> &ensp;&ensp;&ensp;&ensp; dtype: float32 <br> &ensp;&ensp;&ensp;&ensp; label: True <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['dummy'] (shape, low, high, dtype, label, transform=None, filter=None) |
| dummy_v2(input_shape, label_shape, low, high, dtype, transform, filter) | **input_shape** (list or tuple):create single or multi input tensors list represent the sample shape of the dataset, eg and image size should be represented as (224, 224, 3), tuple contains multiple list and represent multi input tensors. <br> **label_shape** (list or tuple):create single or multi label tensors list represent the sample shape of the label, eg and label size should be represented as (1,), tuple contains multiple list and represent multi label tensors. In yaml usage, it offers (1,) as the default value. <br> **low** (list or float, default=-128.):low out the tensor value range from[0, 1] to [0, low] or [low, 0] if low < 0, if float, will implement all tensors with same low value. <br> **high** (list or float, default=127.):high the tensor value by add all tensor element value high. If list, length of list should be same with shape list <br> **dtype** (list or str, default='float32'):support multi tensor dtype setting. If list, length of list should be same with shape list, if str, all tensors will use same dtype. dtype support 'float32', 'float16', 'uint8', 'int8', 'int32', 'int64', 'bool' <br>  **transform** (transform object, default=None): dummy dataset does not need transform. If transform is not None, it will ignore it. <br> **filter** (Filter objects, default=None): filter out examples according to specific conditions | This dataset is to construct a dataset from a specific shape, the value range is calculated from: low * stand_normal(0, 1) + high. | **In yaml file:** <br> dataset: <br> &ensp;&ensp; dummy_v2: <br> &ensp;&ensp;&ensp;&ensp; input_shape: [224, 224, 3] <br> &ensp;&ensp;&ensp;&ensp; label_shape: [1] <br> &ensp;&ensp;&ensp;&ensp; low: 0.0 <br> &ensp;&ensp;&ensp;&ensp; high: 127.0 <br> &ensp;&ensp;&ensp;&ensp; dtype: float32 <br> <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['dummy_v2'] (input_shape, low, high, dtype, transform=None, filter=None) |
| GLUE(data_dir, model_name_or_path, max_seq_length, do_lower_case, task, model_type, dynamic_length, evaluate, transform, filter) | **data_dir** (str): The input data dir <br> **model_name_or_path** (str):  Path to pre-trained student model or shortcut name,  <br> **max_seq_length** (int, default=128): The maximum total input sequence length after tokenization. Sequences longer than this will be truncated, sequences shorter will be padded. <br> **do_lower_case** (bool, default=True): Whether or not to lowercase the input. <br> **task** (bool, default=True): The name of the task to fine-tune. Choices include mrpc, qqp, qnli, rte, sts-b, cola, mnli, wnli. <br> **model_type** (str, default='bert'): model type, support 'distilbert', 'bert', 'mobilebert', 'roberta'. <br> **dynamic_length** (bool, default=False): Whether to use fixed sequence length. <br> **evaluate** (bool, default=True): Whether do evaluation or training. <br> **transform** (bool, default=True): If true, <br> **filter** (bool, default=True): If true, | Refer to [this example](/examples/onnxrt/language_translation/bert) on how to prepare dataset | **In yaml file:** <br> dataset: <br> &ensp;&ensp; bert: <br> &ensp;&ensp;&ensp;&ensp; data_dir: False <br> &ensp;&ensp;&ensp;&ensp; model_name_or_path: True <br> (transform and filter are not set in the range of dataset) <br> **In user code:** <br> from neural_compressor.experimental.data import DATASETS <br> datasets = DATASETS(framework) <br> dataset = datasets['bert'] (data_dir='/path/to/data/', model_name_or_path='bert-base-uncased', max_seq_length=128, task='mrpc', model_type='bert', dynamic_length=True, transform=None, filter=None) |

## User-specific dataset

Users can register their own datasets as follows:

```python
class Dataset(object):
    def __init__(self, args):
        # init code here

    def __getitem__(self, idx):
        # use idx to get data and label
        return data, label

    def __len__(self):
        return len

```

After defining the dataset class, pass it to the quantizer:

```python
from neural_compressor import Quantization, common
quantizer = Quantization(yaml_file)
quantizer.calib_dataloader = common.DataLoader(dataset) # user can pass more optional args to dataloader such as batch_size and collate_fn
quantizer.model = graph
quantizer.eval_func = eval_func
q_model = quantizer.fit() 

```
