From e1ed5f6b67ad99e22f401a380cdcffbe0e3f460a Mon Sep 17 00:00:00 2001 From: Maxim Zhiltsov Date: Sat, 23 Jan 2021 11:34:43 +0300 Subject: [PATCH] Update changelog and docs (#98) * Update changelog * Update docs --- CHANGELOG.md | 8 ++--- README.md | 9 +++--- docs/design.md | 17 ++++++----- docs/developer_guide.md | 67 ++++++++++++++++++++++++++--------------- docs/user_manual.md | 6 +++- 5 files changed, 66 insertions(+), 41 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2d1271c63470..ab020d5cc507 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,20 +6,20 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## 01/19/2021 - Release v0.1.5 +## 01/23/2021 - Release v0.1.5 ### Added - `WiderFace` dataset format (, ) - Function to transform annotations to labels () -- Task-specific Splitter (, ) +- Dataset splits for classification, detection and re-id tasks (, ) - `VGGFace2` dataset format (, ) - Unique image count statistic () +- Installation with pip by name `datumaro` ### Changed - `Dataset` class extended with new operations: `save`, `load`, `export`, `import_from`, `detect`, `run_model` () -- `Dataset` operations return `Dataset` instances, allowing to chain operations () - Allowed importing `Extractor`-only defined formats (in `Project.import_from`, `dataset.import_from` and CLI/`project import`) () - `datum project ...` commands replaced with `datum ...` commands () -- Supported more image formats in `ImageNet` extractor () +- Supported more image formats in `ImageNet` extractors () - Allowed adding `Importer`-defined formats as project sources (`source add`) () - Added max search depth in `ImageDir` format and importers () diff --git a/README.md b/README.md index 9fd34c3e3ec7..4694f4ec6b15 100644 --- a/README.md +++ b/README.md @@ -122,7 +122,7 @@ CVAT annotations ---> Publication, statistics etc. [(Back to top)](#table-of-contents) -- Dataset reading, writing, conversion in any direction. Supported formats: +- Dataset reading, writing, conversion in any direction. [Supported formats](docs/user_manual.md#supported-formats): - [COCO](http://cocodataset.org/#format-data) (`image_info`, `instances`, `person_keypoints`, `captions`, `labels`*) - [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/htmldoc/index.html) (`classification`, `detection`, `segmentation`, `action_classification`, `person_layout`) - [YOLO](https://github.com/AlexeyAB/darknet#how-to-train-pascal-voc-data) (`bboxes`) @@ -188,7 +188,7 @@ python -m virtualenv venv Install Datumaro package: ``` bash -pip install 'git+https://github.com/openvinotoolkit/datumaro' +pip install datumaro ``` ## Usage @@ -234,13 +234,14 @@ dataset = dataset.transform(project.env.transforms.get('remap_labels'), {'cat': 'dog', # rename cat to dog 'truck': 'car', # rename truck to car 'person': '', # remove this label - }, default='delete') + }, default='delete') # remove everything else +# iterate over dataset elements for item in dataset: print(item.id, item.annotations) # export the resulting dataset in COCO format -project.env.converters.get('coco').convert(dataset, save_dir='dst/dir') +dataset.export('dst/dir', 'coco') ``` > Check our [developer guide](docs/developer_guide.md) for additional information. diff --git a/docs/design.md b/docs/design.md index 528b2adf7549..1e520400c0df 100644 --- a/docs/design.md +++ b/docs/design.md @@ -73,9 +73,11 @@ Datumaro is: ## RC 1 vision -In the first version Datumaro should be a project manager for CVAT. -It should only consume data from CVAT. The collected dataset -can be downloaded by user to be operated on with Datumaro CLI. +*CVAT integration* + +Datumaro needs to be integrated with [CVAT](https://github.com/openvinotoolkit/cvat), +extending CVAT UI capabilities regarding task and project operations. +It should be capable of downloading and processing data from CVAT. ``` @@ -94,6 +96,7 @@ can be downloaded by user to be operated on with Datumaro CLI. - [x] Python API for user code - [x] Installation as a package + - [x] Installation with `pip` by name - [x] A command-line tool for dataset manipulations ### Features @@ -106,7 +109,7 @@ can be downloaded by user to be operated on with Datumaro CLI. - [x] YOLO - [x] TF Detection API - [ ] Cityscapes - - [ ] ImageNet + - [x] ImageNet - Dataset visualization (`show`) - [ ] Ability to visualize a dataset @@ -117,7 +120,7 @@ can be downloaded by user to be operated on with Datumaro CLI. - [x] Object counts (detection scenario) - [x] Image-Class distribution (classification scenario) - [x] Pixel-Class distribution (segmentation scenario) - - [ ] Image similarity clusters + - [x] Image similarity clusters - [ ] Custom statistics - Dataset building @@ -164,7 +167,7 @@ can be downloaded by user to be operated on with Datumaro CLI. ### Optional features - Dataset publishing - - [ ] Versioning (for annotations, subsets, sources, etc.) + - [x] Versioning (for annotations, subsets, sources, etc.) - [ ] Blur sensitive areas on images - [ ] Tracking of legal information - [ ] Documentation generation @@ -175,7 +178,7 @@ can be downloaded by user to be operated on with Datumaro CLI. - Dataset and model debugging - [ ] Training visualization - - [ ] Inference explanation (`explain`) + - [x] Inference explanation (`explain`) - [ ] White-box approach ### Properties diff --git a/docs/developer_guide.md b/docs/developer_guide.md index 6317cee99979..52cb09e32782 100644 --- a/docs/developer_guide.md +++ b/docs/developer_guide.md @@ -38,28 +38,27 @@ Datumaro has a number of dataset and annotation features: - various annotation operations ```python -from datumaro.components.project import Environment, Dataset +from datumaro.components.dataset import Dataset from datumaro.components.extractor import Bbox, Polygon, DatasetItem -# Import and save a dataset -env = Environment() -dataset = env.make_importer('voc')('src/dir').make_dataset() -env.converters.get('coco').convert(dataset, save_dir='dst/dir') +# Import and export a dataset +dataset = Dataset.import_from('src/dir', 'voc') +dataset.export('dst/dir', 'coco') # Create a dataset, convert polygons to masks, save in PASCAL VOC format dataset = Dataset.from_iterable([ - DatasetItem(id='image1', annotations=[ - Bbox(x=1, y=2, w=3, h=4, label=1), - Polygon([1, 2, 3, 2, 4, 4], label=2, attributes={'occluded': True}), - ]), + DatasetItem(id='image1', annotations=[ + Bbox(x=1, y=2, w=3, h=4, label=1), + Polygon([1, 2, 3, 2, 4, 4], label=2, attributes={'occluded': True}), + ]), ], categories=['cat', 'dog', 'person']) -dataset = dataset.transform(env.transforms.get('polygons_to_masks')) -env.converters.get('voc').convert(dataset, save_dir='dst/dir') +dataset = dataset.transform('polygons_to_masks') +dataset.export('dst/dir', 'voc') ``` ### The Dataset class -The `Dataset` class from the `datumaro.components.project` module represents +The `Dataset` class from the `datumaro.components.dataset` module represents a dataset, consisting of multiple `DatasetItem`s. Annotations are represented by members of the `datumaro.components.extractor` module, such as `Label`, `Mask` or `Polygon`. A dataset can contain items from one or @@ -80,16 +79,19 @@ The main operation for a dataset is iteration over its elements. An item corresponds to a single image, a video sequence, etc. There are also few other operations available, such as filtration (`dataset.select`) and transformations (`dataset.transform`). A dataset can be created from extractors -or other datasets with `dataset.from_extractors` and directly from items with -`dataset.from_iterable`. A dataset is an extractor itself. If it is created from -multiple extractors, their categories must match, and their contents will be -merged. +or other datasets with `Dataset.from_extractors()` and directly from items with +`Dataset.from_iterable()`. A dataset is an extractor itself. If it is created +from multiple extractors, their categories must match, and their contents +will be merged. A dataset item is an element of a dataset. Its `id` is a name of a corresponding image. There can be some image `attributes`, an `image` and `annotations`. ```python +from datumaro.components.dataset import Dataset +from datumaro.components.extractor import Bbox, Polygon, DatasetItem + # create a dataset from other datasets dataset = Dataset.from_extractors(dataset1, dataset2) @@ -105,7 +107,7 @@ dataset = Dataset.from_iterable([ dataset = dataset.select(lambda item: len(item.annotations) != 0) # change dataset labels -dataset = dataset.transform(project.env.transforms.get('remap_labels'), +dataset = dataset.transform('remap_labels', {'cat': 'dog', # rename cat to dog 'truck': 'car', # rename truck to car 'person': '', # remove this label @@ -116,8 +118,7 @@ for item in dataset: print(item.id, item.annotations) # iterate over subsets -for subset_name in dataset.subsets(): - subset = dataset.get_subset(subset_name) # a dataset, again +for subset_name, subset in dataset.subsets().items(): for item in subset: print(item.id, item.annotations) ``` @@ -129,6 +130,7 @@ persistence, of extending, and CLI operation for Datasets. A project can be converted to a Dataset with `project.make_dataset`. Project datasets can have multiple data sources, which are merged on dataset creation. They can have a hierarchy. Project configuration is available in `project.config`. +A dataset can be saved in `datumaro_project` format. The `Environment` class is responsible for accessing built-in and project-specific plugins. For a project, there is an instance of @@ -204,11 +206,12 @@ YoloConverter.convert(dataset, save_dir=dst_dir) ### Writing a plugin -A plugin is a Python module with any name, which exports some symbols. -To export a symbol, inherit it from one of special classes: +A plugin is a Python module with any name, which exports some symbols. Symbols, +starting with `_` are not exported by default. To export a symbol, +inherit it from one of the special classes: ```python -from datumaro.components.extractor import Importer, SourceExtractor, Transform +from datumaro.components.extractor import Importer, Extractor, Transform from datumaro.components.launcher import Launcher from datumaro.components.converter import Converter ``` @@ -224,6 +227,19 @@ There is also an additional class to modify plugin appearance in command line: ```python from datumaro.components.cli_plugin import CliPlugin + +class MyPlugin(Converter, CliPlugin): + """ + Optional documentation text, which will appear in command-line help + """ + + NAME = 'optional_custom_plugin_name' + + def build_cmdline_parser(self, **kwargs): + parser = super().build_cmdline_parser(**kwargs) + # set up argparse.ArgumentParser instance + # the parsed args are supposed to be used as invocation options + return parser ``` #### Plugin example @@ -269,13 +285,14 @@ class MyTransform(Transform, CliPlugin): `my_plugin2.py` contents: ```python -from datumaro.components.extractor import SourceExtractor +from datumaro.components.extractor import Extractor class MyFormat: ... -class MyFormatExtractor(SourceExtractor): ... +class _MyFormatConverter(Converter): ... +class MyFormatExtractor(Extractor): ... exports = [MyFormat] # explicit exports declaration -# MyFormatExtractor won't be exported +# MyFormatExtractor and _MyFormatConverter won't be exported ``` ## Command-line diff --git a/docs/user_manual.md b/docs/user_manual.md index df93d7845dd9..675c7c9242ee 100644 --- a/docs/user_manual.md +++ b/docs/user_manual.md @@ -45,11 +45,15 @@ python -m virtualenv venv Install: ``` bash +# From PyPI: +pip install datumaro + +# From the GitHub repository: pip install 'git+https://github.com/openvinotoolkit/datumaro' ``` > You can change the installation branch with `...@` -> Also note `--force-reinstall` parameter in this case. +> Also use `--force-reinstall` parameter in this case. ## Interfaces