Restore docs directory (cvat-ai#419)

TOsmanov · Aug 19, 2021 · 935fb0e · 935fb0e
1 parent 4502695
commit 935fb0e
Show file tree

Hide file tree

Showing 18 changed files with 4,563 additions and 13 deletions.
diff --git a/README.md b/README.md
@@ -16,15 +16,116 @@ CVAT annotations                             ---> Publication, statistics etc.
 ```
 <!--lint enable fenced-code-flag-->
 
-- [Getting started](https://openvinotoolkit.github.io/datumaro/getting_started)
-- [Examples](https://openvinotoolkit.github.io/datumaro/examples)
+# Table of Contents
+
+- [Examples](#examples)
 - [Features](#features)
-- [User manual](https://openvinotoolkit.github.io/datumaro/docs/user-manual)
+- [Installation](#installation)
+- [Usage](#usage)
+- [User manual](docs/user_manual.md)
 - [Contributing](#contributing)
 
+## Examples
+
+[(Back to top)](#table-of-contents)
+
+<!--lint disable list-item-indent-->
+<!--lint disable list-item-bullet-indent-->
+
+- Convert PASCAL VOC dataset to COCO format, keep only images with `cat` class
+  presented:
+  ```bash
+  # Download VOC dataset:
+  # http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
+  datum convert --input-format voc --input-path <path/to/voc> \
+                --output-format coco \
+                --filter '/item[annotation/label="cat"]' \
+                -- --reindex 1 # avoid annotation id conflicts
+  ```
+
+- Convert only non-`occluded` annotations from a
+  [CVAT](https://github.com/opencv/cvat) project to TFrecord:
+  ```bash
+  # export Datumaro dataset in CVAT UI, extract somewhere, go to the project dir
+  datum filter -e '/item/annotation[occluded="False"]' \
+    --mode items+anno --output-dir not_occluded
+  datum export --project not_occluded \
+    --format tf_detection_api -- --save-images
+  ```
+
+- Annotate MS COCO dataset, extract image subset, re-annotate it in
+  [CVAT](https://github.com/opencv/cvat), update old dataset:
+  ```bash
+  # Download COCO dataset http://cocodataset.org/#download
+  # Put images to coco/images/ and annotations to coco/annotations/
+  datum import --format coco --input-path <path/to/coco>
+  datum export --filter '/image[images_I_dont_like]' --format cvat \
+    --output-dir reannotation
+  # import dataset and images to CVAT, re-annotate
+  # export Datumaro project, extract to 'reannotation-upd'
+  datum merge reannotation-upd
+  datum export --format coco
+  ```
+
+- Annotate instance polygons in [CVAT](https://github.com/opencv/cvat), export
+  as masks in COCO:
+  ```bash
+  datum convert --input-format cvat --input-path <path/to/cvat.xml> \
+                --output-format coco -- --segmentation-mode masks
+  ```
+
+- Apply an OpenVINO detection model to some COCO-like dataset,
+  then compare annotations with ground truth and visualize in TensorBoard:
+  ```bash
+  datum import --format coco --input-path <path/to/coco>
+  # create model results interpretation script
+  datum model add mymodel openvino \
+    --weights model.bin --description model.xml \
+    --interpretation-script parse_results.py
+  datum model run --model mymodel --output-dir mymodel_inference/
+  datum diff mymodel_inference/ --format tensorboard --output-dir diff
+  ```
+
+- Change colors in PASCAL VOC-like `.png` masks:
+  ```bash
+  datum import --format voc --input-path <path/to/voc/dataset>
+
+  # Create a color map file with desired colors:
+  #
+  # label : color_rgb : parts : actions
+  # cat:0,0,255::
+  # dog:255,0,0::
+  #
+  # Save as mycolormap.txt
+
+  datum export --format voc_segmentation -- --label-map mycolormap.txt
+  # add "--apply-colormap=0" to save grayscale (indexed) masks
+  # check "--help" option for more info
+  # use "datum --loglevel debug" for extra conversion info
+  ```
+
+- Create a custom COCO-like dataset:
+  ```python
+  import numpy as np
+  from datumaro.components.extractor import (DatasetItem,
+    Bbox, LabelCategories, AnnotationType)
+  from datumaro.components.dataset import Dataset
+
+  dataset = Dataset(categories={
+    AnnotationType.label: LabelCategories.from_iterable(['cat', 'dog'])
+  })
+  dataset.put(DatasetItem(id=0, image=np.ones((5, 5, 3)), annotations=[
+    Bbox(1, 2, 3, 4, label=0),
+  ]))
+  dataset.export('test_dataset', 'coco')
+  ```
+
+<!--lint enable list-item-bullet-indent-->
+<!--lint enable list-item-indent-->
+
 ## Features
 
-[(Back to top)](#dataset-management-framework-datumaro)
+[(Back to top)](#table-of-contents)
 
 - Dataset reading, writing, conversion in any direction. [Supported formats](docs/user_manual.md#supported-formats):
   - [COCO](http://cocodataset.org/#format-data) (`image_info`, `instances`, `person_keypoints`, `captions`, `labels`, `panoptic`, `stuff`)
@@ -43,7 +144,7 @@ CVAT annotations                             ---> Publication, statistics etc.
   - [Cityscapes](https://www.cityscapes-dataset.com/)
   - [Kitti](http://www.cvlibs.net/datasets/kitti/index.php) (`segmentation`, `detection`, `3D raw` / `velodyne points`)
   - [Supervisely](https://docs.supervise.ly/data-organization/00_ann_format_navi) (`point cloud`)
-  - [CVAT](https://github.com/openvinotoolkit/cvat/blob/develop/cvat/apps/documentation/xml_format.md)
+  - [CVAT](https://github.com/opencv/cvat/blob/develop/cvat/apps/documentation/xml_format.md)
   - [LabelMe](http://labelme.csail.mit.edu/Release3.0)
   - [ICDAR13/15](https://rrc.cvc.uab.es/?ch=2) (`word_recognition`, `text_localization`, `text_segmentation`)
   - [Market-1501](https://www.aitribune.com/dataset/2018051063) (`person re-identification`)
@@ -86,19 +187,93 @@ CVAT annotations                             ---> Publication, statistics etc.
     - RISE for classification
     - RISE for object detection
 
-> Check
-  [the design document](https://openvinotoolkit.github.io/datumaro/docs/design)
-  for a full list of features.
-> Check
-  [the user manual](https://openvinotoolkit.github.io/datumaro/docs/user-manual)
-  for usage instructions.
+> Check [the design document](docs/design.md) for a full list of features.
+> Check [the user manual](docs/user_manual.md) for usage instructions.
+
+## Installation
+
+[(Back to top)](#table-of-contents)
+
+### Dependencies
+
+- Python (3.6+)
+- Optional: OpenVINO, TensorFlow, PyTorch, MxNet, Caffe, Accuracy Checker
+
+Optionally, create a virtual environment:
+
+``` bash
+python -m pip install virtualenv
+python -m virtualenv venv
+. venv/bin/activate
+```
+
+Install Datumaro package:
+
+``` bash
+pip install datumaro
+```
+
+## Usage
+
+[(Back to top)](#table-of-contents)
+
+There are several options available:
+- [A standalone command-line tool](#standalone-tool)
+- [A python module](#python-module)
+
+### Standalone tool
+
+Datuaro as a standalone tool allows to do various dataset operations from
+the command line interface:
+
+``` bash
+datum --help
+python -m datumaro --help
+```
+
+### Python module
+
+Datumaro can be used in custom scripts as a Python module. Used this way, it
+allows to use its features from an existing codebase, enabling dataset
+reading, exporting and iteration capabilities, simplifying integration of custom
+formats and providing high performance operations:
+
+``` python
+from datumaro.components.project import Project
+
+# load a Datumaro project
+project = Project.load('directory')
+
+# create a dataset
+dataset = project.make_dataset()
+
+# keep only annotated images
+dataset.select(lambda item: len(item.annotations) != 0)
+
+# change dataset labels
+dataset.transform('remap_labels',
+  {'cat': 'dog', # rename cat to dog
+    'truck': 'car', # rename truck to car
+    'person': '', # remove this label
+  }, default='delete') # remove everything else
+
+# iterate over dataset elements
+for item in dataset:
+  print(item.id, item.annotations)
+
+# export the resulting dataset in COCO format
+dataset.export('dst/dir', 'coco')
+```
+
+> Check our [developer guide](docs/developer_guide.md) for additional
+  information.
 
 ## Contributing
 
-[(Back to top)](#dataset-management-framework-datumaro)
+[(Back to top)](#table-of-contents)
 
 Feel free to
 [open an Issue](https://github.com/openvinotoolkit/datumaro/issues/new), if you
 think something needs to be changed. You are welcome to participate in
 development, instructions are available in our
-[contribution guide](https://openvinotoolkit.github.io/datumaro/contribution-guide/).
+[contribution guide](CONTRIBUTING.md).
diff --git a/docs/cli_design.mm b/docs/cli_design.mm
@@ -0,0 +1,65 @@
+<map version="1.0.1">
+<!-- To view this file, download free mind mapping software FreeMind from http://freemind.sourceforge.net -->
+<node CREATED="1562588909441" ID="ID_362065379" MODIFIED="1562594436169" TEXT="datum">
+<node COLOR="#669900" CREATED="1562588926230" ID="ID_392208345" MODIFIED="1562594653553" POSITION="right" STYLE="fork" TEXT="project">
+<node CREATED="1562592021703" ID="ID_1131736910" MODIFIED="1579775533832" TEXT="create">
+<node CREATED="1574330157737" ID="ID_507280937" MODIFIED="1574330158757" TEXT="Creates a Datumaro project"/>
+</node>
+<node CREATED="1562592669910" ID="ID_1273417784" MODIFIED="1579775533832" TEXT="import">
+<node CREATED="1562592677270" ID="ID_1205701076" MODIFIED="1574330175510" TEXT="Generates a project from other project or dataset in a specific format"/>
+</node>
+<node CREATED="1562592764462" ID="ID_724395644" MODIFIED="1579775533832" TEXT="export">
+<node CREATED="1562592918908" ID="ID_44929477" MODIFIED="1574330221398" TEXT="Saves dataset in a specific format"/>
+</node>
+<node CREATED="1562593914751" ID="ID_378739335" MODIFIED="1579775533832" TEXT="extract">
+<node CREATED="1562593918968" ID="ID_424607257" MODIFIED="1569929409897" TEXT="Extracts subproject by filter"/>
+</node>
+<node CREATED="1569928239212" ID="ID_1246336762" MODIFIED="1579775533832" TEXT="merge">
+<node CREATED="1569928465766" ID="ID_96716547" MODIFIED="1569928867634" TEXT="Adds new items to project"/>
+</node>
+<node CREATED="1562594882533" ID="ID_487465081" MODIFIED="1579775533832" TEXT="diff">
+<node CREATED="1562594886583" ID="ID_1671375265" MODIFIED="1569928079633" TEXT="Compares two projects"/>
+</node>
+<node CREATED="1563435039037" ID="ID_97578583" MODIFIED="1579775533832" TEXT="transform">
+<node CREATED="1563435074116" ID="ID_695576446" MODIFIED="1574330414686" TEXT="Applies specific transformation to the dataset"/>
+</node>
+<node CREATED="1569928386605" ID="ID_493330514" MODIFIED="1579775533832" TEXT="info">
+<node CREATED="1569928423173" ID="ID_1273620035" MODIFIED="1569928429050" TEXT="Outputs valuable info"/>
+</node>
+</node>
+<node COLOR="#669900" CREATED="1562592073422" ID="ID_1793909666" MODIFIED="1569928300945" POSITION="right" STYLE="fork" TEXT="source">
+<node CREATED="1562592085302" ID="ID_199597063" MODIFIED="1579775563506" TEXT="add">
+<node CREATED="1562592138228" ID="ID_1202153971" MODIFIED="1579775556115" TEXT="Adds data source by its URL"/>
+</node>
+<node CREATED="1562592088238" ID="ID_744367784" MODIFIED="1579775533831" TEXT="remove">
+<node CREATED="1562592316435" ID="ID_810859340" MODIFIED="1574330377694" TEXT="Removes source dataset"/>
+</node>
+</node>
+<node COLOR="#669900" CREATED="1563434979149" ID="ID_782927311" MODIFIED="1563435233504" POSITION="right" TEXT="model">
+<node CREATED="1563434987574" ID="ID_290716982" MODIFIED="1579775533831" TEXT="add">
+<node CREATED="1563435018178" ID="ID_1059015375" MODIFIED="1574330372326" TEXT="Registers model for inference"/>
+</node>
+<node CREATED="1564500174410" ID="ID_451702794" MODIFIED="1579775533831" TEXT="remove">
+<node CREATED="1569928809165" ID="ID_1093915022" MODIFIED="1574330359950" TEXT="Removes model from project"/>
+</node>
+<node CREATED="1562593758235" ID="ID_1984980861" MODIFIED="1579775533831" STYLE="fork" TEXT="run">
+<node CREATED="1562593765978" ID="ID_918840812" MODIFIED="1574330356630" TEXT="Executes network for inference"/>
+</node>
+</node>
+<node CREATED="1562594240501" ID="ID_1530017548" MODIFIED="1579775542142" POSITION="right" STYLE="fork" TEXT="create">
+<node CREATED="1562594244868" ID="ID_1309935216" MODIFIED="1562594591882" TEXT="Calls project create"/>
+</node>
+<node CREATED="1562594254667" ID="ID_190882752" MODIFIED="1579775542142" POSITION="right" STYLE="fork" TEXT="add">
+<node CREATED="1562594262484" ID="ID_949937557" MODIFIED="1579701784334" TEXT="Calls source add"/>
+</node>
+<node CREATED="1562594276540" ID="ID_1430572506" MODIFIED="1579775542142" POSITION="right" STYLE="fork" TEXT="remove">
+<node CREATED="1562594281180" ID="ID_124160415" MODIFIED="1562594591248" TEXT="Calls source remove"/>
+</node>
+<node CREATED="1562594289395" ID="ID_1608995178" MODIFIED="1579775542142" POSITION="right" STYLE="fork" TEXT="export">
+<node CREATED="1579775617910" ID="ID_1698217727" MODIFIED="1579775622801" TEXT="Calls project export"/>
+</node>
+<node CREATED="1567594310257" ID="ID_995434490" MODIFIED="1579775542141" POSITION="right" TEXT="explain">
+<node CREATED="1567594365942" ID="ID_1529218756" MODIFIED="1567594404172" TEXT="Runs inference explanation"/>
+</node>
+</node>
+</map>