Data streaming using chunks (#1007)

Huge feature (200+ commits from different developers). It completely changes layout of data (please expect very long DB migration process if you have a lot of tasks). The primary idea is to send data as zip chunks (e.g. 36 images in one chunk) or encoded video chunks and decode them on the client side. It helps to solve the problem with latency  when you try to view a separate frame in the UI quickly (play mode).
Another important feature of the patch is to provide access to the original images. Thus for annotations the client uses compressed chunks but if you want to export a dataset Datumaro will use original chunks (but video will be decoded with original quality and encoded with maximum/optimal quality in any case).

.codacy.yml

@@ -1,4 +1,5 @@
 exclude_paths:
  - '**/3rdparty/**'
  - '**/engine/js/cvat-core.min.js'
+ - '**/engine/js/unzip_imgs.js'
  - CHANGELOG.md

.travis.yml

@@ -14,4 +14,4 @@ before_script:
 script:
   - docker-compose -f docker-compose.yml -f docker-compose.ci.yml run cvat_ci /bin/bash -c 'python3 manage.py test cvat/apps utils/cli'
   - docker-compose -f docker-compose.yml -f docker-compose.ci.yml run cvat_ci /bin/bash -c 'python3 manage.py test datumaro/'
-  - docker-compose -f docker-compose.yml -f docker-compose.ci.yml run cvat_ci /bin/bash -c 'cd cvat-core && npm install && npm run test && npm run coveralls'
+  - docker-compose -f docker-compose.yml -f docker-compose.ci.yml run cvat_ci /bin/bash -c 'cd cvat-data && npm install && cd ../cvat-core && npm install && npm run test && npm run coveralls'

CONTRIBUTING.md

@@ -12,52 +12,60 @@ patches and features.
 
 Next steps should work on clear Ubuntu 18.04.
 
-- Install necessary dependencies:
-
-```sh
-$ sudo apt-get update && sudo apt-get --no-install-recommends install -y ffmpeg build-essential nodejs npm curl redis-server python3-dev python3-pip python3-venv libldap2-dev libsasl2-dev
-```
+-   Install necessary dependencies:
+    ```sh
+    $ sudo apt-get update && sudo apt-get --no-install-recommends install -y ffmpeg build-essential nodejs npm curl redis-server python3-dev python3-pip python3-venv libldap2-dev libsasl2-dev
+    ```
+    Also please make sure that you have installed ffmpeg with all necessary libav* libraries and pkg-config package.
+    ```sh
+    # General dependencies
+    sudo apt-get install -y pkg-config
+
+    # Library components
+    sudo apt-get install -y \
+        libavformat-dev libavcodec-dev libavdevice-dev \
+        libavutil-dev libswscale-dev libswresample-dev libavfilter-dev
+    ```
+    See [PyAV Dependencies installation guide](http://docs.mikeboers.com/pyav/develop/overview/installation.html#dependencies)
+    for details.
 
 -   Install [Visual Studio Code](https://code.visualstudio.com/docs/setup/linux#_debian-and-ubuntu-based-distributions)
 for development
 
 -   Install CVAT on your local host:
-
-```sh
-git clone https://github.com/opencv/cvat
-cd cvat && mkdir logs keys
-python3 -m venv .env
-. .env/bin/activate
-pip install -U pip wheel setuptools
-pip install -r cvat/requirements/development.txt
-pip install -r datumaro/requirements.txt
-python manage.py migrate
-python manage.py collectstatic
-```
-
-- Create a super user for CVAT:
-
-```sh
-$ python manage.py createsuperuser
-Username (leave blank to use 'django'): ***
-Email address: ***
-Password: ***
-Password (again): ***
-```
-
-- Install npm packages for UI and start UI debug server (run the following command from CVAT root directory):
-```sh
-npm install && \
-cd cvat-core && npm install && \
-cd ../cvat-canvas && npm install && \
-cd ../cvat-ui && npm install && npm start
-```
-
-- Open new terminal (Ctrl + Shift + T), run Visual Studio Code from the virtual environment
-
-```sh
- cd .. && source .env/bin/activate && code
-```
+    ```sh
+    git clone https://github.com/opencv/cvat
+    cd cvat && mkdir logs keys
+    python3 -m venv .env
+    . .env/bin/activate
+    pip install -U pip wheel setuptools
+    pip install -r cvat/requirements/development.txt
+    pip install -r datumaro/requirements.txt
+    python manage.py migrate
+    python manage.py collectstatic
+    ```
+
+-   Create a super user for CVAT:
+    ```sh
+    $ python manage.py createsuperuser
+    Username (leave blank to use 'django'): ***
+    Email address: ***
+    Password: ***
+    Password (again): ***
+    ```
+
+-   Install npm packages for UI and start UI debug server (run the following command from CVAT root directory):
+    ```sh
+    cd cvat-core && npm install && \
+    cd ../cvat-canvas && npm install && \
+    cd ../cvat-data && npm install && \
+    cd ../cvat-ui && npm install && npm start
+    ```
+
+-   Open new terminal (Ctrl + Shift + T), run Visual Studio Code from the virtual environment
+    ```sh
+    cd .. && source .env/bin/activate && code
+    ```
 
 -   Install followig vscode extensions:
     - [Debugger for Chrome](https://marketplace.visualstudio.com/items?itemName=msjsdiag.debugger-for-chrome)

Dockerfile

@@ -35,8 +35,17 @@ RUN apt-get update && \
         supervisor \
         ffmpeg \
         gstreamer0.10-ffmpeg \
+        libavcodec-dev \
+        libavdevice-dev \
+        libavfilter-dev \
+        libavformat-dev \
+        libavutil-dev \
+        libldap2-dev \
+        libswresample-dev \
+        libswscale-dev \
         libldap2-dev \
         libsasl2-dev \
+        pkg-config \
         python3-dev \
         python3-pip \
         tzdata \
@@ -57,7 +66,8 @@ RUN apt-get update && \
     dpkg-reconfigure -f noninteractive tzdata && \
     add-apt-repository --remove ppa:mc3man/gstffmpeg-keep -y && \
     add-apt-repository --remove ppa:mc3man/xerus-media -y && \
-    rm -rf /var/lib/apt/lists/*
+    rm -rf /var/lib/apt/lists/* && \
+    echo 'application/wasm wasm' >> /etc/mime.types
 
 # Add a non-root user
 ENV USER=${USER}
@@ -123,6 +133,7 @@ COPY ssh ${HOME}/.ssh
 COPY utils ${HOME}/utils
 COPY cvat/ ${HOME}/cvat
 COPY cvat-core/ ${HOME}/cvat-core
+COPY cvat-data/ ${HOME}/cvat-data
 COPY tests ${HOME}/tests
 COPY datumaro/ ${HOME}/datumaro
 

Dockerfile.ui

@@ -17,6 +17,11 @@ ENV TERM=xterm \
 COPY cvat-core/package*.json /tmp/cvat-core/
 COPY cvat-canvas/package*.json /tmp/cvat-canvas/
 COPY cvat-ui/package*.json /tmp/cvat-ui/
+COPY cvat-data/package*.json /tmp/cvat-data/
+
+# Install cvat-data dependencies
+WORKDIR /tmp/cvat-data/
+RUN npm install
 
 # Install cvat-core dependencies
 WORKDIR /tmp/cvat-core/
@@ -31,12 +36,14 @@ WORKDIR /tmp/cvat-ui/
 RUN npm install
 
 # Build source code
+COPY cvat-data/ /tmp/cvat-data/
 COPY cvat-core/ /tmp/cvat-core/
 COPY cvat-canvas/ /tmp/cvat-canvas/
 COPY cvat-ui/ /tmp/cvat-ui/
 RUN npm run build
 
 FROM nginx:stable-alpine
 # Replace default.conf configuration to remove unnecessary rules
+RUN sed -i "s/}/application\/wasm wasm;\n}/g" /etc/nginx/mime.types
 COPY cvat-ui/react_nginx.conf /etc/nginx/conf.d/default.conf
 COPY --from=cvat-ui /tmp/cvat-ui/dist /usr/share/nginx/html/