Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data streaming using chunks #787

Closed
wants to merge 125 commits into from
Closed
Show file tree
Hide file tree
Changes from 124 commits
Commits
Show all changes
125 commits
Select commit Hold shift + click to select a range
accc88b
add video stream feature
Oct 18, 2019
a048e4a
fix vscode settings
Oct 18, 2019
fcdb153
fix server part
Oct 21, 2019
54b2625
fix server part
Oct 21, 2019
403ccff
fix check is_ node part in frames.js
Oct 21, 2019
d18c69e
remove unnessosary modules. replave cvat-data with mini version
Oct 21, 2019
17d38a6
delete pttjpeg.js, update package-lock
Oct 21, 2019
e6eb0ce
add cvat-data to Dockerfile for tests
Oct 21, 2019
ea757cf
fising
Oct 21, 2019
f2a11ab
fix pr comments
Oct 21, 2019
043c625
fix pr comments
Oct 21, 2019
b7f8971
fix pr comments
Oct 22, 2019
6663dc8
delete unnecessary cvat-data.min.js
Oct 22, 2019
8eab70d
fix docs
Oct 22, 2019
5a28f81
fix problem with preview loading
Oct 22, 2019
be75b7e
fix deleting old frames, update files to min version
Oct 22, 2019
6b583ff
commit unpatched 3rdparty src and update diff
Oct 23, 2019
07863cd
tar -> zip
Oct 23, 2019
ab48d82
fixed config.backendAPI value for annotationUI
Oct 23, 2019
e810538
added new models, migrations
Oct 24, 2019
63155bc
updated cvat-core min
Oct 24, 2019
4326292
fixes
Oct 24, 2019
b3f9d58
fixes & create preview in migration
Oct 24, 2019
12305ec
fixed annotation layout
Oct 24, 2019
3eaf4dc
merge with last develop
Oct 25, 2019
69aa8a5
Merge pull request #1 from ygnn123/yg/mbranch
ygnn123 Oct 25, 2019
3ad96a5
extract frames form video by pyAV
Oct 28, 2019
572aa17
fixed start, stop , step filters for video
Oct 28, 2019
8deca39
save original images
Oct 28, 2019
d8eb30f
delete Data object if it has no related tasks
Oct 28, 2019
c05ca63
several fixes
Oct 29, 2019
dc42963
fix problems with idling loading, fix shift method
Nov 1, 2019
3e14f11
fix problem with last frame
Nov 1, 2019
5b8cc0b
fix problems with zip-images
Nov 5, 2019
a5791a4
Merge pull request #2 from ygnn123/yg/mbranch
ygnn123 Nov 5, 2019
ca0c694
fix merging problems
Nov 5, 2019
a2799db
Merge pull request #3 from ygnn123/yg/mbranch
ygnn123 Nov 5, 2019
f168b65
Merge branch 'develop' into develop
ygnn123 Nov 5, 2019
f879828
fix dockerfile
Nov 5, 2019
08fd03a
unified paths in image meta
Nov 5, 2019
98210b7
fixed pdf extractor
Nov 5, 2019
3e81d5d
Added migration for the auto_annotation models
Nov 5, 2019
4fdcbbe
copy -> move
Nov 5, 2019
6593435
Merge remote-tracking branch 'origin/develop' into az/migration
Nov 5, 2019
369a1bd
fixed migration
Nov 5, 2019
05b0004
decode frames based on chunk type
Nov 5, 2019
e7af854
try catch for migration
Nov 5, 2019
4b71c63
fixed build of cvat ui container
Nov 5, 2019
61ab2c6
migration improvement
Nov 6, 2019
c52be5a
fixed image drawing for annotation mode
Nov 6, 2019
f0ef4cc
Merge pull request #5 from azhavoro/fix_annotation_mode
ygnn123 Nov 6, 2019
2dc3211
fix problems with image play idle
Nov 6, 2019
6da262b
Merge pull request #4 from azhavoro/cvat_az_migration
ygnn123 Nov 6, 2019
033b635
fixed unit tests
Nov 7, 2019
41ca049
Merge pull request #6 from azhavoro/az/fix_unittests
ygnn123 Nov 8, 2019
7e0ee34
fix chunk loading
Nov 8, 2019
ec7bfaf
VideoExtractor:filter video stream only
Nov 8, 2019
a79455e
wip
Nov 11, 2019
b2050ec
working sample
Nov 11, 2019
5af9dd4
video media extractor: remove ffmpy and encode video chunks directly via
Nov 12, 2019
950e751
fixed import of MP4Reader
Nov 13, 2019
151651c
using worker for decoder
Nov 13, 2019
61fa2d0
cleanup
Nov 14, 2019
5e7488a
fix last frame loss
Nov 14, 2019
ea69acf
recompile decoder to increase memory size
Nov 14, 2019
2a58cbd
performance logs
Nov 15, 2019
c86bd85
fix filters
Nov 18, 2019
19103a2
fixed Broadway YUV->RGBA conversation
Nov 18, 2019
3539872
fixed preview creating
Nov 18, 2019
b5607ed
asm.js -> wasm
Nov 19, 2019
025506e
added cropImage function
Nov 19, 2019
9a2c04e
added mimetype for wasm ext
Nov 19, 2019
87cb87a
set crf=10 for original chunks
Nov 20, 2019
38ef91d
Merge remote-tracking branch 'develop' into h264
Nov 20, 2019
acbeb87
missed Decoder.js
Nov 20, 2019
600fef7
downscale compressed video chunks
Nov 26, 2019
ccdaf6d
move upscale to player (it works much faster)
Nov 26, 2019
75b8dde
two chunk types
Nov 26, 2019
6df2ed3
added flag to create tasks with zip chunks for video data
Nov 26, 2019
e5c1c86
fixed player
Nov 29, 2019
b644989
wip
Dec 2, 2019
5a9f335
couple of fixes
Dec 2, 2019
8a0ace3
uncomment downscaling
Dec 3, 2019
0511b6a
fixed player
Dec 3, 2019
776b6ed
removed comments
Dec 3, 2019
f75db83
fixed cleanup method
Dec 5, 2019
7c18c40
little code refactoring
Dec 5, 2019
dc6c319
added decoded frames cache limits
Dec 5, 2019
79bb02a
wip
Dec 10, 2019
3ca071c
added filtering of data chunk requests, fixes
Dec 11, 2019
e781938
fixed onmessage handler for archives
Dec 12, 2019
ffa0344
wip
Dec 12, 2019
4159269
refactored media_extractors
Dec 12, 2019
e0cda34
added zip media reader
Dec 12, 2019
abedac5
added direct zip reader
Dec 13, 2019
6a78dee
direct access to frames
Dec 13, 2019
b746e8d
changed REST API endpoints to get chunks, frames and preview. Also
Dec 13, 2019
2f112e0
removed jsmpeg
Dec 16, 2019
5720d73
enable cli tests,
Dec 16, 2019
3ae467a
merge cvat_develop into develop
Dec 17, 2019
ee8a74d
Merge remote-tracking branch 'cvat_dev/develop' into develop
Dec 17, 2019
387a6af
fixed auto_segmentation
Dec 18, 2019
388cb0c
fixed reid application
Dec 18, 2019
de80790
fixed datumaro
Dec 19, 2019
1ef8c9b
wip
Dec 20, 2019
9a691f5
Merge remote-tracking branch 'cvat_dev/develop' into develop
Dec 21, 2019
a1da83d
wip
Dec 21, 2019
ccbda7d
wip
Dec 23, 2019
1f25caa
added frame buffer
Dec 23, 2019
be881e8
Merge branch 'h264_frame_buffer' into develop
Dec 23, 2019
8cf272e
fix codacy issues
Dec 24, 2019
6da5100
fix codacy issues part1
Dec 24, 2019
7f01b6a
fix codacy issues part2
Dec 24, 2019
7c26da9
fix codacy issues part3
Dec 24, 2019
38503c7
fix codacy issues part4
Dec 24, 2019
60483a8
fix codacy issues part5
Dec 24, 2019
96f0dce
fix codacy issues part6
Dec 24, 2019
f00001f
fix codacy issues part7
Dec 24, 2019
375ef75
fix codacy issues part8
Dec 24, 2019
0c55566
fixed build
Dec 24, 2019
8ccb940
fixed incorect frameNumber resolve
Dec 24, 2019
2d7f075
fixed tests
Dec 25, 2019
8d46ec0
removed 'cache' form Data's path
Dec 25, 2019
5bb9062
re-enabled fast rewind
Dec 25, 2019
6d651b4
fixed tests
Dec 25, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .codacy.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
exclude_paths:
- '**/3rdparty/**'
- '**/engine/js/cvat-core.min.js'
- '**/engine/js/unzip_imgs.js'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is a reason to exclude the file from scanning?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is bundled and minified version of cvat-data/src/js/unzip_imgs.js which included to scanning.

- CHANGELOG.md
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ before_script:
script:
- docker exec -it cvat /bin/bash -c 'python3 manage.py test cvat/apps utils/cli'
- docker exec -it cvat /bin/bash -c 'python3 manage.py test datumaro/'
- docker exec -it cvat /bin/bash -c 'cd cvat-core && npm install && npm run test && npm run coveralls'
- docker exec -it cvat /bin/bash -c 'cd cvat-data && npm install && cd ../cvat-core && npm install && npm run test && npm run coveralls'
27 changes: 18 additions & 9 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -31,29 +31,37 @@ RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -yq \
apache2 \
apache2-dev \
libapache2-mod-xsendfile \
supervisor \
ffmpeg \
git-core \
gstreamer0.10-ffmpeg \
libapache2-mod-xsendfile \
libavcodec-dev \
libavdevice-dev \
libavfilter-dev \
libavformat-dev \
libavutil-dev \
libldap2-dev \
libsasl2-dev \
libswresample-dev \
libswscale-dev \
libsm6 \
libxext6 \
p7zip-full \
pkg-config \
python3-dev \
python3-pip \
supervisor \
tzdata \
unzip \
unrar \
p7zip-full \
vim \
git-core \
libsm6 \
libxext6 && \
vim && \
python3 -m pip install -U pip && \
python3 -m pip install -U setuptools && \
ln -fs /usr/share/zoneinfo/${TZ} /etc/localtime && \
dpkg-reconfigure -f noninteractive tzdata && \
add-apt-repository --remove ppa:mc3man/gstffmpeg-keep -y && \
add-apt-repository --remove ppa:mc3man/xerus-media -y && \
rm -rf /var/lib/apt/lists/*
rm -rf /var/lib/apt/lists/* && \
echo 'application/wasm wasm' >> /etc/mime.types

# Add a non-root user
ENV USER=${USER}
Expand Down Expand Up @@ -154,6 +162,7 @@ COPY ssh ${HOME}/.ssh
COPY utils ${HOME}/utils
COPY cvat/ ${HOME}/cvat
COPY cvat-core/ ${HOME}/cvat-core
COPY cvat-data/ ${HOME}/cvat-data
COPY tests ${HOME}/tests
COPY datumaro/ ${HOME}/datumaro

Expand Down
8 changes: 7 additions & 1 deletion Dockerfile.ui
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,16 @@ RUN apt update && apt install -yq nodejs npm curl && \
npm install -g n && n 10.16.3

# Create output directories
RUN mkdir /tmp/cvat-ui /tmp/cvat-core
RUN mkdir /tmp/cvat-ui /tmp/cvat-core /tmp/cvat-data

# Install dependencies
COPY cvat-core/package*.json /tmp/cvat-core/
COPY cvat-ui/package*.json /tmp/cvat-ui/
COPY cvat-data/package*.json /tmp/cvat-data/

# Install cvat-data dependencies
WORKDIR /tmp/cvat-data/
RUN npm install

# Install cvat-core dependencies
WORKDIR /tmp/cvat-core/
Expand All @@ -37,6 +42,7 @@ WORKDIR /tmp/cvat-ui/
RUN npm install

# Build source code
COPY cvat-data/ /tmp/cvat-data/
COPY cvat-core/ /tmp/cvat-core/
COPY cvat-ui/ /tmp/cvat-ui/
RUN npm run build
Expand Down
Loading