Enable video stream access

[nirabo]

Hi,

My workflow is such that I have thousands of frames per annotation task, which amounts to extensive disk space usage (e.g. a <15MB video (~40s VGA@30fps) results in ~2GB in jpeg's).

Adding the ability of CVAT to work directly on a video stream will be a significant improvement as it will allow for the user to only specify a URL/path with potentially an optional download and local storage capability.

One way I see this to be done is by usage of OpenCV.js (https://docs.opencv.org/3.4/d5/d10/tutorial_js_root.html).

I could invest time in this. Let me know of your thoughts.

Thanks

[nmanovic]

Hi @bubalazi ,

In general I believe it will be a cool feature. Several comments:

• When CVAT server extracts frames we know that it does the same way for a video file (doesn't depend on a client). The main problem with navigation by frames directly using OpenCV it can be imprecise and in general depends on platform (VideoCapture::set (CAP_PROP_POS_FRAMES, frameNumber) not exact in opencv 3.2 with ffmpeg opencv/opencv#9053). Precise frame navigation is very important to have correct annotations. Is it a problem for OpenCV.js?
• Probably one way to solve the problem on server side is "dynamic frame extraction". It means that optionally server can try to extract frames on the fly with some internal cache and prefetch. Of course performance and server load will suffer but it can be implemented as an optional feature. Thus only small part of server side should be improved. For client the change will be invisible but it doesn't solve problem with big traffic between clients and a server.
• If the feature is implemented in client and we don't have problems with "unreliable navigation" it will be the best option but both client and server code should be updated.

If you could invest time in this it will be awesome!

[wangpichao]

Hi,

My workflow is such that I have thousands of frames per annotation task, which amounts to extensive disk space usage (e.g. a <15MB video (~40s VGA@30fps) results in ~2GB in jpeg's).

Adding the ability of CVAT to work directly on a video stream will be a significant improvement as it will allow for the user to only specify a URL/path with potentially an optional download and local storage capability.

One way I see this to be done is by usage of OpenCV.js (https://docs.opencv.org/3.4/d5/d10/tutorial_js_root.html).

I could invest time in this. Let me know of your thoughts.

Thanks

Have you finished the job?

[nmanovic]

@dogandemir_gitlab ,

Now that I think of it, it kind of is crazy. For a 2MB 10 sec (420 frame) video, it will store and download 143 MB of frames.

Probably in the future it will be good to convert a video file to a "simple video format" (to avoid problems with positioning inside the video stream) and stream it to clients but it is a big task. Found a library which allows to decode video in a browser: https://github.com/phoboslab/jsmpeg (need to check).

[bsekachev]

Just related issue: w3c/media-and-entertainment#4

[nmanovic]

Particular fixed. Let's see how it works and re-open the issue if we need to improve.