Inference hangs during predicting and video loading issues

[nal10]

I have a couple things going on. When I try to run inference on the entire video, the program will hang when it gets to predicting and the progress bar window will stop responding. Here is the command line info:

     [[{{node EagerPyFunc}}]]

2021-03-26 10:16:49.560889: W tensorflow/core/framework/op_kernel.cc:1755] Unknown: KeyError: "Unable to load frame 15 from MediaVideo(filename='C:/Users/as712/Documents/multi_track/OFT_1.mp4', grayscale=True, bgr=True, dataset='', input_format='channels_last')."
Traceback (most recent call last):

File "c:\users\public\anaconda\envs\sleap_env\lib\site-packages\tensorflow\python\ops\script_ops.py", line 242, in call
return func(device, token, args)

File "c:\users\public\anaconda\envs\sleap_env\lib\site-packages\tensorflow\python\ops\script_ops.py", line 131, in call
ret = self._func(*args)

File "c:\users\public\anaconda\envs\sleap_env\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 302, in wrapper
return func(*args, **kwargs)

File "C:\Users\Public\Anaconda\envs\sleap_env\lib\site-packages\sleap\nn\data\providers.py", line 388, in py_fetch_frame
raw_image = self.video.get_frame(frame_ind)

File "C:\Users\Public\Anaconda\envs\sleap_env\lib\site-packages\sleap\io\video.py", line 1017, in get_frame
return self.backend.get_frame(idx)

File "C:\Users\Public\Anaconda\envs\sleap_env\lib\site-packages\sleap\io\video.py", line 478, in get_frame
raise KeyError(f"Unable to load frame {idx} from {self}.")

KeyError: "Unable to load frame 15 from MediaVideo(filename='C:/Users/as712/Documents/multi_track/OFT_1.mp4', grayscale=True, bgr=True, dataset='', input_format='channels_last')."

tracking.of_window_size: None
tracking.of_max_levels: None
tracking.kf_node_indices: None
tracking.kf_init_frame_count: None
Versions:
SLEAP: 1.1.1
TensorFlow: 2.3.1
Numpy: 1.18.5
Python: 3.6.13
OS: Windows-10-10.0.19041-SP0

System:
GPUs: 1/1 available
Device: /physical_device:GPU:0
Available: True
Initalized: False
Memory growth: True

I noticed that the first few frames were throwing errors about not being able to be loaded and went to the frame scroll bar at the bottom of the main GUI window. I found that if I click on the first frame, then to a random frame in the middle of the video and back to the first frame, the first frame is completely different and not the same as if I play the video from a media player. Also, the first ~250 frames are disjointed or frozen, so there seems to be a bug in how the GUI is loading .MP4 files. This occurs on multiple videos of various lengths.

Not sure if these two issues are related and the inability to properly read a bunch of frames is is crashing the GUI.

[talmo]

Hi @nal10,

Yup, these are all indicative of a common issue relating to how videos are encoded. While they may sometimes be playable in media players like VLC, it usually takes a lot of extra heuristics to make sure that you're landing on the frame you expect -- and even then, it might not actually look identical since lots of video compression formats rely on reconstruction from adjacent frames.

In SLEAP, we try to use the most widely compatible video readers, but because we access frames individually and out of order, you'll run into these sorts of issues. Again, this is because most video formats were not designed for this kind of use case -- they are intended for playing back at 25 FPS sequentially.

An easy fix is to re-encode your video using ffmpeg. Try using this command and then going to File -> Replace videos... in the GUI to swap them out:

ffmpeg -i "C:/Users/as712/Documents/multi_track/OFT_1.mp4" -c:v libx264 -crf 20 -preset superfast -pix_fmt yuv420p "C:/Users/as712/Documents/multi_track/OFT_1.sf.mp4"

Give that a spin and let me know if you're still having issues.

Oh and one more thing: you might consider updating to SLEAP v1.1.3 which contains a few bug fixes that may be relevant for your project: https://github.com/murthylab/sleap/releases/tag/v1.1.3

[nal10]

Hi Talmo,

Thank you for the quick response. your re-encoding recommendation along with updating fixed the problem.

[talmo]

Great! I'll close this issue but feel free to comment again if you run into any problems.

Cheers,

Talmo