frame length for training

[cheezecrisp]

Prepared videos of 49 frames for training, set the frame_buckets as [49], but got this error

video with shape torch.Size([3, 41, 256, 256]) is being skipped because it has less than the target_frames

Then I tried several number of frame_buckets, decreased to 41 it's OK, I don't know why.
So, how to train 49 frames or even larger frame length?

[tdrussell]

Videos are assigned to the first frame bucket such that the video is longer than or equal to the frame bucket. Then it cuts one or more clips from the video of the right size, according to the video_clip_mode strategy. This video only has 41 frames, and you don't have a shorter frame bucket, so it gets dropped.

It has to be this way, because when you train you need to be exactly the number of frames as the bucket. You can't upscale in the time dimension like you can when resizing images/videos, but you can cut clips from videos, so the video has to be >= the frame bucket size.

[cheezecrisp]

Videos are assigned to the first frame bucket such that the video is longer than or equal to the frame bucket. Then it cuts one or more clips from the video of the right size, according to the video_clip_mode strategy. This video only has 41 frames, and you don't have a shorter frame bucket, so it gets dropped.

It has to be this way, because when you train you need to be exactly the number of frames as the bucket. You can't upscale in the time dimension like you can when resizing images/videos, but you can cut clips from videos, so the video has to be >= the frame bucket size.

But all those videos have 49 frames, equal to the frame_buckets setting.

[Cubey42]

I just encountered this also, which is strange because my setup for preparing videos has been fine before, did something change? I have 34 frame videos with a 33 bucket but get video with shape torch.Size([3, 29, 640, 416]) is being skipped because it has less than the target_frames
guess its tied to the fps in some way? lowering the fps to 24 resolved the issue

[tdrussell]

Mismatched framerates are probably the reason. The video has to have at least as many frames as the bucket, after loading the video in HV's native framerate (24). If the video's framerate is not 24, the loading will add / drop frames to make it 24 fps. A bit unintuitive, maybe, but I'm not sure of a better way to do this. If you had a 60 fps video, you would not want to load it at 60 fps, because then it would be trained on as if it were 24 fps, meaning it would be effectively slowed down.

In general I recommend making the input video files approximately the desired time length, or slightly longer, and then just making sure you have enough frame buckets to cover everything.