Fix duplicate frames and respect original frame rate

[rsteckler]

A couple of nuances of how ffmpeg works are fixed here:

1. The default ffmpeg frame rate is 25 fps, but many security cameras are set at 5-15 fps. The new fps command (source_fps) sets the sampling rate to the original video's rate.
2. when decoding videos to images, ffmpeg will duplicate frames to match the output files to the input file. This resulted in many duplicate frames. We can set fps_mode to passthrough to tell ffmpeg that we don't want it to create duplicate frames to fill in the gaps. That's why we used select! We want to dump some frames!

Side note: For debugging ffmpeg, it's useful to add another parameter to the command: "2> /config/www/llmvision/out.txt" so we can see what it's doing.

This fixes #118 and also results in nice, smooth frames and drastically improves the descriptions the LLM returns.

[rsteckler]

Also note this is extracting every other frame. If the intent was different, let me know and I can adjust the command to meet the intent

[valentinfrlch]

Thanks so much! It will probably take me a while to test and merge this, but I definitely will!

[valentinfrlch]

Can I ask why you closed this? Sorry it takes this long, I'm very busy until next week. I know its frustrating when it takes so long to get a PR merged but I think your contribution would be a great addition!

[rsteckler]

Oh sorry. I merged the branches and deleted them in my repo without thinking about it. I'll restore the branches

[rsteckler]

I also have a version that I'm using now that allows a parameter to pull out evenly spaced frames from the video if you want that

[rsteckler]

Spent some more time with this. The version in the PR extracts every interval'th frame (e.g. every second frame) then uses the motion filter to pick the top n up to max_frames.

I have two other versions on my own fork, each controlled by params. Let me know if you want PRs for them:

• An option that uses evenly spaced frames. Takes a video of any length and extracts max_Frames, evenly spaced throughout the video. Works for all frame rates. We always keep the first and last frame. For example, if the video is 100 frames and you max_frames is 10, you get frames 1, 12, 23, 34, 45, 56, 67, 78, 89, 100. Note that ffmpeg only works per-frame so we either have to pre-process to determine the number of frames, or post-process and remove frames. I opted for the latter so ffmpeg extracts all frames to the tmp directory, then we just drop all the frames that we don't want. This takes a little more processing and disk space, but immaterial compared to the other processing for llm vision.
• An option that captures one frame every nth seconds. This one is pretty straightforward, but I haven't tested it as much. Here's the ffmpeg command:

                        # Extract frames from video every frame_per_interval seconds
                        ffmpeg_cmd = [
                            "ffmpeg",
                            "-i", video_path,
                            "-vf", f"fps=fps='source_fps',select='bitor(gte(t-prev_selected_t\\,{frame_per_interval})\\,isnan(prev_selected_t))'",
                            "-fps_mode", "passthrough",
                            os.path.join(tmp_frames_dir, "frame%04d.jpg")
                        ]

Let me know if you want PRs for any of that and happy to send them upstream

[valentinfrlch]

Thanks again for your work!