Is writing out to ffmpeg supposed to be this slow?

[camblomquist]

The documentation for offline_frame_processing provides an example where a sketch is written out to video using ffmpeg. Trying this in my own sketch which normally runs a smooth 30ish or 60ish frames per second, the sketch slows down to a painful 0.3 fps.

[hx2A]

Hello, @camblomquist . What is the window resolution for the Sketch you are trying to encode to video?

The example for that function uses skvideo but skvideo is not the best video library. I once looked at the source code and how it uses ffmpeg and I believe it piped the frames to ffmpeg via stdin using a raw format of some kind. It isn't efficient at all. Can you try using opencv instead? Using opencv to encode videos isn't that great because the encoding formats available depend on how opencv was compiled, so I didn't want to use it for the example code.

It happens I've been making a lot of screen recordings lately with py5 and I've been using save_frames() to write image sequences and then encoding the image sequence using ffmpeg. This can cause problems though because if it tries to save frames faster than what my hard drive can handle, there is a backlog that takes up memory and the process gets killed. This happens if I try to write 4K or 8K frames at 30 fps. To manage that I have to slow down the Sketch so it isn't creating new frames faster than they can be saved.

The offline_frame_processing() function was created for a situation I faced where I couldn't use opencv and wasn't able to write an image sequence to disk because of space limitations. I also couldn't use OBS or X11grab + ffmpeg. It wasn't a good situation to be in. Using the offline_frame_processing function plus skvideo was slow but it got the job done.

[camblomquist]

First, sorry for my rudeness earlier. The window resolution is 1920x1080.
I've tried skvdeo as the minimal reproducable example, but my actual project uses ffmpeg-python and a larger pipeline. This still uses stdin and probably isn't any faster than skvideo. Pre-rendering using OpenCV might be the next best thing mostly to avoid writing out a large image sequence. Another option would be to replace my entire ffmpeg pipeline with gstreamer, which can read the frames without having to pipe it into a separate process.
Unfortunately I do have other sketches that aren't able to run in real time (turns out that multiple blurs really murders the frame rate,) so OBS isn't an option.

[camblomquist]

A discovery!
I decided to try giving the following function to offline_frame_processing()

def write(frames):
    pass

I would think that this shouldn't slow things down, but it does! It appears that using offline_frame_processing() in any capacity (at least on Windows, haven't checked on my Macbook yet,) ends up taking the frame rate.
So this isn't an issue with stdin, but with the library. If you want an MRE and a proper Issue created, I can do that.

[hx2A]

First, sorry for my rudeness earlier. The window resolution is 1920x1080.

It's fine, we all have our moments. I should tell you I'm currently swamped preparing for a gallery show in two weeks so I am not able to respond as quickly as I would like.

I would think that this shouldn't slow things down, but it does! It appears that using offline_frame_processing() in any capacity (at least on Windows, haven't checked on my Macbook yet,) ends up taking the frame rate.

Yes, for large windows, that is to be expected. offline_frame_processing() needs to load each frame's pixels and copy them into a new numpy array that gets put onto a Queue. There is a separate thread that is popping arrays out from the Queue and passing them to the user provided function. The slow part is copying all of those pixels. That's why a simple pass function will also appear to be slow. It seems to be OK for small windows but for HD sizes or larger, performance will take a hit.

I have re-read the reference documentation for offline_frame_processing() and should re-write the last sentence in the first paragraph. This function is less useful than I thought it would be when I was writing the code.

Can I have more information about what the end goal is here? You have an HD sketch running on Windows that you want to encode to a video. Why not use py5_tools.save_frames() or py5.save_frame() to save an image sequence and then encode the sequence? It can temporarily chew up a lot of space and will be slow but it will get the job done.

On Linux I can run a Sketch in fullscreen mode and then use ffmpeg and x11grab to pull from the display and encode the video. If the encoder works on the GPU like hevc_nvenc does, it can go from x11grab right to the encoder, without leaving the GPU. As long as the Sketch can still run in real time, this will get good results.

When that approach doesn't work I will use py5_tools.save_frames() or py5.save_frame() instead. The key with py5_tools.save_frames() here is to make sure it is not generating new frames faster than they can be saved. If that happens, I run out of memory. To combat that I'll set the frame rate to a low number and keep an eye on the used memory.l

[camblomquist]

Congrats on the gallery show!

I know I've been doing a bit of a runaround with Y when I should be talking about X. It's partly tunnel vision and mostly stubbornness. My full script takes an audio file and uses ffmpeg to combine it with an overlay image and the sketch as a background (there's no reason why the overlay image can't be part of the sketch itself I suppose, it's just how I ended up structuring it,) then does a simple fade in and out. The audio file dictates the duration of the video.
Trying to render "live" ends up mostly being a convenience to avoid the step of writing frames out individually just to combine them later. As far as I'm aware, I don't expect to be able to run save_frames() while ffmpeg tries to read them so pre-rendering means doubling the processing time of the script compared to an "ideal" runtime. That assumption could be entirely wrong though. If doing the whole pre-render makes it faster though, who am I to argue really.

[hx2A]

Congrats on the gallery show!

Thanks! The show info is here:

https://bitforms.art/exhibition/world-wind/

The two animated works are both Processing sketches, with py5 playing a heavy role in the development and artistic experimentation. One of the Processing sketches uses py5 as a bridge to connect to Python data libraries (sklearn, etc). This is a not-yet-documented feature that adds an easy to use callPython() method to Processing.

The audio file dictates the duration of the video

This seems like a common Processing design pattern and I think there are better ways to approach this that avoid the dependency on time.

I would recommend running the sketch live and then storing any data you derive from the audio file in a pickle file or numpy file, etc, with the data indexed by the frame number. Then you can re-run the sketch but pull info from the stored file. This would decouple your work from needing to run at a certain speed to keep it in sync with the audio.

When I code in py5 or Processing my code often looks kind of like this:

frame_rate = 30
time_rate = 30
t = 0

def setup():
    py5.size(500, 500)
    py5.frame_rate(frame_rate)

def draw():
    global t
    t += 1 / time_rate

Everything time related is directly linked to the variable t and nothing else. If I was using audio, I would use t to get info from that location in the audio file. Using this pattern, I can set frame_rate to 1 but leave time_rate the same, giving me the exact same sequence of frames. Then I can save the frames and slow down the sketch without affecting the output.