Playback Lag Affects Sound Perception

[Jose-Moreno]

Issue Summary

A few weeks ago I and scribblemaniac tested an issue with a project I was using at my work, where there was a significant delay occurring on my work. After a lot of grinding we identified that this delay is perceptual and is influenced by the amount of layers and frames you have.

Basically the more layers / frames, the laggier the editor playback becomes and sound is audibly offset, however in reality what's offset is the playback of the frames trying to compensate for the lag, but the sound is played at real time speed instead so the discrepancy occurs.

This can be further verified by simply scrubbing and exporting the video since in both instances the sound is correctly mapped out to the animation.

The problem is that this is a pervasive problem that has been affecting users for years and at that moment I realized that every time someone complained about sound delays, this could be one of the potential problems (aside from those that were already fixed)

Expected Results

Animation playback is a tricky endeavor without a doubt. none of the commercial software I use ever tries to sell you real time playback speed with the editor, they usually have terms like pre-render, pre-caching, ram-preview, etc where they forcefully cache a range of images to be played or render them in some sort of intermediate or low latency file type to preview either in a separate panel widget or in the main viewer widget. In general there's always a preview feature regardless, sometimes there's even both or a hybrid of these ideas.

I know Pencil2D caches the images for playback but after years of believing this also cached the entire layer stack the current implementation doesn't seem to be enough to replicate a real time preview feature. This idea of caching is hinted in #965

Once of the first enhancements I'm requesting is improving the playback caching in a way that it really helps reduce the latency when there's a lot of elements. Perhaps we could pair it with the frame range feature so whatever is on this range will be guaranteed to play at real time speed. Perfect for shot playback and looping.

It's also been proposed before in #12 to have a separate preview option, which I would also agree to have as well if we consider that this can and should be used for the entire animation. pre-caching, software rendering or otherwise lowering the latency to have real time playback in a different widget (or a canvas mode so it doesn't look alien) is a technique widely used in various animation applications as well.

Video or Image Reference

This is a step by step explanation of how the issue occurs. It is quite involved so apologies if I don't provide the steps in written form.
https://youtu.be/u7PBZetRpN4

Steps to reproduce

Please refer to the video above but for a brief summary:

1. Open Pencil2D
2. Import a movie video to have a base line to test with plenty frames
3. Import the appropriate movie audio
4. Start adding layers on top without any frame. Test the playback latency per every layer you add and record your findings.
5. Now add a few frames on every layer you added and draw something over. Test playback latency again and record the behavior
6. Now hide all the layers except the main bitmap layer for the video. Test the playback latency and notice the difference.

System Information

• Pencil2D Version:

Nightly Build Jan 4 2021

commit: 528aacb
date: 2021-01-04_15:21:03

Operating System: Windows 10 (10.0)
CPU Architecture: x86_64
Qt Version: 5.12.10