WebCodecs and autoplay policies

[tidoust]

Some browsers disallow automatic playback of videos even when muted until some user gesture, be that by default or through user settings. WebCodecs lets applications decode videos and render decoded frames onto a canvas. Unless I missed something, that can be done without triggering any internal check, giving applications an efficient mechanism to bypass autoplay restrictions of videos.

How can a user agent enforce its autoplay policy for videos when WebCodecs is used?

Would that be done at the canvas level, meaning that rendering would not work until user makes a gesture? That seems hard to me, given that canvas can be used for pretty much everything, and not only for video playback (one could argue that a 2D/3D animation is a video, but well, let's focus on "real" ones).

If that cannot easily be achieved at the canvas level, one option would be to amend WebCodecs algorithms to include a concept similar to allowed to start in the Web Audio API. That would allow browsers to defer decoding until the window has sticky activation. However, that would prevent scenarios that could decode frames for other purposes from starting immediately.

[dalecurtis]

Is there a user agent which restricts animation of canvas content? WebCodecs is harder to use than an array of still images drawn to canvas on a timer, so I'm not sure we need any special considerations here.

[tidoust]

I note an old (2016) discussion on the use of animated images to work around autoplay restrictions in the HTML repo: whatwg/html#976

As soon as you have more than a hundred of still images to draw, they likely need to be compressed efficiently for distribution. Existing animated image formats are not fantastic for that, a proper video codec is a better fit. Applications can decode these formats through code already but that is hard and not efficient. WebCodecs provides an efficient mechanism to deal with such codecs. Turning that into a proper player is not easy, but I suppose that player libraries will be able to handle that.

In other words, WebCodecs greatly reduces the cost (in terms of network, memory and processing consumption) of shipping a custom video player to work around policy restrictions that user agents may try to enforce. I wonder how such user agents plan to handle that.

I do realize that the nuance between an animated image format and a video format is thin (my understanding is that AV1 does not make any distinction between the two for instance) and, similarly, one could argue that 2D/3D animations can be regarded as "videos".

[dalecurtis]

If a user agent finds this necessary, I think this is best solved with restrictions at the canvas level (as you suggest) similar to how autoplay restrictions on play out of decoded audio are solved at the WebAudio/audio tag level.

E.g., if a user has indicated they prefer reduced motion or disabled muted autoplay, canvas updates could be limited to 1fps or otherwise stalled transparently to the page. Ideally with some click-to-play like message informing the user or just unlocking upon user gesture like some autoplay restrictions.

[chcunningham]

I agree w/ @dalecurtis take. Doing something in WebCodecs (e.g. throttling decoder output) is more complex (we can't know if you intend to render the frame, and attempts to track that will be fraught) and leaves much of the problem unsolved (canvas usage outside WebCodecs). @tidoust how do you feel about closing this issue? Maybe filing a similar issue against canvas?

[tidoust]

Fine with closing if people don't have further input.

For the sake of recording scenarios impacted by autoplay considerations, on top of autoplay policies at the browser level, this also affects embedded content scenarios for which the autoplay policy-controlled permission was not explicitly given:

<!-- Allowed to autoplay -->
<iframe src="https://example.org/webcodecs-video.html" allow="autoplay">

<!-- Not allowed to autoplay but (muted) autoplay made easier by WebCodecs -->
<iframe src="https://example.org/webcodecs-video.html">

WebCodecs does not break anything: normative statements defined in HTML will continue to apply. In particular, the eligible for autoplay criterion is scoped to media elements, and canvas are not media elements.

From a developer perspective, I would argue that an "autoplay" permission is likely seen as applying more broadly to media content and not only to media elements. The nuance existed before WebCodecs but remained thin. WebCodecs facilitates the ability to play video without a media element.

Autoplay has accessibility implications, although my understanding is that accessibility concerns are more directly targeted at media content with an audio track. For muted playback, criteria are more around providing controls... controls which are incidentally no longer a given in a canvas-based approach.

Anyway, I don't feel strongly about whether, how and where to address this. As mentioned in previous comments, other technologies mentioned also make it easier to create and render things that can be regarded as videos to a canvas. Throttling canvas updates may be the way to go. I just wanted to record the shift of meaning and potential implications for existing autoplay control features. Now done ;)