[css-animationworklet] Sending data to animators in worklet from main thread

[majido]

Many useful effects running inside animation worklet may need to know about specific values from main thread, e.g., size of window or bounding client rects of various objects.

We allow such data to be sent when worklet animation (via options property) is being constructed. However this is limited since these values may change during the lifetime of the animation (e.g., window resize). Currently in these cases one has to cancel the current animation and create a new one. This does not work well and can cause jusmp if the animation has internal state.

A simple logical extension is to allow options to be updated. Here is a proposal that achieves this:

In Animation Worklet context:

registerAnimator('abc', class {
  constructor(options) {
    this.options = options;
  }
  optionsChanged(options) {
    this.options = options;
    // TODO figure out if this should also implicitly invalidated the animation such that animate() is always called or alternatively we have a way to request one frame!
  }
  animate(time, effect) {
    // uses options.height as needed
  }
});

In document context:

let options = { height: top.innerHeight };
const animation = new WorkletAnimation('abc', keyframe, options);
animation.play()

window.addEventListener('resize', e => {
  options.height = top.innerHeight;
  animation.updateOptions(options); 
});

Why method instead of a property?
This means we don't need to keep the options as a live object. This is similar to other Web-animation APIs.

[majido]

An alternative approach is is support postMessage on WorkletAnimation. AudioWorklet has taken this approach.

Using postMessage has a few advantages:

• Well known semantic and API
• Support transfers in addition to sending copies
• Can be used as a bi-directional channel i.e., send data from worklet to main

There are however several concern with using postMessage:

1. Its semantic may require that we always keep the animator instance alive so that it can receive the message. (Our current implementation in Chrome currently keeps animator instances alive but spec allows animator instances to be torn down at any time).

2. We want to ensure that the data sent by main thread is processed before the next animation frame so that animation can respond to it appropriately. Not clear if this is possible with postMessage.

[majido]

Thinking about this a bit more I believe we should limit this type of API to StatefulAnimator. (This may make the postMessage case more appealing)

First it is not clear to me why a stateless animator needs to receive "options" given that its output should only be dependent on its animate() input currentTime. Therefore for StatelessAnimator if options has changed on main thread, the animation should be restarted (cancel/play) without any visual glitch

More over it is not clear how this can be implemented properly for stateless animators where this may be different between calling optionsChanged and animate.

[css-meeting-bot]

The Houdini Task Force just discussed Sending data to animators in worklet from main thread (here since it is closely related to the above), and agreed to the following:

• RESOLVED: Add input properties, expose for the target, should work for input right now. And we'll pursue events / message channel later for output.

The full IRC log of that discussion

<iank_> Topic: Sending data to animators in worklet from main thread (here since it is closely related to the above)
<TabAtkins> github: https://github.com//issues/869
<iank_> majidvp: Similar issue as previous issue, we want to get some data from the main thread to the animation thread. We have this at startup with the options bag.
<iank_> majidvp: The issue is that there might be new data on the main thread that affect the animation, e.g. sizes of elements change.
<TabAtkins> q+
<iank_> majidvp: One idea is that do allow the options to be mutable, the main thread can up date the uptions, and it'll update the options on the animation thread.
<iank_> majidvp: The other options is using a postmessage port with the animator.
<iank_> flackr: If we want harmonize with css - we could do this with a property map.
<iank_> TabAtkins: Yes .
<iank_> flackr: There are several elements in play, and could read properties out of those elements.
<iank_> TabAtkins: When the animate() function get called, are there element references?
<iank_> majidvp: No the effect has a target, we don't expose that right now, but we could expose a styleMap.
<iank_> TabAtkins: You'd want to limit the number of properties.
<iank_> flackr: You could just expose all the same properties?
<iank_> hober: I agree.
<iank_> TabAtkins: Seems like we just want to expose inputProperties on the target for the effect? It doesn't let you pass information out however.
<iank_> majidvp: There are usecases for more stateful effects, and this stage I'm fine with not solving this.
<iank_> majidvp: These usecases are more important once you feed input events into animation worklet. But we don't have this yet, so its not pressing.
<iank_> bkardell_: A message channel is good for JS, but not good for CSS?
<flackr> iank_: another option is event dispatching
<Rossen_> q?
<TabAtkins> ack
<TabAtkins> zakim, ack
<Zakim> I don't understand 'ack', TabAtkins
<TabAtkins> q-
<Rossen_> ack TabAtkins
<iank_> majidvp: Output could be events which fits nicely with the animations API.
<iank_> majidvp: postmessage is more generic, but has its own lifetime concerns, but you might not want to commit to that.
<iank_> TabAtkins: Add input properties, expose for the target, should work for input right now. And we'll pursue events / message channel later for output.
<iank_> Proposal: Add input properties, expose for the target, should work for input right now. And we'll pursue events / message channel later for output.
<iank_> RESOLVED: Add input properties, expose for the target, should work for input right now. And we'll pursue events / message channel later for output.
<iank_> break 15 mins.
<emilio> ScribeNick: emilio

[mattgperry]

@majido

First it is not clear to me why a stateless animator needs to receive "options" given that its output should only be dependent on its animate() input currentTime.

There's a difference between configurable animations and stateful animations. We set options for every keyframes animation but I wouldn't consider them stateful when they can be resolved for t. Likewise there are spring physics implementations that resolve for t (stateless) and those that integrate time into the previous state (stateful).

For instance, we might want to create a new tween/keyframes animation implementation that accepts references to easing functions not already available in the WebAnimations API that can't be described with a bezier curve (https://github.com/Popmotion/popmotion/blob/master/packages/easing/src/index.ts#L49)

Options and state are different for this reason.

[jimmywarting]

I have a case where i have a <video> element but sometimes it dose not contain any video tracks (only audio tracks).
in those cases i would like to show a visualizer. I do not wish to replace the src using something new like video.srcObject = canvas.captureStream() i already have a media playing. and it would not work to seek, pause and play, etc

I would have wished for something like video.videoTracks.addTrack to exist but that is not the case, it's only possible to do so with text tracks 😞

So my 2nd effort will be to show a canvas behind the video element but that is not as much appealing as just adding a background to the video element instead. Therefore i need to 1) create a audio context context.createMediaElementSource(video) 2) analyze the audio and paint a visualizer to a canvas context.
it would feel unnecessary to have to send a postMessage for every frame, if something could be transferable that would be more awesome.

Maybe someone have some tricks and tips for me of how i can solve this today?

---

it would have been better to be able to add a video track cuz than i can show the video with picture in picture. currently the feature is disabled if there is no video tracks - hmm, don't know how to best solve it... i would wish for it to be possible to have some media buttons in the pip view - i know some work is being done to allow arbitrary content in pip so that you can have any html content you want inside of it. but not possible today