Improve potential for generating multiple animation frames in parallel

[majido]

(Note: Issue was created based on feedback from Houdini Paris F2F - irc logs)

There is interest in improving opportunities for user-agent to generate more animation frames in
parallel. Current API only allows running individual animators in parallel but there are still
potential to do more. For example it is useful to generate multiple frames for the same
animations in parallel an/or ahead of time.

Pure Effects

If an animation is pure (i.e., has no internal state), then it is possible to
generate multiple frames for it in parallel by passing new input and recording the output.
Here are few improvements to the API that leverage this:

• Introduce a mode (maybe this should be the default) where animator is assumed to be pure. We can enforce this via the same mechanisms that is used for paint worklet e.g., random assignments to global scope and dropping the animator object. (Strawman API: https://gist.github.com/anonymous/e22250dbdfa92da59508a220b8087d9d)

• Enable even more effects to be written in this mode. For example if we expose scroll velocity and acceleration then a lots of existing effects don't need to have to keep state to compute it.

Additional idea - Speculative execution

Browsers can also speculatively run an animator in parallel as if it is pure and then fallback to normal operation if they detect a conflict.

[majido]

Here are the relevant IRC log discussions:
till: The internal state that has to be transferred to a worklet means that it's not possible to ever parallize this further.
till: If instead we did something like provide a velocity of the current scroll, or last location, we could potentially do away with internal state, and speculatively execute multiple steps in the timline, and use them if necessary.
flackr: It's worth pointing out that you can run each aniamtor independently.
till: Understood.
flackr: We'd have to specify all different pieces of state ...
TabAtkins: Not necessarily all, just some useful stuff. With a properly designed API, could detect when they're only relying on pure state.
till: Or restrict them to only pure state, if that's not too restrictive.
flackr: Another header example is where your action depends on scroll direction in the past...
till: Right, we could pass that in.
TabAtkins: Or just, instead of mutating, always have it run on pure state that it can produce new versions of.
flackr: Yeah, maybe your animation could specify that it's clean, and we could have a policy of regularly disposing of "clean" animations.
till: This isn't about tearing down an animation, it's about running multiple frames interleaved, and reducing the risk of getting jank in this one animation. That's only possible with pure state.
TabAtkins: Yeah, this could easily be an array of pure position/velocity state, and just returning an updated clone.
iank_: A concer was the cost of structred cloning every frame. Would be better if we could just clone every N frames.
flackr: This is coming from the world where devs would usually write this as a main-thread effect, so here at least they can be run separately.
till: Right, so I see that being able to keep internal state is more powerful, but it keeps you from parallelizing.
flackr: I wonder if there's verification possibility - analyze the script to see if it's pure enough.
till: No, we tried that with parallel.js
TabAtkins: And it's easy to fall out of the pit of success for no apparent reason.
iank_: So if we returned the state every frame from animate, and passed it back in, we could do pure state.
till: Not quite - we want to ask for frame N, and start working on frame N+1, without waiting for frame N to finish.
TabAtkins: Ah, so that does require only relying on UA-provided state.
dbaron till: You could increase your frame budget from 16ms to 32ms if you're doing frames on 2 different cores
till: Yeah, letting us take care of animations on multiple cores.
surma: People are already asking for how to depend on user input. That breaks speculation, right?
till: People don't usually change the velocity of their input in short timescales (30ms or so). You'll occasionally mis-speculate, but mostly will be okay.
birtles: And doing multiple frames at slight offsets to produce motion blur.
TabAtkins: I think this is all quite tractable with minimal changes.
iank_: Would you be okay with starting with a mode that says you only use pure state, and we'll tear down your object regularly to preven tyou from relying on impure state?
till: I probably can't convince you to actually produce fresh objects on every frame, so yeah.
TabAtkins: Get all the JS engines to give us cheap immutable state, and we'll talk. ^^
till: And spec some additional pure state info the UA can provide to the animator.
**iank**: Velocity, accel, maybe last N frames of scroll data...
iank_: So thinking maybe we start with a separate mode, see what people are using with it, start adding more data.
jack: Could we pass in velocity and acceleration by default? You pretty much have to calculate that anyway.
till: Problem with the flag is that we can't default to that. Maybe if we just speculate wrongly for the first N frames we can switch to the stateful version?
flackr: We have this idea that we can add info to the timeline that contains this extra info, like user input too.
iank_: We could invert the flag - default to not passing in new state, you can flag it on to get user-defined state passed around.
till: Seems useful regardless of speculative execution.
iank_ https://gist.github.com/anonymous/e22250dbdfa92da59508a220b8087d9d
till: If we always pass in these values, people can rely on it, and we can just assume things are pure unless we fail speculation.
TabAtkins: How do you speculate if people are using mutating state? Seems you'd mess up the state.
astearns "useful regardless of speculative execution" should go into someone's twitter bio
till: Run one thread as source of truth for a while, speculatively run the same frames ahead of time on another thread. If speculation fails for more than N out of the first M frames (speculation gives different result than the "real" thread), assume they're using mutating state and stop speculating. Otherwise, continue speculating.
surma waves at brucel
till: Something that could be helped by sequential speculative execution is power usage - better to spike the CPU in bursts rather than run it continuously.
brucel: And with 120Hz animation, letting the animator idle as much as possible would be good.
surma s/brucel/dino
till: And with 4 cores, you can split frames between them, have better guarantee of hitting every frame.
dino: So I think you're saying - based on what's happening now, you'll be issuing a lot of updates in a short amoutn of time, so you want to use other cores to pregen some frames ahead now, so they're ready when needed.
dino: So the speculative ones return frames, timestamped, the system uses them?
TabAtkins: Yeah. Speculate for the first N frames along with primary thread, compare values; assume speculation is safe if it succeeds.
dino: Why not let the worklet return multiple successive frames?
TabAtkins: That doesn't solve the problem, right? You're still spending the same amount of CPU as calling it multiple times.
dino: Like, if I know the animation wants to run at 120Hz, but I'll be called at 60Hz, you can do two frames at once.
till: That still requires you to calculate frames at 120Hz. With speculation you can actually pump frames at 60Hz, just every other frame on each core.
iank_ thinks this flag should be: "static callMeMaybe = true;"
TabAtkins: Probably useful to have a flag that specifies whether you think you're safe to speculate or not. If you claim you are, we do some things to keep you honest (tear down your object regularly); if you claim you're not, UAs might still do speculation speculatively to see if you really are, as till described earlier.
flackr: This sounds sensible.
till: This is also based on impl experience we're having with Paint Worklet, we're already doing speculation there. Slightly different cases there, but similar API.
till PR for speculative paint worklet execution: servo/servo#17810

[majido]

/cc @flackr @bfgeek who will correct me if I missed anything and if this is not a good summary of the discussion.

[majido]

I think we should go ahead an introduce a mode where the animator has to explicitly declare that it has state then by default effects are pure and engines can produce frame for them in parallel.

Here is some example API:

registerAnimator('my-statefull-effect', class {
  // This ensures the engine produces frame in order. 
  // hasState is false by default which allows multiple frames to be produced in parallel and animator 
  // to be destroyed/migrated without needing to move state around worklet instances.
  static hasState = true; 
                
  animate(currentTime, effects) {
      const delta = currentTime - this.lastTime;  
      this.lastTime = currentTime;
        
     this.acceleration += Math.random();
     this.velocity += this.acceleration * delta;
     effects[0].localTime = this.velocity * currentTime;
  }
});

[tschneidereit]

@asajeffrey, this is different from the speculative execution we discussed, but I'd be interested in hearing your thoughts on this proposal.

[asajeffrey]

Hmm, interesting...

One possible API for stateful animators would be to provide an explicit state object, and in the API say that it might be deep cloned at any point (e.g. when context switching to a different thread).

[majido]

@asajeffrey we have been thinking about similar mechanism for stateful animator but with a callback instead. Rather than a explicit object that may be deep cloned at "any point" we have been favoring a onDestroy callback that returns a object which is then deep cloned. Here is the basic idea.

One advantage of callback is that the animator does not have to maintain the state object all the time but only when we are migrating between contexts. I expect migrations to be infrequent in which case this can lead to a more optimal outcome.

I suppose if we have an state object then its existence can also play the role of hasState as well. Slightly better API but has its own set of complications.

BTW, it was implied in original comment but I want to make it explicit that I think it is best to treat hasState as static property of animator that is considered constant during its lifetime while the actual state is dynamic.

[flackr]

I like having an explicit state object. If the developer stores nothing in that state then we assume that since it is okay to shut down and resume their animation with no state that it should also be okay to speculatively execute on future values (until doing so results in state being used at which point we have to stop, destroy the speculative animation, and restart it at the point before it became stateful).

The nice thing about this is that the developer doesn't have to think about whether their animation is stateful upfront, and could even have animations that transition between being stateful and stateless based on inputs (e.g. stateful while an object has momentum, stateless when it comes to rest).

[birtles]

I wonder if we need an explicit flag or whether we can look at the shape of the passed-in animator to determine if it is stateful or stateless.

e.g. looking at the length of the constructor function:

// Stateful
registerAnimator('stateful', class {
  constructor(options, state) {
  }
  animate(currentTime, effect) {
  }
});

// Stateless
registerAnimator('stateless', class {
  constructor(options) {
  }
  animate(currentTime, effect) {
  }
});

e.g. differentiate between function objects and generic objects

// Stateful
registerAnimator('stateful', class {
  constructor(options, state) {
  }
  animate(currentTime, effect) {
  }
});

// Stateless
registerAnimator('stateless', (currentTime, effect) => {
  // ...
});

Or we could possibly even do both of the above.

[dbaron]

The beginning of the IRC discussion on this topic, during which IRC went down and caused state loss, was this:

See the rest of the IRC log

<dael> Topic: Animation Worklets
<majidvp> https://github.com//issues/87
<dael> majidvp: Right now when you have an animation worklets everything you register is considered stateful. Some of the effects you don't need to have local state. If your animation doesn't need local state it's good to know.
<dael> majidvp: When it's stateful you need to go in sequence.
<dael> majidvp: Ask was to be able to make the distinction explicit.
<dael> majidvp: Proposal from astearns was to have a state object. This is similar to layout API and caching property.
<astearns> s/astearns/asajeffrey/
<astearns> (I think)
<dael> majidvp: Proposal is to have a state object on the animation which the impl uses to 1) if the state object is available it means it's stateful and not a pure animation and 2) previously we had an onDestroy callback which allowed the animation to serialize. This proposal allows us to get rid of that callback. We can just serialize the state and move it when going between.
<dael> smfr: Seems odd to have the state...Once you computed the state the first time you might want it to be read only
<dael> flackr: If you want that you should pass that in rather then computing it. i'ts a static input.
<dael> shane: Is that too restrictive?
<dael> flackr: Maybe? If you had a state that didn't change regularly you could imagine allowing parallel execution until state changes. I don't know if that's a common use case.
<dael> majidvp: If you have a state it means we opt you out of run in parallel. Could be possible we have a state and until it changes you're in parallel. Let's be more conservitive and if there are use cases we can relax.
<dael> flackr: Only concern with relaxing is detecting the state change. You have to detect or have an API to say I changed.
<dael> majidvp: Initial version was to have a state boolean which would indicate if it allows parallel execution. We can do something like that in the future with the boolean
<dael> smfr: isStateful is a better description the hasState
<dael> majidvp: Current proposal is if the state object exists it means it's stateful, not sue isSateful or hasState.
<dael> smfr: Can you show us the API?
<dael> majidvp: First argument to the consctructor.
<dael> majidvp: It would make sure that if you have an actual state you set it on the state object so when you move between global context you keep your state. If we kill your instance your state sticks.
<dael> majidvp: You can hold onto your state.
<dael> birtles: You look at length of constructor funciton?
<dael> majidvp: Every time we animate you we call the state getter. If that provides an object that's your state and it means you're stateful. Every time we destroy the global scope that state will be passed in to you
<dael> majidvp: It's only when you register the animation. First time you're called it's null, as we animate you can change.
<dael> birtles: If you're looking at length of constructor I understand that, but how do you know.
<dael> majidvp: You're looking at spec, this is a proposal. You get the state and the options.
<dael> birtles: The difference between an animator and a pure animator is clear to anyone that works in React.
<dael> shane: It's an import and distinction too.
<dael> shane: Maybe an flag is acceptable
<dael> majidvp: One thing about taking this flag is it can change over time. YOu start as stateless and at some point you do have a state from that point on you're stable.
<dael> shane: Can you clear state?
<dael> majidvp: Yes.
<dael> shane: This would be useful with impl a state machine. You can make a decision about what the next animation is.
<dael> birtles: What if you have several frames at ones and one is stable.
<dael> shane: You're stateless if it's many frames. If one frame is stateful it stops there.
<dael> majidvp: The example from Jan. they run in parallel and if they see state they can't treat it the same.
<dael> birtles: They're saying it's not impl-able. You're either stateful or stateless, choose one.
<dael> majidvp: Personally I'm fine for it to be static. Most of the use cases weren't specifically stafefull or stateless, but it did change.
<dael> flackr: Let's say we go with something static now, how may it change?
<dael> shane: If you register yourself as stateful or stateless you can register yourself as auto and change.
<dael> flackr: Right thing to do is statis route until we have feedback.
<dael> majidvp: iStateful is the proposed name?
<dael> smfr: Yeah.
<dael> shane: Do we want State or Stateful?
<dael> majidvp: It would be a static property on the animator class.
<dael> birtles: In the current spec you ask to register a function and that will change you pass something else?
<dbaron> shane: It should be enum values so it's extensible.
<dael> majidvp: It's a function callback. It's a class similar to Paint. This is a class that implements callback interface. In the class there's a static property of isStateless
<dael> birtles: I'm wondering if we can just look at functions
<dael> majidvp: My understanding is that there is slighly different syntax for stateful and stateless. If you're stateful component it's a class.
<dael> majidvp: I think in this case static attribute on the class seems usable enough
<dael> shane: Makes it explicit
<dael> majidvp: There's nothing like this on other spec where we can extend
<dael> Rossen: What did we narrow down tp?
<dael> Rossen: Do we keep going with isStateful?
<dael> majidvp: I think it was have explicit attribute on the class, I don't know what hte name should be. If you want an enum stateful isn't a good name.
<dael> majidvp: An enum is the right idea, I'll bikeshed the name on the issue.
<dael> Rossen: WE want to have the state state on the class, name pending bikeshed.
<dael> Rossen: Objections?
<dael> birtles: I'd like to look at other options. I'm not 100% sure we need the attribute.
<dael> shane: If we want to preserve possibility of switching we need it.
<dael> birtles: I think there's more enquery to do.
<dael> Rossen: Before we resolve on the big issue, we can resolve on having the state always be one by construction. Either always stateful or not.
<dael> Rossen: For now we're not allowing a dynamic state. Let's resolve on that and then figure out if we need to expose it.
<dael> Rossen: Objections to not having a dynamic ability on animations, they're either stateful or not.
<dael> RESOLVED: not have a dynamic ability on animations, they're either stateful or not.
<dael> Rossen: Do we need to expose this is the next question.
<dael> majidvp: We want to give author a way to differentiate between the two. I'm in favor of exposing and letting authors say what their animation is. Mechanism I'm impartial. WE can have the bool on the class, we can have different parent classes.
<dael> Rossen: Because you're basing things on the fact that people can take advantage of this performance, can you give us an example of that?
<dael> majidvp: If you want to impl parallax. It doesn't have state, just translates input to state. If we say it's stateless the browser can produce more animations and avoid running JS.
<dael> majidvp: It's more obvious when anmation is time based. In one animation worklet you run the thread once and get multiple frames. For stateful you can't do that because producing one frame might not be there in the next seq. You're losing some performance benefit.
<dael> majidvp: Our impl doesn't run in parallel.
<dael> Rossen: By construction you're impl will know if it's stateless or stable
<dael> majidvp: How would you know.
<dael> smfr: Pasing state in constructor.
<dael> Rossen: Oh. So we resolved by construction they are stateful or stateless So you know if this is one or the other. You don't need to ask anymore. Attribute you'd need in case you change your mind, but I don't see anything dynamic.
<dael> flackr: How do you construct it?
<dael> Rossen: You tell me.
<dael> flackr: This was a static class to say.
<dael> majidvp: Other option would be construction for worklet takes in a state object that's passed into an animation instance. If it's undefined or null it's stateless.
<dael> Rossen: Yeah.
<dael> majidvp: We have an opbect called options. We use it to pass things the animation needs to know. It's similar. I wouldn't mind that.
<dael> Rossen: That way everyone is kind of happy. birtles is happier.
<dael> birtles: I was looking for other options, but yeah. I put on the issues two other things about the constructor. May be other possiblilites with switching.
Connection closed unexpectedly: Server going down
<surma> \o/
<dael> majidvp: You don't have to provide when you do the animation that you have 2 arguments it's stateful
<TabAtkins> esprehn: Can you op rossen or astearns and de-op yourself?
<RRSAgent> logging to https://www.w3.org/2018/04/09-houdini-irc
You invited logbot to join #houdini
<dael> majidvp: birtles proposal in the issue is nice and ergonomic. If it has 2 arguments it means second is tate
<dael> majidvp: Number of arguments.
<dbaron> Topic: Animation Worklets, continued
<dael> iank_: You can't look at the shape of arguments.

[dbaron]

The rest of the discussion ended with one resolution:

• RESOLVED: introduce 2 classes as a mean to listing stateful and stateless types

See the full IRC log

<dbaron> Topic: Animation Worklets, continued
<dael> iank_: You can't look at the shape of arguments.
<dael> Rossen: Your versioning becomes crazy.
<dael> birtles: It's common in JS libraries.
<dbaron> github: https://github.com//issues/87
<dael> iank_: Are you switching inside of native code here?
<dael> iank_: When you call the register animation, you're looking for arguments, that doesn't work. That's not failable.
<dael> birtles: Call .legth
<dael> majidvp: If you call .length it gives you number of required params
<dael> majidvp: If you have opional parameters, in webIDL at least.
<dael> birtles: There's another option further down
<dael> majidvp: I don't think second is possible. Can you differentiate between class and function?
<dael> TabAtkins: No, class returns a fuction
<dael> smfr: I think check is constructbale?
<dael> majidvp: Any is constructable
<dael> gsnedders: Not in webIDL
<dael> majidvp: This is user provided.
<dael> birtles: Point I was trying to make is that we don't nec need an attribute.
<gsnedders> gsnedders: then it does have construct
<dael> Rossen: Checking number of arguments is defined. You can check the length.
<dael> shane: I think there is a more desciplined argument about having it as a param because it's clear to people making the code.
<dael> birtles: I think it's quite intuitive to check the callback function
<dael> Rossen: One of the versioning schemas we have in our API is based on number of parameters
<dael> shane: It's a technique. But we're designing an API. It's bad design to change fundimentally different behaviors based on magic.
<fremy> q+
<dael> birtles: I hear you, but I see it so much I think it's idomatic.
<Rossen> a/in our API is based/in our ABI is based/
<dael> fremy: Relying on number or arguments is a problem when you need state and it's in the constructor...people use [missed] and if you don't put something in the argument it throws an error. Also if we ever need a 3rd argument in the future you're screwed. Maybe we have a default value so if you have that you don't need to do anything special.
<dbaron> Topic: Animation Worklets, continued
<fremy> q-
<dbaron> github: https://github.com//issues/87
<dael> iank_: It works on top of the class, but if you extend something it doesn't work.
<dael> majidvp: Number of arguments works in webIDL because it has syntax to say which params are optional. normal JS doesn't have that concept. When you call a function at run time you know # arguments passed through. You use that to choose one behavior or another. As an engine I think you'll have a hard time.
<dael> shane: THe experiment from iank_ shows you can't do it at top level.
<dael> birtles: I'm not sure we want to decide now, it's discussion to have.
<dael> Rossen: One of the things...the whole discussion was do we need the attribute or not.
<dael> shane: Isn't going back to the issue likely to push it to the WICG?
<dael> birtles: We can take it offline.
<dael> shane: F2F is when we do things in person.
<dael> birtles: WEdnesday afternoon, then, during the breakout?
<dael> shane: You want to think about it more.
<dael> Rossen: Let's summarize. we don't want the attribute. To avoid that we can pass the info throught he param list and base the statefulness on that. That was mildly argued against since it's not clean design from internal API, though it's common for JS developers.
<dael> Rossen: We can resolve on nothing and keep the param for now, or we fast forward and resolve on the parameter and then bring other ways to do this if it's hard to impl or bad behavior.
<dbaron> FWIW, IRC bot issues: https://github.com/dbaron/wgmeeting-github-ircbot/issues/27 and https://github.com/dbaron/wgmeeting-github-ircbot/issues/28
<dael> Rossen: What do we want to do. Stay with static or number of param on construction
<dael> smfr: Two super classes was an option.
<dael> majidvp: We haven't discussed that
<dael> flackr: If statefullness defaulted to stateless when not spec you get behavior that you want.
<dael> majidvp: Which is a good default to have
<dael> shane: There's no signal it's a sure thing. I think you're expecting a state and it doesn't appear. I think being explicit is more clear.
<dael> flackr: You could throw if there is no state.
<dael> shane: That would be okay....it covers knowledge.
<dael> Rossen: Going back to three choices, I liked the 3rd, have the stateful and stateless classes. It makes it one or the other and satisfies the pushback against the attribute.
<dael> Rossen: Going forward if we discuss again and have something better we can re-discuss. But let's not lose the opportunity to resolve
<dael> majidvp: The super class if you want to allow dynamic it doesn't allow for that
<dael> shane: Not just a third class?
<dael> majidvp: Okay....you're using a heavy hammer for something that could be an attribute. Attributes are very common.
<dael> shane: If thre's a state object and you're in stateless version it's there. If you do have the hybrid later you can have explicit functions for switching.
<dael> Rossen: You can always have a conversion function based on object data
<dael> majidvp: Okay...that's not bad. I buy having the state object as a super class and then have setters.
<dael> Rossen: Other opinions?
<dael> Rossen: Objections to introducing 2 classes as a mean to listing stateful and stateless types?
<dael> RESOLVED: introduce 2 classes as a mean to listing stateful and stateless types

[majido]

This issue was moved to w3c/css-houdini-drafts#812