Unconditionally exposing domain specific API for all scripts

[padenot]

I'm opening this off of whatwg/webidl#468 (comment), that suggests that more API will be available in ECMAScript in the future, and therefore (I think, not knowing in detail whether it is possible to opt-out at the WebIDL level) in all *GlobalScopes on the Web platform.

First I'll provide a bit of context, then summarize the concerns I (and probably some other folks in the AudioWG) have here.

Context: real-time audio programming on the web

The Audio Working Group has standardized the AudioWorklet API that allows running script in a special environment, with the explicit and sole goal of being able to do real-time audio processing. A summary of the processing model of this API is summarized below.

Considering you have an AudioContext that has instantiated an AudioWorklet object, that loads a script in an AudioWorkletGlobalScope:

• The operating system fires some sort of event, saying that audio output buffers are running low and should be refilled (or that audio input buffers are available), often passing down audio output buffer to be filled, and audio input buffer to be processed
• The UA picks this event up, runs some internal code on a very high priority thread (or sometimes even on a real-time scheduler, depending on the platform)
• According to the Web Audio API processing model, this ends up calling the process method of the AudioWorkletProcessor associated to the AudioWorkletNode, passing it an input buffer and an output buffer (both Float32Arrays).
• The script runs doing some math, filling out the output buffer, that gets returned to the UA for further processing
• The UA returns the platform-level output buffer, that gets queued to be played back somehow

The steps above happen continuously. To get a sense of scale, in a typical situation, such a system would be processing 10ms buffers each time. With a sound card clocked at 48kHz (again very typical), it is necessary to process a 480 frames buffer 100 times a second. Failure in returning on time even once is considered a critical failure of the system. This 10ms period is called a rendering quantum.

We have some code examples in https://webaudio.github.io/web-audio-api/#AudioWorklet-Examples.

Concerns

The concerns here can be split into two groups: implementation concerns and authoring concerns

Implementation concerns

Interruptiblity: it is desirable to be able to brutally interrupt the execution of a script running in the AudioWorkletGlobalScope, for example if it's running for too long (much more than a rendering quantum), to both prevent a denial of service or a resource abuse (which would be tempting: this script is likely to be running on a high priority thread).
If the script calls into browser APIs, it's likely that it won't be interruptible until it has returned.

Blockiness: it is critical to never block the thread that is processing the audio. Allocating memory, taking locks, loading in a shared library, making system calls can all be blocking operations. I don't think there are strong guarantees on how to implement browser API so that they don't block, but in practice, what is available today is more or less OK (Regular expressions and the Date object being the two obvious exceptions).

Authoring concerns

Having high-level APIs (the Internationalization is an example that has been brought up in the issue linked at the beginning of this issue) exposed in the AudioWorkletGlobalScope sends the wrong message to authors. An AudioWorkletGlobalScope (and I'd go as far as saying other Worklets, such as the PaintWorklet) is never the right location to do some i18n work, regardless of the use-case or application, so it seems wrong to be able to access it there.

Now, we can always say "don't do this and that in an AudioWorklet", and maybe that's fine but at least there should be a discussion about it.

[littledan]

Thanks for filing this issue here so that the discussion can reach more people who are interested in JavaScript.

For interruptability: I wonder if Promises interact with this concern. My understanding is that, for HTML in general, all Promise-queued tasks ("microtasks") run to completion before resuming the HTML-level task queue. Is this the case for AudioWorklets as well? Is this appropriate? If not, I'm not sure if anything needs to be changed at the JavaScript level, or just the way it's embedded. Other than that, it's hard for me to think of other JS functionality which may be added to the JavaScript specification in the future which may want to call into browser APIs or block (just synchronous computation work).

For authoring concerns: This one seems a bit tricker. How do you draw the line exactly? Do you have any examples from other specifications which made a deliberate decision to include or exclude an interface in AudioWorklet?

[padenot]

Is this the case for AudioWorklets as well? Is this appropriate? If not, I'm not sure if anything needs to be changed at the JavaScript level, or just the way it's embedded. Other than that, it's hard for me to think of other JS functionality which may be added to the JavaScript specification in the future which may want to call into browser APIs or block (just synchronous computation work).

I reckon that Promises don't make too much sense at the minute for AudioWorklet, and I don't know how they would work just yet. AudioWorklets are intentionnaly very much synchronous (the process method gets call, it computes for a bit, it returns), and it would be ideal to disallow or at least define precisely what they do.

I need to read up on Promises in the ECMAspec. Since AudioWorklets don't have a traditional event loop (in the web platform sense), I think we need to define some bindings here.

For authoring concerns: This one seems a bit tricker. How do you draw the line exactly? Do you have any examples from other specifications which made a deliberate decision to include or exclude an interface in AudioWorklet?

SharedArrayBuffer sharing has been allowed but the wait operation has been disallowed. This is the only ECMAScript feature we've somehow been able to control for now (to my knowledge). This is possible because there are concepts in the web platform that allow this, such as [Sharable] and agent clusters.

Other than that, exposition of a Web Platform API is controled by [Exposed=Worklet] or [Exposed=AudioWorklet] (it would be better to be a bit more precise, it does seems like some AnimationWorklet bleed into AudioWorklet at the minute), and everything is opt-in. console is available for all worklets, but for example MessagePort is available to AudioWorklet (but not PaintWorklet).

I think you're right that this is partly an embedding issue (for now, granted there are now other ECMAScript feature that are going to be a bit incompatible with the AudioWorklet model), but it might also be an issue about the scope of what is available unconditionally to a script. But maybe we can restrict feature at the embedding level, and this is not really an ECMAScript issue.

[ljharb]

@padenot do you think there's something actionable here for ECMAScript, or can this be closed?

[padenot]

We can probably close this, thanks.

[annevk]

I think the main takeaway is that taking on large dependencies as is sometimes talked about, such as importing TextDecoder into ECMAScript, is not actually desirable for all agents. (And note that this remains true if instead you made it a built-in module. It still shouldn't be available.)

[littledan]

I think this got closed a bit quickly. This is an important perspective that we might want to discuss further. We are considering more expansion of the JS standard library, and this is a big proposed design goal.

[ljharb]

Since https://github.com/tc39/proposal-javascript-standard-library now exists, I think tc39/proposal-built-in-modules#16 is a better place for further discussion.