Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How decodeQueueSize and encodeQueueSize behave will introduce web compatibility issues, and need to be better defined. #864

Open
jyavenard opened this issue Dec 12, 2024 · 0 comments
Open

How decodeQueueSize and encodeQueueSize behave will introduce web compatibility issues, and need to be better defined. #864

jyavenard opened this issue Dec 12, 2024 · 0 comments

Comments

@jyavenard
Copy link
Member

Whenever decode() or encode() is called, we are to increase respectively decodeQueueSize or encodeQueueSize and enqueue a control message while immediately processing the control message.

If the codec is saturated, then we are to abort.

However "Codec Saturation" is defined as:
**_Codec Saturation

The state of an underlying codec implementation where the number of active decoding or encoding requests has reached an implementation specific maximum such that it is temporarily unable to accept more work. The maximum may be any value greater than 1, including infinity (no maximum). While saturated, additional calls to decode() or encode() will be buffered in the control message queue, and will increment the respective decodeQueuSize and encodeQueueSize attributes. The codec implementation will become unsaturated after making sufficient progress on the current workload._**

If a UA choose for a maximum a value of infinity; a side effect is that decodeQueueSize and encodeQueueSize will only ever be observed with a value of 0.

WebCodecs' demo player https://w3c.github.io/webcodecs/samples/audio-video-player/audio_video_player.html uses the value of decodeQueueSize as a rate limiter on how quickly it needs to decode ahead of time to fill its ring buffer for audio.

If the UA chose to have an unlimited queue and decodeQueueSize be always 0, then such rate limitation will be ineffective.

In the demo above, this causes the ring buffer to become full almost instantly and decoded audio frames are all dropped, and playback never progress.

While it's obviously an issue in this reference player itself, it will be used by others as an example (I already know of a few parties that based their work on it)

Chrome uses a maximum queue size of between 1 and 8 for video (Default: https://source.chromium.org/chromium/chromium/src/+/main:media/base/video_decoder.cc;drc=0e516eb08aa56163bfe649cebbba13da1fcf5f3c;bpv=1;bpt=1;l=30 max: https://source.chromium.org/chromium/chromium/src/+/main:media/gpu/chromeos/oop_video_decoder.h;drc=0e516eb08aa56163bfe649cebbba13da1fcf5f3c;bpv=1;bpt=1;l=207) and just 1 for audio (https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/modules/webcodecs/audio_decoder.cc;drc=0e516eb08aa56163bfe649cebbba13da1fcf5f3c;bpv=1;bpt=1;l=132)
Firefox uses a maximum queue size of 1 (https://searchfox.org/mozilla-central/source/dom/media/webcodecs/DecoderTemplate.cpp#564-567) and won't enqueue a new decode/encode if one is pending.

And so the demo video player works fine on Firefox and Chrome; but it wouldn't had they not constricted the limit to such a low value.

It may be beneficial to provide guidelines of best practices and amend the demo player to cater for such condition.
@jyavenard jyavenard changed the title How decodeQueueSize and encodeQueueSize behaves will introduce web compatibility issues, and need to be better defined. How decodeQueueSize and encodeQueueSize behave will introduce web compatibility issues, and need to be better defined. Dec 12, 2024
jyavenard added a commit to jyavenard/WebKit that referenced this issue Dec 12, 2024
@jyavenard
[WebCodec|AudioDecoder] Limit the number of codec operations we can e…
9502a07 
…nqueue

https://bugs.webkit.org/show_bug.cgi?id=284448
rdar://141272065

Reviewed by NOBODY (OOPS!).

While the WebCodecs specs clearly allows for boundless queue for calculating
"Codec Saturdation", a side effect is that both decodeQueueSize and encodeQueueSize
can only ever be observed with a size of 0.
The value of decodeQueueSize and encodeQueueSize is used by some site (including W3C's WebCodecs's own example)
as a way to limit how much in advance it will attempt to encode or decode.

Both Chrome and Firefox have a limit on how many codecs operations (1 for Firefox, between 1 and 8 for Chrome)
they will enqueue concurrently.
To minimise web compatibility issue, in addition to lodging a spec bug (w3c/webcodecs#864)
we also introduce a limit on how many frames with submit to the internal decoder/encoder
before submitting more:
- 1 for Audio
- 4 for Video.

In order of not having to replicate 4 times the code in {Audio|Video}{Encoder|Decoder}
we create a new WebCodecsBase that handles all the control message queue operations
as well as handle how many codec operations have been submitted, and of which
all WebCodecs inherit from.
Doing so allows to remove a lot of similar code across all webcodecs and get
us closer to the verbiage of the specs.

Another benefit is that WebCodecsControlMessage no longer needs to be a templated class,
and we make it return a "Processed" or "Not Processed" value as the specs does.

No change in observable behaviours with existing WPT.
Tested that it allows the WebCodecs reference player to work.
No new tests as none of the work above is per spec.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp:
(WebCore::WebCodecsAudioDecoder::WebCodecsAudioDecoder):
(WebCore::WebCodecsAudioDecoder::configure):
(WebCore::WebCodecsAudioDecoder::decode):
(WebCore::WebCodecsAudioDecoder::flush):
(WebCore::WebCodecsAudioDecoder::closeDecoder):
(WebCore::WebCodecsAudioDecoder::resetDecoder):
(WebCore::WebCodecsAudioDecoder::stop):
(WebCore::WebCodecsAudioDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h:
(WebCore::WebCodecsAudioDecoder::decodeQueueSize const):
(WebCore::WebCodecsAudioDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::WebCodecsAudioEncoder::WebCodecsAudioEncoder):
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode):
(WebCore::WebCodecsAudioEncoder::flush):
(WebCore::WebCodecsAudioEncoder::closeEncoder):
(WebCore::WebCodecsAudioEncoder::resetEncoder):
(WebCore::WebCodecsAudioEncoder::stop):
(WebCore::WebCodecsAudioEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.h:
(WebCore::WebCodecsAudioEncoder::encodeQueueSize const):
(WebCore::WebCodecsAudioEncoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsBase.cpp: Added.
(WebCore::WebCodecsBase::WebCodecsBase):
(WebCore::WebCodecsBase::queueControlMessageAndProcess):
(WebCore::WebCodecsBase::queueControlMessageForCodecOperationAndProcess):
(WebCore::WebCodecsBase::scheduleDequeueEvent):
(WebCore::WebCodecsBase::processControlMessageQueue):
(WebCore::WebCodecsBase::incrementOperationQueueSize):
(WebCore::WebCodecsBase::decrementOperationQueueSizeAndScheduleDequeueEvent):
(WebCore::WebCodecsBase::decreaseCodecOperationCountAndMaybeProcessControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueueAndMaybeScheduleDequeueEvent):
(WebCore::WebCodecsBase::blockControlMessageQueue):
(WebCore::WebCodecsBase::unblockControlMessageQueue):
(WebCore::WebCodecsBase::virtualHasPendingActivity const):
* Source/WebCore/Modules/webcodecs/WebCodecsBase.h: Copied from Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h.
(WebCore::WebCodecsBase::state const):
(WebCore::WebCodecsBase::setState):
(WebCore::WebCodecsBase::operationQueueSize const):
(WebCore::WebCodecsBase::maximumCodecOperationsEnqueued const):
(WebCore::WebCodecsBase::increaseCodecOperationCount):
(WebCore::WebCodecsBase::isCodecSaturated const):
* Source/WebCore/Modules/webcodecs/WebCodecsControlMessage.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::WebCodecsVideoDecoder::WebCodecsVideoDecoder):
(WebCore::WebCodecsVideoDecoder::configure):
(WebCore::WebCodecsVideoDecoder::decode):
(WebCore::WebCodecsVideoDecoder::flush):
(WebCore::WebCodecsVideoDecoder::closeDecoder):
(WebCore::WebCodecsVideoDecoder::resetDecoder):
(WebCore::WebCodecsVideoDecoder::stop):
(WebCore::WebCodecsVideoDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h:
(WebCore::WebCodecsVideoDecoder::decodeQueueSize const):
(WebCore::WebCodecsVideoDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.cpp:
(WebCore::WebCodecsVideoEncoder::WebCodecsVideoEncoder):
(WebCore::WebCodecsVideoEncoder::updateRates):
(WebCore::WebCodecsVideoEncoder::configure):
(WebCore::WebCodecsVideoEncoder::encode):
(WebCore::WebCodecsVideoEncoder::flush):
(WebCore::WebCodecsVideoEncoder::closeEncoder):
(WebCore::WebCodecsVideoEncoder::resetEncoder):
(WebCore::WebCodecsVideoEncoder::stop):
(WebCore::WebCodecsVideoEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.h:
(WebCore::WebCodecsVideoEncoder::encodeQueueSize const):
(WebCore::WebCodecsVideoEncoder::state const): Deleted.
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
@jyavenard jyavenard mentioned this issue Dec 12, 2024
Merged
jyavenard added a commit to jyavenard/WebKit that referenced this issue Dec 12, 2024
@jyavenard
[WebCodecs] Limit the number of codec operations we can enqueue
c7c7c22 
https://bugs.webkit.org/show_bug.cgi?id=284448
rdar://141272065

Reviewed by NOBODY (OOPS!).

While the WebCodecs specs clearly allows for boundless queue for calculating
"Codec Saturdation", a side effect is that both decodeQueueSize and encodeQueueSize
can only ever be observed with a size of 0.
The value of decodeQueueSize and encodeQueueSize is used by some site (including W3C's WebCodecs's own example)
as a way to limit how much in advance it will attempt to encode or decode.

Both Chrome and Firefox have a limit on how many codecs operations (1 for Firefox, between 1 and 8 for Chrome)
they will enqueue concurrently.
To minimise web compatibility issue, in addition to lodging a spec bug (w3c/webcodecs#864)
we also introduce a limit on how many frames with submit to the internal decoder/encoder
before submitting more:
- 1 for Audio
- 4 for Video.

In order of not having to replicate 4 times the code in {Audio|Video}{Encoder|Decoder}
we create a new WebCodecsBase that handles all the control message queue operations
as well as handle how many codec operations have been submitted, and of which
all WebCodecs inherit from.
Doing so allows to remove a lot of similar code across all webcodecs and get
us closer to the verbiage of the specs.

Another benefit is that WebCodecsControlMessage no longer needs to be a templated class,
and we make it return a "Processed" or "Not Processed" value as the specs does.

No change in observable behaviours with existing WPT.
Tested that it allows the WebCodecs reference player to work.
No new tests as none of the work above is per spec.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp:
(WebCore::WebCodecsAudioDecoder::WebCodecsAudioDecoder):
(WebCore::WebCodecsAudioDecoder::configure):
(WebCore::WebCodecsAudioDecoder::decode):
(WebCore::WebCodecsAudioDecoder::flush):
(WebCore::WebCodecsAudioDecoder::closeDecoder):
(WebCore::WebCodecsAudioDecoder::resetDecoder):
(WebCore::WebCodecsAudioDecoder::stop):
(WebCore::WebCodecsAudioDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h:
(WebCore::WebCodecsAudioDecoder::decodeQueueSize const):
(WebCore::WebCodecsAudioDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::WebCodecsAudioEncoder::WebCodecsAudioEncoder):
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode):
(WebCore::WebCodecsAudioEncoder::flush):
(WebCore::WebCodecsAudioEncoder::closeEncoder):
(WebCore::WebCodecsAudioEncoder::resetEncoder):
(WebCore::WebCodecsAudioEncoder::stop):
(WebCore::WebCodecsAudioEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.h:
(WebCore::WebCodecsAudioEncoder::encodeQueueSize const):
(WebCore::WebCodecsAudioEncoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsBase.cpp: Added.
(WebCore::WebCodecsBase::WebCodecsBase):
(WebCore::WebCodecsBase::queueControlMessageAndProcess):
(WebCore::WebCodecsBase::queueControlMessageForCodecOperationAndProcess):
(WebCore::WebCodecsBase::scheduleDequeueEvent):
(WebCore::WebCodecsBase::processControlMessageQueue):
(WebCore::WebCodecsBase::incrementOperationQueueSize):
(WebCore::WebCodecsBase::decrementOperationQueueSizeAndScheduleDequeueEvent):
(WebCore::WebCodecsBase::decreaseCodecOperationCountAndMaybeProcessControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueueAndMaybeScheduleDequeueEvent):
(WebCore::WebCodecsBase::blockControlMessageQueue):
(WebCore::WebCodecsBase::unblockControlMessageQueue):
(WebCore::WebCodecsBase::virtualHasPendingActivity const):
* Source/WebCore/Modules/webcodecs/WebCodecsBase.h: Copied from Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h.
(WebCore::WebCodecsBase::state const):
(WebCore::WebCodecsBase::setState):
(WebCore::WebCodecsBase::operationQueueSize const):
(WebCore::WebCodecsBase::maximumCodecOperationsEnqueued const):
(WebCore::WebCodecsBase::increaseCodecOperationCount):
(WebCore::WebCodecsBase::isCodecSaturated const):
* Source/WebCore/Modules/webcodecs/WebCodecsControlMessage.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::WebCodecsVideoDecoder::WebCodecsVideoDecoder):
(WebCore::WebCodecsVideoDecoder::configure):
(WebCore::WebCodecsVideoDecoder::decode):
(WebCore::WebCodecsVideoDecoder::flush):
(WebCore::WebCodecsVideoDecoder::closeDecoder):
(WebCore::WebCodecsVideoDecoder::resetDecoder):
(WebCore::WebCodecsVideoDecoder::stop):
(WebCore::WebCodecsVideoDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h:
(WebCore::WebCodecsVideoDecoder::decodeQueueSize const):
(WebCore::WebCodecsVideoDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.cpp:
(WebCore::WebCodecsVideoEncoder::WebCodecsVideoEncoder):
(WebCore::WebCodecsVideoEncoder::updateRates):
(WebCore::WebCodecsVideoEncoder::configure):
(WebCore::WebCodecsVideoEncoder::encode):
(WebCore::WebCodecsVideoEncoder::flush):
(WebCore::WebCodecsVideoEncoder::closeEncoder):
(WebCore::WebCodecsVideoEncoder::resetEncoder):
(WebCore::WebCodecsVideoEncoder::stop):
(WebCore::WebCodecsVideoEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.h:
(WebCore::WebCodecsVideoEncoder::encodeQueueSize const):
(WebCore::WebCodecsVideoEncoder::state const): Deleted.
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
jyavenard added a commit to jyavenard/WebKit that referenced this issue Dec 12, 2024
@jyavenard
[WebCodecs] Limit the number of codec operations we can enqueue
e7c4ce2 
https://bugs.webkit.org/show_bug.cgi?id=284448
rdar://141272065

Reviewed by NOBODY (OOPS!).

While the WebCodecs specs clearly allows for boundless queue for calculating
"Codec Saturdation", a side effect is that both decodeQueueSize and encodeQueueSize
can only ever be observed with a size of 0.
The value of decodeQueueSize and encodeQueueSize is used by some site (including W3C's WebCodecs's own example)
as a way to limit how much in advance it will attempt to encode or decode.

Both Chrome and Firefox have a limit on how many codecs operations (1 for Firefox, between 1 and 8 for Chrome)
they will enqueue concurrently.
To minimise web compatibility issue, in addition to lodging a spec bug (w3c/webcodecs#864)
we also introduce a limit on how many frames with submit to the internal decoder/encoder
before submitting more:
- 1 for Audio
- 4 for Video.

In order of not having to replicate 4 times the code in {Audio|Video}{Encoder|Decoder}
we create a new WebCodecsBase that handles all the control message queue operations
as well as handle how many codec operations have been submitted, and of which
all WebCodecs inherit from.
Doing so allows to remove a lot of similar code across all webcodecs and get
us closer to the verbiage of the specs.

Another benefit is that WebCodecsControlMessage no longer needs to be a templated class,
and we make it return a "Processed" or "Not Processed" value as the specs does.

No change in observable behaviours with existing WPT.
Tested that it allows the WebCodecs reference player to work.
No new tests as none of the work above is per spec.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp:
(WebCore::WebCodecsAudioDecoder::WebCodecsAudioDecoder):
(WebCore::WebCodecsAudioDecoder::configure):
(WebCore::WebCodecsAudioDecoder::decode):
(WebCore::WebCodecsAudioDecoder::flush):
(WebCore::WebCodecsAudioDecoder::closeDecoder):
(WebCore::WebCodecsAudioDecoder::resetDecoder):
(WebCore::WebCodecsAudioDecoder::stop):
(WebCore::WebCodecsAudioDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h:
(WebCore::WebCodecsAudioDecoder::decodeQueueSize const):
(WebCore::WebCodecsAudioDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::WebCodecsAudioEncoder::WebCodecsAudioEncoder):
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode):
(WebCore::WebCodecsAudioEncoder::flush):
(WebCore::WebCodecsAudioEncoder::closeEncoder):
(WebCore::WebCodecsAudioEncoder::resetEncoder):
(WebCore::WebCodecsAudioEncoder::stop):
(WebCore::WebCodecsAudioEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.h:
(WebCore::WebCodecsAudioEncoder::encodeQueueSize const):
(WebCore::WebCodecsAudioEncoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsBase.cpp: Added.
(WebCore::WebCodecsBase::WebCodecsBase):
(WebCore::WebCodecsBase::queueControlMessageAndProcess):
(WebCore::WebCodecsBase::queueControlMessageForCodecOperationAndProcess):
(WebCore::WebCodecsBase::scheduleDequeueEvent):
(WebCore::WebCodecsBase::processControlMessageQueue):
(WebCore::WebCodecsBase::incrementOperationQueueSize):
(WebCore::WebCodecsBase::decrementOperationQueueSizeAndScheduleDequeueEvent):
(WebCore::WebCodecsBase::decreaseCodecOperationCountAndMaybeProcessControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueueAndMaybeScheduleDequeueEvent):
(WebCore::WebCodecsBase::blockControlMessageQueue):
(WebCore::WebCodecsBase::unblockControlMessageQueue):
(WebCore::WebCodecsBase::virtualHasPendingActivity const):
* Source/WebCore/Modules/webcodecs/WebCodecsBase.h: Copied from Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h.
(WebCore::WebCodecsBase::state const):
(WebCore::WebCodecsBase::setState):
(WebCore::WebCodecsBase::operationQueueSize const):
(WebCore::WebCodecsBase::maximumCodecOperationsEnqueued const):
(WebCore::WebCodecsBase::increaseCodecOperationCount):
(WebCore::WebCodecsBase::isCodecSaturated const):
* Source/WebCore/Modules/webcodecs/WebCodecsControlMessage.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::WebCodecsVideoDecoder::WebCodecsVideoDecoder):
(WebCore::WebCodecsVideoDecoder::configure):
(WebCore::WebCodecsVideoDecoder::decode):
(WebCore::WebCodecsVideoDecoder::flush):
(WebCore::WebCodecsVideoDecoder::closeDecoder):
(WebCore::WebCodecsVideoDecoder::resetDecoder):
(WebCore::WebCodecsVideoDecoder::stop):
(WebCore::WebCodecsVideoDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h:
(WebCore::WebCodecsVideoDecoder::decodeQueueSize const):
(WebCore::WebCodecsVideoDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.cpp:
(WebCore::WebCodecsVideoEncoder::WebCodecsVideoEncoder):
(WebCore::WebCodecsVideoEncoder::updateRates):
(WebCore::WebCodecsVideoEncoder::configure):
(WebCore::WebCodecsVideoEncoder::encode):
(WebCore::WebCodecsVideoEncoder::flush):
(WebCore::WebCodecsVideoEncoder::closeEncoder):
(WebCore::WebCodecsVideoEncoder::resetEncoder):
(WebCore::WebCodecsVideoEncoder::stop):
(WebCore::WebCodecsVideoEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.h:
(WebCore::WebCodecsVideoEncoder::encodeQueueSize const):
(WebCore::WebCodecsVideoEncoder::state const): Deleted.
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
jyavenard added a commit to jyavenard/WebKit that referenced this issue Dec 13, 2024
@jyavenard
[WebCodecs] Limit the number of codec operations we can enqueue
4101f3f 
https://bugs.webkit.org/show_bug.cgi?id=284448
rdar://141272065

Reviewed by NOBODY (OOPS!).

While the WebCodecs specs clearly allows for boundless queue for calculating
"Codec Saturdation", a side effect is that both decodeQueueSize and encodeQueueSize
can only ever be observed with a size of 0.
The value of decodeQueueSize and encodeQueueSize is used by some site (including W3C's WebCodecs's own example)
as a way to limit how much in advance it will attempt to encode or decode.

Both Chrome and Firefox have a limit on how many codecs operations (1 for Firefox, between 1 and 8 for Chrome)
they will enqueue concurrently.
To minimise web compatibility issue, in addition to lodging a spec bug (w3c/webcodecs#864)
we also introduce a limit on how many frames with submit to the internal decoder/encoder
before submitting more:
- 1 for Audio
- 4 for Video.

In order of not having to replicate 4 times the code in {Audio|Video}{Encoder|Decoder}
we create a new WebCodecsBase that handles all the control message queue operations
as well as handle how many codec operations have been submitted, and of which
all WebCodecs inherit from.
Doing so allows to remove a lot of similar code across all webcodecs and get
us closer to the verbiage of the specs.

Another benefit is that WebCodecsControlMessage no longer needs to be a templated class,
and we make it return a "Processed" or "Not Processed" value as the specs does.

No change in observable behaviours with existing WPT.
Tested that it allows the WebCodecs reference player to work.
No new tests as none of the work above is per spec.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp:
(WebCore::WebCodecsAudioDecoder::WebCodecsAudioDecoder):
(WebCore::WebCodecsAudioDecoder::configure):
(WebCore::WebCodecsAudioDecoder::decode):
(WebCore::WebCodecsAudioDecoder::flush):
(WebCore::WebCodecsAudioDecoder::closeDecoder):
(WebCore::WebCodecsAudioDecoder::resetDecoder):
(WebCore::WebCodecsAudioDecoder::stop):
(WebCore::WebCodecsAudioDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h:
(WebCore::WebCodecsAudioDecoder::decodeQueueSize const):
(WebCore::WebCodecsAudioDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::WebCodecsAudioEncoder::WebCodecsAudioEncoder):
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode):
(WebCore::WebCodecsAudioEncoder::flush):
(WebCore::WebCodecsAudioEncoder::closeEncoder):
(WebCore::WebCodecsAudioEncoder::resetEncoder):
(WebCore::WebCodecsAudioEncoder::stop):
(WebCore::WebCodecsAudioEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.h:
(WebCore::WebCodecsAudioEncoder::encodeQueueSize const):
(WebCore::WebCodecsAudioEncoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsBase.cpp: Added.
(WebCore::WebCodecsBase::WebCodecsBase):
(WebCore::WebCodecsBase::queueControlMessageAndProcess):
(WebCore::WebCodecsBase::queueControlMessageForCodecOperationAndProcess):
(WebCore::WebCodecsBase::scheduleDequeueEvent):
(WebCore::WebCodecsBase::processControlMessageQueue):
(WebCore::WebCodecsBase::incrementOperationQueueSize):
(WebCore::WebCodecsBase::decrementOperationQueueSizeAndScheduleDequeueEvent):
(WebCore::WebCodecsBase::decreaseCodecOperationCountAndMaybeProcessControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueueAndMaybeScheduleDequeueEvent):
(WebCore::WebCodecsBase::blockControlMessageQueue):
(WebCore::WebCodecsBase::unblockControlMessageQueue):
(WebCore::WebCodecsBase::virtualHasPendingActivity const):
* Source/WebCore/Modules/webcodecs/WebCodecsBase.h: Copied from Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h.
(WebCore::WebCodecsBase::state const):
(WebCore::WebCodecsBase::setState):
(WebCore::WebCodecsBase::operationQueueSize const):
(WebCore::WebCodecsBase::maximumCodecOperationsEnqueued const):
(WebCore::WebCodecsBase::increaseCodecOperationCount):
(WebCore::WebCodecsBase::isCodecSaturated const):
* Source/WebCore/Modules/webcodecs/WebCodecsControlMessage.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::WebCodecsVideoDecoder::WebCodecsVideoDecoder):
(WebCore::WebCodecsVideoDecoder::configure):
(WebCore::WebCodecsVideoDecoder::decode):
(WebCore::WebCodecsVideoDecoder::flush):
(WebCore::WebCodecsVideoDecoder::closeDecoder):
(WebCore::WebCodecsVideoDecoder::resetDecoder):
(WebCore::WebCodecsVideoDecoder::stop):
(WebCore::WebCodecsVideoDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h:
(WebCore::WebCodecsVideoDecoder::decodeQueueSize const):
(WebCore::WebCodecsVideoDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.cpp:
(WebCore::WebCodecsVideoEncoder::WebCodecsVideoEncoder):
(WebCore::WebCodecsVideoEncoder::updateRates):
(WebCore::WebCodecsVideoEncoder::configure):
(WebCore::WebCodecsVideoEncoder::encode):
(WebCore::WebCodecsVideoEncoder::flush):
(WebCore::WebCodecsVideoEncoder::closeEncoder):
(WebCore::WebCodecsVideoEncoder::resetEncoder):
(WebCore::WebCodecsVideoEncoder::stop):
(WebCore::WebCodecsVideoEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.h:
(WebCore::WebCodecsVideoEncoder::encodeQueueSize const):
(WebCore::WebCodecsVideoEncoder::state const): Deleted.
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
jyavenard added a commit to jyavenard/WebKit that referenced this issue Dec 13, 2024
@jyavenard
[WebCodecs] Limit the number of codec operations we can enqueue
d1702d6 
https://bugs.webkit.org/show_bug.cgi?id=284448
rdar://141272065

Reviewed by NOBODY (OOPS!).

While the WebCodecs specs clearly allows for boundless queue for calculating
"Codec Saturdation", a side effect is that both decodeQueueSize and encodeQueueSize
can only ever be observed with a size of 0.
The value of decodeQueueSize and encodeQueueSize is used by some site (including W3C's WebCodecs's own example)
as a way to limit how much in advance it will attempt to encode or decode.

Both Chrome and Firefox have a limit on how many codecs operations (1 for Firefox, between 1 and 8 for Chrome)
they will enqueue concurrently.
To minimise web compatibility issue, in addition to lodging a spec bug (w3c/webcodecs#864)
we also introduce a limit on how many frames with submit to the internal decoder/encoder
before submitting more:
- 1 for Audio
- 4 for Video.

In order of not having to replicate 4 times the code in {Audio|Video}{Encoder|Decoder}
we create a new WebCodecsBase that handles all the control message queue operations
as well as handle how many codec operations have been submitted, and of which
all WebCodecs inherit from.
Doing so allows to remove a lot of similar code across all webcodecs and get
us closer to the verbiage of the specs.

Another benefit is that WebCodecsControlMessage no longer needs to be a templated class,
and we make it return a "Processed" or "Not Processed" value as the specs does.

No change in observable behaviours with existing WPT.
Tested that it allows the WebCodecs reference player to work.
No new tests as none of the work above is per spec.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp:
(WebCore::WebCodecsAudioDecoder::WebCodecsAudioDecoder):
(WebCore::WebCodecsAudioDecoder::configure):
(WebCore::WebCodecsAudioDecoder::decode):
(WebCore::WebCodecsAudioDecoder::flush):
(WebCore::WebCodecsAudioDecoder::closeDecoder):
(WebCore::WebCodecsAudioDecoder::resetDecoder):
(WebCore::WebCodecsAudioDecoder::stop):
(WebCore::WebCodecsAudioDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h:
(WebCore::WebCodecsAudioDecoder::decodeQueueSize const):
(WebCore::WebCodecsAudioDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::WebCodecsAudioEncoder::WebCodecsAudioEncoder):
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode):
(WebCore::WebCodecsAudioEncoder::flush):
(WebCore::WebCodecsAudioEncoder::closeEncoder):
(WebCore::WebCodecsAudioEncoder::resetEncoder):
(WebCore::WebCodecsAudioEncoder::stop):
(WebCore::WebCodecsAudioEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.h:
(WebCore::WebCodecsAudioEncoder::encodeQueueSize const):
(WebCore::WebCodecsAudioEncoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsBase.cpp: Added.
(WebCore::WebCodecsBase::WebCodecsBase):
(WebCore::WebCodecsBase::queueControlMessageAndProcess):
(WebCore::WebCodecsBase::queueControlMessageForCodecOperationAndProcess):
(WebCore::WebCodecsBase::scheduleDequeueEvent):
(WebCore::WebCodecsBase::processControlMessageQueue):
(WebCore::WebCodecsBase::incrementOperationQueueSize):
(WebCore::WebCodecsBase::decrementOperationQueueSizeAndScheduleDequeueEvent):
(WebCore::WebCodecsBase::decreaseCodecOperationCountAndMaybeProcessControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueueAndMaybeScheduleDequeueEvent):
(WebCore::WebCodecsBase::blockControlMessageQueue):
(WebCore::WebCodecsBase::unblockControlMessageQueue):
(WebCore::WebCodecsBase::virtualHasPendingActivity const):
* Source/WebCore/Modules/webcodecs/WebCodecsBase.h: Copied from Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h.
(WebCore::WebCodecsBase::state const):
(WebCore::WebCodecsBase::setState):
(WebCore::WebCodecsBase::operationQueueSize const):
(WebCore::WebCodecsBase::maximumCodecOperationsEnqueued const):
(WebCore::WebCodecsBase::increaseCodecOperationCount):
(WebCore::WebCodecsBase::isCodecSaturated const):
* Source/WebCore/Modules/webcodecs/WebCodecsControlMessage.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::WebCodecsVideoDecoder::WebCodecsVideoDecoder):
(WebCore::WebCodecsVideoDecoder::configure):
(WebCore::WebCodecsVideoDecoder::decode):
(WebCore::WebCodecsVideoDecoder::flush):
(WebCore::WebCodecsVideoDecoder::closeDecoder):
(WebCore::WebCodecsVideoDecoder::resetDecoder):
(WebCore::WebCodecsVideoDecoder::stop):
(WebCore::WebCodecsVideoDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h:
(WebCore::WebCodecsVideoDecoder::decodeQueueSize const):
(WebCore::WebCodecsVideoDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.cpp:
(WebCore::WebCodecsVideoEncoder::WebCodecsVideoEncoder):
(WebCore::WebCodecsVideoEncoder::updateRates):
(WebCore::WebCodecsVideoEncoder::configure):
(WebCore::WebCodecsVideoEncoder::encode):
(WebCore::WebCodecsVideoEncoder::flush):
(WebCore::WebCodecsVideoEncoder::closeEncoder):
(WebCore::WebCodecsVideoEncoder::resetEncoder):
(WebCore::WebCodecsVideoEncoder::stop):
(WebCore::WebCodecsVideoEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.h:
(WebCore::WebCodecsVideoEncoder::encodeQueueSize const):
(WebCore::WebCodecsVideoEncoder::state const): Deleted.
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
jyavenard added a commit to jyavenard/WebKit that referenced this issue Dec 13, 2024
@jyavenard
[WebCodecs] Limit the number of codec operations we can enqueue
682f314 
https://bugs.webkit.org/show_bug.cgi?id=284448
rdar://141272065

Reviewed by NOBODY (OOPS!).

While the WebCodecs specs clearly allows for boundless queue for calculating
"Codec Saturdation", a side effect is that both decodeQueueSize and encodeQueueSize
can only ever be observed with a size of 0.
The value of decodeQueueSize and encodeQueueSize is used by some site (including W3C's WebCodecs's own example)
as a way to limit how much in advance it will attempt to encode or decode.

Both Chrome and Firefox have a limit on how many codecs operations (1 for Firefox, between 1 and 8 for Chrome)
they will enqueue concurrently.
To minimise web compatibility issue, in addition to lodging a spec bug (w3c/webcodecs#864)
we also introduce a limit on how many frames with submit to the internal decoder/encoder
before submitting more:
- 1 for Audio
- 4 for Video.

In order of not having to replicate 4 times the code in {Audio|Video}{Encoder|Decoder}
we create a new WebCodecsBase that handles all the control message queue operations
as well as handle how many codec operations have been submitted, and of which
all WebCodecs inherit from.
Doing so allows to remove a lot of similar code across all webcodecs and get
us closer to the verbiage of the specs.

Another benefit is that WebCodecsControlMessage no longer needs to be a templated class,
and we make it return a "Processed" or "Not Processed" value as the specs does.

No change in observable behaviours with existing WPT.
Tested that it allows the WebCodecs reference player to work.
No new tests as none of the work above is per spec.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp:
(WebCore::WebCodecsAudioDecoder::WebCodecsAudioDecoder):
(WebCore::WebCodecsAudioDecoder::configure):
(WebCore::WebCodecsAudioDecoder::decode):
(WebCore::WebCodecsAudioDecoder::flush):
(WebCore::WebCodecsAudioDecoder::closeDecoder):
(WebCore::WebCodecsAudioDecoder::resetDecoder):
(WebCore::WebCodecsAudioDecoder::stop):
(WebCore::WebCodecsAudioDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h:
(WebCore::WebCodecsAudioDecoder::decodeQueueSize const):
(WebCore::WebCodecsAudioDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::WebCodecsAudioEncoder::WebCodecsAudioEncoder):
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode):
(WebCore::WebCodecsAudioEncoder::flush):
(WebCore::WebCodecsAudioEncoder::closeEncoder):
(WebCore::WebCodecsAudioEncoder::resetEncoder):
(WebCore::WebCodecsAudioEncoder::stop):
(WebCore::WebCodecsAudioEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.h:
(WebCore::WebCodecsAudioEncoder::encodeQueueSize const):
(WebCore::WebCodecsAudioEncoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsBase.cpp: Added.
(WebCore::WebCodecsBase::WebCodecsBase):
(WebCore::WebCodecsBase::queueControlMessageAndProcess):
(WebCore::WebCodecsBase::queueControlMessageForCodecOperationAndProcess):
(WebCore::WebCodecsBase::scheduleDequeueEvent):
(WebCore::WebCodecsBase::processControlMessageQueue):
(WebCore::WebCodecsBase::incrementOperationQueueSize):
(WebCore::WebCodecsBase::decrementOperationQueueSizeAndScheduleDequeueEvent):
(WebCore::WebCodecsBase::decreaseCodecOperationCountAndMaybeProcessControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueueAndMaybeScheduleDequeueEvent):
(WebCore::WebCodecsBase::blockControlMessageQueue):
(WebCore::WebCodecsBase::unblockControlMessageQueue):
(WebCore::WebCodecsBase::virtualHasPendingActivity const):
* Source/WebCore/Modules/webcodecs/WebCodecsBase.h: Copied from Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h.
(WebCore::WebCodecsBase::state const):
(WebCore::WebCodecsBase::setState):
(WebCore::WebCodecsBase::operationQueueSize const):
(WebCore::WebCodecsBase::maximumCodecOperationsEnqueued const):
(WebCore::WebCodecsBase::increaseCodecOperationCount):
(WebCore::WebCodecsBase::isCodecSaturated const):
* Source/WebCore/Modules/webcodecs/WebCodecsControlMessage.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::WebCodecsVideoDecoder::WebCodecsVideoDecoder):
(WebCore::WebCodecsVideoDecoder::configure):
(WebCore::WebCodecsVideoDecoder::decode):
(WebCore::WebCodecsVideoDecoder::flush):
(WebCore::WebCodecsVideoDecoder::closeDecoder):
(WebCore::WebCodecsVideoDecoder::resetDecoder):
(WebCore::WebCodecsVideoDecoder::stop):
(WebCore::WebCodecsVideoDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h:
(WebCore::WebCodecsVideoDecoder::decodeQueueSize const):
(WebCore::WebCodecsVideoDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.cpp:
(WebCore::WebCodecsVideoEncoder::WebCodecsVideoEncoder):
(WebCore::WebCodecsVideoEncoder::updateRates):
(WebCore::WebCodecsVideoEncoder::configure):
(WebCore::WebCodecsVideoEncoder::encode):
(WebCore::WebCodecsVideoEncoder::flush):
(WebCore::WebCodecsVideoEncoder::closeEncoder):
(WebCore::WebCodecsVideoEncoder::resetEncoder):
(WebCore::WebCodecsVideoEncoder::stop):
(WebCore::WebCodecsVideoEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.h:
(WebCore::WebCodecsVideoEncoder::encodeQueueSize const):
(WebCore::WebCodecsVideoEncoder::state const): Deleted.
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
jyavenard added a commit to jyavenard/WebKit that referenced this issue Dec 13, 2024
@jyavenard
[WebCodecs] Limit the number of codec operations we can enqueue
d38cfb6 
https://bugs.webkit.org/show_bug.cgi?id=284448
rdar://141272065

Reviewed by NOBODY (OOPS!).

While the WebCodecs specs clearly allows for boundless queue for calculating
"Codec Saturdation", a side effect is that both decodeQueueSize and encodeQueueSize
can only ever be observed with a size of 0.
The value of decodeQueueSize and encodeQueueSize is used by some site (including W3C's WebCodecs's own example)
as a way to limit how much in advance it will attempt to encode or decode.

Both Chrome and Firefox have a limit on how many codecs operations (1 for Firefox, between 1 and 8 for Chrome)
they will enqueue concurrently.
To minimise web compatibility issue, in addition to lodging a spec bug (w3c/webcodecs#864)
we also introduce a limit on how many frames with submit to the internal decoder/encoder
before submitting more:
- 1 for Audio
- 4 for Video.

In order of not having to replicate 4 times the code in {Audio|Video}{Encoder|Decoder}
we create a new WebCodecsBase that handles all the control message queue operations
as well as handle how many codec operations have been submitted, and of which
all WebCodecs inherit from.
Doing so allows to remove a lot of similar code across all webcodecs and get
us closer to the verbiage of the specs.

Another benefit is that WebCodecsControlMessage no longer needs to be a templated class,
and we make it return a "Processed" or "Not Processed" value as the specs does.

No change in observable behaviours with existing WPT.
Tested that it allows the WebCodecs reference player to work.
No new tests as none of the work above is per spec.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp:
(WebCore::WebCodecsAudioDecoder::WebCodecsAudioDecoder):
(WebCore::WebCodecsAudioDecoder::configure):
(WebCore::WebCodecsAudioDecoder::decode):
(WebCore::WebCodecsAudioDecoder::flush):
(WebCore::WebCodecsAudioDecoder::closeDecoder):
(WebCore::WebCodecsAudioDecoder::resetDecoder):
(WebCore::WebCodecsAudioDecoder::stop):
(WebCore::WebCodecsAudioDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h:
(WebCore::WebCodecsAudioDecoder::decodeQueueSize const):
(WebCore::WebCodecsAudioDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::WebCodecsAudioEncoder::WebCodecsAudioEncoder):
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode):
(WebCore::WebCodecsAudioEncoder::flush):
(WebCore::WebCodecsAudioEncoder::closeEncoder):
(WebCore::WebCodecsAudioEncoder::resetEncoder):
(WebCore::WebCodecsAudioEncoder::stop):
(WebCore::WebCodecsAudioEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.h:
(WebCore::WebCodecsAudioEncoder::encodeQueueSize const):
(WebCore::WebCodecsAudioEncoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsBase.cpp: Added.
(WebCore::WebCodecsBase::WebCodecsBase):
(WebCore::WebCodecsBase::queueControlMessageAndProcess):
(WebCore::WebCodecsBase::queueControlMessageForCodecOperationAndProcess):
(WebCore::WebCodecsBase::scheduleDequeueEvent):
(WebCore::WebCodecsBase::processControlMessageQueue):
(WebCore::WebCodecsBase::incrementOperationQueueSize):
(WebCore::WebCodecsBase::decrementOperationQueueSizeAndScheduleDequeueEvent):
(WebCore::WebCodecsBase::decreaseCodecOperationCountAndMaybeProcessControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueueAndMaybeScheduleDequeueEvent):
(WebCore::WebCodecsBase::blockControlMessageQueue):
(WebCore::WebCodecsBase::unblockControlMessageQueue):
(WebCore::WebCodecsBase::virtualHasPendingActivity const):
* Source/WebCore/Modules/webcodecs/WebCodecsBase.h: Copied from Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h.
(WebCore::WebCodecsBase::state const):
(WebCore::WebCodecsBase::setState):
(WebCore::WebCodecsBase::operationQueueSize const):
(WebCore::WebCodecsBase::maximumCodecOperationsEnqueued const):
(WebCore::WebCodecsBase::increaseCodecOperationCount):
(WebCore::WebCodecsBase::isCodecSaturated const):
* Source/WebCore/Modules/webcodecs/WebCodecsControlMessage.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::WebCodecsVideoDecoder::WebCodecsVideoDecoder):
(WebCore::WebCodecsVideoDecoder::configure):
(WebCore::WebCodecsVideoDecoder::decode):
(WebCore::WebCodecsVideoDecoder::flush):
(WebCore::WebCodecsVideoDecoder::closeDecoder):
(WebCore::WebCodecsVideoDecoder::resetDecoder):
(WebCore::WebCodecsVideoDecoder::stop):
(WebCore::WebCodecsVideoDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h:
(WebCore::WebCodecsVideoDecoder::decodeQueueSize const):
(WebCore::WebCodecsVideoDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.cpp:
(WebCore::WebCodecsVideoEncoder::WebCodecsVideoEncoder):
(WebCore::WebCodecsVideoEncoder::updateRates):
(WebCore::WebCodecsVideoEncoder::configure):
(WebCore::WebCodecsVideoEncoder::encode):
(WebCore::WebCodecsVideoEncoder::flush):
(WebCore::WebCodecsVideoEncoder::closeEncoder):
(WebCore::WebCodecsVideoEncoder::resetEncoder):
(WebCore::WebCodecsVideoEncoder::stop):
(WebCore::WebCodecsVideoEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.h:
(WebCore::WebCodecsVideoEncoder::encodeQueueSize const):
(WebCore::WebCodecsVideoEncoder::state const): Deleted.
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
jyavenard added a commit to jyavenard/WebKit that referenced this issue Dec 13, 2024
@jyavenard
[WebCodecs] Limit the number of codec operations we can enqueue
4818979 
https://bugs.webkit.org/show_bug.cgi?id=284448
rdar://141272065

Reviewed by NOBODY (OOPS!).

While the WebCodecs specs clearly allows for boundless queue for calculating
"Codec Saturdation", a side effect is that both decodeQueueSize and encodeQueueSize
can only ever be observed with a size of 0.
The value of decodeQueueSize and encodeQueueSize is used by some site (including W3C's WebCodecs's own example)
as a way to limit how much in advance it will attempt to encode or decode.

Both Chrome and Firefox have a limit on how many codecs operations (1 for Firefox, between 1 and 8 for Chrome)
they will enqueue concurrently.
To minimise web compatibility issue, in addition to lodging a spec bug (w3c/webcodecs#864)
we also introduce a limit on how many frames with submit to the internal decoder/encoder
before submitting more:
- 1 for Audio
- 4 for Video.

In order of not having to replicate 4 times the code in {Audio|Video}{Encoder|Decoder}
we create a new WebCodecsBase that handles all the control message queue operations
as well as handle how many codec operations have been submitted, and of which
all WebCodecs inherit from.
Doing so allows to remove a lot of similar code across all webcodecs and get
us closer to the verbiage of the specs.

Another benefit is that WebCodecsControlMessage no longer needs to be a templated class,
and we make it return a "Processed" or "Not Processed" value as the specs does.

No change in observable behaviours with existing WPT.
Tested that it allows the WebCodecs reference player to work.
No new tests as none of the work above is per spec.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp:
(WebCore::WebCodecsAudioDecoder::WebCodecsAudioDecoder):
(WebCore::WebCodecsAudioDecoder::configure):
(WebCore::WebCodecsAudioDecoder::decode):
(WebCore::WebCodecsAudioDecoder::flush):
(WebCore::WebCodecsAudioDecoder::closeDecoder):
(WebCore::WebCodecsAudioDecoder::resetDecoder):
(WebCore::WebCodecsAudioDecoder::stop):
(WebCore::WebCodecsAudioDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h:
(WebCore::WebCodecsAudioDecoder::decodeQueueSize const):
(WebCore::WebCodecsAudioDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::WebCodecsAudioEncoder::WebCodecsAudioEncoder):
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode): FlyBy: we need to check the configuration of the AudioData
against the original configuration of the encoder, not the internal configuration which may not always match.
(WebCore::WebCodecsAudioEncoder::flush):
(WebCore::WebCodecsAudioEncoder::closeEncoder):
(WebCore::WebCodecsAudioEncoder::resetEncoder):
(WebCore::WebCodecsAudioEncoder::stop):
(WebCore::WebCodecsAudioEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.h:
(WebCore::WebCodecsAudioEncoder::encodeQueueSize const):
(WebCore::WebCodecsAudioEncoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsBase.cpp: Added.
(WebCore::WebCodecsBase::WebCodecsBase):
(WebCore::WebCodecsBase::queueControlMessageAndProcess):
(WebCore::WebCodecsBase::queueControlMessageForCodecOperationAndProcess):
(WebCore::WebCodecsBase::scheduleDequeueEvent):
(WebCore::WebCodecsBase::processControlMessageQueue):
(WebCore::WebCodecsBase::incrementOperationQueueSize):
(WebCore::WebCodecsBase::decrementOperationQueueSizeAndScheduleDequeueEvent):
(WebCore::WebCodecsBase::decreaseCodecOperationCountAndMaybeProcessControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueueAndMaybeScheduleDequeueEvent):
(WebCore::WebCodecsBase::blockControlMessageQueue):
(WebCore::WebCodecsBase::unblockControlMessageQueue):
(WebCore::WebCodecsBase::virtualHasPendingActivity const):
* Source/WebCore/Modules/webcodecs/WebCodecsBase.h: Copied from Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h.
(WebCore::WebCodecsBase::state const):
(WebCore::WebCodecsBase::setState):
(WebCore::WebCodecsBase::operationQueueSize const):
(WebCore::WebCodecsBase::maximumCodecOperationsEnqueued const):
(WebCore::WebCodecsBase::increaseCodecOperationCount):
(WebCore::WebCodecsBase::isCodecSaturated const):
* Source/WebCore/Modules/webcodecs/WebCodecsControlMessage.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::WebCodecsVideoDecoder::WebCodecsVideoDecoder):
(WebCore::WebCodecsVideoDecoder::configure):
(WebCore::WebCodecsVideoDecoder::decode):
(WebCore::WebCodecsVideoDecoder::flush):
(WebCore::WebCodecsVideoDecoder::closeDecoder):
(WebCore::WebCodecsVideoDecoder::resetDecoder):
(WebCore::WebCodecsVideoDecoder::stop):
(WebCore::WebCodecsVideoDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h:
(WebCore::WebCodecsVideoDecoder::decodeQueueSize const):
(WebCore::WebCodecsVideoDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.cpp:
(WebCore::WebCodecsVideoEncoder::WebCodecsVideoEncoder):
(WebCore::WebCodecsVideoEncoder::updateRates):
(WebCore::WebCodecsVideoEncoder::configure):
(WebCore::WebCodecsVideoEncoder::encode):
(WebCore::WebCodecsVideoEncoder::flush):
(WebCore::WebCodecsVideoEncoder::closeEncoder):
(WebCore::WebCodecsVideoEncoder::resetEncoder):
(WebCore::WebCodecsVideoEncoder::stop):
(WebCore::WebCodecsVideoEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.h:
(WebCore::WebCodecsVideoEncoder::encodeQueueSize const):
(WebCore::WebCodecsVideoEncoder::state const): Deleted.
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
@jyavenard jyavenard mentioned this issue Dec 13, 2024
Merged
jyavenard added a commit to jyavenard/WebKit that referenced this issue Dec 13, 2024
@jyavenard
[WebCodecs] Limit the number of codec operations we can enqueue
93922fa 
https://bugs.webkit.org/show_bug.cgi?id=284448
rdar://141272065

Reviewed by NOBODY (OOPS!).

While the WebCodecs specs clearly allows for boundless queue for calculating
"Codec Saturdation", a side effect is that both decodeQueueSize and encodeQueueSize
can only ever be observed with a size of 0.
The value of decodeQueueSize and encodeQueueSize is used by some site (including W3C's WebCodecs's own example)
as a way to limit how much in advance it will attempt to encode or decode.

Both Chrome and Firefox have a limit on how many codecs operations (1 for Firefox, between 1 and 8 for Chrome)
they will enqueue concurrently.
To minimise web compatibility issue, in addition to lodging a spec bug (w3c/webcodecs#864)
we also introduce a limit on how many frames with submit to the internal decoder/encoder
before submitting more:
- 1 for Audio
- 4 for Video.

In order of not having to replicate 4 times the code in {Audio|Video}{Encoder|Decoder}
we create a new WebCodecsBase that handles all the control message queue operations
as well as handle how many codec operations have been submitted, and of which
all WebCodecs inherit from.
Doing so allows to remove a lot of similar code across all webcodecs and get
us closer to the verbiage of the specs.

Another benefit is that WebCodecsControlMessage no longer needs to be a templated class,
and we make it return a "Processed" or "Not Processed" value as the specs does.

No change in observable behaviours with existing WPT.
Tested that it allows the WebCodecs reference player to work.
No new tests as none of the work above is per spec.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp:
(WebCore::WebCodecsAudioDecoder::WebCodecsAudioDecoder):
(WebCore::WebCodecsAudioDecoder::configure):
(WebCore::WebCodecsAudioDecoder::decode):
(WebCore::WebCodecsAudioDecoder::flush):
(WebCore::WebCodecsAudioDecoder::closeDecoder):
(WebCore::WebCodecsAudioDecoder::resetDecoder):
(WebCore::WebCodecsAudioDecoder::stop):
(WebCore::WebCodecsAudioDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h:
(WebCore::WebCodecsAudioDecoder::decodeQueueSize const):
(WebCore::WebCodecsAudioDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::WebCodecsAudioEncoder::WebCodecsAudioEncoder):
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode): FlyBy: we need to check the configuration of the AudioData
against the original configuration of the encoder, not the internal configuration which may not always match.
(WebCore::WebCodecsAudioEncoder::flush):
(WebCore::WebCodecsAudioEncoder::closeEncoder):
(WebCore::WebCodecsAudioEncoder::resetEncoder):
(WebCore::WebCodecsAudioEncoder::stop):
(WebCore::WebCodecsAudioEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.h:
(WebCore::WebCodecsAudioEncoder::encodeQueueSize const):
(WebCore::WebCodecsAudioEncoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsBase.cpp: Added.
(WebCore::WebCodecsBase::WebCodecsBase):
(WebCore::WebCodecsBase::queueControlMessageAndProcess):
(WebCore::WebCodecsBase::queueCodecControlMessageAndProcess):
(WebCore::WebCodecsBase::scheduleDequeueEvent):
(WebCore::WebCodecsBase::processControlMessageQueue):
(WebCore::WebCodecsBase::incrementCodecQueueSize):
(WebCore::WebCodecsBase::decrementCodecQueueSizeAndScheduleDequeueEvent):
(WebCore::WebCodecsBase::decrementCodecOperationCountAndMaybeProcessControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueueAndMaybeScheduleDequeueEvent):
(WebCore::WebCodecsBase::blockControlMessageQueue):
(WebCore::WebCodecsBase::unblockControlMessageQueue):
(WebCore::WebCodecsBase::virtualHasPendingActivity const):
* Source/WebCore/Modules/webcodecs/WebCodecsBase.h: Copied from Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h.
(WebCore::WebCodecsBase::state const):
(WebCore::WebCodecsBase::setState):
(WebCore::WebCodecsBase::codecQueueSize const):
(WebCore::WebCodecsBase::maximumCodecOperationsEnqueued const):
(WebCore::WebCodecsBase::incrementCodecOperationCount):
(WebCore::WebCodecsBase::isCodecSaturated const):
* Source/WebCore/Modules/webcodecs/WebCodecsControlMessage.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::WebCodecsVideoDecoder::WebCodecsVideoDecoder):
(WebCore::WebCodecsVideoDecoder::configure):
(WebCore::WebCodecsVideoDecoder::decode):
(WebCore::WebCodecsVideoDecoder::flush):
(WebCore::WebCodecsVideoDecoder::closeDecoder):
(WebCore::WebCodecsVideoDecoder::resetDecoder):
(WebCore::WebCodecsVideoDecoder::stop):
(WebCore::WebCodecsVideoDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h:
(WebCore::WebCodecsVideoDecoder::decodeQueueSize const):
(WebCore::WebCodecsVideoDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.cpp:
(WebCore::WebCodecsVideoEncoder::WebCodecsVideoEncoder):
(WebCore::WebCodecsVideoEncoder::updateRates):
(WebCore::WebCodecsVideoEncoder::configure):
(WebCore::WebCodecsVideoEncoder::encode):
(WebCore::WebCodecsVideoEncoder::flush):
(WebCore::WebCodecsVideoEncoder::closeEncoder):
(WebCore::WebCodecsVideoEncoder::resetEncoder):
(WebCore::WebCodecsVideoEncoder::stop):
(WebCore::WebCodecsVideoEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.h:
(WebCore::WebCodecsVideoEncoder::encodeQueueSize const):
(WebCore::WebCodecsVideoEncoder::state const): Deleted.
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
webkit-commit-queue pushed a commit to jyavenard/WebKit that referenced this issue Dec 13, 2024
@jyavenard
[WebCodecs] Limit the number of codec operations we can enqueue
53a8065 
https://bugs.webkit.org/show_bug.cgi?id=284448
rdar://141272065

Reviewed by Youenn Fablet.

While the WebCodecs specs clearly allows for boundless queue for calculating
"Codec Saturdation", a side effect is that both decodeQueueSize and encodeQueueSize
can only ever be observed with a size of 0.
The value of decodeQueueSize and encodeQueueSize is used by some site (including W3C's WebCodecs's own example)
as a way to limit how much in advance it will attempt to encode or decode.

Both Chrome and Firefox have a limit on how many codecs operations (1 for Firefox, between 1 and 8 for Chrome)
they will enqueue concurrently.
To minimise web compatibility issue, in addition to lodging a spec bug (w3c/webcodecs#864)
we also introduce a limit on how many frames with submit to the internal decoder/encoder
before submitting more:
- 1 for Audio
- 4 for Video.

In order of not having to replicate 4 times the code in {Audio|Video}{Encoder|Decoder}
we create a new WebCodecsBase that handles all the control message queue operations
as well as handle how many codec operations have been submitted, and of which
all WebCodecs inherit from.
Doing so allows to remove a lot of similar code across all webcodecs and get
us closer to the verbiage of the specs.

Another benefit is that WebCodecsControlMessage no longer needs to be a templated class,
and we make it return a "Processed" or "Not Processed" value as the specs does.

No change in observable behaviours with existing WPT.
Tested that it allows the WebCodecs reference player to work.
No new tests as none of the work above is per spec.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp:
(WebCore::WebCodecsAudioDecoder::WebCodecsAudioDecoder):
(WebCore::WebCodecsAudioDecoder::configure):
(WebCore::WebCodecsAudioDecoder::decode):
(WebCore::WebCodecsAudioDecoder::flush):
(WebCore::WebCodecsAudioDecoder::closeDecoder):
(WebCore::WebCodecsAudioDecoder::resetDecoder):
(WebCore::WebCodecsAudioDecoder::stop):
(WebCore::WebCodecsAudioDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h:
(WebCore::WebCodecsAudioDecoder::decodeQueueSize const):
(WebCore::WebCodecsAudioDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::WebCodecsAudioEncoder::WebCodecsAudioEncoder):
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode): FlyBy: we need to check the configuration of the AudioData
against the original configuration of the encoder, not the internal configuration which may not always match.
(WebCore::WebCodecsAudioEncoder::flush):
(WebCore::WebCodecsAudioEncoder::closeEncoder):
(WebCore::WebCodecsAudioEncoder::resetEncoder):
(WebCore::WebCodecsAudioEncoder::stop):
(WebCore::WebCodecsAudioEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.h:
(WebCore::WebCodecsAudioEncoder::encodeQueueSize const):
(WebCore::WebCodecsAudioEncoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsBase.cpp: Added.
(WebCore::WebCodecsBase::WebCodecsBase):
(WebCore::WebCodecsBase::queueControlMessageAndProcess):
(WebCore::WebCodecsBase::queueCodecControlMessageAndProcess):
(WebCore::WebCodecsBase::scheduleDequeueEvent):
(WebCore::WebCodecsBase::processControlMessageQueue):
(WebCore::WebCodecsBase::incrementCodecQueueSize):
(WebCore::WebCodecsBase::decrementCodecQueueSizeAndScheduleDequeueEvent):
(WebCore::WebCodecsBase::decrementCodecOperationCountAndMaybeProcessControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueueAndMaybeScheduleDequeueEvent):
(WebCore::WebCodecsBase::blockControlMessageQueue):
(WebCore::WebCodecsBase::unblockControlMessageQueue):
(WebCore::WebCodecsBase::virtualHasPendingActivity const):
* Source/WebCore/Modules/webcodecs/WebCodecsBase.h: Copied from Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h.
(WebCore::WebCodecsBase::state const):
(WebCore::WebCodecsBase::setState):
(WebCore::WebCodecsBase::codecQueueSize const):
(WebCore::WebCodecsBase::maximumCodecOperationsEnqueued const):
(WebCore::WebCodecsBase::incrementCodecOperationCount):
(WebCore::WebCodecsBase::isCodecSaturated const):
* Source/WebCore/Modules/webcodecs/WebCodecsControlMessage.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::WebCodecsVideoDecoder::WebCodecsVideoDecoder):
(WebCore::WebCodecsVideoDecoder::configure):
(WebCore::WebCodecsVideoDecoder::decode):
(WebCore::WebCodecsVideoDecoder::flush):
(WebCore::WebCodecsVideoDecoder::closeDecoder):
(WebCore::WebCodecsVideoDecoder::resetDecoder):
(WebCore::WebCodecsVideoDecoder::stop):
(WebCore::WebCodecsVideoDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h:
(WebCore::WebCodecsVideoDecoder::decodeQueueSize const):
(WebCore::WebCodecsVideoDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.cpp:
(WebCore::WebCodecsVideoEncoder::WebCodecsVideoEncoder):
(WebCore::WebCodecsVideoEncoder::updateRates):
(WebCore::WebCodecsVideoEncoder::configure):
(WebCore::WebCodecsVideoEncoder::encode):
(WebCore::WebCodecsVideoEncoder::flush):
(WebCore::WebCodecsVideoEncoder::closeEncoder):
(WebCore::WebCodecsVideoEncoder::resetEncoder):
(WebCore::WebCodecsVideoEncoder::stop):
(WebCore::WebCodecsVideoEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.h:
(WebCore::WebCodecsVideoEncoder::encodeQueueSize const):
(WebCore::WebCodecsVideoEncoder::state const): Deleted.
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:

Canonical link: https://commits.webkit.org/287798@main
Jarred-Sumner pushed a commit to oven-sh/WebKit that referenced this issue Dec 18, 2024
@jyavenard @Jarred-Sumner
[WebCodecs] Limit the number of codec operations we can enqueue
6a4cfa4 
https://bugs.webkit.org/show_bug.cgi?id=284448
rdar://141272065

Reviewed by Youenn Fablet.

While the WebCodecs specs clearly allows for boundless queue for calculating
"Codec Saturdation", a side effect is that both decodeQueueSize and encodeQueueSize
can only ever be observed with a size of 0.
The value of decodeQueueSize and encodeQueueSize is used by some site (including W3C's WebCodecs's own example)
as a way to limit how much in advance it will attempt to encode or decode.

Both Chrome and Firefox have a limit on how many codecs operations (1 for Firefox, between 1 and 8 for Chrome)
they will enqueue concurrently.
To minimise web compatibility issue, in addition to lodging a spec bug (w3c/webcodecs#864)
we also introduce a limit on how many frames with submit to the internal decoder/encoder
before submitting more:
- 1 for Audio
- 4 for Video.

In order of not having to replicate 4 times the code in {Audio|Video}{Encoder|Decoder}
we create a new WebCodecsBase that handles all the control message queue operations
as well as handle how many codec operations have been submitted, and of which
all WebCodecs inherit from.
Doing so allows to remove a lot of similar code across all webcodecs and get
us closer to the verbiage of the specs.

Another benefit is that WebCodecsControlMessage no longer needs to be a templated class,
and we make it return a "Processed" or "Not Processed" value as the specs does.

No change in observable behaviours with existing WPT.
Tested that it allows the WebCodecs reference player to work.
No new tests as none of the work above is per spec.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp:
(WebCore::WebCodecsAudioDecoder::WebCodecsAudioDecoder):
(WebCore::WebCodecsAudioDecoder::configure):
(WebCore::WebCodecsAudioDecoder::decode):
(WebCore::WebCodecsAudioDecoder::flush):
(WebCore::WebCodecsAudioDecoder::closeDecoder):
(WebCore::WebCodecsAudioDecoder::resetDecoder):
(WebCore::WebCodecsAudioDecoder::stop):
(WebCore::WebCodecsAudioDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h:
(WebCore::WebCodecsAudioDecoder::decodeQueueSize const):
(WebCore::WebCodecsAudioDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::WebCodecsAudioEncoder::WebCodecsAudioEncoder):
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode): FlyBy: we need to check the configuration of the AudioData
against the original configuration of the encoder, not the internal configuration which may not always match.
(WebCore::WebCodecsAudioEncoder::flush):
(WebCore::WebCodecsAudioEncoder::closeEncoder):
(WebCore::WebCodecsAudioEncoder::resetEncoder):
(WebCore::WebCodecsAudioEncoder::stop):
(WebCore::WebCodecsAudioEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.h:
(WebCore::WebCodecsAudioEncoder::encodeQueueSize const):
(WebCore::WebCodecsAudioEncoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsBase.cpp: Added.
(WebCore::WebCodecsBase::WebCodecsBase):
(WebCore::WebCodecsBase::queueControlMessageAndProcess):
(WebCore::WebCodecsBase::queueCodecControlMessageAndProcess):
(WebCore::WebCodecsBase::scheduleDequeueEvent):
(WebCore::WebCodecsBase::processControlMessageQueue):
(WebCore::WebCodecsBase::incrementCodecQueueSize):
(WebCore::WebCodecsBase::decrementCodecQueueSizeAndScheduleDequeueEvent):
(WebCore::WebCodecsBase::decrementCodecOperationCountAndMaybeProcessControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueueAndMaybeScheduleDequeueEvent):
(WebCore::WebCodecsBase::blockControlMessageQueue):
(WebCore::WebCodecsBase::unblockControlMessageQueue):
(WebCore::WebCodecsBase::virtualHasPendingActivity const):
* Source/WebCore/Modules/webcodecs/WebCodecsBase.h: Copied from Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h.
(WebCore::WebCodecsBase::state const):
(WebCore::WebCodecsBase::setState):
(WebCore::WebCodecsBase::codecQueueSize const):
(WebCore::WebCodecsBase::maximumCodecOperationsEnqueued const):
(WebCore::WebCodecsBase::incrementCodecOperationCount):
(WebCore::WebCodecsBase::isCodecSaturated const):
* Source/WebCore/Modules/webcodecs/WebCodecsControlMessage.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::WebCodecsVideoDecoder::WebCodecsVideoDecoder):
(WebCore::WebCodecsVideoDecoder::configure):
(WebCore::WebCodecsVideoDecoder::decode):
(WebCore::WebCodecsVideoDecoder::flush):
(WebCore::WebCodecsVideoDecoder::closeDecoder):
(WebCore::WebCodecsVideoDecoder::resetDecoder):
(WebCore::WebCodecsVideoDecoder::stop):
(WebCore::WebCodecsVideoDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h:
(WebCore::WebCodecsVideoDecoder::decodeQueueSize const):
(WebCore::WebCodecsVideoDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.cpp:
(WebCore::WebCodecsVideoEncoder::WebCodecsVideoEncoder):
(WebCore::WebCodecsVideoEncoder::updateRates):
(WebCore::WebCodecsVideoEncoder::configure):
(WebCore::WebCodecsVideoEncoder::encode):
(WebCore::WebCodecsVideoEncoder::flush):
(WebCore::WebCodecsVideoEncoder::closeEncoder):
(WebCore::WebCodecsVideoEncoder::resetEncoder):
(WebCore::WebCodecsVideoEncoder::stop):
(WebCore::WebCodecsVideoEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.h:
(WebCore::WebCodecsVideoEncoder::encodeQueueSize const):
(WebCore::WebCodecsVideoEncoder::state const): Deleted.
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:

Canonical link: https://commits.webkit.org/287798@main
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jyavenard