Fix FPS and latency bugs with VP9 screenshare. #1069
Conversation
Also restored L3T3_KEY publishing for screen share. It was disabled previously. Chrome does not allow more than 5 fps with L1T3, and it has encoding bugs with L3T3 It has a different path for screenshare handling and it seems to be untested/buggy As a workaround, we are setting contentHint to force it to go through the same path as regular camera video. While this is not optimal, it delivers the performance that we need
🦋 Changeset detectedLatest commit: 7f7ba5a The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
size-limit report 📦
|
| @@ -736,7 +736,16 @@ export default class LocalParticipant extends Participant { | |||
| if (isSVCCodec(videoCodec)) { | |||
| // vp9 svc with screenshare has problem to encode, always use L1T3 here | |||
| if (track.source === Track.Source.ScreenShare && videoCodec === 'vp9') { | |||
| opts.scalabilityMode = 'L1T3'; | |||
| // Chrome does not allow more than 5 fps with L1T3, and it has encoding bugs with L3T3 | |||
There was a problem hiding this comment.
I think it allows 30 fps. The array in Chrome code is for three temporal layers. If we do T3, it should do 30 fps if I am not mistaken (that layer array is for temporal layers as it is FPS). But, my interpretation could be incorrect.
There was a problem hiding this comment.
yes, it would if L3T3 functioned correctly. I think both you and @cnderrauber confirmed that it does not?
There was a problem hiding this comment.
From what I can see, they are using spatial layer as the index into the array: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/codecs/vp9/svc_config.cc;l=63;drc=b8a4daa31c86ef61f7a78ea73711b86ab732db32
So I think the T3 is indicating 3 temporal bounded by selected maxFramerate
There was a problem hiding this comment.
I could not get either L3T3 or L3T3_KEY to work. I must have done something incorrect with L3T3_KEY as it is working for you.
I meant that the array is setting FPS for temporal layers. With L1T3, it should still be one spatial and three temporal layers. So, it should do 30 fps and allow temporal scale down to lower frame rate. I am confused as to how it is limiting to only at 5 fps (which looks like temporal layer 0 in that array). But, if it is actually doing three temporal layers with one spatial, it should still do 30 fps. That's how I was reading that array.
There was a problem hiding this comment.
From what I can see, they are using spatial layer as the index into the array: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/codecs/vp9/svc_config.cc;l=63;drc=b8a4daa31c86ef61f7a78ea73711b86ab732db32
So I think the T3 is indicating 3 temporal bounded by selected maxFramerate
oh okay, that is interesting.
There was a problem hiding this comment.
does this branch work for you? the main difference that's needed is setting contentHint to motion.. which bypasses the screenshare code path in libwebrtc
There was a problem hiding this comment.
From what I can see, they are using spatial layer as the index into the array: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/codecs/vp9/svc_config.cc;l=63;drc=b8a4daa31c86ef61f7a78ea73711b86ab732db32
So I think the T3 is indicating 3 temporal bounded by selected maxFramerateoh okay, that is interesting.
Did not realise that the fps is scoped to the spatial layer. Sorry, my bad.
There was a problem hiding this comment.
does this branch work for you? the main difference that's needed is setting
contentHintto motion.. which bypasses the screenshare code path in libwebrtc
Definitely much better. Seemed to start off with low fps/high latency, but got much smoother after a few seconds.
There was a problem hiding this comment.
The initial choppiness could be due to some server side changes I am trying to test.
Also restored L3T3_KEY publishing for screen share. It was disabled previously.
Chrome does not allow more than 5 fps with L1T3, and it has encoding bugs with L3T3 It has a different path for screenshare handling and it seems to be untested/buggy As a workaround, we are setting contentHint to force it to go through the same path as regular camera video. While this is not optimal, it delivers the performance that we need.
The problem is here. where layer 0 always gets capped at 5fps, and it doesn't get reset to 30 even if there's only a single layer published.