Automatic ABR quality restrictions based on size

[alejandroiglesias]

Have you read the FAQ and checked for duplicate open issues?
This was discussed as a part of #565, but the limit based on video container size wasn't implemented.

Is your feature request related to a problem? Please describe.

A functionality that Dash.js and hls.js have is the ability to limit which quality levels are selected based on video element size.

Describe the solution you'd like

On the ABR settings, an extra flag could be added to instruct the ABR manager to limit the quality level based on the video element. For example:

{
  abr: {
    capLevelToPlayerSize: true
  }
}

Or:

{
  abr: {
    limitBitrateByPortal: true
  }
}

Those are the options used in Dash.js and hls.js. You can choose any name fits for this project.

Describe alternatives you've considered

Without this feature, a possible solution would be to introduce a ResizeObserver on the video element to react to changes in size and dynamically set the ABR restrictions on Shaka player.

Additional context

This would allow saving time (and potential problems with the orchestration of the ResizeObserver and the dynamic setting of restrictions) that can arise in the implementation of a video player that handles bandwidth in a smart way not to load a video quality that is higher than required at a given moment.

[joeyparrish]

ResizeObserver is not available on Edge (until it becomes Chromium-based), IE (which we will drop support for some day), or Safari (which is a bigger issue). If we restrict ABR based on the size of the video element at the level of Shaka Player, our options seem to be limited to:

1. ResizeObserver (except on IE, Edge, Safari)
2. Polling getClientRects() (fallback for IE, Edge, & Safari perhaps?)

I worry about the expense of polling, especially on embedded devices.

A cheap and easy time to check the size is when we go into or out of fullscreen mode, which there are universal events for.

It's not clear to me in what other circumstances an application would change the size of the video element. But it does seem clear the application always knows when that happens, because it is the thing changing the size of the element.

So I have always felt that the most efficient way to do this is at the app level. But I'll look into what dash.js and hls.js do. Perhaps there's something else I've overlooked.

[joeyparrish]

hls.js polls the video size once per second to implement the capLevelToPlayerSize option. They use clientWidth and clientHeight to determine the size.

---

dash.js adds a resize event listener on the window, which only reacts to changes in the window size, not other layout changes. This could be useful in some cases, but it wouldn't, for example, be able to account for pure app-driven layout changes, such as something equivalent to YouTube's "theater mode" button.

It's not obvious that this is connected to their limitBitrateByPortal option, though. It seems that their approach is just to recompute the size every time they make an ABR decision, which is pretty clever compared to polling every second.

---

Neither relies on ResizeObserver, which I've just found will be supported in Safari TP according to caniuse.com.

One last note from my research: I've also discovered that there is a resize event on the video element, which only reacts to changes in the decoded frame size, not layout changes. This is equivalent to getting an event fired when you make an ABR decision, so this is misleading and not useful.

[joeyparrish]

In general, I'm in favor of this option, and I'm in favor of the default being true. It seems very helpful to the service provider to help them more easily avoid wasting bandwidth on higher resolutions just to downscale them on the device, not to mention saving the end-user's battery.

An event-driven solution, such as ResizeObserver, would seem ideal. It would require no changes to the plugin interface, just a new option. But we probably can't get that on all platforms, especially when we consider TVs.

I'd really like to avoid polling, since polling turns out to be surprisingly expensive in specific cases on devices. (See #2252)

Another alternative is to make AbrManager instances aware of the video element and measure its size every time it makes a decision, as dash.js does. This is probably our best option for robustness, but may require changing the plugin interface.

[alejandroiglesias]

@joeyparrish I appreciate the time you took into investigating this. Thank you. I looked at the smallest ResizeObserver ponyfill I found, the resize-observer npm module, which is based on polling window.getComputedStyle() on requestAnimationFrame callbacks. Do you think this could be a performance issue on certain devices? Why? Shouldn't requestAnimationFrame callbacks happen on spare cycles?

Otherwise, a mix of ResizeObserver for supporting browsers and making AbrManager instances aware of the video element and measure its size every time it makes a decision, as dash.js does, for non-supporting browsers would be ideal. I assume this would mean reacting instantly on supporting browsers and reacting with a delay on non-supporting ones. Regarding the latter, how often are ABR decisions made? And are they constantly made or it stops after reaching the upper limits?

[joeyparrish]

I suspect that making AbrManager aware of the video element should be enough. I'm not sure we should need to poll. The default AbrManager (by design) doesn't make decisions all the time, but only once every so many seconds (8 by default).

A custom AbrManager can already make decisions whenever it likes, ignoring the config value for abr.switchInterval, and it can also poll the video element size or add ResizeObservers.

So I think checking the element size when we make decisions should be good enough for the default AbrManager. The only risk in this approach is changing a plugin interface, which must be done carefully and with an eye toward backward compatibility if possible.

[alejandroiglesias]

@joeyparrish I agree that checking the video element size every few seconds (8 by default) seems more than enough. What changes in the interface do you expect for this to happen? And do you expect this functionality to be implemented by you or any member of the team in the short term?
Also, can you provide a quick intro on how to create a custom AbrManager that works as the default one but in which I can add polling of the video element size on every decision so that I can work around this functionality until it's implemented in the core?

[matvp91]

@alejandroiglesias if you want to introduce polling outside of shaka, I guess it would make sense to use the restrictions config in AbrConfiguration to match the size of your video element. This way, there is no need to extend the abr manager's functionality.

[Maxou44]

This feature will be amazing! 🥳

I think a mix of ResizeObserver and a fallback could be a great solution to get the player size 🤔

With a dash manifest with 720p + 1080p streams and a player resolution of 850p, it could be nice to choose if we want to use the biggest or the smallest video stream (if we don't match the exact resolution) 🙂

[Wuschba]

Thanks @TheModMaker to point me to this issue. He suggested using:

player.configure({
  	abr: {
		restrictions: {
  			maxWidth: 400,
                        maxHeight: 300
  		}
  	}
  });

to limit the size of the video choosen by the abr-manager. This works inside a webpage which is not changing in size, but doesn't work for example when switching to fullscreen-mode.

In my opinion a polling or continuous tracking of size-changes isn't necessary, it would be greate if maxWidth and maxHeight just adapt when shaka.abr.SimpleAbrManager.chooseVariant is called the next time.

[joeyparrish]

You make a good point. That would update the restrictions every abr decision interval, which by default is every 8 seconds. That would be sure to be more performant than polling, and at such a long interval, probably close enough in battery performance to listening for size change events.

[Doolali]

I fixed the problem using restrictions:

this.player.configure({
    restrictions: {
        maxHeight: height
    }
});

Where height comes from a window resize event.

[sunknudsen]

Did any of you manage to set ABR restrictions that adapt to player size?

[joeyparrish]

ResizeObserver is not available on all supported platforms, but is now available on all evergreen browsers. And the platforms that don't support ResizeObserver (TVs, set top boxes, etc) also don't support resizing.

So I think it would be reasonable to add this as a built-in feature, inside something like if (window.ResizeObserver).

[EyMaddis]

@Doolali the problem is that the restrictions might actually prevent playback if the video element is smaller.
This is not relevant for full screen players, but for example for small preview videos.

In a perfect world I would like to give a suggested or target resolution where shaka can for example decide to use a larger version if the screen is around 1000px in height, it could choose 1080p.

[joeyparrish]

If you use abr.restrictions instead, playback will still be possible. These are restrictions on adaptation, whereas restrictions is a hard playback restriction (meant to match up with DRM policies the player wouldn't know about).

@EyMaddis, in your example, if abr.restrictions.maxHeight is set to 1000, but the smallest resolution available is 1080p, then 1080p will still be played.