Use Media Capabilities Decoding Info API

[beaufortfrancois]

The Media Capabilities API is an API for querying the user agent about decoding and encoding capabilities of the device, providing better information about the display properties, and real-time feedback about playback so that adaptive streaming can better alter the quality of content based on user perceived quality. The intent of this API is to give more control for content delivery while providing a better user experience.

I think this would be great to integrate it into Shaka Player as it is going to be enabled by default in Chrome 66 and is already used by YouTube player.

Example:

const mediaConfig = {
  type: 'media-source', // or 'file'
  audio: {
    contentType: 'audio/webm; codecs=opus',
    channels: '2', // audio channels used by the track
    bitrate: 132266, // number of bits used to encode a second of audio
    samplerate: 48000 // number of samples of audio carried per second
  },
  video: {
    contentType: 'video/webm; codecs="vp09.00.10.08"',
    width: 1920,
    height: 1080,
    bitrate: 2646242, // number of bits used to encode a second of video
    framerate: '25' // number of frames used in one second
  }
};

navigator.mediaCapabilities.decodingInfo(mediaConfig).then(result => {
  log('This configuration is' +
      (result.supported ? '' : ' NOT') + ' supported,' +
      (result.smooth ? '' : ' NOT') + ' smooth and' +
      (result.powerEfficient ? '' : ' NOT') + ' power efficient.');
});

Resources:

• https://wicg.github.io/media-capabilities/
• https://developers.google.com/web/updates/2017/12/chrome-63-64-media-updates#media-capabilities-decoding-info-api
• https://googlechrome.github.io/samples/media-capabilities/decoding-info.html
• https://www.chromestatus.com/feature/5869632707624960

[joeyparrish]

Scheduled for v2.5. Thanks!

[joeyparrish]

Rough plan: Use this API (if present) to drive an additional filtering pass on the media. If any variants are smooth, we avoid all non-smooth variants.

We may want to consider using the powerEfficient field as well if the data-saver flag is on. See #855.

[beaufortfrancois]

That's great to hear!
FYI @chcunningham @mounirlamouri

[mounirlamouri]

Out of curiosity, how would you use powerEfficient with data-saver?

[joeyparrish]

To clarify, I was thinking data-saver being enabled would be a proxy to tell us it's a mobile device.

But now I realize that there's a battery API as well. MDN says it's obsolete, but it's not clear why, and they don't recommend an alternative.

Mounir, I see your name on the battery status spec. Can you clarify its status? Is there another way we should determine that powerEfficient is an important bit?

[joeyparrish]

I tried out MediaCapabilities in Shaka Player this week, but I think my plan to use it for an additional filtering pass needs to change.

For one, MC is async, and our existing filtering is synchronous. This means we may want to refactor a bit before introducing it.

For another, this is different from our filtering based on isTypeSupported, codec compatibility (to avoid unsupported codec switches in MSE), and DRM restrictions (to remove things we can't decrypt). Those things are "hard" restrictions that we should never be flexible with. If a codec is not supported at all, we fail if we try. But just because something is smooth doesn't mean we should never play it. smooth is a "soft" signal, indicating what we should do.

We could still use it as a filter, in some circumstances. For example, if playback were smooth for 480p & 720p, but not 1080p, we could easily exclude 1080p without much harm.

But what if stereo audio were smooth and 5.1 surround were not? We don't know if that would ever happen in practice, but from the API itself, this is a possible outcome. Now what if the 5.1 surround track were the original English language version of a TV show, but the French dub were only available in stereo? I've seen this sort of thing on some region 2 DVDs. In those cases, we might be filtering out an entire language based on smoothness.

So the only safe way to use this as a filter is to first split up the tracks by things like language (and probably other metadata), then filter in each group. If nothing in a group is smooth, you can't get rid of the whole group. This sounds like a giant pain.

The alternative is to use MC to give ABR hints, rather than filter tracks. For example, we could annotate the variants with smoothness, and we could disallow AbrManager from choosing anything that is not smooth. (If MC is missing, we would assume everything to be smooth.)

I'm leaning toward using MC for ABR hints, as this is a much smaller refactor.

[chcunningham]

But what if stereo audio were smooth and 5.1 surround were not? We don't know if that would ever happen in practice, but from the API itself, this is a possible outcome.

Adding a little color here - in chrome, especially for unencrypted, the audio configuration will not affect smoothness.

There is a hypothetical scenario where UAs in very constrained devices (e.g. TVs, cast devices, etc) could theoretically be impacted by the audio configuration. For instance, say audioConfigA means can choose internal pipeline A which leverages their hw chip to smoothly decode the corresponding video configuration. But audioConfigB might only be available in a SW pipeline and they don't support running both pipelines in parallel -> less smooth video. But this is pretty contrived and the real world may never hit this problem.

You might just use the API in whatever way is most valuable to you and keep some metrics to indicate whether you ever encounter this hypothetical audio filtering problem. I think chances are low, maybe zero, but I'd be excited to learn otherwise.

[joeyparrish]

As @michellezhuogg has been working on MediaCapabilities for a couple weeks now, I decided to test the platforms in our lab to see which ones have MediaCapabilities.

I found that IE, Tizen 3, and Xbox One do not have the API. IE is not a big deal, since we are dropping support for IE anyway. But the other two will need a polyfill to fall back on.

What's sadder, though, is that Chromecast has MediaCapabilities, but I found that it is broken for encrypted content. Though both PlayReady and Widevine are present on the device, MediaCapabilities reports that neither is. So we will need to deny-list the implementation on Chromecast based on the firmware version number until MediaCapabilities in the Cast firmware gets fixed.

All other platforms that supported MediaCapabilities seem to provide correct results for encrypted content, meaning that the EME-related parts of the API are actually supported. This means our polyfill will not have to deal with platforms that only have MediaCapabilities for clear content.

[avelad]

Regarding Tizen, the newer versions are based on a Chromium version that should support this. To avoid banning these versions if you need to do tests I can do them to see if it really works.

Links of interest:
https://caniuse.com/mdn-api_mediacapabilities_decodinginfo

https://developer.samsung.com/smarttv/develop/specifications/web-engine-specifications.html

[joeyparrish]

@avelad, we probably won't have to ban anything by version after all. Normally, we will detect the API itself. If navigator.mediaCapabilities is null, we will have to install a polyfill to provide the necessary info.

The results I'm seeing on Chromecast via our test framework are not reproducible in the JS debugger on my Chromecast at home. Not sure what's going on there yet, but it probably means we won't have any Cast-specific bans in the long term.

[joeyparrish]

Just for a quick update, the weird results I get on Chromecast are limited to our test environment. It seems to be triggered by the iframe used to run our tests on the device. I'm talking to the Cast team about this to get it fixed.

[avelad]

@michellezhuogg Is the integration of this functionality already complete?

[joeyparrish]

This is complete, too.