#745 Video playback quality watcher

[tchakabam]

Taking note of #745, I'd like to give a first shot at how we could deal with adaptation related to device capabilities.

We have added such a thing to our product, but it would be great to contribute this upstream to have it in a great shape and well maintained. Also I think the more general purpose we make this solution, the better it will probably work for us as well.

We added an object called DroppedFramesDetector. It takes a StatsProvider to call upon the getStats function of the Player or get dropped-frames in some way. It periodically checks the current dropped-frames-per-seconds average on a given time-window.

A lot of magic constants and opinions can be found in there, all of this could be made configurable i.e is subject to feedback. It's kept very simple intentionally. One could specify more complex detection algorithms and override it in the future, in the same way as it is done for bandwidth estimation elsewhere.

The key is to quantify somehow the device capacity to decode frames in time with a scalar.

Then we map this scalar back to a maximum pixel height, which is then applied to the ABR manager's restrictions.

In case a level-down is detected, we are doing a flushing switch to immediatly help the resources catch up with the workload and eventually lower the total pixels to decode.

The way the mapping is done, and how the levels are determined could be customizable. But I guess it would be important to find a good ground that generally will work.

Now, also one might want to have a cool-down period as sometimes a CPU might get overflowed with work and frames will get dropped, even if it's a device with good resources. Therefore to optimize the QoE one would want to switch up again after the average dropped-fps have not exceeded the heat-up threshold for the given cooldown period.

Similar to other ABR settings for bandwidth estimation, one can choose a low cooldown period and may have a better average quality, but might risk to switch up too high qualities for the CPU capacity, as an analogy to bandwidth estimation where one may smooth more or less the estimation.

Finally, I would say it is important to deactivate the cooldown when playback is buffering or paused!
And especially stop the whole detection of average dropped frames when paused or buffering in fact.
Otherwise it will take a decision based on decoding zero fps, which would be wrong :)

For the last point, we need an interface to the ABR manager where it detects playhead state (buffering/playing/paused...) in order to ensure this.

[tchakabam]

Note that there are a few debug logs in there, and no docs yet, linting/compile will surely fail as well. Only tested in uncompiled mode.

This is to give a base, to see if this approach gets positive resonance, and if yes, docs and format will be added.

[tchakabam]

in order to test this and cause some frame-dropping, i can recommend the stress tool: http://manpages.ubuntu.com/manpages/trusty/man1/stress.1.html

[TheModMaker]

In general, this is a good idea; thanks for the contribution.

But I don't think we should be guessing the capabilities of the device explicitly. Since a number of factors can influence dropped frames, it would be better to treat this the same as the network. Namely, if we have too many dropped frames, we downgrade. If we don't have any dropped frames for a while, we can allow upgrades again.

You may want to use the shaka.abr.Ewma type to get a rolling average of the dropped frames per second. Then use this to determine if we have too many dropped frames.

You could also use the video.playbackRate to determine if we are playing. When we are paused or buffering, this will equal 0.

[tchakabam]

@TheModMaker quoting you: But I don't think we should be guessing the capabilities of the device explicitly. Since a number of factors can influence dropped frames, it would be better to treat this the same as the network. Namely, if we have too many dropped frames, we downgrade. If we don't have any dropped frames for a while, we can allow upgrades again.

In case I did understand what you mean, it is in line with what we are doing in this PR. More precisely, how do you suggest should we abstract away from the specific device capabilities?

I understand we should not set fixed max-pixel values. The defaults in this PR were kind of an p-o-c example. How about one would allow to lib clients to input as configuration a set of pixel-values, over which we up and downgrad as you suggest. For this we already have added here the concept of DeviceCapacityLevels. We can rename it to Capability if it is a more consistent term in this context.

Generally, the dropped-frames detection and the handling should be disabled by default, it might cause unexpected behavior - but when running this with specific content like full-HD and 4K streams on a range of devices that may have issues with decoding, it would be great to be able to configure the behavior to ones needs and enable the watch-interval.

The idea of using a moving average for smoothing the dropped-frames measurements is a good one I agree! That's exactly what I meant by adding more advanced means of generating a decision-driving signal.

Also, was thinking the same about playbackRate, we should probably give the ABR manager access to the Playhead instance?

[TheModMaker]

What I mean is that we should not have fixed pixel sizes at all. It would be better to adapt based on the device rather than just dropping all UHD streams. So rather than having explicit regions, I suggest adjusting to the next lower track.

I would suggest a type to track an estimate for what the device supports, similar to how the bandwidth estimate works. If we start dropping frames, then the estimate should go down below what is being played. If we are not dropping frames, increase the estimate some (being sure to remember old data to avoid switching back and forth). This will result in a dynamic estimate for what the device supports.

Then SimpleAbrManager will poll the bandwidth and the video size estimate and choose the best track it can play (rather than having callbacks from DroppedFramesDetector).

Also, I don't think we should give it Playhead, or even the getStats method. I don't want the abr manager to become too coupled with the rest of the library. Remember this is a general interface that all implementations must use. I would suggest having the SimpleAbrManager accept the <video> element and use that for dropped frames and playbackRate.

[tchakabam]

> So rather than having explicit regions, I suggest adjusting to the next lower track.

Yes that makes much more sense and will universally work from a control systems pov.

We then also need to have the exponential smoothing in there to make it stabilize.

Also agreeing with all your other suggestions. Thanks for the valuable input on the solution here.

[hyurtseven81]

i've fixed the linter and test issues and i've tested as compiled mode, as far as i test i works as expected. i'm still continuing to test, i will try to inform you about updates

[tchakabam]

yes but we should move this towards the direction as discussed with @TheModMaker here. his comments make more sense than my initial approach here.

I will soon modify it in the direction of not restricting on preset resolutions, but rather have a control-system that switches bitrate up/down upon extended period of frame-drops detected by the exponential moving average.

@TheModMaker I figured the smoothing filter will also be good to cancel out high short spikes that might happen on any machine when one moves around windows or other stuff that happens in the background. we wouldn't want to switch down on that but only if there is a prolonged dropped fps rate over a certain time that does not go away.

@hyurtseven81 great if you fixed linter things! could you commit that please? :D 👍

[forbesjo]

Just wanted to add a note for this dropped frames ABR feature. In Edge it seems like dropped and total frames from getVideoPlaybackQuality reset to 0 on seeks. This occurs in regular MP4s as well so this isn't an MSE-only issue.

[tchakabam]

@forbesjo interesting. that is something we could eventually fix via the polyfill (which we already use in this library to enable webkit-prefixed dropped/decoded frame-count) by keeping a count in a reserved property in the video element itself.

but then again, we may not necessarily need it, there should be some kind of differential sampling of these counters going on, so that a reset should not matter to the overall estimate. especially if there is a seek happening, potentially we are starting on a flushed buffer and can assume different payload properties yielding different decoder behavior.

[tchakabam]

generally about the solution to this: the previous approach here is not yet what I think is what we should do.

there will in fact rather be a scalar dimension for "video quality" that should be proportional to the number of decoded frames and the total pixels decoded per frame. when frames are dropped, the quality estimate will drop (smoothed by the ewma, just like for the bandwidth).

then the ABR algorithm should optimize for highest quality possible, just like it already optimizes for highest bandwidth possible. to do so it will have to choose resolutions such that they fulfill a condition against the quality estimate (anologous to how we choose variants by the bandwidth estimate).

we are currently working on a such a direction for a solution to this.

[bitactive]

Are there any plans to include this functionality into release in the near future ?

[hyurtseven81]

Hi all,

When i'm waiting this feature added to core, i implemented a drop frame solution but i need your help and advise about the idea and implementation, i'm sending the code

to sum, i create a 5 seconds interval, and check the current period drop frame percentage if this period exceeds the threshold i limit the max resolution,

i tried the cool down period implementation but when i test on a low-end machine i saw the following behaviour:

Player starts playing on high bitrate drop frames start and the following function limits the max height and the bitrate the lower than max bitrate player can play video without any drop frame and cooldown period starts after that drop frames starts again. so my opinion cooldown period can be useless.

`
var DROPPED_PERCENTAGE_FORBID = 0.20;

var droppedFramesHistory_ = [];
var maxIndex;
setInterval(function () {
try {
var variants = player.getBitrateLevels();
if (!maxIndex) {
maxIndex = variants.length;
}
var stats = player.shaka.getStats();
var droppedFrames = stats.droppedFrames;
var totalFrames = droppedFrames + stats.decodedFrames;

droppedFramesHistory_.push({
  droppedFrames: droppedFrames,
  totalFrames: totalFrames
});

for (var i = 1; i < droppedFramesHistory_.length; i++) { 
  if (droppedFramesHistory_[i]) {
    droppedFrames = droppedFramesHistory_[i].droppedFrames - droppedFramesHistory_[i - 1].droppedFrames;
    if (droppedFrames < 0) {
      droppedFrames = 0;
    }
    totalFrames = droppedFramesHistory_[i].totalFrames - droppedFramesHistory_[i - 1].totalFrames;
    if (totalFrames < 0) {
      totalFrames = 0;
    }

    if (
      droppedFrames / totalFrames > DROPPED_PERCENTAGE_FORBID
    ) {
      maxIndex = maxIndex - 1;
      if (maxIndex <= 1) {
        maxIndex = 1;
      }
      console.log('DroppedFramesRule, index: ' + maxIndex + ' Dropped Frames: ' + droppedFrames + ' Total Frames: ' + totalFrames);
      player.shaka.setMaxHardwareResolution(variants[maxIndex - 1].width, variants[maxIndex - 1].height);
      // to clear buffer i set a static bitrate and change that to auto again
      player.setBitrateLevel(maxIndex);
      player.setBitrateLevel(-1);
      droppedFramesHistory_ = [];
      break;
    }
  }
}

} catch (e) {
console.error(e);
}
}, 5000);

`

[bitactive]

@hyurtseven81 thank you for sharing.
I have checked this code and seems working.

I would just change one thing:

player.shaka.setMaxHardwareResolution(variants[maxIndex - 1].width, variants[maxIndex - 1].height);
player.setBitrateLevel(maxIndex);
player.setBitrateLevel(-1);

into:

player.shaka.configure({ abr: {restrictions: { maxHeight: variants[maxIndex - 1].height } }});

This way player do buffer clean automatically and also do not drop other variants from its list, but only do not take them into consideration during abr selection (so user still can force to select higher resolution).

I'll check this, test in some A/B scenarios and share some results.

[hyurtseven81]

@bitactive Thank you for your reply, i'll try to change the line that you suggested and i'm waiting your test results.

[joeyparrish]

I'm sorry that we have fallen behind in reviewing pull requests. I promise to take a look at this as soon as I can make time. Please accept my apologies for keeping you waiting for so long.

[itaykinnrot]

Hi @joeyparrish - i think this is a critical issue especially in chrome DRM when it not using the GPU we see stalls due to drop frames.
Any plan to add this functionality? or we should measure it from the application?
Thanks!

[tchakabam]

@itaykinnrot +1 exactly what we observed with many HD+ and 4K streams, CDM processing creates a bottle-neck on certain hardware configs... It would be good if Shaka could detect this, and generally solve the problem of "hardware capabilities adaptation" - while browser API that tells us better about CPU / GPU resources still needs to be spec'd :)

[TheModMaker]

Is this still something you are interested in? If so, please rebase on the latest master and fix any linter errors for our new style guidelines. Note that all the new files need to have copyright headers and need to be added to one of the files in build/types (probably core).

I think having the EwmaVideoQualityEstimator is a good idea (maybe squash the public interface into a single getSmoothness method?), but it may be better as part of SimpleAbrManager's suggestStreams_ logic. So that method would determine the current smoothness or expected highest quality and use that to adjust the possible choices.

[TheModMaker]

I think it would be better to not require passing in the video element to AbrManager. What the app can do is pass a closure to player.configure that will capture the current video element and pass it to their own constructor. This is what we do for the TextDisplayer.

For the Player's default abrFactory, you could set it to shaka.abr.SimpleAbrManager.bind(null, this.video_). See how we do this for textDisplayFactory.

[TheModMaker]

This needs the full type shaka.abr.Ewma.

[TheModMaker]

Please namespace global constants. (you also can't use var under our new style rules)

shaka.abr.EwmaVideoQualityEstimator.FAST_WINDOW = 1;

[TheModMaker]

This file needs a copyright header.

[TheModMaker]

All fields need a type using either @type for public fields or @private for private ones.

[TheModMaker]

We have a utility class for this. shaka.util.Timer

[TheModMaker]

First, you should only throw shaka.util.Error (the compiler will yell at you). Second, it would be better to use feature detection and avoid using this type instead of throwing.

[TheModMaker]

We have a utility class for event listeners, shaka.util.EventManager.

[tchakabam]

@TheModMaker Hey, yes, this is an actual issue, and I would like to continue to work on a solution for it. I hope thanks to your reviewing we can get it to an actual patch to master this time 👍 I had interrupted these efforts as there was a lack of response from the maintainers side here from my pov.

[joeyparrish]

This is badly out of date, which is our fault for not reviewing it in a timely manner. I'm going to close it now, but please feel free to rebase and/or reimplement on the latest code and send a new PR. Thanks!

[tchakabam]

@joeyparrish understood & agreed :) it is also due on our side: we haven't seen the need to pursue the idea as far and/or merge it into upstream Shaka version at the time (in the end). which is really why back then i may have not continued to provide necessary changes to progress in the review. no worries and thanks for the feedback provided so far, and the clear communication right now! best regards!!