Request floods in 1.4.1

[spinitron]

What version of Hls.js are you using?

1.4.1

What browser (including version) are you using?

Several including 13.0.2 Safari , SamsungBrowser/20.0 and others

What OS (including version) are you using?

Several including Intel Mac OS X 10_14_6 , Android 13; SAMSUNG SM-A146U and others

Test stream

https://ark3.spinitron.com/ark2/WZBC-20230509T090000Z/index.m3u8

Configuration

{debug: enableDebug}

Additional player setup steps

https://github.com/spinitron/ark-player/blob/main/src/arkPlayer.js#L168-L183

Checklist

• The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
• The issue occurs in the stable client (latest release) on https://hlsjs.video-dev.org/demo and not just on my page
• The issue occurs in the latest client (main branch) on https://hlsjs-dev.video-dev.org/demo and not just on my page
• The stream has correct Access-Control-Allow-Origin headers (CORS)
• There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

Sorry, I had to lie in the 3rd and 5th form checkbox for Github to submit the form. I have not tested with main. There are some 404s involved. Please read on...

I operate a production audio stream service that provides an HLS server and a web player. I use the player on spinitron.com e.g. https://spinitron.com/WZBC/ The player uses hls.js if (Hls.isSupported().

I do not have a test env to reproduce the error.

Expected behaviour

Normal HLS client traffic visible on the HLS server.

What actually happened?

Our web pages with the player were set to load hls.js@latest. 6 days ago we started seeing floods of requests for .m3u8 playlists and for expired .ts segments. A flood from a single client could sustain over 1000 requests per second for a few minutes. The number of these floods increased over a period of 5 days. Yesterday evening I changed the asset URL to [email protected]. The floods have almost stopped and in the last 8 hours only 3 clients exhibit anomalous persistent requests.

The request floods were mostly for valid .m3u8 URLs. There were also floods of requests for expired .ts segments resulting in 404 responses. The nginx server responded to cache until I added rate limiting when it responded with 429 for the excess. Rate limiting didn't change the request flood patterns that I could see.

Console output

I don't have a dev env to reproduce the error. I'm describing an incident on a production system.

Chrome media internals output

No response

[robwalch]

Additional player setup steps
https://github.com/spinitron/ark-player/blob/main/src/arkPlayer.js#L168-L183

myhls.attachMedia(theMediaElement);
myhls.on(Hls.Events.MEDIA_ATTACHED, function () {
    myhls.loadSource(playlistUrl);
    pleaseReload = false;
    myhls.on(Hls.Events.ERROR, handleHlsError);
    myhls.on(Hls.Events.MANIFEST_PARSED, function (event, data) {
        pressPlay();
    });
});

Do not call loadSource or attach event listeners in attached callbacks. This opens up the possibility of recursion, and attaching the same callback multiple times to the same event. Instead, do all the setup at once at the top level:

myhls.loadSource(playlistUrl);
myhls.attachMedia(theMediaElement);
myhls.on(Hls.Events.MEDIA_ATTACHED, function () {
    pleaseReload = false;
});
myhls.on(Hls.Events.ERROR, handleHlsError);
myhls.on(Hls.Events.MANIFEST_PARSED, function (event, data) {
    pressPlay();
});

[robwalch]

There are some 404s involved.
6 days ago we started seeing floods of requests for .m3u8 playlists and for expired .ts segments.

    function handleHlsError(event, data) {
        if (data.fatal) {
            //telemetry(data);

            switch (data.type) {
                case Hls.ErrorTypes.NETWORK_ERROR:
                    debug('fatal network error');
                    myhls.startLoad();
                    break;
                case Hls.ErrorTypes.MEDIA_ERROR:
                    debug('fatal media error');
                    myhls.recoverMediaError();
                    break;
                default:

The issue could be your error handling combined with the event listeners:

• recoverMediaError detaches and reattaches the media, which will add all of your event listeners again, so now when one of the events fires the callbacks are called multiple times
• startLoad can reload playlists and segments in previously loaded playlists

When an error is fatal, all internal attempts to recover have failed, and startLoad should not be used. If all you have is a single media playlist like the example above, what you want to do is configure fragment retries with reasonable timeouts and intervals based on the errors you are getting (see API.md). If a fatal error occurs, you can destroy the player and create a new instance with a startPosition matching video.currentTime or the end of the fragment that errored or last FRAG_CHANGED fragment.

[spinitron]

Wow. Thanks, @robwalch !

[robwalch]

Np. I'm sorry that this came up as a regression. The most likely scenario is that you've got some recursion happening where with each bad request, you trigger a new one:

  case Hls.ErrorTypes.NETWORK_ERROR:
      debug('fatal network error');
      myhls.startLoad();

In earlier versions, the stream controllers may have gotten into a state where startLoad would do nothing after a fatal error. Now they work and this code reveals a very bad side-effect. startLoad is not intended for recovery purposes. It is there to control buffering and halt and resume network activity in combination with stopLoad. Use the legacy *LoaderConfig retry settings or new LoadPolicy settings to make the player make only a certain number of appropriately timed reload attempts.

[spinitron]

To confirm your assessment I'll try removing that startLoad call in the NETWORK_ERROR case and making no other changes. Might take a day or more.

[robwalch]

Created #5478 to update the docs as this should only have been recommended in very specific scenarios with warning about loop loading.

v1.4.2 might behave better for you as some errors are now marked as resolved to prevent fatal errors for live streams. This may change a bit as it may have introduced cases where an error with no follow up is not-fatal which is less than ideal.

[spinitron]

Thanks (again) @robwalch for the really tremendous attention and support you've given.

I'm going to try to push to production a build of our public page bundle with a trial fix that comments out the myhls.startLoad(); line. And I can control the hls.js version very easily so I should be able to confirm soon and will report here.

[spinitron]

I deployed a player update with the myhls.startLoad(); removed and that was running for about 24 hours with no appreciable difference in the flooding on the server. But I don't know for sure what player build was actually running in the clients causing the floods. So today I deployed another update that adds player and hls.js versions to the UA. If over the next days we see no flood requests with the test-fix versions then we'll have probably confirmed the fix.

---

Btw: I though y'all might be curious to learn why we use hls.js in an audio streaming service. Without it some mobile devices stop playing HLS audio streams 5 minutes after the screen is locked. It's no small thing to be able to play audio continuously from a player in a web page running in the user's chosen or default browser on a phone when the browser is not the foreground app, the page isn't isn't the foreground tab, and the device screen is locked. On some devices/browsers that works with the browser's naked HTMLMediaElement but with hls.js we can make it work for nearly everyone. Thanks!

[spinitron]

v1.4.2 might behave better for you as some errors are now marked as resolved to prevent fatal errors for live streams.

Fyi, we are using VOD streams and (very rarely [if ever]) EVENT streams.

[robwalch]

Any luck with 1.4.3?

[spinitron]

I haven't tried 1.4.3 yet. In our ark-player I pinned hls.js at 1.4.1 while testing your suggested improvments. I wanted it to be robust whatever 1.4.1 was doing.

That stabilized over the last few days. It's slow because some people keep browser tabs open for a long time (surprisingly long) and we just have to wait until they unload.

Anyway, with the suggestions you made last week I see no weird behavior with 1.4.1.

In the next days I'll push the fixed ark-player to npm and then change our site back to loading hls.js@latest. I don't expect trouble with that but it's going to be a few more days.

[spinitron]

Sorry I forgot to come report back.

I configured my player to use 1.4.1, the version that caused the trouble, and corrected it using the advice you gave on May 9. After a few days I was satisfied things looked better. Then I made the player use the latest tagged release. No problems.

FWIW, maybe someone finds this useful, I made the player report its version and that of hls.js by configuring the Hls object like this. Idk if the fetchSetup works, dunno how to test it, but the XHR version does.

[robwalch]

Idk if the fetchSetup works, dunno how to test it, but the XHR version does.

Looks correct. The easiest way to have hls.js use the fetch loader by default is to initialize the player in progressive mode (such that enableStreamingMode is invoked) and then disable progressive loading:

const hls = new Hls({ progressive: true });
hls.config.progressive = hls.config.enableSoftwareAES = false;

The correct way would be to override the default loader (or fLoader or pLoader) with the FetchLoader, but the public library does not expose a reference to FetchLoader. If this is something you would like, please file an issue.