Long 608 Captions are are not displayed properly, and sometimes not at all.

[jcekstrom]

What version of Hls.js are you using?

1.4.12

What browser (including version) are you using?

Firefox 119.0, Edge 118.0.2088.69, Chrome 119.6045.105

What OS (including version) are you using?

Win11

Test stream

https://storage.ekstech.net/video/long_caption_test/master.m3u8

Configuration

{
  "debug": true,
  "enableWorker": true,
  "lowLatencyMode": true,
  "backBufferLength": 90
}

Additional player setup steps

https://hlsjs.video-dev.org/demo defaults

Checklist

• The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
• The issue occurs in the stable client (latest release) on https://hlsjs.video-dev.org/demo and not just on my page
• The issue occurs in the latest client (main branch) on https://hlsjs-dev.video-dev.org/demo and not just on my page
• The stream has correct Access-Control-Allow-Origin headers (CORS)
• There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

1. Open https://hlsjs.video-dev.org/demo
2. Paste https://storage.ekstech.net/video/long_caption_test/master.m3u8 into the stream bar.
3. Click Apply
4. Pause and turn on English Captions

Expected behaviour

1
00:00:00,534 --> 00:00:02,001
A long 3 line caption is coming...

00:00:02,334 --> 00:00:04,501

• There are three lines to
• this caption. Can you see
• ALL THREE LINE?!?!?!

2
00:00:04,803 --> 00:00:06,002
Did you see the 3 lines?

What actually happened?

It displayed caption #1 and #3, but the long caption, #2 is lost.

This is due to the fact that cea-608 is embedded into 2 separate cea-708 packets as which are then embedded as user_data_registered_itu_t_t35 into the sei_rbsp() as sei_message() according to the ITU-T H.264 spec.

There are a few small problems with the implementation found in src/utils/mp4-tools.ts:974 parseSEIMesageFromNALu(...):
1- All sei_message payload_type are restricted to 8 bits/1 byte, yet the parsing logic ignores this, allowing for payloadType to have values larger than 255.
2- The parsing logic for the payloadSize will not always parse all the data associated with a single sei_message, which will cause the parsing to be off, and the corruption of parsing of any sei_message following the first.

Console output

Using Hls.js config: Object
logger.ts:74 [log] > Debug logs enabled for "Hls instance" in hls.js version 1.4.12
hls.ts:410 [log] > stopLoad
hls.ts:379 [log] > loadSource:https://storage.ekstech.net/video/long_caption_test/master.m3u8
stream-controller.ts:569 [log] > [stream-controller]: Trigger BUFFER_RESET
hls.ts:351 [log] > attachMedia
level-controller.ts:269 [log] > [level-controller]: manifest loaded, 1 level(s) found, first bitrate: 545965
buffer-controller.ts:148 [log] > 1 bufferCodec event(s) expected
hls.ts:400 [log] > startLoad(-1)
level-controller.ts:351 [log] > [level-controller]: Switching to level 0 from level -1
level-controller.ts:520 [log] > [level-controller]: Loading level index 0 with URI 1/1 https://storage.ekstech.net/video/long_caption_test/profile.m3u8
base-stream-controller.ts:1777 [log] > [stream-controller]: STOPPED->IDLE
base-stream-controller.ts:1777 [log] > [subtitle-stream-controller]: STOPPED->IDLE
stream-controller.ts:634 [log] > [stream-controller]: Level 0 loaded [0,0], cc [0, 0] duration:6.006
buffer-controller.ts:800 [log] > [buffer-controller]: Media source opened
buffer-controller.ts:692 [log] > [buffer-controller]: Updating Media Source duration to 6.006
base-stream-controller.ts:743 [log] > [stream-controller]: Loading fragment 0 cc: 0 of [0-0] level: 0, target: 0
base-stream-controller.ts:1777 [log] > [stream-controller]: IDLE->FRAG_LOADING
transmuxer-interface.ts:85 [log] > injecting Web Worker for "main"
transmuxer-interface.ts:227 [log] > [transmuxer-interface, main]: Starting new transmux session for sn: 0 p: -1 level: 0 id: 1
        discontinuity: true
        trackSwitch: true
        contiguous: false
        accurateTimeOffset: true
        timeOffset: 0
        initSegmentChange: true
base-stream-controller.ts:386 [log] > [stream-controller]: Loaded fragment 0 of level 0
timeline-chart.ts:754 Canvas2D: Multiple readback operations using getImageData are faster with the willReadFrequently attribute set to true. See: https://html.spec.whatwg.org/multipage/canvas.html#concept-canvas-will-read-frequently
drawLineX @ timeline-chart.ts:754
/favicon.ico:1 
        
        
       Failed to load resource: the server responded with a status of 404 ()
6408d0c0-7602-49c8-bfc2-f77096b95cfb:542 [log] > Debug logs enabled for "main" in hls.js version 1.4.12
transmuxer-interface.ts:379 [log] > [mp4-remuxer]: ISGenerated flag reset
transmuxer-interface.ts:379 [log] > [mp4-remuxer]: initPTS & initDTS reset
transmuxer-interface.ts:379 [log] > [mp4-remuxer]: reset next timestamp
transmuxer-interface.ts:379 [log] > manifest codec:mp4a.40.2, ADTS type:2, samplingIndex:3
transmuxer-interface.ts:379 [log] > parsed codec:mp4a.40.5, rate:48000, channels:2
base-stream-controller.ts:1777 [log] > [stream-controller]: FRAG_LOADING->PARSING
stream-controller.ts:1275 [log] > [stream-controller]: Init audio buffer, container:audio/mp4, codecs[selected/level/parsed]=[mp4a.40.2/mp4a.40.2/mp4a.40.5]
stream-controller.ts:1286 [log] > [stream-controller]: Init video buffer, container:video/mp4, codecs[level/parsed]=[avc1.64001e/avc1.64001e]
buffer-controller.ts:765 [log] > [buffer-controller]: creating sourceBuffer(audio/mp4;codecs=mp4a.40.2)
buffer-controller.ts:765 [log] > [buffer-controller]: creating sourceBuffer(video/mp4;codecs=avc1.64001e)
audio-stream-controller.ts:128 [log] > [audio-stream-controller]: InitPTS for cc: 0 found from main: 0
transmuxer-interface.ts:379 [log] > [transmuxer.ts]: Flushed fragment 0 of level 0
base-stream-controller.ts:1777 [log] > [stream-controller]: PARSING->PARSED
base-stream-controller.ts:572 [log] > [stream-controller]: Buffered main sn: 0 of level 0 (frag:[0.000-6.059] > buffer:[0.000-6.040])
base-stream-controller.ts:1777 [log] > [stream-controller]: PARSED->IDLE
buffer-controller.ts:544 [log] > [buffer-controller]: audio sourceBuffer now EOS
buffer-controller.ts:544 [log] > [buffer-controller]: video sourceBuffer now EOS
buffer-controller.ts:551 [log] > [buffer-controller]: Queueing mediaSource.endOfStream()
base-stream-controller.ts:1777 [log] > [stream-controller]: IDLE->ENDED
buffer-controller.ts:568 [log] > [buffer-controller]: Calling mediaSource.endOfStream()
buffer-controller.ts:819 [log] > [buffer-controller]: Media source ended

Chrome media internals output

No response

[jcekstrom]

long_caption_test.zip

Here are the relevant files for the test, should the link to the test go missing in the future.

[robwalch]

Hi Joey,

Thanks for the detailed bug report and sample.

This is due to the fact that cea-608 is embedded into 2 separate cea-708 packets as which are then embedded as user_data_registered_itu_t_t35 into the sei_rbsp() as sei_message() according to the ITU-T H.264 spec.

There are a few small problems with the implementation found in src/utils/mp4-tools.ts:974 parseSEIMesageFromNALu(...):
1- All sei_message payload_type are restricted to 8 bits/1 byte, yet the parsing logic ignores this, allowing for payloadType to have values larger than 255.
2- The parsing logic for the payloadSize will not always parse all the data associated with a single sei_message, which will cause the parsing to be off, and the corruption of parsing of any sei_message following the first.

As you seem to have a good handle on the issue and what is unique about these captions would you consider providing a fix in the form of a PR?

[robwalch]

Closing as fixed in v1.5.0-beta.1