IPTV/HLS/M3U: Failing to parse EXT-X-VERSION 7

[angelaschmid]

Running latest master.

Streams from Zattoo, Teleboy, famous IPTV providers in Germany/Switzerland, are failing to play in MythTV. They play fine in VLC.
MythTV does not have version 7 M3U implemented.

The stream contains the following M3U data:

#EXTM3U
#EXT-X-VERSION:7
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio",NAME="deutsch",DEFAULT=YES,AUTOSELECT=YES,LANGUAGE="deu",URI="https://zba6-0-hls7-live.zahs.tv/telebielingue/t_track_audio_bw_128000000_num_0_tid_2_p_10_l_de_nd_1600_mbr_2999.m3u8?z32=MF2......"
#EXT-X-STREAM-INF:BANDWIDTH=2999000,CODECS="avc1.4d4020,mp4a.40.2",RESOLUTION=1024x576,FRAME-RATE=50,AUDIO="audio",CLOSED-CAPTIONS=NONE
https://zba6-0-hls7-live.zahs.tv/telebielingue/t_track_video_bw_2799000_num_0_tid_1_nd_1600_mbr_2999.m3u8?z32=MF2......

• MythTV version:
  Latest master

• Component:
  mythbackend

What do you see instead?

Logfile mythbackend:
StreamHandler m3u.cpp:106 (ParseVersion) - #EXT-X-VERSION is 7, but we only understand 0 through 3

Changing the source to also allow version 7, gives the following failures, and the stream does not play:

2024-08-05 10:41:19.523190 I [344463/352425] StreamHandler m3u.cpp:120 (ParseStreamInformation) - Parsing stream from https://zba6-1-hls7-live.zahs.tv/HD_s1/t_track_video_bw_7800000_num_0_tid_1_nd_1600_mbr_8000.m3u8?z32=MF2WI2L
2024-08-05 10:41:19.523196 I [344463/352425] StreamHandler m3u.cpp:132 (ParseStreamInformation) - #EXT-X-STREAM-INF: expected PROGRAM-ID=, using -1
2024-08-05 10:41:19.523200 I [344463/352425] StreamHandler m3u.cpp:157 (ParseStreamInformation) - bandwidth adaptation detected (program-id=-1, bandwidth=8000000
2024-08-05 10:41:20.590946 D [344463/352428] HLSPlaylist m3u.cpp:367 (ParseProgramDateTime) - https://zba6-1-hls7-live.zahs.tv/HD_s1/t_track_video_bw_7800000_num_0_tid_1_nd_1600_mbr_8000.m3u8?z32=MF2WI2L tag not supported: #EXT-X-PROGRAM-DATE-TIME #EXT-X-PROGRAM-DATE-TIME:2024-08-05T08:39:56.800+00:00
2024-08-05 10:41:21.603634 D [344463/352428] HLSPlaylist m3u.cpp:367 (ParseProgramDateTime) - https://zba6-1-hls7-live.zahs.tv/HD_s1/t_track_video_bw_7800000_num_0_tid_1_nd_1600_mbr_8000.m3u8?z32=MF2WI2L tag not supported: #EXT-X-PROGRAM-DATE-TIME #EXT-X-PROGRAM-DATE-TIME:2024-08-05T08:39:56.800+00:00
2024-08-05 10:41:22.615515 D [344463/352428] HLSPlaylist m3u.cpp:367 (ParseProgramDateTime) - https://zba6-1-hls7-live.zahs.tv/HD_s1/t_track_video_bw_7800000_num_0_tid_1_nd_1600_mbr_8000.m3u8?z32=MF2WI2L tag not supported: #EXT-X-PROGRAM-DATE-TIME #EXT-X-PROGRAM-DATE-TIME:2024-08-05T08:39:58.400+00:00
2024-08-05 10:44:24.025961 I [344463/358923] StreamHandler m3u.cpp:120 (ParseStreamInformation) - Parsing stream from https://zh4-11-hls7-live.zahs.tv/HD_s1/t_track_video_bw_7800000_num_0_tid_1_nd_1600_mbr_8000.m3u8?z32=MF2W
2024-08-05 10:44:24.025965 I [344463/358923] StreamHandler m3u.cpp:132 (ParseStreamInformation) - #EXT-X-STREAM-INF: expected PROGRAM-ID=, using -1
2024-08-05 10:44:24.025969 I [344463/358923] StreamHandler m3u.cpp:157 (ParseStreamInformation) - bandwidth adaptation detected (program-id=-1, bandwidth=8000000
2024-08-05 10:44:25.103642 D [344463/358926] HLSPlaylist m3u.cpp:367 (ParseProgramDateTime) - https://zh4-11-hls7-live.zahs.tv/HD_s1/t_track_video_bw_7800000_num_0_tid_1_nd_1600_mbr_8000.m3u8?z32=MF2Wtag not supported: #EXT-X-PROGRAM-DATE-TIME #EXT-X-PROGRAM-DATE-TIME:2024-08-05T08:43:00.800+00:00
2024-08-05 10:44:26.117132 D [344463/358926] HLSPlaylist m3u.cpp:367 (ParseProgramDateTime) - https://zh4-11-hls7-live.zahs.tv/HD_s1/t_track_video_bw_7800000_num_0_tid_1_nd_1600_mbr_8000.m3u8?z32=MF2Wtag not supported: #EXT-X-PROGRAM-DATE-TIME #EXT-X-PROGRAM-DATE-TIME:2024-08-05T08:43:02.400+00:00

[kmdewaal]

Can you please provide a complete version 7 m3u file?
Actually playing something from Zattoo might be difficult in The Netherlands because of geofencing but a fix for the parser can be tested.

[kmdewaal]

HLS version 7 is described here https://datatracker.ietf.org/doc/html/rfc8216 and it looks to be non-trivial undertaking...

[angelaschmid]

Hi Klaas

Please find attached mythtv_936_m3u.zip

• hls_init7_list.m3u, program list of my local internet provider, only available on their network
• hls_init7_ext6_stream.m3u, EXT6 stream
• hls_zattoo_list.m3u, program list of Zattoo Switzerland, GEO-blocked
• hls_zattoo_ext7_stream.m3u, EXT7 stream

Zattoo is offered only in Germany, Switzerland and Austria. Each having their own account and GEO-blocking.
To use it, I use "telerising", a daemon who acts in the middle to login to Zattoo (cookie handling) and produces a program.m3u and stream.m3u:

• Use a VPN to one of the countries
• Create a free account, allows 30 hours of streaming/month.
• download, https://github.com/sunsettrack4/telerising-api: telerising-v0.13.4_x86-64_linux.zip
• unpack
• exec ./api, shows the available web server ports
• browser, e.g. http://192.168.1.11:5000
• configure: choose admin password, choose e.g. Zattoo CH (Smart TV), press SELECT, add account details, press "Add TV Setup", use M3U playlist URL

https://www.kodinerds.net/thread/72127-telerising-api-zattoo-blue-tv-sky-ch-f%C3%BCr-tvheadend-und-vlc-web-app/
https://github.com/applesh00ter/telerising-api

Currently I am stuck on master of ~August, due to #955, but will be available to test.

Cheers, Angela

[kmdewaal]

Testing with a local IPTV stream for channel EWTN Europe did show up one bug.

The tag EXT-X-DISCONTINUITY-SEQUENCE is parsed and executed as EXT-X-DISCONTINUITY which is another tag, and this leads to a total reset every time the tag is encountered. This then causes downloading at maximum bandwidth and in my case this then downloads 60Mbyte per second. Playback however is OK.

I vaguely recall that this problem has been reported by somebody long ago.

Adding parsing of EXT-X-DISCONTINUITY-SEQUENCE does fix this even when not yet doing anything with the value.
I intend to add a fix for this in the near future.

To be continued.

[kmdewaal]

@angelaschmid I have committed a fix to master. With this fix many channels that previously could not be received are now playing good. I have not yet tested with your channels as I could produce already many problems with the readily available IPTV channels. It is possible that your issue is now resolved. Testing is appreciated.

[kmdewaal]

The above fix does errors in playing the RTVE channel https://rtvelivestream.akamaized.net/rtvesec/24h/24h_main_dvr_720.m3u8.
Reason for the overflow is that the segment list for this channel contains all segments for the next two hours, in segments of 5 seconds. The size of a segment times the number of segments causes the integer overflow. This makes sense; the total size of 2 hours recording can easily be far more than 2Gbyte, the limit of a 32-bit integer.

However, the remaining issue is that IPTV/HLS does read all the segments in the segment list. This means it does a readahead of the next two hours. This gives really big recording files even if you do only a few minutes of Live TV watching on that channel. I think that this is not the behavior that we want, especially not with Live TV but maybe also not when making recordings.

One possible approach is to limit the read-ahead time to about 1 minute or so (12 segments of 5 seconds each) but how to do that exactly needs to be investigated.

[angelaschmid]

A snapshot of https://rtvelivestream.akamaized.net/rtvesec/24h/24h_main_dvr_720.m3u8
produced the last line: https://rtvelivestream.akamaized.net/rtvesec/24h/GL0/2_v_5_2024_11_24_23_02_02_12256.ts
the scene showed a time of 23:02:02, so the last line is the livetv scene, the previous chunks allow to skip back.

vlc shows the total time as 2 hours, forwards to the end (starting with the last chunk), and allows the user to skip back.

HLSPlaylist HLSReader.cpp:457 (ParseM3U8), is executed every second, it should keep track what already was read. LiveTV would mean, only reading the last, but that might be fragile, so reading some chunks back would make sense. Somebody who want to have near realtime LiveTV, can skip forward. I think the same is expected with recordings, as it should start near time, forgetting too many available previous chunks.

Besides your fix, as the channel list contain URLs without an extension, I had to change:

diff --git a/mythtv/libs/libmythtv/iptvtuningdata.h b/mythtv/libs/libmythtv/iptvtuningdata.h
index a1cf9b951b..9a287703fc 100644
--- a/mythtv/libs/libmythtv/iptvtuningdata.h
+++ b/mythtv/libs/libmythtv/iptvtuningdata.h
@@ -241,7 +241,7 @@ class MTV_PUBLIC IPTVTuningData
         auto path = m_dataUrl.path();

         // check url is valid for a playlist before downloading (see trac ticket #12856)
-        if(path.endsWith(".m3u8", Qt::CaseInsensitive) ||
+        if(true || path.endsWith(".m3u8", Qt::CaseInsensitive) ||
            path.endsWith(".m3u", Qt::CaseInsensitive))
         {
             LOG(VB_RECORD, LOG_INFO, QString("IsHLSPlaylist url ends with either .m3u8 or .m3u %1").arg(url));

I have seen URLs like 'http://.....m3u8/?....." (from https://github.com/iptv-org/iptv?tab=readme-ov-file#playlists).

Within GuessProtocol(void)::IsHLSPlaylist(void), the if/else can/should be removed, as downloader.DownloadURL(url, &buffer, 5s, 0, 10000); reads the content, which is verified in HLSReader::IsValidPlaylist(text). It checks that the first line starts with: #EXTM3U. I doubt if it makes sense to read next lines to be checked for other tags, but will for sure be available on the second line. Maybe even reducing it to some 1000 characters, so that IPTVTuningData::http_ts channels are not badly influenced.

[kmdewaal]

About the IsHLSPlaylist function, I do agree with your analysis. The presence of the substring "m3u" either at the end or halfway the URL should not be used to decide on the content of the file.
I did check trac ticket 12856 mentioned in the comment and this has to do with channel up and down from a Kodi plugin with channels received from a VBox. As the check on the presence of m3u does not change the logic but only the timing it has likely been a race condition or so back then. To be ignored for now and if it is reported again it should be properly fixed and not by a time delay.
I also agree that reading 1000 bytes instead of 10000 should be enough and this then also reduces the amount of time spent in this function.
A fix is forthcoming. Thanks.

[kmdewaal]

Reducing the number of bytes read from 10000 to 1000 does cause failure on playing that RTVE channel so this will remain at 10000 for now. Size of the RTVE file is 7.7 kbyte so to get the last chunk we need to read the whole file.

[kmdewaal]

Status update:

EXT-X-VERSION:7 files can be decoded now. Most of the channels present in https://iptv-org.github.io/iptv/countries/nl.m3u can be played; the ones that cannot do not exist anymore, are geo-blocked or are encrypted. All the fixes to get this far have been fairly trivial.

The biggest remaining issue is that some channels do have video playback but no sound. This is because the EXT-X-MEDIA tag is not yet implemented.
Usually the HTTP stream does contain a transport stream with both audio and video. However, it can also happen that the HTTP stream does contain only video and that the tag EXT-X-MEDIA contains the URL for the audio.
This means that two streams must be downloaded in parallel and that they must be remuxed into a single stream. It is clear what needs to be done but it is not trivial.

@angelaschmid I expect that the streams that you mention in this issue can al be played by MythTV but al the streams that use the EXT-X-MEDIA tag to specify the audio URL will play without sound. Can you please check?

To be continued.