Twitcasting not working

[Yarn]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2020.03.24
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

> youtube-dl -v 'https://twitcasting.tv/ivetesangalo/movie/2357609'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://twitcasting.tv/ivetesangalo/movie/2357609']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.03.24
[debug] Python version 3.7.7 (CPython) - Darwin-19.3.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2, rtmpdump 2.4
[debug] Proxy map: {}
[TwitCasting] 2357609: Downloading webpage
[TwitCasting] 2357609: Downloading m3u8 information
ERROR: Failed to download m3u8 information: <urlopen error no host given> (caused by URLError('no host given'))
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2238, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 523, in open
    req = meth(req)
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1242, in do_request_
    raise URLError('no host given')

Description

TLDR: it looks like most of the below doesn't actually need to be addressed since live streams are handled differently and there is an alternative for VODs. I'm planning to make a PR once I get stuff working. I could use guidance on how to handle data coming from a websocket within youtube-dl.
Live streams use websockets so as for as I can tell they can't be supported in ytdl, not sure how to support vods anymore without fixing the issue below.

I've fixed the extractor to get the m3u8 file (url changes, can be found in chrome inspector) but I'm still getting broken videos. A video file is created but it does not play back correctly. Manually downloading and concatenating the segments or passing the m3u8 url directly to ffmpeg produces the same result. I've attached one of the resulting files.
out.mp4.zip

For me in mpv the file plays a frame every 6 or 7 seconds. The frames look like they are probably the correct ones for a given time. Audio seems to work (it cuts out in the original video viewed on twitcasting).
During playback mpv gives warnings like Reset playback due to audio timestamp reset. and Invalid audio PTS: 0.325500 -> 7.402667.

ffmpeg -i out.mp4 -vn -acodec copy out.aac produces a mostly playable audio file. mpv gives a warning [ffmpeg/demuxer] aac: Estimating duration from bitrate, this may be inaccurate when played back.

Twitcasting provides mp4 downloads and I modified the extractor to download those but that doesn't work for live broadcasts. Just checked and it looks like twitcasting uses websockets to transfer live video so I guess it will be a different extractor anyway. I'll see if I can get this working but I have no idea how to integrate that into youtube-dl.

ffmpeg logs

> ffmpeg -i 'https://dl01.twitcasting.tv/tc.vod/v/2357609-1587258621-1587287421-d9c7710d-6ea92f959c9728bb/fmp4/index.m3u8' -codec copy out_two.mp4
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple clang version 11.0.0 (clang-1100.0.33.17)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.2.2_2 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[hls @ 0x7fbaaa001600] Skip ('#EXT-X-ALLOW-CACHE:YES')
[hls @ 0x7fbaaa001600] Skip ('#EXT-X-VERSION:6')
[hls @ 0x7fbaaa001600] Opening 'https://dl01.twitcasting.tv/tc.vod/v/2357609-1587258621-1587287421-d9c7710d-6ea92f959c9728bb/fmp4/init-v1-a1.mp4' for reading
[hls @ 0x7fbaaa001600] Opening 'https://dl01.twitcasting.tv/tc.vod/v/2357609-1587258621-1587287421-d9c7710d-6ea92f959c9728bb/fmp4/seg-1-v1-a1.m4s' for reading
[hls @ 0x7fbaaa001600] Opening 'https://dl01.twitcasting.tv/tc.vod/v/2357609-1587258621-1587287421-d9c7710d-6ea92f959c9728bb/fmp4/seg-2-v1-a1.m4s' for reading
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fbaa9806c00] DTS 0 < 18000 out of order
[hls @ 0x7fbaaa001600] decoding for stream 0 failed
Input #0, hls, from 'https://dl01.twitcasting.tv/tc.vod/v/2357609-1587258621-1587287421-d9c7710d-6ea92f959c9728bb/fmp4/index.m3u8':
  Duration: 00:00:31.00, start: 0.000000, bitrate: 0 kb/s
  Program 0 
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 320x180 [SAR 1:1 DAR 16:9], 30 kb/s, 10 tbr, 90k tbn, 20 tbc
    Metadata:
      variant_bitrate : 0
      major_brand     : isom
      minor_version   : 1
      compatible_brands: isomavc1
    Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, mono, fltp, 11 kb/s
    Metadata:
      variant_bitrate : 0
Output #0, mp4, to 'out_two.mp4':
  Metadata:
    encoder         : Lavf58.29.100
    Stream #0:0: Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 320x180 [SAR 1:1 DAR 16:9], q=2-31, 30 kb/s, 10 tbr, 90k tbn, 90k tbc
    Metadata:
      variant_bitrate : 0
      major_brand     : isom
      minor_version   : 1
      compatible_brands: isomavc1
    Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, mono, fltp, 11 kb/s
    Metadata:
      variant_bitrate : 0
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[mp4 @ 0x7fbaa9007200] Non-monotonous DTS in output stream 0:1; previous: 11264, current: 0; changing to 11265. This may result in incorrect timestamps in the output file.
... ((a lot of lines similar to above and a few about opening a new segment file))
frame=  310 fps= 51 q=-1.0 Lsize=    1015kB time=00:00:30.59 bitrate= 271.7kbits/s speed=5.02x    
video:764kB audio:241kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.914374%

Related Issues:
#21225 similar ffmeg output but I do get a playable, if incorrect, file.

[Yarn]

It looks like the way I was getting the video is broken now (along with the code to get the m3u8 url). Using the m3u8 url in ffmpeg still produces a broken video file.

[Yarn]

It looks like VLC can play the video and avconv produces a normal file so it looks like an ffmpeg issue of some sort. I'll keep working on the extractor. Is there an established convention for what to do if the output isn't expected to work with mpv/ffmpeg? It seems like there should at least be some sort of warning.

A side note this looks like potentially the same underlying issue as #24933 (https://trac.ffmpeg.org/ticket/6191) but I'm not sure.

[igorcmoura]

About the URLError: it looks like the slashes in the m3u8 URL are escaped with backslashes and urllib is parsing the entire url (without the scheme) as the path, with netloc (host) being returned empty. Removing the backslashes seems to fix this issue but, is it common to have escaped slashes in URLs? Should this be normalized in the sanitize_url function?

e.g.: https:\/\/dl01.twitcasting.tv\/tc.vod\/v\/2357609-1592727937-1592756737-105ac9c0-6d794cc72e7398a9\/fmp4\/index.m3u8

Still, the HLS native downloader doesn't seem to handle the file correctly as it ends up corrupted. I didn't look too much into it, but using ffmpeg (with the --hls-prefer-ffmpeg option) works fine.

[rivke41levp656]

Did you ever get this working? I tried your pull request but just get URLErrors.

[Yarn]

is it common to have escaped slashes in URLs
Extremely slow response but no.

Did you ever get this working?
Yes but the pushed version may not be up to date (or a twitcasting change broke it). I'll check and fix/push it.

It also looks like ffmpeg is able to play the files it couldn't before so that issue may be fixed.

[Yarn]

@rivke41levp656 I pushed a change so it can download a video now. That said I tried downloading a video and it was unplayable in mpv and vlc so it looks like there is some new issue.

[Yarn]

It appears to work with --hls-prefer-ffmpeg. It also works with my bare bones downloader script so there is nothing super special going on with it. I suspect ytdl isn't picking up #EXT-X-MAP:URI="init-v1-a1.mp4" (which needs to be prepended to the rest of the segments). Searching the code "EXT-X-MAP" doesn't show up.

[rivke41levp656]

@Yarn I tried your new commit but get the same results you describe in OP.
hlsnative ==> unplayable output file
ffmpeg ==> Non-monotonous DTS in output stream 0:1; previous: 11264, current: 0; changing to 11265. This may result in incorrect timestamps in the output file. Tons of these resulting in a playable but useless file.

[Yarn]

@rivke41levp656 What version of ffmpeg are you using? Try a new version from ffmpeg's site (https://ffmpeg.org/download.html) if you don't have a new version available through your package manager.

--ffmpeg-location PATH           Location of the ffmpeg/avconv binary;
                                 either the path to the binary or its
                                 containing directory.

I'm on 4.3.1 (the latest stable release).

[rivke41levp656]

@Yarn Indeed my ffmpeg was quite old, didn't consider that. It seems to work now, thanks!