[vrtnu] Requesting video info has new end-point requiring a new vrtPlayerToken.

[Twilkie]

Please follow the guide below

• You will be asked some questions, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your pull request (like that [x])
• Use Preview tab to see how your pull request will actually look like

---

Before submitting a pull request make sure you have:

• At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
• Searched the bugtracker for similar pull requests
• Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

• I am the original author of this code and I am willing to release it under Unlicense
• I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

• Bug fix
• Improvement
• New extractor
• New feature

---

Description of your pull request and other information

Explanation of your pull request in arbitrary form goes here. Please make sure the description explains the purpose and effect of your pull request and is worded well enough to be understood. Provide as much context and examples as possible.

[Twilkie]

Fixes: #17680

[michaelarnauts]

This need more work. I get a 400 after it is fetching the vrtPlayerToken.

root@media:~/tmp# youtube-dl --verbose --write-pages "https://www.vrt.be/vrtnu/a-z/over-water/1/over-water-s1a1/"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'--write-pages', u'https://www.vrt.be/vrtnu/a-z/over-water/1/over-water-s1a1/']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.12.17
[debug] Python version 2.7.15rc1 (CPython) - Linux-4.15.0-42-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.4-0ubuntu0.18.04.1, ffprobe 3.4.4-0ubuntu0.18.04.1
[debug] Proxy map: {}
[VrtNU] over-water-s1a1: Downloading webpage
[VrtNU] Saving request to over-water-s1a1_https_-_www.vrt.be_vrtnu_a-z_over-water_1_over-water-s1a1_.dump
[VrtNU] over-water-s1a1: Downloading JSON metadata
[VrtNU] Saving request to over-water-s1a1_https_-_www.vrt.be_vrtnu_a-z_over-water_1_over-water-s1a1.mssecurevideo.json.dump
[Canvas] Downloading JSON metadata
[Canvas] Saving request to None_https_-_media-services-public.vrt.be_vualto-video-aggregator-web_rest_external_v1_tokens.dump
[Canvas] pbs-pub-76cdcf43-7667-46cd-86c7-ec6367450e10$vid-466347f4-9010-4aac-85f7-fc28a71ca023: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 400: Bad Request (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 605, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2212, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

[Twilkie]

You didn't provide your credentials? You can't download the vrtToken without credentials.

…

On Fri, 28 Dec 2018, 15:59 Michaël Arnauts ***@***.*** wrote: This need more work. I get a 400 when it is fetching the vrtPlayerToken. ***@***.***:~/tmp# youtube-dl --verbose --write-pages "https://www.vrt.be/vrtnu/a-z/over-water/1/over-water-s1a1/" [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'--verbose', u'--write-pages', u'https://www.vrt.be/vrtnu/a-z/over-water/1/over-water-s1a1/'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2018.12.17 [debug] Python version 2.7.15rc1 (CPython) - Linux-4.15.0-42-generic-x86_64-with-Ubuntu-18.04-bionic [debug] exe versions: ffmpeg 3.4.4-0ubuntu0.18.04.1, ffprobe 3.4.4-0ubuntu0.18.04.1 [debug] Proxy map: {} [VrtNU] over-water-s1a1: Downloading webpage [VrtNU] Saving request to over-water-s1a1_https_-_www.vrt.be_vrtnu_a-z_over-water_1_over-water-s1a1_.dump [VrtNU] over-water-s1a1: Downloading JSON metadata [VrtNU] Saving request to over-water-s1a1_https_-_www.vrt.be_vrtnu_a-z_over-water_1_over-water-s1a1.mssecurevideo.json.dump [Canvas] Downloading JSON metadata [Canvas] Saving request to None_https_-_media-services-public.vrt.be_vualto-video-aggregator-web_rest_external_v1_tokens.dump [Canvas] pbs-pub-76cdcf43-7667-46cd-86c7-ec6367450e10$vid-466347f4-9010-4aac-85f7-fc28a71ca023: Downloading JSON metadata ERROR: Unable to download JSON metadata: HTTP Error 400: Bad Request (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 605, in _request_webpage return self._downloader.urlopen(url_or_request) File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2212, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/usr/lib/python2.7/urllib2.py", line 435, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 548, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 473, in error return self._call_chain(*args) File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#18629 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ar-kYjw0c58mHnVapDYtiIdapV7ubhIXks5u9jHsgaJpZM4Zgs2Z> .

[Twilkie]

@michaelarnauts
I tried it myself, i can reproduce your problem if I remove my username and password flags from my testing command.

It seems VRT is making logging in required to fetch the meta data.

I know raise an error if user didn't provide user credentials

[xplorr]

Thanks for your efforts, I tested, but works only for some vrtnu url's.

[kevmns]

How can I implement this correction ?

[Twilkie]

@xplorr Thanks for the feedback. I'll check into those links that are not working.

[Twilkie]

@xplorr Tested it. Cirque du Soleil is no longer available.
My brilliant friend seems to fail because the playlist file seems to indicate Apple Fairplay. My guess is that the episode is protected by DRM.
So this seems to be an other issue then the one I fixed.

[xplorr]

Thx @Twilkie!

[kevmns]

@xplorr

Can you check why following video is not working ? https://www.vrt.be/vrtnu/a-z/thuis/24/thuis-s24a4535/

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.vrt.be/vrtnu/a-z/thuis/24/thuis-s24a4534/', '--username', 'PRIVATE', '--password', 'PRIVATE', '-f', 'best']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.12.17
[debug] Python version 3.7.2 (CPython) - Darwin-16.7.0-x86_64-i386-64bit
[debug] exe versions: avconv 12.3, avprobe 12.3, ffmpeg N-89803-g856b7cae9f-tessus
[debug] Proxy map: {}
[VrtNU] Logging in
[VrtNU] Requesting a token
[VrtNU] thuis-s24a4534: Downloading webpage
[VrtNU] thuis-s24a4534: Downloading JSON metadata
[Canvas] pbs-pub-c425bc72-e4c8-4ceb-973e-d78de7fb4cc4$vid-5d01e94a-0d77-4631-b0ff-302c938f778a: Downloading JSON metadata
[Canvas] pbs-pub-c425bc72-e4c8-4ceb-973e-d78de7fb4cc4$vid-5d01e94a-0d77-4631-b0ff-302c938f778a: Downloading m3u8 information
WARNING: Failed to download m3u8 information: <urlopen error [Errno 8] nodename nor servname provided, or not known>
[Canvas] pbs-pub-c425bc72-e4c8-4ceb-973e-d78de7fb4cc4$vid-5d01e94a-0d77-4631-b0ff-302c938f778a: Downloading MPD manifest
WARNING: Failed to download MPD manifest: <urlopen error [Errno 8] nodename nor servname provided, or not known>
[Canvas] pbs-pub-c425bc72-e4c8-4ceb-973e-d78de7fb4cc4$vid-5d01e94a-0d77-4631-b0ff-302c938f778a: Downloading ISM manifest
WARNING: Failed to download ISM manifest: <urlopen error [Errno 8] nodename nor servname provided, or not known>
[Canvas] pbs-pub-c425bc72-e4c8-4ceb-973e-d78de7fb4cc4$vid-5d01e94a-0d77-4631-b0ff-302c938f778a: Downloading f4m manifest
WARNING: Unable to download f4m manifest: <urlopen error [Errno 8] nodename nor servname provided, or not known>
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 793, in extract_info
ie_result = ie.extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 508, in extract
ie_result = self._real_extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/canvas.py", line 73, in _real_extract
self._sort_formats(formats)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 1292, in _sort_formats
raise ExtractorError('No video formats found')
youtube_dl.utils.ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

[xplorr]

@kevmns working ok

[kevmns]

@kevmns working ok

Strange... Any idea why I'm getting this message ?

[xplorr]

Did you compile and test with a version of youtube-dl with the changes of Twilkie included?

[kevmns]

I've installed y-dl via Brew; how should I include those changes ?

[Twilkie]

@kevmns You can't, as far as I know. Youtube-DL in brew is a single compiled file.
You can download the source from this GitHub repo and apply the patch yourself to canvas.py
Or you can download my repo as a zip file
The zip file comes with a executable.

[kevmns]

It works, many thanks @Twilkie !

[brechtm]

Doesn't seem to work for https://www.vrt.be/vrtnu/a-z/louis-theroux-s-altered-states--choosing-death/2018/louis-theroux-s-altered-states--choosing-death-s2018/
It only downloads an .mpd file.

[Twilkie]

@brechtm Tried the video, it's protected by DRM. I was able to get the best-quality playlist but after downloading the first part it just gibberish/garbage when played.

Does anyone know the strategy of youtube-dl concerning pull-request. My fix works for all non-DRM videos. Yet, after three weeks it's still not merged. This way you scare away potential developers. Was thinking in looking into the vier extractor, but if even this simple fix doesn't get merged.. why bother.

[xplorr]

I know it can take a long time before the changes are accepted. That's why I compile my own custom version aimed at Dutch/Belgian sites and include it in my TVDownloader front-end (which you can find elsewhere). I also make my own changes. So please do continue with Vier extractor. I appreciate it a lot and will include it in my custom version as soon as it is available.

[ntphx]

This fixed my issue, thank you!

[madebr]

Verified working on a recent episode.

[dstftw]

1. Breaks current canvas extractor test.
2. Read coding conventions.