Funimation: Unable to extract al:web:url

[maobowen]

Please follow the guide below

• You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your issue (like this: [x])
• Use the Preview tab to see what your issue will actually look like

---

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.01.14. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2018.01.14

Before submitting an issue make sure you have:

• At least skimmed through the README, most notably the FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

This might be a duplicate of #14569; however, there's no update. Not sure if Funimation's using Incapsula's robot detection that causes the issue (cuz I was writing a program and was unable to get HTML content because of being identified as a robot).

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-F', 'https://www.funimation.com/shows/date-a-live/april-1x/', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.01.14
[debug] Python version 3.6.4 (CPython) - Darwin-17.3.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.4.1, ffprobe 3.4.1
[debug] Proxy map: {}
[Funimation] april-1x: Downloading webpage
WARNING: [Funimation] april-1x: Failed to parse JSON Expecting value: line 1 column 1 (char 0)
ERROR: Unable to extract al:web:url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/Users/Steven/anaconda/envs/django/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 784, in extract_info
    ie_result = ie.extract(url)
  File "/Users/Steven/anaconda/envs/django/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 438, in extract
    ie_result = self._real_extract(url)
  File "/Users/Steven/anaconda/envs/django/lib/python3.6/site-packages/youtube_dl/extractor/funimation.py", line 97, in _real_extract
    ], webpage, fatal=True)
  File "/Users/Steven/anaconda/envs/django/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 924, in _html_search_meta
    html, display_name, fatal=fatal, group='content', **kwargs)
  File "/Users/Steven/anaconda/envs/django/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 803, in _html_search_regex
    res = self._search_regex(pattern, string, name, default, fatal, flags, group)
  File "/Users/Steven/anaconda/envs/django/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 794, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract al:web:url; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

[maobowen]

Hi @yan12125, I don't think this is a geo issue. I'm in US and I can watch that anime series without any problems using Chrome. I've tested on another Windows machine and the error is the same. If you have a look at what _download_webpage returns, it should show that the program is detected as a bot provided that you use a US IP.

[yan12125]

FYI: The label is for developers. It means a proxy or VPN is necessary for debugging.

[opensiriusfox]

It looks like robot detection is exactly the issue. When poking around I dig out the following:

Request unsuccessful. Incapsula incident ID:

It also looks like there are some people addressing parts of this issue in #13515.

[ElonSatoshi]

Will Youtube-dl have to display a CAPTCHA for Funimation now?

[remitamine]

Duplicate of #14569.