[prosieben] Endless redirect, failure to download on austrian IP

[SkoricIT]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2019.09.01
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

igor@igor-linux:~/Videos$ youtube-dl https://www.prosieben.at/tv/joko-gegen-klaas/videos/13-das-duell-um-die-welt-team-joko-gegen-team-klaas-folge-16-ganze-folge -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://www.prosieben.at/tv/joko-gegen-klaas/videos/13-das-duell-um-die-welt-team-joko-gegen-team-klaas-folge-16-ganze-folge', u'-v']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.09.01
[debug] Python version 2.7.16 (CPython) - Linux-5.0.0-25-generic-x86_64-with-Ubuntu-19.04-disco
[debug] exe versions: ffmpeg 4.1.3, ffprobe 4.1.3
[debug] Proxy map: {}
[debug] Using fake IP 53.107.178.173 (DE) as X-Forwarded-For.
[prosiebensat1] tv/joko-gegen-klaas/videos/13-das-duell-um-die-welt-team-joko-gegen-team-klaas-folge-16-ganze-folge: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 301: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
Moved Permanently (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2229, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 644, in http_error_302
    self.inf_msg + msg, headers, fp)

Description

The site https://www.prosieben.de redirects users from Austria to https://www.prosieben.at which might be the reason that this results in a endless loop.

Using an IP in Germany (ssh on server) shows no errors. I am using Ubuntu 19.04 with Xubuntu packages.

[mpra]

I think the problem is the X-Forwarded-For header. It is (always) set to a German IP (53.107.178.173 in the output above) by youtube-dl. But when you try to download with an Austrian IP you get in this redirect loop and it doesn't work. Not setting this header field at all solves the problem for me.

[SkoricIT]

@dstftw cant-reproduce -> Does this mean that I have to fix it myself and then submit a PR or that it is not going to be fixed at all, or... ?

[mpra]

I think this problem could be easily fixed by removing the _GEO_COUNTRIES restriction implemented in the prosiebensat1 extractor, e.g. by removing line 19 (https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/prosiebensat1.py#L19). But I don't know why this is there in the first place and if removing it would break downloads for other (officially unsupported) countries.

[SkoricIT]

How about adding 'AT' ?

[mpra]

I don't think this would work as expected. If there are multiple countries defined a random one gets selected. Therefore it would only work if you have "luck" (hey, fifty-fifty are good odds), or if you call the command multiple times until it works. Additionally this would maybe create the same problem for users with a German IP.

Off the cuff I can think of multiple ways of solving this problem:

1. (As mentioned) Remove the _GEO_COUNTRIES completely. This is a viable option if this doesn't break the download for some regions. Somebody from a non German speaking/unsupported country would have to try this.

2. Detect the country of the users IP and don't set the header if he/she is from a supported country. This would probably require an external service to determine the (external) IP of the user. Therefore I don't think this is something that's gonna happen.

3. Always try to download it without setting the header first and only set the header if it doesn't work otherwise.

Unfortunately I don't really know the code base, I just clicked through a bunch of files to find the root of this problem. Therefore I don't know if other plugins/extractors already had a similar problem and if there are already solutions available. Maybe the (original) developer of the prosiebensat1 plugin/extractor can weigh in? Additionally it would be interesting if this problem also exists for users with a Swiss IP.

[SkoricIT]

Thanks, @mpra. I am now confident that this is not actually a bug but a configuration issue. I will probably resort to manual tuning of geo-restriction by using --no-geo-bypass or similar flag. I have tested this and it is working as intended. I no longer think that this is a bug and therefore am closing.

There could be an argument for trying to download without geo-restrictions if an endless loop is detected.

[mpra]

I wouldn't say that. I think it's indeed a bug, because the geo bypass mechanism, which should allow more users to download stuff, breaks download for others. But using the "--no-geo-bypass" flag is a viable and easy workaround.

[SkoricIT]

Would you like me to leave the issue open?