Whether https proxy `--proxy=https://user:[email protected]` is not supported? [it isn't] #32549

wisetc · 2023-09-08T15:29:28Z

Checklist

I'm asking a question
I've looked through the README and FAQ for similar questions
I've searched the bugtracker for similar questions including closed ones

Question

I've checked that https proxy with username authenticated can not work well, for example https://user:[email protected] will output 'Connection reset by peer'.

for security, proxy address can not be provided here.

I'm using macOS to run this program.

Do I wrongly setup somewhere?

The text was updated successfully, but these errors were encountered:

dirkf · 2023-09-08T16:30:09Z

From the doc of the Python API that yt-dl uses:

Proxies which require authentication for use are not currently supported; this is considered an implementation limitation.

Maybe this works in yt-dlp, where the Python3 API does allow for authenticated proxies. It may depend on what happens with/without the recently reworked network stack.

wisetc · 2023-09-09T05:30:51Z

@dirkf Thank you, I will give it a try later.

wisetc · 2023-09-10T07:53:16Z

@Vangelis66 I'm calling youtube-dl not yt-dl. Do I use wrong command? I will try to install yt-dl to test. thank you.

dirkf · 2023-09-10T10:00:49Z

"yt-dl" is just an abbreviation for "youtube-dl".

Vangelis66 · 2023-09-10T14:40:39Z

Yes, yt-dl is short (or, in many of my posted logs, an "alias" for) youtube-dl 😄 ...
Also, in your first post (OP), you should really have included the FULL yt-dl command you originally used and the complete (with "sensitive" details redacted) debug log it generated 😉 ; BTW, posting image attachments is generally to be avoided, when posting plain text (encapsulated in code ``` tags) is possible...

As for my recommendation here, care should be taken to properly escape "special" characters for your shell, in case such characters are part of your "userName" and/or "yourPassword" strings; of course, this is OT for this issue tracker 😜 ...

coletdjnz · 2023-09-11T19:14:20Z

https proxies are not supported by either youtube-dl or yt-dlp (yet). Due to a bug in Python, they are treated as http in youtube-dl.

related: yt-dlp/yt-dlp#1890 #26709

dirkf · 2023-09-12T10:50:31Z

yt-dlp/yt-dlp#1890 is very useful, these especially.

From yt-dlp/yt-dlp#1890 (comment):

Can confirm, HTTP proxies work but HTTPS ones are broken, whether via --proxy or environment variable.

From yt-dlp/yt-dlp#1890 (comment):

One quick workaround would be to use a network repeater like stone (https://www.gcd.org/sengoku/stone/) that is capable of translation such as:

[python]<--(plain-text proxy)-->[repeater]<--(TLS proxy)-->[secure web (HTTPS) proxy server]

Run stone as

$ stone uk2344.nordvpn.com:89/ssl 8080

and then invoke yt-dlp in another terminal as

$ yt-dlp --proxy "http://xxxxxxxxx%40xxxxxxx.com:xxxxxxxxxxxxxxxx@localhost:8080" "https://www.youtube.com/watch?v=Fh7UgrHMjbQ"
[confirmed working with sufficiently recent OpenSSL].

According to yt-dlp/yt-dlp#1890 (comment), yt-dlp/yt-dlp#3668 adds support in yt-dlp.

But there's not likely to be a back-port of that because of dependencies that would mean you could just run yt-dlp anyway.

Vangelis66 · 2023-09-12T21:40:57Z

So, according to this testament:

but HTTPS ones are broken, whether via --proxy or environment variable.

my previous recommendation above shouldn't/wouldn't work after all 😞 ...

It's a pity, though, we haven't heard anything additional from OP 😉 ...

Vangelis66 · 2023-09-18T01:58:28Z

It's a pity, though, we haven't heard anything additional from OP 😉 ...

... At long last, yesterday I got my hands on a US HTTPS proxy server with authentication, so I can now conduct some tests myself on the yt-dl issue at hand...

Youtube title/id 7LBYrDx_784 is BLOCKED in my location (purportedly due to "rights" reasons), but is available in the US...

Trying a DIRECT fetch command yields the uninformative message This video is not available:

No Proxy

yt-dl -vF "7LBYrDx_784" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-vF', '7LBYrDx_784']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.09.03.2215
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k  26 Jan 2017
[debug] exe versions: ffmpeg n6.1-dev-1945-N-111829-g3c9dc0, ffprobe n6.1-dev-1945-N-111829-g3c9dc0, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] 7LBYrDx_784: Downloading webpage
ERROR: Video unavailable
This video is not available
Traceback (most recent call last):
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 859, in wrapper
    return func(self, *args, **kwargs)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 955, in __extract_info
    ie_result = ie.extract(url)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 565, in extract
    ie_result = self._real_extract(url)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\youtube.py", line 2120, in _real_extract
    raise ExtractorError(reason, expected=True)
youtube_dl.utils.ExtractorError: Video unavailable
This video is not available

Trying the HTTPS proxy via the --proxy cmdline option (use case reported by OP):

--proxy cmdline option

yt-dl --proxy "https://ProxyUserName:ProxyPassword@77.*.*.*:443" -vF "7LBYrDx_784" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '--proxy', 'https://ProxyUserName:ProxyPassword@77.*.*.*:443', '-vF', '7LBYrDx_784']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.09.03.2215
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k  26 Jan 2017
[debug] exe versions: ffmpeg n6.1-dev-1945-N-111829-g3c9dc0, ffprobe n6.1-dev-1945-N-111829-g3c9dc0, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {'https': 'https://ProxyUserName:ProxyPassword@77.*.*.*:443', 'http': 'https://ProxyUserName:ProxyPassword@77.*.*.*:443'}
[youtube] 7LBYrDx_784: Downloading webpage
WARNING: Unable to download webpage: ''
[youtube] 7LBYrDx_784: Downloading API JSON
ERROR: Unable to download API page: '' (caused by BadStatusLine("''",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 665, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 2461, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\Python3410-32\lib\urllib\request.py", line 464, in open
    response = self._open(req, data)
  File "C:\Python3410-32\lib\urllib\request.py", line 482, in _open
    '_open', req)
  File "C:\Python3410-32\lib\urllib\request.py", line 442, in _call_chain
    result = func(*args)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\utils.py", line 2889, in https_open
    req, **kwargs)
  File "C:\Python3410-32\lib\urllib\request.py", line 1183, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "C:\Python3410-32\lib\http\client.py", line 1137, in request
    self._send_request(method, url, body, headers)
  File "C:\Python3410-32\lib\http\client.py", line 1182, in _send_request
    self.endheaders(body)
  File "C:\Python3410-32\lib\http\client.py", line 1133, in endheaders
    self._send_output(message_body)
  File "C:\Python3410-32\lib\http\client.py", line 963, in _send_output
    self.send(msg)
  File "C:\Python3410-32\lib\http\client.py", line 898, in send
    self.connect()
  File "C:\Python3410-32\lib\http\client.py", line 1279, in connect
    super().connect()
  File "C:\Python3410-32\lib\http\client.py", line 874, in connect
    self._tunnel()
  File "C:\Python3410-32\lib\http\client.py", line 852, in _tunnel
    (version, code, message) = response._read_status()
  File "C:\Python3410-32\lib\http\client.py", line 356, in _read_status
    raise BadStatusLine(line)

the ERROR produced is:

ERROR: Unable to download API page: '' (caused by BadStatusLine("''",))

IOW, the connection to the HTTPS proxy cannot be established 😭 ...

Sadly, I can also confirm that specifying the HTTPS proxy via the HTTPS_PROXY env var doesn't work 😞 either (as reported in the "downstream" issue):

HTTPS_PROXY env var

set HTTPS_PROXY=https://ProxyUserName:ProxyPassword@77.*.*.*:443 && yt-dl -vF "7LBYrDx_784" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-vF', '7LBYrDx_784']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.09.03.2215
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k  26 Jan 2017
[debug] exe versions: ffmpeg n6.1-dev-1945-N-111829-g3c9dc0, ffprobe n6.1-dev-1945-N-111829-g3c9dc0, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {'https': 'https://ProxyUserName:ProxyPassword@77.*.*.*:443'}
[youtube] 7LBYrDx_784: Downloading webpage
WARNING: Unable to download webpage: ''
[youtube] 7LBYrDx_784: Downloading API JSON
ERROR: Unable to download API page: '' (caused by BadStatusLine("''",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 665, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 2461, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\Python3410-32\lib\urllib\request.py", line 464, in open
    response = self._open(req, data)
  File "C:\Python3410-32\lib\urllib\request.py", line 482, in _open
    '_open', req)
  File "C:\Python3410-32\lib\urllib\request.py", line 442, in _call_chain
    result = func(*args)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\utils.py", line 2889, in https_open
    req, **kwargs)
  File "C:\Python3410-32\lib\urllib\request.py", line 1183, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "C:\Python3410-32\lib\http\client.py", line 1137, in request
    self._send_request(method, url, body, headers)
  File "C:\Python3410-32\lib\http\client.py", line 1182, in _send_request
    self.endheaders(body)
  File "C:\Python3410-32\lib\http\client.py", line 1133, in endheaders
    self._send_output(message_body)
  File "C:\Python3410-32\lib\http\client.py", line 963, in _send_output
    self.send(msg)
  File "C:\Python3410-32\lib\http\client.py", line 898, in send
    self.connect()
  File "C:\Python3410-32\lib\http\client.py", line 1279, in connect
    super().connect()
  File "C:\Python3410-32\lib\http\client.py", line 874, in connect
    self._tunnel()
  File "C:\Python3410-32\lib\http\client.py", line 852, in _tunnel
    (version, code, message) = response._read_status()
  File "C:\Python3410-32\lib\http\client.py", line 356, in _read_status
    raise BadStatusLine(line)

Thus, I've hidden my previous suggestion above ...

One quick workaround would be to use a network repeater

One such application I'm familiar with from past usage is GOST (GoSecureTunnel):

https://github.com/ginuerzh/gost

The app (written in golang) has seen more recent development compared to stone referenced in the "downstream" issue tracker 😉 ; gost can be configured to launch a local HTTP proxy (that yt-dl has support for and can be pointed to) and, at the same time, forward the requests made through that proxy to a remote HTTPS proxy; gost has no problem authenticating to that remote HTTPS proxy, if basic authentication has been enforced on the proxy; in a nutshell, the command to use would look like below:

gost -L="http://localhost:8080" -F="https://ProxyUserName:ProxyPassword@ProxyHostname(|ProxyIP):ProxyPort"

Indeed, in my testing I proved it works as advertised 👍 :

Using gost as MITM

(first cmdline window, shall remain open for the duration)
gost -L="http://localhost:8080" -F="https://ProxyUserName:ProxyPassword@77.*.*.*:443"

(second cmdline window)
yt-dl --proxy "http://localhost:8080" -vF "7LBYrDx_784" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '--proxy', 'http://localhost:8080', '-vF', '7LBYrDx_784']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.09.03.2215
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k  26 Jan 2017
[debug] exe versions: ffmpeg n6.1-dev-1945-N-111829-g3c9dc0, ffprobe n6.1-dev-1945-N-111829-g3c9dc0, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {'https': 'http://localhost:8080', 'http': 'http://localhost:8080'}
[youtube] 7LBYrDx_784: Downloading webpage
[debug] [youtube] Decrypted nsig -Sq7NdL_EDAFmZ_X => vQkYN_oLLglXcA
[debug] [youtube] Decrypted nsig pTv43VKeQKiF6Nma => uz32DjmqPXMKhA
[info] Available formats for 7LBYrDx_784:
format code  extension  resolution note
249          webm       audio only tiny   57k , webm_dash container, opus  (48000Hz), 897.81KiB
250          webm       audio only tiny   76k , webm_dash container, opus  (48000Hz), 1.17MiB
140          m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2 (44100Hz), 1.98MiB
251          webm       audio only tiny  151k , webm_dash container, opus  (48000Hz), 2.31MiB
160          mp4        146x144    144p    9k , mp4_dash container, avc1.4d400b, 25fps, video only, 151.79KiB
278          webm       146x144    144p   21k , webm_dash container, vp9, 25fps, video only, 330.81KiB
394          mp4        146x144    144p   21k , mp4_dash container, av01.0.00M.08, 25fps, video only, 332.29KiB
133          mp4        244x240    240p   13k , mp4_dash container, avc1.4d400c, 25fps, video only, 209.02KiB
395          mp4        244x240    240p   29k , mp4_dash container, av01.0.00M.08, 25fps, video only, 468.01KiB
242          webm       244x240    240p   38k , webm_dash container, vp9, 25fps, video only, 606.50KiB
134          mp4        366x360    360p   23k , mp4_dash container, avc1.4d4015, 25fps, video only, 364.02KiB
396          mp4        366x360    360p   44k , mp4_dash container, av01.0.00M.08, 25fps, video only, 701.91KiB
243          webm       366x360    360p   62k , webm_dash container, vp9, 25fps, video only, 979.65KiB
135          mp4        488x480    480p   38k , mp4_dash container, avc1.4d401e, 25fps, video only, 608.24KiB
397          mp4        488x480    480p   65k , mp4_dash container, av01.0.01M.08, 25fps, video only, 1.00MiB
244          webm       488x480    480p   99k , webm_dash container, vp9, 25fps, video only, 1.52MiB
136          mp4        732x720    720p   70k , mp4_dash container, avc1.4d401f, 25fps, video only, 1.08MiB
398          mp4        732x720    720p  124k , mp4_dash container, av01.0.04M.08, 25fps, video only, 1.90MiB
247          webm       732x720    720p  200k , webm_dash container, vp9, 25fps, video only, 3.07MiB
18           mp4        366x360    360p   71k , avc1.42001E, 25fps, mp4a.40.2 (22050Hz) (best)

... and this is what the gost (first) window prints:

2023/09/17 06:47:06 route.go:695: http://localhost:8080 on 127.0.0.1:8080
2023/09/17 06:48:46 http.go:161: [http] 127.0.0.1:13561 -> http://localhost:8080 -> www.youtube.com:443
2023/09/17 06:48:46 http.go:256: [route] 127.0.0.1:13561 -> http://localhost:8080 -> 1@https://77.*.*.*:443 -> www.youtube.com:443
2023/09/17 06:48:47 http.go:311: [http] 127.0.0.1:13561 <-> www.youtube.com:443
2023/09/17 06:48:48 http.go:313: [http] 127.0.0.1:13561 >-< www.youtube.com:443

Since OP is on MacOS, gost does come in precompiled binaries for that platform:

https://github.com/ginuerzh/gost/releases/download/v2.11.5/gost-darwin-amd64-2.11.5.gz
https://github.com/ginuerzh/gost/releases/download/v2.11.5/gost-darwin-arm64-2.11.5.gz

But there's not likely to be a back-port of that because of dependencies

... Is this going to be a won't/can't fix then? Could a "requests" yt-dl branch be created and maintained for those able to run yt-dl on py3.7+ ???

dirkf · 2023-09-18T10:43:44Z

Probably, but more research needed.

One downstream hack (for a settop box whose Python2 SSL is too old) delegates web access for extraction to wget; this could be worked up as an option (eg, --external-webpage-request[eo]r ..., to match --external-downloader ...); using curl would also be possible and might offer more functionality, such as better fingerprinting resistance. It might also be possible to offer a built-in alternative request[eo]r using Requests if available, but this is an even bluer sky idea.

But in the end a local proxy is enough here.

Vangelis66 · 2023-09-20T00:57:14Z

FWIW, and I'm not sure whether this is related to the recent networking overhaul 😉 , "downstream" do display currently a helpful/informative message 👍 when a yt-dlp user has configured it to use a HTTPS proxy (with or without authentication):

yt-dlp --proxy "https://ProxyHostName:ProxyPort" -vF "7LBYrDx_784" => 

[debug] Command-line config: ['--ffmpeg-location', '..\\FFmpeg', '--downloader-args', 'ffmpeg:-v 8 -stats', '--proxy', 'https://ProxyHostName:ProxyPort', '-vF', '7LBYrDx_784']
[debug] Encodings: locale cp1253, fs utf-8, pref cp1253, out utf-8 (No VT), error utf-8 (No VT), screen utf-8 (No VT)
[debug] yt-dlp version [email protected] [b532556d0] (win_x86_exe)
[debug] Python 3.7.9 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 (OpenSSL 1.1.1g  21 Apr 2020)
[debug] exe versions: ffmpeg n6.1-dev-1945-N-111829-g3c9dc0 (setts), ffprobe n6.1-dev-1945-N-111829-g3c9dc0, phantomjs 2.1.1, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.19.0, brotli-1.1.0, certifi-2023.07.22, mutagen-1.47.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {'all': 'https://ProxyHostName:ProxyPort'}
[debug] Extractor Plugins: AGB+NSIG (YoutubeIE)
[debug] Plugin directories: [redacted]
[debug] Loaded 1875 extractors
[youtube+AGB+NSIG] Extracting URL: 7LBYrDx_784
[youtube+AGB+NSIG] 7LBYrDx_784: Downloading webpage
ERROR: Unable to handle request: Unsupported proxy type: "https" (urllib)
Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 1575, in wrapper
  File "yt_dlp\YoutubeDL.py", line 1710, in __extract_info
  File "yt_dlp\extractor\common.py", line 715, in extract
  File "yt_dlp\extractor\youtube.py", line 4059, in _real_extract
  File "<redacted>\extractor\yt_agb.py", line 44, in _download_player_responses
    ret = super()._download_player_responses(url, smuggled_data, video_id, *args, **kwargs)
  File "yt_dlp\extractor\youtube.py", line 4019, in _download_player_responses
  File "yt_dlp\extractor\common.py", line 1118, in _download_webpage
  File "yt_dlp\extractor\common.py", line 1069, in download_content
  File "yt_dlp\extractor\common.py", line 903, in _download_webpage_handle
  File "yt_dlp\extractor\common.py", line 847, in _request_webpage
  File "yt_dlp\YoutubeDL.py", line 4059, in urlopen
  File "yt_dlp\networking\common.py", line 127, in send
yt_dlp.networking.exceptions.NoSupportingHandlers: Unable to handle request: Unsupported proxy type: "https" (urllib)

Is such an "error reporting" similarly beyond the realms of possibility for yt-dl ?

dirkf · 2023-09-20T10:14:55Z

Eg:

--- old/youtube-dl/youtube_dl/utils.py
+++ new/youtube-dl/youtube_dl/utils.py
@@ -5833,10 +5833,13 @@
 
         if proxy == '__noproxy__':
             return None  # No Proxy
-        if compat_urllib_parse.urlparse(proxy).scheme.lower() in ('socks', 'socks4', 'socks4a', 'socks5'):
+        scheme = compat_urllib_parse.urlparse(proxy).scheme.lower()
+        if scheme in ('socks', 'socks4', 'socks4a', 'socks5'):
             req.add_header('Ytdl-socks-proxy', proxy)
             # youtube-dl's http/https handlers do wrapping the socket with socks
             return None
+        elif scheme == 'https':
+        	raise NotImplementedError('''urllib2 doesn't support https proxies''')
         return compat_urllib_request.ProxyHandler.proxy_open(
             self, req, proxy, type)

Vangelis66 · 2023-09-20T23:59:08Z

Thanks, it works as expected when latest nightly build has been patched accordingly:

youtube-dl --proxy "https://ProxyHostName:443" -vF "Fh7UgrHMjbQ" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--proxy', 'https://ProxyHostName:443', '-vF', 'Fh7UgrHMjbQ']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.09.07 [66ab0814c] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {'https': 'https://ProxyHostName:443', 'http': 'https://ProxyHostName:443'}
[youtube] Fh7UgrHMjbQ: Downloading webpage
Traceback (most recent call last):
  File "__main__.py", line 19, in <module>
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\__init__.py", line 473, in main
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\__init__.py", line 463, in _real_main
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 2223, in download
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 856, in extract_info
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 863, in wrapper
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 959, in __extract_info
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 565, in extract
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 1850, in _real_extract
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 832, in _download_webpage
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 698, in _download_webpage_handle
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 665, in _request_webpage
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 2465, in urlopen
  File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 464, in open
  File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 482, in _open
  File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 442, in _call_chain
  File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 735, in <lambda>
  File "utils.py", line 5843, in proxy_open
NotImplementedError: urllib2 doesn't support https proxies

Vangelis66 · 2023-09-21T14:03:29Z

... Nit picking 😜 , I know, but yt-dlp in non-debug mode just prints a single ERROR line,

yt-dlp --proxy "https://ProxyHostName:ProxyPort" -F "7LBYrDx_784" => 

[youtube+AGB+NSIG] Extracting URL: 7LBYrDx_784
[youtube+AGB+NSIG] 7LBYrDx_784: Downloading webpage
ERROR: Unable to handle request: Unsupported proxy type: "https" (urllib)

whereas yt-dl, with the posted patch, throws the Traceback even in non-debug mode:

yt-dl --proxy "https://ProxyHostName:ProxyPort" -F "Fh7UgrHMjbQ" => 

[youtube] Fh7UgrHMjbQ: Downloading webpage
Traceback (most recent call last):
  File "__main__.py", line 19, in <module>
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\__init__.py", line 473, in main
    _real_main(argv)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\__init__.py", line 463, in _real_main
    retcode = ydl.download(all_urls)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 2219, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 852, in extract_info
    return self.__extract_info(url, ie, download, extra_info, process)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 859, in wrapper
    return func(self, *args, **kwargs)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 955, in __extract_info
    ie_result = ie.extract(url)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 565, in extract
    ie_result = self._real_extract(url)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\youtube.py", line 1844, in _real_extract
    webpage_url + '&bpctr=9999999999&has_verified=1', video_id, fatal=False)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 832, in _download_webpage
    expected_status=expected_status)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 698, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 665, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 2461, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\Python3410-32\lib\urllib\request.py", line 464, in open
    response = self._open(req, data)
  File "C:\Python3410-32\lib\urllib\request.py", line 482, in _open
    '_open', req)
  File "C:\Python3410-32\lib\urllib\request.py", line 442, in _call_chain
    result = func(*args)
  File "C:\Python3410-32\lib\urllib\request.py", line 735, in <lambda>
    meth(r, proxy, type))
  File "utils.py", line 5843, in proxy_open
NotImplementedError: urllib2 doesn't support https proxies

(NB: ProxyHostName, ProxyPort have real, valid, values, which have been omitted here, for privacy reasons 😉 ...)

Easy way to get a yt-dlp-like behaviour here? 😉

Many thanks, keep well 😄 !

dirkf · 2023-09-21T15:31:46Z

Maybe:

-        	raise NotImplementedError('''urllib2 doesn't support https proxies''')
+        	err = NotImplementedError('''urllib2 doesn't support https proxies''')
+        	raise ExtractorError('Proxy error', cause=err, expected=True)

I suppose the message also appears for PROXY environment vars?

Vangelis66 · 2023-09-21T17:49:17Z

... Right! 👍

non-debug mode

youtube-dl --proxy "https://ProxyHostName:443" -F "Fh7UgrHMjbQ" => 

[youtube] Fh7UgrHMjbQ: Downloading webpage
ERROR: Proxy error (caused by NotImplementedError("urllib2 doesn't support https proxies",))

... and:

debug mode

youtube-dl --proxy "https://ProxyHostName:443" -vF "Fh7UgrHMjbQ" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--proxy', 'https://ProxyHostName:443', '-vF', 'Fh7UgrHMjbQ']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.09.03.2215
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k  26 Jan 2017
[debug] exe versions: none
[debug] Proxy map: {'http': 'https://ProxyHostName:443', 'https': 'https://ProxyHostName:443'}
[youtube] Fh7UgrHMjbQ: Downloading webpage
ERROR: Proxy error (caused by NotImplementedError("urllib2 doesn't support https proxies",))
Traceback (most recent call last):
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 859, in wrapper
    return func(self, *args, **kwargs)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 955, in __extract_info
    ie_result = ie.extract(url)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 565, in extract
    ie_result = self._real_extract(url)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\youtube.py", line 1844, in _real_extract
    webpage_url + '&bpctr=9999999999&has_verified=1', video_id, fatal=False)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 832, in _download_webpage
    expected_status=expected_status)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 698, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 665, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 2461, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\Python3410-32\lib\urllib\request.py", line 464, in open
    response = self._open(req, data)
  File "C:\Python3410-32\lib\urllib\request.py", line 482, in _open
    '_open', req)
  File "C:\Python3410-32\lib\urllib\request.py", line 442, in _call_chain
    result = func(*args)
  File "C:\Python3410-32\lib\urllib\request.py", line 735, in <lambda>
    meth(r, proxy, type))
  File "utils.py", line 5844, in proxy_open
youtube_dl.utils.ExtractorError: Proxy error (caused by NotImplementedError("urllib2 doesn't support https proxies",))

I suppose the message also appears for PROXY environment vars?

Yes:

HTTPS proxy configured via env var

set HTTPS_PROXY=https://ProxyHostName:443

youtube-dl -vF "Fh7UgrHMjbQ" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vF', 'Fh7UgrHMjbQ']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.09.03.2215
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k  26 Jan 2017
[debug] exe versions: none
[debug] Proxy map: {'https': 'https://ProxyHostName:443'}
[youtube] Fh7UgrHMjbQ: Downloading webpage
ERROR: Proxy error (caused by NotImplementedError("urllib2 doesn't support https proxies",))
Traceback (most recent call last):
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 859, in wrapper
    return func(self, *args, **kwargs)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 955, in __extract_info
    ie_result = ie.extract(url)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 565, in extract
    ie_result = self._real_extract(url)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\youtube.py", line 1844, in _real_extract
    webpage_url + '&bpctr=9999999999&has_verified=1', video_id, fatal=False)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 832, in _download_webpage
    expected_status=expected_status)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 698, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\common.py", line 665, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\YoutubeDL.py", line 2461, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\Python3410-32\lib\urllib\request.py", line 464, in open
    response = self._open(req, data)
  File "C:\Python3410-32\lib\urllib\request.py", line 482, in _open
    '_open', req)
  File "C:\Python3410-32\lib\urllib\request.py", line 442, in _call_chain
    result = func(*args)
  File "C:\Python3410-32\lib\urllib\request.py", line 735, in <lambda>
    meth(r, proxy, type))
  File "utils.py", line 5844, in proxy_open
youtube_dl.utils.ExtractorError: Proxy error (caused by NotImplementedError("urllib2 doesn't support https proxies",))

Thanks ❤️ ...

Adds support for HTTPS proxies and persistent connections (keep-alive) Closes #1890 Resolves #4070 Resolves ytdl-org/youtube-dl#32549 Resolves ytdl-org/youtube-dl#14523 Resolves ytdl-org/youtube-dl#13734 Authored by: coletdjnz, Grub4K, bashonly

wisetc · 2023-11-24T14:31:16Z

requests==2.31.0
urllib3==1.26.17

urllib3 is required for https to work? isn't it?

yt-dlp/yt-dlp#8627

dirkf · 2023-11-24T15:43:47Z

Certainly one way to support https proxies in Python requires urllib3>=1.26.17. There may be other ways given the size of the Python module ecosystem, but the refined error message is the change that's likely to be implemented here, for reasons discussed above.

I'll keep the issue open to remind me of the patch.

dirkf · 2024-03-22T17:11:33Z

Also https://github.com/SeaHOH/extproxy as an optional dependency is another possibility.

Adds support for HTTPS proxies and persistent connections (keep-alive) Closes yt-dlp#1890 Resolves yt-dlp#4070 Resolves ytdl-org/youtube-dl#32549 Resolves ytdl-org/youtube-dl#14523 Resolves ytdl-org/youtube-dl#13734 Authored by: coletdjnz, Grub4K, bashonly

wisetc added the question label Sep 8, 2023

dirkf changed the title ~~Wether https proxy --proxy=https://user:[email protected] is not supported?~~ Whether https proxy --proxy=https://user:[email protected] is not supported? [it isn't] Sep 9, 2023

This comment was marked as outdated.

Sign in to view

coletdjnz mentioned this issue Sep 29, 2023

[rh:requests] Add support for requests yt-dlp/yt-dlp#3668

Merged

9 tasks

bashonly mentioned this issue Oct 9, 2023

Regression: unable to handle request: Unsupported proxy type: "https" when using system proxy on Windows yt-dlp/yt-dlp#7663

Closed

10 tasks

wisetc closed this as completed Nov 24, 2023

dirkf reopened this Nov 24, 2023

dirkf mentioned this issue Mar 22, 2024

socks5 with DNS-requests #32696

Open

3 tasks

dirkf mentioned this issue Sep 5, 2024

Fails to utilize proxy #32924

Closed

3 tasks

dirkf mentioned this issue Nov 16, 2024

[Suggestion] Keep proxy connection open through the task #32966

Closed

3 tasks

dirkf mentioned this issue Dec 5, 2024

YouTube blocking: 加上--proxy "socks5://127.0.0.1:11223"依旧出现由于目标计算机积极拒绝，无法连接 #31438

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Whether https proxy `--proxy=https://user:[email protected]` is not supported? [it isn't] #32549

Whether https proxy `--proxy=https://user:[email protected]` is not supported? [it isn't] #32549

wisetc commented Sep 8, 2023

dirkf commented Sep 8, 2023

wisetc commented Sep 9, 2023

This comment was marked as outdated.

wisetc commented Sep 10, 2023

dirkf commented Sep 10, 2023

Vangelis66 commented Sep 10, 2023

coletdjnz commented Sep 11, 2023 •

edited

Loading

dirkf commented Sep 12, 2023

Vangelis66 commented Sep 12, 2023

Vangelis66 commented Sep 18, 2023 •

edited

Loading

dirkf commented Sep 18, 2023

Vangelis66 commented Sep 20, 2023 •

edited

Loading

dirkf commented Sep 20, 2023

Vangelis66 commented Sep 20, 2023 •

edited

Loading

Vangelis66 commented Sep 21, 2023

dirkf commented Sep 21, 2023

Vangelis66 commented Sep 21, 2023

wisetc commented Nov 24, 2023

dirkf commented Nov 24, 2023

dirkf commented Mar 22, 2024

Whether https proxy --proxy=https://user:[email protected] is not supported? [it isn't] #32549

Whether https proxy --proxy=https://user:[email protected] is not supported? [it isn't] #32549

Comments

wisetc commented Sep 8, 2023

Checklist

Question

dirkf commented Sep 8, 2023

wisetc commented Sep 9, 2023

This comment was marked as outdated.

wisetc commented Sep 10, 2023

dirkf commented Sep 10, 2023

Vangelis66 commented Sep 10, 2023

coletdjnz commented Sep 11, 2023 • edited Loading

dirkf commented Sep 12, 2023

Vangelis66 commented Sep 12, 2023

Vangelis66 commented Sep 18, 2023 • edited Loading

dirkf commented Sep 18, 2023

Vangelis66 commented Sep 20, 2023 • edited Loading

dirkf commented Sep 20, 2023

Vangelis66 commented Sep 20, 2023 • edited Loading

Vangelis66 commented Sep 21, 2023

dirkf commented Sep 21, 2023

Vangelis66 commented Sep 21, 2023

wisetc commented Nov 24, 2023

dirkf commented Nov 24, 2023

dirkf commented Mar 22, 2024

Whether https proxy `--proxy=https://user:[email protected]` is not supported? [it isn't] #32549

Whether https proxy `--proxy=https://user:[email protected]` is not supported? [it isn't] #32549

coletdjnz commented Sep 11, 2023 •

edited

Loading

Vangelis66 commented Sep 18, 2023 •

edited

Loading

Vangelis66 commented Sep 20, 2023 •

edited

Loading

Vangelis66 commented Sep 20, 2023 •

edited

Loading