Unable to use system proxy with HTTPS connection on Windows and Python 3.9

[kotori2]

On Windows you can only set a "host:port" http proxy as system proxy.

PS > Get-ItemProperty -Path 'HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings' | findstr ProxyServer
ProxyServer              : 127.0.0.1:7890

But in Python, it assumes that this proxy supports both http and https on the same port:

>>> import urllib
>>> urllib.request.getproxies()
{'http': 'http://127.0.0.1:7890', 'https': 'https://127.0.0.1:7890', 'ftp': 'ftp://127.0.0.1:7890'}

Which will try to handshake httpps on http port.

Expected Result

On Python 3.8.x:

> .\python38.exe
Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib, requests
>>> urllib.request.getproxies()
{'http': 'http://127.0.0.1:7890', 'https': 'https://127.0.0.1:7890', 'ftp': 'ftp://127.0.0.1:7890'}
>>> requests.get("http://www.google.com")
<Response [200]>
>>> requests.get("https://www.google.com")
<Response [200]>

Actual Result

> python3
Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec  7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib, requests
>>> urllib.request.getproxies()
{'http': 'http://127.0.0.1:7890', 'https': 'https://127.0.0.1:7890', 'ftp': 'ftp://127.0.0.1:7890'}
>>> requests.get("http://www.google.com")
<Response [200]>
>>> requests.get("https://www.google.com")
Traceback (most recent call last):
  File "C:\Python39\lib\site-packages\urllib3\connectionpool.py", line 696, in urlopen
    self._prepare_proxy(conn)
  File "C:\Python39\lib\site-packages\urllib3\connectionpool.py", line 964, in _prepare_proxy
    conn.connect()
  File "C:\Python39\lib\site-packages\urllib3\connection.py", line 359, in connect
    conn = self._connect_tls_proxy(hostname, conn)
  File "C:\Python39\lib\site-packages\urllib3\connection.py", line 496, in _connect_tls_proxy
    return ssl_wrap_socket(
  File "C:\Python39\lib\site-packages\urllib3\util\ssl_.py", line 432, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
  File "C:\Python39\lib\site-packages\urllib3\util\ssl_.py", line 474, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
  File "C:\Python39\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\Python39\lib\ssl.py", line 1040, in _create
    self.do_handshake()
  File "C:\Python39\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python39\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "C:\Python39\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "C:\Python39\lib\site-packages\urllib3\util\retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1123)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python39\lib\site-packages\requests\api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Python39\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python39\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python39\lib\site-packages\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python39\lib\site-packages\requests\adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1123)')))

Reproduction Steps

Set a http Windows system proxy then execute:

import requests
requests.get("https://www.google.com")

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "4.0.0"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.10"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.1"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.25.1"
  },
  "system_ssl": {
    "version": "1010107f"
  },
  "urllib3": {
    "version": "1.26.2"
  },
  "using_pyopenssl": false
}

[davycloud]

see: pypa/pip#9216 (comment)

reinstall urllib3==1.25.11 or pass proxies to requests

proxies={
'http': 'http://127.0.0.1:7890',
'https': 'http://127.0.0.1:7890'  # https -> http
}

r = requests.get(url, proxies=proxies)

[kotori2]

@davycloud
So is that means it is a urllib3 issue?
But from this line

Users that have their proxy configured as proxy_url=https://... and connecting to https://pypi.org before were actually using their proxy in an HTTP configuration

Since Windows only support http proxy as system proxy, I think it's more like a system proxy parsing issue. So I have no idea if the issue happens on urllib3 side or requests side.
Also of course setting proxies in request will work but I smh don't want to detect system proxy manually in every script.

[davycloud]

@kotori2

I think it's more like a system proxy parsing issue.

I think you are right.
Since the proxy server is http or https is nothing about the target url.
So in Windows, when we config the proxy 127.0.0.1 and 7890，it should means:

# http proxy
proxies={
'http': 'http://127.0.0.1:7890',
'https': 'http://127.0.0.1:7890' 
}

If the proxy is https, we should config the address manually: https://127.0.0.1 ，and the result should be:

# https proxy
proxies={
'http': 'https://127.0.0.1:7890',
'https': 'https://127.0.0.1:7890' 
}

BUT，in case 1，the actual result is :

>>> import urllib
>>> urllib.request.getproxies()
{'http': 'http://127.0.0.1:7890', 'https': 'https://127.0.0.1:7890', 'ftp': 'ftp://127.0.0.1:7890'}

which cause the error in new urllib3 .

Then I manually config the address http://127.0.0.1

>>> urllib.request.getproxies()
{'http': 'http://127.0.0.1:7890'}

😂

BTW，set Environment variable https_proxy=http://127.0.0.1:7890 also work，but it will ignore the system setting.

[kotori2]

https://bugs.python.org/issue42627