ClientConnectorCertificateError raised when using aiohttp Client

[alviezhang]

🐞 Describe the bug

Recently I moved from homebrew to macports for some reasons, I soon found that aiohttp cannot connect to https sites by using ports python-39 with py39-aiohttp, the same result as the pip installed aiohttp.

I tested with requests and it performed very well to connect https sites.

What is wrong with macports and its builtin python package, how can I find out what is going wrong?

💡 To Reproduce

OS: macOS 10.15.7
macport: 2.6.4
port package installed: python39, py-aiohttp

Code to reproduce, the official document example

import aiohttp
import asyncio

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('https://httpbin.org/get') as resp:
            print(resp.status)
            print(await resp.text())

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

💡 Expected behavior

Successfully get the API response.

📋 Logs/tracebacks

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/aiohttp/connector.py", line 969, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 1081, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 1111, in _create_connection_transport
    await waiter
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/sslproto.py", line 528, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/sslproto.py", line 188, in feed_ssldata
    self._sslobj.do_handshake()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 944, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "<stdin>", line 3, in main
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/aiohttp/client.py", line 1117, in __aenter__
    self._resp = await self._coro
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/aiohttp/client.py", line 520, in _request
    conn = await self._connector.connect(
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/aiohttp/connector.py", line 535, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/aiohttp/connector.py", line 892, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/aiohttp/connector.py", line 1051, in _create_direct_connection
    raise last_exc
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/aiohttp/connector.py", line 1020, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/aiohttp/connector.py", line 971, in _wrap_create_connection
    raise ClientConnectorCertificateError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host httpbin.org:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')]

📋 Your version of the Python

$ python --version
Python 3.9.1

📋 Your version of the aiohttp/yarl/multidict distributions

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.7.3
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: Nikolay Kim
Author-email: [email protected]
License: Apache 2
Location: /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages
Requires: attrs, chardet, multidict, async-timeout, yarl, typing-extensions
Required-by:

$ python -m pip show multidict
Name: multidict
Version: 5.1.0
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages
Requires:
Required-by: yarl, aiohttp

$ python -m pip show yarl
Name: yarl
Version: 1.6.3
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages
Requires: multidict, idna
Required-by: aiohttp

📋 Additional context

Same environment but use requests or httpx module instead behaves normally

import requests
resp = requests.get('https://httpbin.org/get')

import httpx
async with httpx.AsyncClient() as client:
    resp = await client.get('https://httpbin.org')

And I also tested them with self signed URL too, they both raise SSL error as expected.

[Dreamsorcerer]

Judging by the first traceback, I'd say this was an issue in Python's ssl module, not something to do with aiohttp. So, I'd suggest looking for that error more generally. At a guess, it may be that the installation is missing some CA certificates or something similar.

I recall similar issues in the past with MacOS generally, due to Apple having not updated the CA certs for a couple of years (so websites using newer CAs would end up failing).

[alviezhang]

I checked the source code of requests and httpx, it will use the certificates in the python-certifi package. I will try to use it first in my code, but it would be better for aiohttp to have this feature built in. :)

[alviezhang]

Also, I tested with macOS built in Python and pyenv installed Python, they both work well, it must be something wrong with port installed Python , I have no idea about how to solve it.

[derlih]

So may be it makes sense to convert this bug to a feature request for soft dependency for python-certifi.
We have such soft deps for example for brotli compression support.

[Dreamsorcerer]

I've just encountered this myself on a Mac, after installing the 3.9.2 binary from python.org.
It was working fine before (with 3.7.5, probably the pre-installed version...).

[Dreamsorcerer]

So, I can run them explicitly.
No problems with:
/usr/local/Cellar/python/3.7.5/bin/python3
SSL issue with:
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3

[Dreamsorcerer]

With old install:

>>> ssl.create_default_context().cert_store_stats()
{'x509': 168, 'crl': 0, 'x509_ca': 168}

With new install:

>>> ssl.create_default_context().cert_store_stats()
{'x509': 0, 'crl': 0, 'x509_ca': 0}

Clearly, installing the binary from python.org results in it being unable to find the system installed certificates.

So, this is definetely not an issue with aiohttp, but with the Python installer. Not sure how to fix it yet though.

[webknjaz]

@Dreamsorcerer the official DMGs come with a post-install script: /Applications/Python 3.7/Install Certificates.command.

It should be executed upon install: https://github.com/ansible/pylibssh/blob/fc51b62/.github/workflows/build-test-n-publish.yml#L247-L253.

Ref: https://flaviocopes.com/python-installation-macos/.

This will also manifest itself with the bundled pip install IIRC so it doesn't make sense to make this a dependency of aiohttp or require any other specific CA bundle. The users should be able to choose how they set up their certs.

P.S. It looks like @alviezhang didn't hit this TLS problem with pip install because he didn't use pip install and used some other packaging software (macports?) which means that (1) this install method is unsupported and we don't even know how well some unknown third-party packaged it for that ecosystem and (2) any other software requiring TLS via the built-in ssl module will hit exactly the same issue. It is not a bug in aiohttp.

[webknjaz]

@alviezhang You can either run the post-install script to set up the trust chain or use a custom SSLContext and pass it to aiohttp explicitly. This is documented: https://docs.aiohttp.org/en/stable/client_advanced.html#ssl-control-for-tcp-sockets.

[Dreamsorcerer]

Uhh, OK, this is what happens when you click through an installer without reading.

The command mentioned by @webknjaz is not run automatically when installing with the python.org installer. The last step of the installer tells you that you should run the command yourself.

[abd-shouman]

I wanted to highlight the Issue Comment referenced in @webknjaz's comment as it was pretty helpful.

It shows the steps to install the certificate.
After the installation, you might have to restart your editor (or Jupyter Kernel)