aiohttp raises SSLError when requesting URLs with FQDN #3636
Labels
Comments
|
GitMate.io thinks the contributor most likely able to help you is @asvetlov. Possibly related issues are #2920 (AIOHttp failing after some requests), #660 (aiohttp.request hangs on some URLs), #206 (SSL issue with aiohttp.request), #1403 ([QUESTION] aiohttp.ClientSession.request('GET') issue), and #3523 (aiohttp not forwarding cookies with Session requests). |
Your trusted CA chain is probably broken/invalid/misconfigured. It's not aiohttp's fault. |
|
@webknjaz Thank you for taking the time to look at this. If you are correct, why does the following code work correctly (with the same environment)? import aiohttp
import asyncio
async def fetch(session, url):
async with session.get(url) as response:
return await response.text()
async def main():
async with aiohttp.ClientSession() as session:
html = await fetch(session, 'https://github.com')
print(html)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())Note that the only difference is And Python-Requests with |
|
Ah, ok. But strictly speaking, certificate has |
|
Yes, it doesn't match. Where do you think this should be fixed if not in |
|
Yea, I saw that. So I decided to do some research with what I have on my machine. Google Chrome:
curl:
So it looks like there's no agreement on what clients should do but a single-dot case is handled gracefully. |
|
Ref: curl/curl#716 |
|
SNI note: https://tools.ietf.org/html/rfc6066#section-3
|
Right, this clears things up about who should handle trailing dot which is application layer, according to @tiran. This seems fair. Another excerpt:
|
Verdict
|
martin-sucha
added a commit
to kiwicom/aiohttp
that referenced
this issue
Jul 12, 2023
The TLS verification fails with an exception if the client uses a fully-qualified domain name with a trailing dot, like https://github.com./ : aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host github.com.:443 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'github.com.'. (_ssl.c:1051)")] The reason is that TLS certificates do not contain the trailing dot, as per RFC 6066: "HostName" contains the fully qualified DNS hostname of the server, as understood by the client. The hostname is represented as a byte string using ASCII encoding without a trailing dot. We need to strip the trailing dot for TLS context and Host header, where trailing dots are not present. For DNS resolution, we need to include the trailing dot as it signifies a fully-qualified domain name (FQDN). DNS lookups of FQDNs are faster as the resolver does not need to check DNS search path, like for relative DNS names. Closes aio-libs#3636
martin-sucha
added a commit
to kiwicom/aiohttp
that referenced
this issue
Jul 12, 2023
The TLS verification fails with an exception if the client uses a fully-qualified domain name with a trailing dot, like https://github.com./ : aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host github.com.:443 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'github.com.'. (_ssl.c:1051)")] The reason is that TLS certificates do not contain the trailing dot, as per RFC 6066: "HostName" contains the fully qualified DNS hostname of the server, as understood by the client. The hostname is represented as a byte string using ASCII encoding without a trailing dot. We need to strip the trailing dot for TLS context and Host header, where trailing dots are not present. For DNS resolution, we need to include the trailing dot as it signifies a fully-qualified domain name (FQDN). DNS lookups of FQDNs are faster as the resolver does not need to check DNS search path, like for relative DNS names. Closes aio-libs#3636
5 tasks
|
FTR, @martin-sucha is taking a stab at this @ #7364 if anyone watching this issue is interested in participating. |
webknjaz
added a commit
that referenced
this issue
Sep 11, 2023
Before this patch, the TLS verification fails with an exception if the client uses a fully-qualified domain name with a trailing dot, like https://github.com./ : ```console aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host github.com.:443 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'github.com.'. (_ssl.c:1051)")] ``` The reason is that TLS certificates do not contain the trailing dot, as per RFC 6066: "HostName" contains the fully qualified DNS hostname of the server, as understood by the client. The hostname is represented as a byte string using ASCII encoding without a trailing dot. This change makes aiohttp strip the trailing dot for TLS context and Host header, where trailing dots are not present. For DNS resolution, we include the trailing dot as it signifies a fully-qualified domain name (FQDN). DNS lookups of FQDNs are faster as the resolver does not need to check DNS search path, like for relative DNS names. This effectively allows clients to connect to server if URL has dot at the end of the hostname, e.g. `https://example.com./. Fixes #3636 PR #7364 Co-authored-by: Sviatoslav Sydorenko <[email protected]>
martin-sucha
added a commit
to kiwicom/aiohttp
that referenced
this issue
Sep 12, 2023
Before this patch, the TLS verification fails with an exception if the client uses a fully-qualified domain name with a trailing dot, like https://github.com./ : ```console aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host github.com.:443 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'github.com.'. (_ssl.c:1051)")] ``` The reason is that TLS certificates do not contain the trailing dot, as per RFC 6066: "HostName" contains the fully qualified DNS hostname of the server, as understood by the client. The hostname is represented as a byte string using ASCII encoding without a trailing dot. This change makes aiohttp strip the trailing dot for TLS context and Host header, where trailing dots are not present. For DNS resolution, we include the trailing dot as it signifies a fully-qualified domain name (FQDN). DNS lookups of FQDNs are faster as the resolver does not need to check DNS search path, like for relative DNS names. This effectively allows clients to connect to server if URL has dot at the end of the hostname, e.g. `https://example.com./. Fixes aio-libs#3636 PR aio-libs#7364 Co-authored-by: Sviatoslav Sydorenko <[email protected]> (cherry picked from commit d84fcf7)
5 tasks
Dreamsorcerer
pushed a commit
that referenced
this issue
Sep 30, 2023
Before this patch, the TLS verification fails with an exception if the client uses a fully-qualified domain name with a trailing dot, like https://github.com./ : ```console aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host github.com.:443 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'github.com.'. (_ssl.c:1051)")] ``` The reason is that TLS certificates do not contain the trailing dot, as per RFC 6066: "HostName" contains the fully qualified DNS hostname of the server, as understood by the client. The hostname is represented as a byte string using ASCII encoding without a trailing dot. This change makes aiohttp strip the trailing dot for TLS context and Host header, where trailing dots are not present. For DNS resolution, we include the trailing dot as it signifies a fully-qualified domain name (FQDN). DNS lookups of FQDNs are faster as the resolver does not need to check DNS search path, like for relative DNS names. This effectively allows clients to connect to server if URL has dot at the end of the hostname, e.g. `https://example.com./. Fixes #3636 PR #7364 Co-authored-by: Sviatoslav Sydorenko <[email protected]> (cherry picked from commit d84fcf7) <!-- Thank you for your contribution! --> ## What do these changes do? Backport #7364 into 3.9 <!-- Please give a short brief about these changes. --> ## Are there changes in behavior for the user? <!-- Outline any notable behaviour for the end users. --> ## Related issue number <!-- Are there any issues opened that will be resolved by merging this change? --> ## Checklist - [ ] I think the code is well written - [ ] Unit tests for the changes exist - [ ] Documentation reflects the changes - [ ] If you provide code modification, please add yourself to `CONTRIBUTORS.txt` * The format is <Name> <Surname>. * Please keep alphabetical order, the file is sorted by names. - [ ] Add a new news fragment into the `CHANGES` folder * name it `<issue_id>.<type>` for example (588.bugfix) * if you don't have an `issue_id` change it to the pr id after creating the pr * ensure type is one of the following: * `.feature`: Signifying a new feature. * `.bugfix`: Signifying a bug fix. * `.doc`: Signifying a documentation improvement. * `.removal`: Signifying a deprecation or removal of public API. * `.misc`: A ticket has been closed, but it is not of interest to users. * Make sure to use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files."
renovate bot
referenced
this issue
in allenporter/pyrainbird
Nov 20, 2023
[](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [aiohttp](https://togithub.com/aio-libs/aiohttp) | `==3.8.6` -> `==3.9.0` | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>aio-libs/aiohttp (aiohttp)</summary> ### [`v3.9.0`](https://togithub.com/aio-libs/aiohttp/blob/HEAD/CHANGES.rst#390-2023-11-18) [Compare Source](https://togithub.com/aio-libs/aiohttp/compare/v3.8.6...v3.9.0) \================== ## Features - Introduced `AppKey` for static typing support of `Application` storage. See https://docs.aiohttp.org/en/stable/web_advanced.html#application-s-config `#​5864 <https://github.com/aio-libs/aiohttp/issues/5864>`\_ - Added a graceful shutdown period which allows pending tasks to complete before the application's cleanup is called. The period can be adjusted with the `shutdown_timeout` parameter. -- by :user:`Dreamsorcerer`. See https://docs.aiohttp.org/en/latest/web_advanced.html#graceful-shutdown `#​7188 <https://github.com/aio-libs/aiohttp/issues/7188>`\_ - Added `handler_cancellation <https://docs.aiohttp.org/en/stable/web_advanced.html#web-handler-cancellation>`\_ parameter to cancel web handler on client disconnection. -- by :user:`mosquito` This (optionally) reintroduces a feature removed in a previous release. Recommended for those looking for an extra level of protection against denial-of-service attacks. `#​7056 <https://github.com/aio-libs/aiohttp/issues/7056>`\_ - Added support for setting response header parameters `max_line_size` and `max_field_size`. `#​2304 <https://github.com/aio-libs/aiohttp/issues/2304>`\_ - Added `auto_decompress` parameter to `ClientSession.request` to override `ClientSession._auto_decompress`. -- by :user:`Daste745` `#​3751 <https://github.com/aio-libs/aiohttp/issues/3751>`\_ - Changed `raise_for_status` to allow a coroutine. `#​3892 <https://github.com/aio-libs/aiohttp/issues/3892>`\_ - Added client brotli compression support (optional with runtime check). `#​5219 <https://github.com/aio-libs/aiohttp/issues/5219>`\_ - Added `client_max_size` to `BaseRequest.clone()` to allow overriding the request body size. -- :user:`anesabml`. `#​5704 <https://github.com/aio-libs/aiohttp/issues/5704>`\_ - Added a middleware type alias `aiohttp.typedefs.Middleware`. `#​5898 <https://github.com/aio-libs/aiohttp/issues/5898>`\_ - Exported `HTTPMove` which can be used to catch any redirection request that has a location -- :user:`dreamsorcerer`. `#​6594 <https://github.com/aio-libs/aiohttp/issues/6594>`\_ - Changed the `path` parameter in `web.run_app()` to accept a `pathlib.Path` object. `#​6839 <https://github.com/aio-libs/aiohttp/issues/6839>`\_ - Performance: Skipped filtering `CookieJar` when the jar is empty or all cookies have expired. `#​7819 <https://github.com/aio-libs/aiohttp/issues/7819>`\_ - Performance: Only check origin if insecure scheme and there are origins to treat as secure, in `CookieJar.filter_cookies()`. `#​7821 <https://github.com/aio-libs/aiohttp/issues/7821>`\_ - Performance: Used timestamp instead of `datetime` to achieve faster cookie expiration in `CookieJar`. `#​7824 <https://github.com/aio-libs/aiohttp/issues/7824>`\_ - Added support for passing a custom server name parameter to HTTPS connection. `#​7114 <https://github.com/aio-libs/aiohttp/issues/7114>`\_ - Added support for using Basic Auth credentials from :file:`.netrc` file when making HTTP requests with the :py:class:`~aiohttp.ClientSession` `trust_env` argument is set to `True`. -- by :user:`yuvipanda`. `#​7131 <https://github.com/aio-libs/aiohttp/issues/7131>`\_ - Turned access log into no-op when the logger is disabled. `#​7240 <https://github.com/aio-libs/aiohttp/issues/7240>`\_ - Added typing information to `RawResponseMessage`. -- by :user:`Gobot1234` `#​7365 <https://github.com/aio-libs/aiohttp/issues/7365>`\_ - Removed `async-timeout` for Python 3.11+ (replaced with `asyncio.timeout()` on newer releases). `#​7502 <https://github.com/aio-libs/aiohttp/issues/7502>`\_ - Added support for `brotlicffi` as an alternative to `brotli` (fixing Brotli support on PyPy). `#​7611 <https://github.com/aio-libs/aiohttp/issues/7611>`\_ - Added `WebSocketResponse.get_extra_info()` to access a protocol transport's extra info. `#​7078 <https://github.com/aio-libs/aiohttp/issues/7078>`\_ - Allow `link` argument to be set to None/empty in HTTP 451 exception. `#​7689 <https://github.com/aio-libs/aiohttp/issues/7689>`\_ ## Bugfixes - Implemented stripping the trailing dots from fully-qualified domain names in `Host` headers and TLS context when acting as an HTTP client. This allows the client to connect to URLs with FQDN host name like `https://example.com./`. \-- by :user:`martin-sucha`. `#​3636 <https://github.com/aio-libs/aiohttp/issues/3636>`\_ - Fixed client timeout not working when incoming data is always available without waiting. -- by :user:`Dreamsorcerer`. `#​5854 <https://github.com/aio-libs/aiohttp/issues/5854>`\_ - Fixed `readuntil` to work with a delimiter of more than one character. `#​6701 <https://github.com/aio-libs/aiohttp/issues/6701>`\_ - Added `__repr__` to `EmptyStreamReader` to avoid `AttributeError`. `#​6916 <https://github.com/aio-libs/aiohttp/issues/6916>`\_ - Fixed bug when using `TCPConnector` with `ttl_dns_cache=0`. `#​7014 <https://github.com/aio-libs/aiohttp/issues/7014>`\_ - Fixed response returned from expect handler being thrown away. -- by :user:`Dreamsorcerer` `#​7025 <https://github.com/aio-libs/aiohttp/issues/7025>`\_ - Avoided raising `UnicodeDecodeError` in multipart and in HTTP headers parsing. `#​7044 <https://github.com/aio-libs/aiohttp/issues/7044>`\_ - Changed `sock_read` timeout to start after writing has finished, avoiding read timeouts caused by an unfinished write. -- by :user:`dtrifiro` `#​7149 <https://github.com/aio-libs/aiohttp/issues/7149>`\_ - Fixed missing query in tracing method URLs when using `yarl` 1.9+. `#​7259 <https://github.com/aio-libs/aiohttp/issues/7259>`\_ - Changed max 32-bit timestamp to an aware datetime object, for consistency with the non-32-bit one, and to avoid a `DeprecationWarning` on Python 3.12. `#​7302 <https://github.com/aio-libs/aiohttp/issues/7302>`\_ - Fixed `EmptyStreamReader.iter_chunks()` never ending. -- by :user:`mind1m` `#​7616 <https://github.com/aio-libs/aiohttp/issues/7616>`\_ - Fixed a rare `RuntimeError: await wasn't used with future` exception. -- by :user:`stalkerg` `#​7785 <https://github.com/aio-libs/aiohttp/issues/7785>`\_ - Fixed issue with insufficient HTTP method and version validation. `#​7700 <https://github.com/aio-libs/aiohttp/issues/7700>`\_ - Added check to validate that absolute URIs have schemes. `#​7712 <https://github.com/aio-libs/aiohttp/issues/7712>`\_ - Fixed unhandled exception when Python HTTP parser encounters unpaired Unicode surrogates. `#​7715 <https://github.com/aio-libs/aiohttp/issues/7715>`\_ - Updated parser to disallow invalid characters in header field names and stop accepting LF as a request line separator. `#​7719 <https://github.com/aio-libs/aiohttp/issues/7719>`\_ - Fixed Python HTTP parser not treating 204/304/1xx as an empty body. `#​7755 <https://github.com/aio-libs/aiohttp/issues/7755>`\_ - Ensure empty body response for 1xx/204/304 per RFC 9112 sec 6.3. `#​7756 <https://github.com/aio-libs/aiohttp/issues/7756>`\_ - Fixed an issue when a client request is closed before completing a chunked payload. -- by :user:`Dreamsorcerer` `#​7764 <https://github.com/aio-libs/aiohttp/issues/7764>`\_ - Edge Case Handling for ResponseParser for missing reason value. `#​7776 <https://github.com/aio-libs/aiohttp/issues/7776>`\_ - Fixed `ClientWebSocketResponse.close_code` being erroneously set to `None` when there are concurrent async tasks receiving data and closing the connection. `#​7306 <https://github.com/aio-libs/aiohttp/issues/7306>`\_ - Added HTTP method validation. `#​6533 <https://github.com/aio-libs/aiohttp/issues/6533>`\_ - Fixed arbitrary sequence types being allowed to inject values via version parameter. -- by :user:`Dreamsorcerer` `#​7835 <https://github.com/aio-libs/aiohttp/issues/7835>`\_ - Performance: Fixed increase in latency with small messages from websocket compression changes. `#​7797 <https://github.com/aio-libs/aiohttp/issues/7797>`\_ ## Improved Documentation - Fixed the `ClientResponse.release`'s type in the doc. Changed from `comethod` to `method`. `#​5836 <https://github.com/aio-libs/aiohttp/issues/5836>`\_ - Added information on behavior of base_url parameter in `ClientSession`. `#​6647 <https://github.com/aio-libs/aiohttp/issues/6647>`\_ - Fixed `ClientResponseError` docs. `#​6700 <https://github.com/aio-libs/aiohttp/issues/6700>`\_ - Updated Redis code examples to follow the latest API. `#​6907 <https://github.com/aio-libs/aiohttp/issues/6907>`\_ - Added a note about possibly needing to update headers when using `on_response_prepare`. -- by :user:`Dreamsorcerer` `#​7283 <https://github.com/aio-libs/aiohttp/issues/7283>`\_ - Completed `trust_env` parameter description to honor `wss_proxy`, `ws_proxy` or `no_proxy` env. `#​7325 <https://github.com/aio-libs/aiohttp/issues/7325>`\_ - Expanded SSL documentation with more examples (e.g. how to use certifi). -- by :user:`Dreamsorcerer` `#​7334 <https://github.com/aio-libs/aiohttp/issues/7334>`\_ - Fix, update, and improve client exceptions documentation. `#​7733 <https://github.com/aio-libs/aiohttp/issues/7733>`\_ ## Deprecations and Removals - Added `shutdown_timeout` parameter to `BaseRunner`, while deprecating `shutdown_timeout` parameter from `BaseSite`. -- by :user:`Dreamsorcerer` `#​7718 <https://github.com/aio-libs/aiohttp/issues/7718>`\_ - Dropped Python 3.6 support. `#​6378 <https://github.com/aio-libs/aiohttp/issues/6378>`\_ - Dropped Python 3.7 support. -- by :user:`Dreamsorcerer` `#​7336 <https://github.com/aio-libs/aiohttp/issues/7336>`\_ - Removed support for abandoned `tokio` event loop. -- by :user:`Dreamsorcerer` `#​7281 <https://github.com/aio-libs/aiohttp/issues/7281>`\_ ## Misc - Made `print` argument in `run_app()` optional. `#​3690 <https://github.com/aio-libs/aiohttp/issues/3690>`\_ - Improved performance of `ceil_timeout` in some cases. `#​6316 <https://github.com/aio-libs/aiohttp/issues/6316>`\_ - Changed importing Gunicorn to happen on-demand, decreasing import time by ~53%. -- :user:`Dreamsorcerer` `#​6591 <https://github.com/aio-libs/aiohttp/issues/6591>`\_ - Improved import time by replacing `http.server` with `http.HTTPStatus`. `#​6903 <https://github.com/aio-libs/aiohttp/issues/6903>`\_ - Fixed annotation of `ssl` parameter to disallow `True`. -- by :user:`Dreamsorcerer`. `#​7335 <https://github.com/aio-libs/aiohttp/issues/7335>`\_ *** </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/allenporter/pyrainbird). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy41OS44IiwidXBkYXRlZEluVmVyIjoiMzcuNTkuOCIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
](https://renovatebot.com){kind=link}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Long story short
It seems that aiohttp doesn't work properly when requesting URLs with Fully Qualified Domain Name. This was fixed in urllib3, see urllib3/urllib3#1255 It should probably by fixed also in aiohttp. May be related to #3171
Expected behaviour
aiohttp works when requesting URLs with FQDN
Actual behaviour
aiohttp raises SSL error:
Steps to reproduce
Run the following code:
Your environment
Python 3.7.1
Ubuntu 18.04
pip freeze
The text was updated successfully, but these errors were encountered: