❇️ Add WebSocket support over HTTP/1, HTTP/2 and HTTP/3 (#153)

In addition to that, we also provide a way to deal with other sub
protocol by providing a secure I/O for the given stream

.github/workflows/ci.yml

@@ -44,7 +44,7 @@ jobs:
       matrix:
         python-version: ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]
         os:
-          - macos-12
+          - macos-13
           - windows-2022
           - ubuntu-20.04  # OpenSSL 1.1.1
           - ubuntu-latest  # OpenSSL 3.0
@@ -85,11 +85,13 @@ jobs:
           # https://github.com/python/cpython/issues/83001
           - python-version: "3.7"
             os: ubuntu-22.04
+          - python-version: "3.7"
+            os: macos-13
           - python-version: "3.8"
             os: ubuntu-22.04
 
     runs-on: ${{ matrix.os }}
-    name: ${{ fromJson('{"macos-12":"macOS","windows-2022":"Windows","ubuntu-latest":"Ubuntu","ubuntu-20.04":"Ubuntu 20.04 (OpenSSL 1.1.1)","ubuntu-latest":"Ubuntu Latest (OpenSSL 3+)"}')[matrix.os] }} ${{ matrix.python-version }} ${{ matrix.nox-session }}
+    name: ${{ fromJson('{"macos-13":"macOS","windows-2022":"Windows","ubuntu-latest":"Ubuntu","ubuntu-20.04":"Ubuntu 20.04 (OpenSSL 1.1.1)","ubuntu-latest":"Ubuntu Latest (OpenSSL 3+)"}')[matrix.os] }} ${{ matrix.python-version }} ${{ matrix.nox-session }}
     continue-on-error: ${{ matrix.experimental }}
     timeout-minutes: 40
     steps:

CHANGES.rst

@@ -1,10 +1,25 @@
-2.10.900 (2024-09-??)
+2.10.900 (2024-10-06)
 =====================
 
 - Added complete support for Informational Response whether it's an early response or not. We introduced a callback named
   ``on_early_response`` that takes exactly one parameter, namely a ``HTTPResponse``. You may start leveraging Early Hints!
   This works regardless of the negotiated protocol: HTTP/1.1, HTTP/2 or HTTP/3! As always, you may use that feature
   in a synchronous or asynchronous context.
+- Changed ``qh3`` lower bound version to v1.2 in order to support Informational Response in HTTP/3 also.
+- Added full automated support for WebSocket through HTTP/1.1, HTTP/2 or HTTP/3.
+  In order to leverage this feature, urllib3-future now recognize url scheme ``ws://`` (insecure) and ``wss://`` (secure).
+  The response will be of status 101 (Switching Protocol) and the body will be None.
+  Most servers out there only support WebSocket through HTTP/1.1, and using HTTP/2 or HTTP/3 usually ends up in stream (reset) error.
+  By default, connecting to ``wss://`` or ``ws://`` use HTTP/1.1, but if you desire to leverage the WebSocket through a multiplexed connection,
+  use ``wss+rfc8441://`` or ``ws+rfc8441://``.
+  A new property has been introduced in ``HTTPResponse``, namely ``extension`` to be able to interact with the websocket
+  server. Everything is handled automatically, from thread safety to all the protocol logic. See the documentation for more.
+  This will require the installation of an optional dependency ``wsproto``, to do so, please install urllib3-future with
+  ``pip install urllib3-future[ws]``.
+- Fixed a rare issue where the ``:authority`` (special header) value might be malformed.
+- Fixed an issue where passing ``Upgrade: websocket`` would be discarded without effect, thus smuggling the original user
+  intent. This is an issue due to our strict backward compatibility with our predecessor. Now, passing this header
+  will automatically disable HTTP/2 and HTTP/3 support for the given request.
 
 2.9.900 (2024-09-24)
 ====================

README.md

@@ -30,6 +30,7 @@
 - Support for Python/PyPy 3.7+, no compromise.
 - Early (Informational) Responses / Hints.
 - HTTP/1.1, HTTP/2 and HTTP/3 support.
+- WebSocket over HTTP/2+ (RFC8441).
 - Proxy support for HTTP and SOCKS.
 - Post-Quantum Security with QUIC.
 - Detailed connection inspection.
@@ -38,6 +39,7 @@
 - Mirrored Sync & Async.
 - Trailer Headers.
 - Amazingly Fast.
+- WebSocket.
 
 urllib3.future is powerful and easy to use:
 

docs/advanced-usage.rst

@@ -1556,3 +1556,33 @@ This example works whether you enable manual multiplexing or using asyncio.
 .. note:: The status 101 (Switching Protocol) is never considered "early", therefor "response" will be the 101 one and "early_response" will be worth "None".
 
 .. note:: Any responses yielded through the "on_early_response" callback will never have a body per standards.
+
+Switching Protocol
+------------------
+
+.. note:: Available since urllib3-future version 2.10 or greater.
+
+Manually passing from HTTP to a sub protocol can be achieved easily thanks to our ``DirectStreamAccess`` policy.
+If, for any reason, you wanted to negotiate ``WebSocket`` manually or any other proprietary protocol after receiving
+a ``101 Switching Protocol`` response or alike; you may access the RawExtensionFromHTTP that is available on your
+response object.
+
+.. code-block:: python
+
+    import urllib3
+
+    with urllib3.PoolManager() as pm:
+        resp = pm.urlopen("GET", "https://example.tld", headers={...})
+
+        print(resp.status)  # output '101' for 'Switching Protocol' response status
+
+        print(resp.extension)  # output <class 'urllib3.contrib.webextensions.RawExtensionFromHTTP'>
+
+        print(resp.extension.next_payload())  # read from the stream
+        resp.extension.send_payload(b"..")  # write in the stream
+
+        # gracefully close the sub protocol.
+        resp.extension.close()
+
+
+.. note:: The important thing here, is that, when the server agrees to stop speaking HTTP in favor of something else, the ``response.extension`` is set and you will be able to exchange raw data at will.

docs/conf.py

@@ -158,4 +158,6 @@
     ("py:attr", "HTTPResponse.data"),
     ("py:class", "_TYPE_PEER_CERT_RET_DICT"),
     ("py:class", "_TYPE_ASYNC_BODY"),
+    ("py:class", "ExtensionFromHTTP"),
+    ("py:class", "AsyncExtensionFromHTTP"),
 ]

docs/user-guide.rst

@@ -423,6 +423,70 @@ recommended to set the ``Content-Type`` header:
 
 .. _ssl:
 
+WebSocket
+---------
+
+Using non-multiplexed mode
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. note:: Available since urllib3-future version 2.10 or greater by installing urllib3-future with the ``ws`` extra. Like this: ``pip install urllib3-future[ws]`` or by installing ``wsproto`` by itself.
+
+The WebSocket protocol is an extremely popular extension of HTTP nowadays, and thanks to
+latest achievements in urllib3-future, we're able to serve that capability without even
+breaking a sweat!
+
+In the following example, we will explore how to interact with a basic, but well known echo server.
+
+.. code-block:: python
+
+    import urllib3
+
+    with urllib3.PoolManager() as pm:
+        resp = pm.urlopen("GET", "wss://echo.websocket.org")  # be sure to have installed the required extra prior to this.
+
+        print(resp.status)  # output '101' for 'Switching Protocol' response status
+
+        print(resp.extension)  # output <class 'urllib3.contrib.webextensions.WebSocketExtensionFromHTTP'>
+
+        print(resp.extension.next_payload())  # output a greeting message from the echo webserver.
+
+        # send two example payloads, one of type string, one of type bytes.
+        resp.extension.send_payload("Hello World!")
+        resp.extension.send_payload(b"Foo Bar Baz!")
+
+        # they should be echoed in order.
+        assert resp.extension.next_payload() == "Hello World!"
+        assert resp.extension.next_payload() == b"Foo Bar Baz!"
+
+        resp.extension.ping()  # send a ping to server
+
+        # gracefully close the sub protocol.
+        resp.extension.close()
+
+That is it! That easy.
+
+.. note:: Historically, urllib3 only accepted ``http://`` and ``https://`` as schemes. But now, you may use ``wss://`` for WebSocket Secure or ``ws://`` for WebSocket over PlainText.
+
+.. warning:: In case anything goes wrong (e.g. server denies us access), ``resp.extension`` will be worth ``None``! Be careful.
+
+Using multiplexed mode
+~~~~~~~~~~~~~~~~~~~~~~
+
+urllib3-future can leverage a multiplexed connection using HTTP/2 or HTTP/3, but often enough, server aren't quite ready
+to bootstrap WebSocket over HTTP/2 or HTTP/3.
+
+For this exact reason, we won't try to negotiate WebSocket over HTTP/2 or HTTP/3 by default. But if you were
+aware of a particular server capable of it, you would simply do as follow:
+
+.. code-block:: python
+
+    import urllib3
+
+    with urllib3.PoolManager() as pm:
+        resp = pm.urlopen("GET", "wss+rfc8441://example.test")
+
+The rest of the code is identical to the previous subsection. You may also append ``multiplexed=True`` to urlopen.
+
 Certificate Verification
 ------------------------
 

mypy-requirements.txt

@@ -12,3 +12,4 @@ qh3>=1.0.1,<2.0.0
 h11>=0.11.0,<1.0.0
 jh2>=5.0.0,<6.0.0
 python_socks>=2.0,<3.0
+wsproto>=1.2,<2

noxfile.py

@@ -196,7 +196,7 @@ def traefik_boot(session: nox.Session) -> typing.Generator[None, None, None]:
 
 def tests_impl(
     session: nox.Session,
-    extras: str = "socks,brotli,zstd",
+    extras: str = "socks,brotli,zstd,ws",
     byte_string_comparisons: bool = False,
 ) -> None:
     with traefik_boot(session):
@@ -468,7 +468,7 @@ def mypy(session: nox.Session) -> None:
 @nox.session
 def docs(session: nox.Session) -> None:
     session.install("-r", "docs/requirements.txt")
-    session.install(".[socks,brotli,zstd]")
+    session.install(".[socks,brotli,zstd,ws]")
 
     session.chdir("docs")
     if os.path.exists("_build"):

pyproject.toml

@@ -40,7 +40,7 @@ classifiers = [
 requires-python = ">=3.7"
 dynamic = ["version"]
 dependencies = [
-  "qh3>=1.0.3,<2.0.0; (platform_python_implementation != 'CPython' or python_full_version > '3.7.10') and (platform_system == 'Darwin' or platform_system == 'Windows' or platform_system == 'Linux') and (platform_machine == 'x86_64' or platform_machine == 's390x' or platform_machine == 'aarch64' or platform_machine == 'armv7l' or platform_machine == 'ppc64le' or platform_machine == 'ppc64' or platform_machine == 'AMD64' or platform_machine == 'arm64' or platform_machine == 'ARM64') and (platform_python_implementation == 'CPython' or (platform_python_implementation == 'PyPy' and python_version < '3.11'))",
+  "qh3>=1.2.0,<2.0.0; (platform_python_implementation != 'CPython' or python_full_version > '3.7.10') and (platform_system == 'Darwin' or platform_system == 'Windows' or platform_system == 'Linux') and (platform_machine == 'x86_64' or platform_machine == 's390x' or platform_machine == 'aarch64' or platform_machine == 'armv7l' or platform_machine == 'ppc64le' or platform_machine == 'ppc64' or platform_machine == 'AMD64' or platform_machine == 'arm64' or platform_machine == 'ARM64') and (platform_python_implementation == 'CPython' or (platform_python_implementation == 'PyPy' and python_version < '3.11'))",
   "h11>=0.11.0,<1.0.0",
   "jh2>=5.0.3,<6.0.0",
 ]
@@ -60,6 +60,9 @@ socks = [
 qh3 = [
   "qh3>=1.0.3,<2.0.0",
 ]
+ws = [
+  "wsproto>=1.2,<2",
+]
 
 [project.urls]
 "Changelog" = "https://github.com/jawah/urllib3.future/blob/main/CHANGES.rst"

src/urllib3/_async/connectionpool.py

@@ -32,6 +32,7 @@
     AsyncManyResolver,
     AsyncResolverDescription,
 )
+from ..contrib.webextensions._async import load_extension
 from ..exceptions import (
     BaseSSLError,
     ClosedPoolError,
@@ -76,6 +77,8 @@
 
     from typing_extensions import Literal
 
+    from ..contrib.webextensions._async import AsyncExtensionFromHTTP
+
 log = logging.getLogger(__name__)
 
 _SelfT = typing.TypeVar("_SelfT")
@@ -849,6 +852,17 @@ async def get_response(
 
             return await self.get_response(promise=new_promise if promise else None)
 
+        extension = from_promise.get_parameter("extension")
+
+        if extension is not None:
+            if response.status == 101 or (
+                200 <= response.status < 300 and method == "CONNECT"
+            ):
+                if extension is None:
+                    extension = load_extension(None)()
+
+                await response.start_extension(extension)
+
         return response
 
     @typing.overload
@@ -873,6 +887,7 @@ async def _make_request(
         ] = ...,
         on_early_response: typing.Callable[[AsyncHTTPResponse], typing.Awaitable[None]]
         | None = ...,
+        extension: AsyncExtensionFromHTTP | None = ...,
         *,
         multiplexed: Literal[True],
     ) -> ResponsePromise:
@@ -900,6 +915,7 @@ async def _make_request(
         ] = ...,
         on_early_response: typing.Callable[[AsyncHTTPResponse], typing.Awaitable[None]]
         | None = ...,
+        extension: AsyncExtensionFromHTTP | None = ...,
         *,
         multiplexed: Literal[False] = ...,
     ) -> AsyncHTTPResponse:
@@ -927,6 +943,7 @@ async def _make_request(
         | None = None,
         on_early_response: typing.Callable[[AsyncHTTPResponse], typing.Awaitable[None]]
         | None = None,
+        extension: AsyncExtensionFromHTTP | None = None,
         multiplexed: Literal[False] | Literal[True] = False,
     ) -> AsyncHTTPResponse | ResponsePromise:
         """
@@ -1047,6 +1064,31 @@ async def _make_request(
             # overruling
             multiplexed = False
 
+        if (
+            extension is not None
+            and conn.conn_info is not None
+            and conn.conn_info.http_version is not None
+        ):
+            extension_headers = extension.headers(conn.conn_info.http_version)
+
+            if extension_headers:
+                if headers is None:
+                    headers = extension_headers
+                elif hasattr(headers, "copy"):
+                    headers = headers.copy()
+                    headers.update(extension_headers)  # type: ignore[union-attr]
+                else:
+                    merged_headers = HTTPHeaderDict()
+
+                    for k, v in headers.items():
+                        merged_headers.add(k, v)
+                    for k, v in extension_headers.items():
+                        merged_headers.add(k, v)
+
+                    headers = merged_headers
+        else:
+            extension = None
+
         try:
             rp = await conn.request(
                 method,
@@ -1080,6 +1122,7 @@ async def _make_request(
                 raise OSError
             rp.set_parameter("read_timeout", read_timeout)
             rp.set_parameter("on_early_response", on_early_response)
+            rp.set_parameter("extension", extension)
             return rp
 
         if not conn.is_closed:
@@ -1111,6 +1154,13 @@ async def _make_request(
         response.retries = retries
         response._pool = self
 
+        if response.status == 101 or (
+            200 <= response.status < 300 and method == "CONNECT"
+        ):
+            if extension is None:
+                extension = load_extension(None)()
+            await response.start_extension(extension)
+
         log.debug(
             '%s://%s:%s "%s %s %s" %s %s',
             self.scheme,
@@ -1189,6 +1239,7 @@ async def urlopen(
         ] = ...,
         on_early_response: typing.Callable[[AsyncHTTPResponse], typing.Awaitable[None]]
         | None = ...,
+        extension: AsyncExtensionFromHTTP | None = ...,
         *,
         multiplexed: Literal[False] = ...,
         **response_kw: typing.Any,
@@ -1219,6 +1270,7 @@ async def urlopen(
         ] = ...,
         on_early_response: typing.Callable[[AsyncHTTPResponse], typing.Awaitable[None]]
         | None = ...,
+        extension: AsyncExtensionFromHTTP | None = ...,
         *,
         multiplexed: Literal[True],
         **response_kw: typing.Any,
@@ -1249,6 +1301,7 @@ async def urlopen(
         | None = None,
         on_early_response: typing.Callable[[AsyncHTTPResponse], typing.Awaitable[None]]
         | None = None,
+        extension: AsyncExtensionFromHTTP | None = None,
         multiplexed: bool = False,
         **response_kw: typing.Any,
     ) -> AsyncHTTPResponse | ResponsePromise:
@@ -1470,6 +1523,7 @@ async def urlopen(
                 on_post_connection=on_post_connection,
                 on_upload_body=on_upload_body,
                 on_early_response=on_early_response,
+                extension=extension,
                 multiplexed=multiplexed,
             )