13.1: pytest fails in 3 units

[kloczek]

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix> using installer module
• run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
• build is performed in env which is cut off from access to the public network (pytest is executed with -m "not network")

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-websockets-13.1-2.fc37.x86_64/usr/lib64/python3.10/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-websockets-13.1-2.fc37.x86_64/usr/lib/python3.10/site-packages
+ /usr/bin/pytest -ra -m 'not network' tests
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.10.14, pytest-8.2.2, pluggy-1.5.0
rootdir: /home/tkloczko/rpmbuild/BUILD/websockets-13.1
configfile: pyproject.toml
collected 1725 items

tests/asyncio/test_client.py .............................................                                                                                                            [  2%]
tests/asyncio/test_connection.py ...........s.ss.....s.............F.........................................................................................s.ss.....s.........s..sF [ 11%]
s.............................................................................                                                                                                        [ 15%]
tests/asyncio/test_messages.py ....................................................                                                                                                   [ 18%]
tests/asyncio/test_server.py .........F.................................................                                                                                              [ 22%]
tests/extensions/test_base.py .....                                                                                                                                                   [ 22%]
tests/extensions/test_permessage_deflate.py ...............................                                                                                                           [ 24%]
tests/legacy/test_auth.py ..................                                                                                                                                          [ 25%]
tests/legacy/test_client_server.py .................................................................................................................................................. [ 33%]
......................................................                                                                                                                                [ 36%]
tests/legacy/test_exceptions.py .                                                                                                                                                     [ 36%]
tests/legacy/test_framing.py ...................................                                                                                                                      [ 38%]
tests/legacy/test_handshake.py ........................                                                                                                                               [ 40%]
tests/legacy/test_http.py ...................                                                                                                                                         [ 41%]
tests/legacy/test_protocol.py .........s.ss.....s..............................................................................................................................s.ss.. [ 50%]
...s.....................................................................................................................                                                             [ 57%]
tests/sync/test_client.py ..............................                                                                                                                              [ 58%]
tests/sync/test_connection.py ...................................s..............................................s..s.....................s....................................        [ 67%]
tests/sync/test_messages.py ..................................                                                                                                                        [ 69%]
tests/sync/test_server.py ................................................                                                                                                            [ 72%]
tests/sync/test_utils.py ....                                                                                                                                                         [ 72%]
tests/test_client.py ..............................................                                                                                                                   [ 74%]
tests/test_connection.py .                                                                                                                                                            [ 75%]
tests/test_datastructures.py .................................................                                                                                                        [ 77%]
tests/test_exceptions.py ...                                                                                                                                                          [ 78%]
tests/test_exports.py ..                                                                                                                                                              [ 78%]
tests/test_frames.py .....................................................                                                                                                            [ 81%]
tests/test_headers.py ..................                                                                                                                                              [ 82%]
tests/test_http.py .                                                                                                                                                                  [ 82%]
tests/test_http11.py ...............................                                                                                                                                  [ 84%]
tests/test_imports.py ....                                                                                                                                                            [ 84%]
tests/test_protocol.py .............................................................................................................................................................. [ 93%]
........................                                                                                                                                                              [ 94%]
tests/test_server.py ......................................................                                                                                                           [ 98%]
tests/test_streams.py ..................                                                                                                                                              [ 99%]
tests/test_uri.py ....                                                                                                                                                                [ 99%]
tests/test_utils.py ............                                                                                                                                                      [100%]

========================================================================================= FAILURES ==========================================================================================
__________________________________________________________________ ClientConnectionTests.test_close_waits_for_close_frame ___________________________________________________________________

self = <tests.asyncio.test_connection.ClientConnectionTests testMethod=test_close_waits_for_close_frame>

    async def test_close_waits_for_close_frame(self):
        """close waits for a close frame (then EOF) before returning."""
        async with self.delay_frames_rcvd(MS), self.delay_eof_rcvd(MS):
            await self.connection.close()

        with self.assertRaises(ConnectionClosedOK) as raised:
            await self.connection.recv()

        exc = raised.exception
        self.assertEqual(str(exc), "sent 1000 (OK); then received 1000 (OK)")
>       self.assertIsNone(exc.__cause__)
E       AssertionError: TimeoutError('timed out while closing connection') is not None

tests/asyncio/test_connection.py:663: AssertionError
__________________________________________________________________ ServerConnectionTests.test_close_waits_for_close_frame ___________________________________________________________________

self = <tests.asyncio.test_connection.ServerConnectionTests testMethod=test_close_waits_for_close_frame>

    async def test_close_waits_for_close_frame(self):
        """close waits for a close frame (then EOF) before returning."""
        async with self.delay_frames_rcvd(MS), self.delay_eof_rcvd(MS):
            await self.connection.close()

        with self.assertRaises(ConnectionClosedOK) as raised:
            await self.connection.recv()

        exc = raised.exception
        self.assertEqual(str(exc), "sent 1000 (OK); then received 1000 (OK)")
>       self.assertIsNone(exc.__cause__)
E       AssertionError: TimeoutError('timed out while closing connection') is not None

tests/asyncio/test_connection.py:663: AssertionError
___________________________________________________________________ ServerTests.test_close_server_keeps_handlers_running ____________________________________________________________________

self = <tests.asyncio.test_server.ServerTests testMethod=test_close_server_keeps_handlers_running>

    async def test_close_server_keeps_handlers_running(self):
        """Server waits for connection handlers to terminate."""
        async with serve(*args) as server:
            async with connect(get_uri(server) + "/delay") as client:
                # Delay termination of connection handler.
                await client.send(str(3 * MS))

                server.close()

                # The server waits for the connection handler to terminate.
                with self.assertRaises(TimeoutError):
                    async with asyncio_timeout(2 * MS):
                        await server.wait_closed()

                async with asyncio_timeout(3 * MS):
>                   await server.wait_closed()

tests/asyncio/test_server.py:500:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <websockets.asyncio.server.Server object at 0x7f0be1375930>

    async def wait_closed(self) -> None:
        """
        Wait until the server is closed.

        When :meth:`wait_closed` returns, all TCP connections are closed and
        all connection handlers have returned.

        To ensure a fast shutdown, a connection handler should always be
        awaiting at least one of:

        * :meth:`~ServerConnection.recv`: when the connection is closed,
          it raises :exc:`~websockets.exceptions.ConnectionClosedOK`;
        * :meth:`~ServerConnection.wait_closed`: when the connection is
          closed, it returns.

        Then the connection handler is immediately notified of the shutdown;
        it can clean up and exit.

        """
>       await asyncio.shield(self.closed_waiter)
E       asyncio.exceptions.CancelledError

../../BUILDROOT/python-websockets-13.1-2.fc37.x86_64/usr/lib64/python3.10/site-packages/websockets/asyncio/server.py:492: CancelledError

During handling of the above exception, another exception occurred:

self = <tests.asyncio.test_server.ServerTests testMethod=test_close_server_keeps_handlers_running>

    async def test_close_server_keeps_handlers_running(self):
        """Server waits for connection handlers to terminate."""
        async with serve(*args) as server:
            async with connect(get_uri(server) + "/delay") as client:
                # Delay termination of connection handler.
                await client.send(str(3 * MS))

                server.close()

                # The server waits for the connection handler to terminate.
                with self.assertRaises(TimeoutError):
                    async with asyncio_timeout(2 * MS):
                        await server.wait_closed()

>               async with asyncio_timeout(3 * MS):

tests/asyncio/test_server.py:499:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../BUILDROOT/python-websockets-13.1-2.fc37.x86_64/usr/lib64/python3.10/site-packages/websockets/asyncio/async_timeout.py:181: in __aexit__
    self._do_exit(exc_type)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <websockets.asyncio.async_timeout.Timeout object at 0x7f0be129e520>, exc_type = <class 'asyncio.exceptions.CancelledError'>

    def _do_exit(self, exc_type: Optional[Type[BaseException]]) -> None:
        if exc_type is asyncio.CancelledError and self._state == _State.TIMEOUT:
            assert self._task is not None
            _uncancel_task(self._task)
            self._timeout_handler = None
            self._task = None
>           raise asyncio.TimeoutError
E           asyncio.exceptions.TimeoutError

../../BUILDROOT/python-websockets-13.1-2.fc37.x86_64/usr/lib64/python3.10/site-packages/websockets/asyncio/async_timeout.py:268: TimeoutError
================================================================================== short test summary info ==================================================================================
SKIPPED [2] tests/asyncio/test_connection.py:1168: raise_exceptions requires Python 3.11+
SKIPPED [2] tests/asyncio/test_connection.py:1277: raise_exceptions requires Python 3.11+
SKIPPED [2] tests/asyncio/test_connection.py:1237: raise_exceptions requires Python 3.11+
SKIPPED [2] tests/asyncio/test_connection.py:1154: raise_exceptions requires Python 3.11+
SKIPPED [1] tests/asyncio/test_connection.py:694: only relevant on the client-side
SKIPPED [1] tests/asyncio/test_connection.py:723: only relevant on the client-side
SKIPPED [1] tests/asyncio/test_connection.py:665: only relevant on the client-side
SKIPPED [2] tests/legacy/test_protocol.py:1477: raise_exceptions requires Python 3.11+
SKIPPED [2] tests/legacy/test_protocol.py:1560: raise_exceptions requires Python 3.11+
SKIPPED [2] tests/legacy/test_protocol.py:1530: raise_exceptions requires Python 3.11+
SKIPPED [2] tests/legacy/test_protocol.py:1465: raise_exceptions requires Python 3.11+
SKIPPED [2] tests/sync/test_connection.py:735: works only on BSD
SKIPPED [1] tests/sync/test_connection.py:458: only relevant on the client-side
SKIPPED [1] tests/sync/test_connection.py:431: only relevant on the client-side
FAILED tests/asyncio/test_connection.py::ClientConnectionTests::test_close_waits_for_close_frame - AssertionError: TimeoutError('timed out while closing connection') is not None
FAILED tests/asyncio/test_connection.py::ServerConnectionTests::test_close_waits_for_close_frame - AssertionError: TimeoutError('timed out while closing connection') is not None
FAILED tests/asyncio/test_server.py::ServerTests::test_close_server_keeps_handlers_running - asyncio.exceptions.TimeoutError
======================================================================== 3 failed, 1699 passed, 23 skipped in 8.63s =========================================================================

List of installed modules in build env:

Package                       Version
----------------------------- -----------
alabaster                     0.7.16
Babel                         2.15.0
build                         1.2.2
charset-normalizer            3.3.2
contourpy                     1.2.1
cycler                        0.12.1
defusedxml                    0.7.1
distro                        1.9.0
docutils                      0.20.1
exceptiongroup                1.1.3
fonttools                     4.53.1
imagesize                     1.4.1
importlib_metadata            8.0.0
iniconfig                     2.0.0
installer                     0.7.0
Jinja2                        3.1.4
kiwisolver                    1.4.5
MarkupSafe                    2.1.5
matplotlib                    3.9.0
numpy                         1.26.4
olefile                       0.47
packaging                     24.0
pillow                        10.4.0
pluggy                        1.5.0
pyenchant                     3.2.2
Pygments                      2.18.0
pyparsing                     3.1.2
pyproject_hooks               1.0.0
pytest                        8.2.2
python-dateutil               2.9.0.post0
requests                      2.32.3
setuptools                    75.0.0
snowballstemmer               2.2.0
Sphinx                        8.0.2
sphinx-copybutton             0.5.2
sphinx_inline_tabs            2023.4.21
sphinxcontrib-applehelp       2.0.0
sphinxcontrib-devhelp         1.0.6
sphinxcontrib-htmlhelp        2.1.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          2.0.0
sphinxcontrib-serializinghtml 2.0.0
sphinxcontrib-spelling        8.0.0
sphinxcontrib-trio            1.1.2
sphinxext-opengraph           0.9.1
tokenize_rt                   6.0.0
tomli                         2.0.1
urllib3                       2.2.2
wheel                         0.44.0
zipp                          3.19.2

Please let me know if you need more details or want me to perform some diagnostics.

[kloczek]

After add those tree units to --deselect list ne units popped oud so looks like there are some dependencies between units (probably it would be good to test test suite using https://github.com/mrbean-bremen/pytest-find-dependencies/).
I was able to pass pytest only with below list of deselected units

        --deselect tests/asyncio/test_client.py::ClientTests::test_keepalive_is_enabled \
        --deselect tests/asyncio/test_connection.py::ClientConnectionTests::test_close_waits_for_close_frame \
        --deselect tests/asyncio/test_connection.py::ClientConnectionTests::test_close_waits_for_connection_closed \
        --deselect tests/asyncio/test_connection.py::ServerConnectionTests::test_close_waits_for_close_frame \
        --deselect tests/asyncio/test_server.py::ServerTests::test_close_server_keeps_handlers_running \

[aaugustin]

run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

☝️ Just skip this step — it doesn't serve a purpose.

[aaugustin]

Probably you're taking this step because you believe that it serves a purpose. We can disagree on this.

As consequence of this disagreement, I'm choosing not to invest my time into debugging this problem.

Perhaps you have constraints like "if a git repository related to the library contains various test suites, you must pick at least one of them and run it, although it doesn't matter very much which one you choose". (There are three test suites in the repo.) If you have that kind of constraint, then the test suite that you chose is the easiest to run, so stick with that one, even if it's completely inadequate to the environment where you're running it.

You may re-run until you get a non-flaky run. You may or may not get a successful run eventually, depending on details of your environment. You may try setting the WEBSOCKETS_TESTS_TIMEOUT_FACTOR environment variable to 100. I just pushed a change that will do this automatically in the next release.

[aaugustin]

looks like there are some dependencies between units

I don't think that's the problem. The problem is that I'm using extremely short timings for testing in order to keep the test suite fast; obviously that makes it flaky if you run it in a slower or less stable environment. All failing tests that you observe involve timings or timeouts.

[kloczek]

I've tested with WEBSOCKETS_TESTS_TIMEOUT_FACTOR=100 and pytest os OK.
In other words it is issue with to short timeout.