deprecate passing stopped loops to LoopRunner (and therefore Client/Cluster) #6680

graingert · 2022-07-06T14:55:18Z

Tests added / passed
Passes pre-commit run --all-files

github-actions · 2022-07-06T18:08:11Z

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

      15 files ±  0       15 suites ±0 6h 30m 25s ⏱️ + 25m 31s
  2 979 tests +  3   2 890 ✔️ +  2     87 💤 ±0 2 ❌ +1
22 086 runs +22 21 053 ✔️ +21 1 031 💤 +1 2 ❌ ±0

For more details on these failures, see this check.

Results for commit 24ea2d5. ± Comparison against base commit a53858a.

♻️ This comment has been updated with latest results.

graingert · 2022-07-07T10:47:13Z

Tests / test (macos-latest, 3.8, not ci1) (pull_request) #6683

…ationWarning

…alled

graingert · 2022-07-27T09:56:51Z

the test_ssh tests are xfailed on macos, however because they are xfailed rather than skipped they still run, and when they fail they leave an unclosed SSHCluster object alive. Which is then garbage collected in the scope of the next test:

macos-latest 3.8 not c1:

=================================== FAILURES ===================================
___________________________ test_cluster_dump_plugin ___________________________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7fdc2d5d8940>
when = 'call'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], TResult]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        """Call func, wrapping the result in a CallInfo.
    
        :param func:
            The function to call. Called without arguments.
        :param when:
            The phase in which the function is called.
        :param reraise:
            Exception or exceptions that shall propagate if raised by the
            function, instead of being wrapped in the CallInfo.
        """
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

../../../miniconda3/envs/dask-distributed/lib/python3.8/site-packages/_pytest/runner.py:338: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../miniconda3/envs/dask-distributed/lib/python3.8/site-packages/_pytest/runner.py:259: in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
../../../miniconda3/envs/dask-distributed/lib/python3.8/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
../../../miniconda3/envs/dask-distributed/lib/python3.8/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
../../../miniconda3/envs/dask-distributed/lib/python3.8/site-packages/_pytest/unraisableexception.py:88: in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            yield
            if cm.unraisable:
                if cm.unraisable.err_msg is not None:
                    err_msg = cm.unraisable.err_msg
                else:
                    err_msg = "Exception ignored in"
                msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                msg += "".join(
                    traceback.format_exception(
                        cm.unraisable.exc_type,
                        cm.unraisable.exc_value,
                        cm.unraisable.exc_traceback,
                    )
                )
>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Cluster.__del__ at 0x7fdc4ef0ac10>
E               
E               Traceback (most recent call last):
E                 File "/Users/runner/work/distributed/distributed/distributed/deploy/cluster.py", line 223, in __del__
E                   _warn(f"unclosed cluster {self_r}", ResourceWarning, source=self)
E               ResourceWarning: unclosed cluster SSHCluster(SSHCluster, 'tcp://10.212.0.104:54852', workers=0, threads=0, memory=0 B)

../../../miniconda3/envs/dask-distributed/lib/python3.8/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning
----------------------------- Captured stderr call -----------------------------
2022-07-26 22:12:54,678 - distributed.client - ERROR - 
Traceback (most recent call last):
  File "/Users/runner/work/distributed/distributed/distributed/utils.py", line 799, in wrapper
    return await func(*args, **kwargs)
  File "/Users/runner/work/distributed/distributed/distributed/client.py", line 1243, in _reconnect
    await self._ensure_connected(timeout=timeout)
  File "/Users/runner/work/distributed/distributed/distributed/client.py", line 1273, in _ensure_connected
    comm = await connect(
  File "/Users/runner/work/distributed/distributed/distributed/comm/core.py", line 291, in connect
    comm = await asyncio.wait_for(
  File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/asyncio/tasks.py", line 481, in wait_for
    await waiter
asyncio.exceptions.CancelledError
------------------------------ Captured log call -------------------------------
ERROR    asyncio.events:utils.py:825 
Traceback (most recent call last):
  File "/Users/runner/work/distributed/distributed/distributed/utils.py", line 799, in wrapper
    return await func(*args, **kwargs)
  File "/Users/runner/work/distributed/distributed/distributed/client.py", line 1432, in _handle_report
    await self._reconnect()
  File "/Users/runner/work/distributed/distributed/distributed/utils.py", line 799, in wrapper
    return await func(*args, **kwargs)
  File "/Users/runner/work/distributed/distributed/distributed/client.py", line 1243, in _reconnect
    await self._ensure_connected(timeout=timeout)
  File "/Users/runner/work/distributed/distributed/distributed/client.py", line 1273, in _ensure_connected
    comm = await connect(
  File "/Users/runner/work/distributed/distributed/distributed/comm/core.py", line 291, in connect
    comm = await asyncio.wait_for(
  File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/asyncio/tasks.py", line 481, in wait_for
    await waiter
asyncio.exceptions.CancelledError
- generated xml file: /Users/runner/work/distributed/distributed/reports/pytest.xml -
================================ RESOURCE LEAKS ================================

distributed/deploy/tests/test_ssh.py::test_list_of_remote_python_raises XPASS [ 24%]
distributed/diagnostics/tests/test_cluster_dump_plugin.py::test_cluster_dump_plugin FAILED [ 24%]

graingert · 2022-07-27T09:59:15Z

I'm seeing: test (macos-latest, 3.8, ci1)

__________________ test_restart_timeout_on_long_running_task ___________________

args = (), kwds = {}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)

  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Couldn't use data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.29793.407325': database disk image is malformed
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Couldn't use data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.19630.300069': database disk image is malformed
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.47663.687920' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Couldn't use data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.45900.171500': database disk image is malformed
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.33602.172653' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.20947.592818' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.25153.883363' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.79538.320101' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.45069.942084' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.47617.189130' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.26817.002892' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Couldn't use data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.27732.904873': database disk image is malformed
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.28136.201786' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.35809.607860' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.25925.321607' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.23827.869941' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.42560.328112' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Couldn't use data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.36545.463240': database disk image is malformed
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.46218.289226' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.29005.480101' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.19256.528728' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.42073.315280' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.19186.133119' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.48057.069764' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.78896.495836' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Couldn't use data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.32370.578552': database disk image is malformed
  data._warn(str(exc))
/Users/runner/miniconda3/envs/dask-distributed/lib/python3.8/site-packages/coverage/data.py:130: CoverageWarning: Data file '/Users/runner/work/distributed/distributed/.coverage.Mac-1658869966720.local.26028.054393' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
  data._warn(str(exc))
../../../miniconda3/envs/dask-distributed/lib/python3.8/contextlib.py:75: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
distributed/utils_test.py:1170: in test_func
    return _run_and_close_tornado(async_fn_outer)
distributed/utils_test.py:451: in _run_and_close_tornado
    return asyncio.run(inner_fn())
../../../miniconda3/envs/dask-distributed/lib/python3.8/asyncio/runners.py:44: in run
    return loop.run_until_complete(main)
../../../miniconda3/envs/dask-distributed/lib/python3.8/asyncio/base_events.py:616: in run_until_complete
    return future.result()
distributed/utils_test.py:448: in inner_fn
    return await async_fn(*args, **kwargs)
distributed/utils_test.py:1167: in async_fn_outer
    return await asyncio.wait_for(async_fn(), timeout=timeout * 2)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

fut = <Task cancelled name='Task-60329' coro=<gen_cluster.<locals>._.<locals>.test_func.<locals>.async_fn() done, defined at /Users/runner/work/distributed/distributed/distributed/utils_test.py:1027>>
timeout = 120

    async def wait_for(fut, timeout, *, loop=None):
        """Wait for the single Future or coroutine to complete, with timeout.
    
        Coroutine will be wrapped in Task.
    
        Returns result of the Future or coroutine.  When a timeout occurs,
        it cancels the task and raises TimeoutError.  To avoid the task
        cancellation, wrap it in shield().
    
        If the wait is cancelled, the task is also cancelled.
    
        This function is a coroutine.
        """
        if loop is None:
            loop = events.get_running_loop()
        else:
            warnings.warn("The loop argument is deprecated since Python 3.8, "
                          "and scheduled for removal in Python 3.10.",
                          DeprecationWarning, stacklevel=2)
    
        if timeout is None:
            return await fut
    
        if timeout <= 0:
            fut = ensure_future(fut, loop=loop)
    
            if fut.done():
                return fut.result()
    
            await _cancel_and_wait(fut, loop=loop)
            try:
                fut.result()
            except exceptions.CancelledError as exc:
                raise exceptions.TimeoutError() from exc
            else:
                raise exceptions.TimeoutError()
    
        waiter = loop.create_future()
        timeout_handle = loop.call_later(timeout, _release_waiter, waiter)
        cb = functools.partial(_release_waiter, waiter)
    
        fut = ensure_future(fut, loop=loop)
        fut.add_done_callback(cb)
    
        try:
            # wait until the future completes or the timeout
            try:
                await waiter
            except exceptions.CancelledError:
                if fut.done():
                    return fut.result()
                else:
                    fut.remove_done_callback(cb)
                    # We must ensure that the task is not running
                    # after wait_for() returns.
                    # See https://bugs.python.org/issue32751
                    await _cancel_and_wait(fut, loop=loop)
                    raise
    
            if fut.done():
                return fut.result()
            else:
                fut.remove_done_callback(cb)
                # We must ensure that the task is not running
                # after wait_for() returns.
                # See https://bugs.python.org/issue32751
                await _cancel_and_wait(fut, loop=loop)
>               raise exceptions.TimeoutError()
E               asyncio.exceptions.TimeoutError

../../../miniconda3/envs/dask-distributed/lib/python3.8/asyncio/tasks.py:501: TimeoutError

distributed/client.py

distributed/deploy/tests/test_cluster.py

distributed/utils_test.py

gjoseph92

@graingert if you think this is ready, I'll merge

graingert · 2022-08-01T14:34:49Z

@gjoseph92 yeah I think this is ready, thanks!

hendrikmakait

LGTM, thanks!

…luster) (dask#6680)

graingert force-pushed the deprecate-loop-handling branch 2 times, most recently from 1bd380a to d017f39 Compare July 8, 2022 08:48

graingert added 12 commits July 8, 2022 14:39

deprecate incorrect usage of LoopRunner

bd6c7ec

avoid accessing self.loop in Client

dd0debd

run loop() fixtures' loop in a thread

865e62a

cache the loop property on first access

5d615d6

avoid deprecated io_loop property

208f67a

avoid deprecated use of LoopRunner in TextProgressBar

ab1cd03

test_client_async_before_loop_starts now intentionally warns a Deprec…

5072a20

…ationWarning

ignore deprecation warnings in LoopRunner tests

f8bb440

unignore deprecated asyncio.get_event_loop()

a056847

cluster attributes may not have been assigned by the time atexit is c…

31145dc

…alled

mark deprecated those tests that use loop=None

bc98b4e

test deprecated loop and io_loop properties

8a99fa3

graingert force-pushed the deprecate-loop-handling branch from d017f39 to 8a99fa3 Compare July 8, 2022 14:02

graingert mentioned this pull request Jul 26, 2022

Add Cluster.get_client() method #6745

Merged

3 tasks

Merge branch 'main' into deprecate-loop-handling

24ea2d5

graingert marked this pull request as ready for review July 27, 2022 09:59

graingert requested a review from gjoseph92 July 27, 2022 09:59

gjoseph92 reviewed Jul 28, 2022

View reviewed changes

distributed/client.py Show resolved Hide resolved

distributed/client.py Show resolved Hide resolved

distributed/deploy/tests/test_cluster.py Show resolved Hide resolved

distributed/deploy/tests/test_cluster.py Show resolved Hide resolved

distributed/utils_test.py Show resolved Hide resolved

gjoseph92 approved these changes Jul 28, 2022

View reviewed changes

hendrikmakait self-requested a review August 1, 2022 16:27

hendrikmakait approved these changes Aug 1, 2022

View reviewed changes

gjoseph92 merged commit 172e37f into dask:main Aug 1, 2022

jrbourbeau mentioned this pull request Aug 2, 2022

Import loop_in_thread fixture in tests dask/dask#9337

Merged

wence- mentioned this pull request Oct 13, 2022

Enable support for tornado 6.2 #7134

Closed

2 tasks

gjoseph92 pushed a commit to gjoseph92/distributed that referenced this pull request Oct 31, 2022

deprecate passing stopped loops to LoopRunner (and therefore Client/C…

110f444

…luster) (dask#6680)

fjetter mentioned this pull request Nov 2, 2022

Remove deprecated LoopRunner API #7239

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

deprecate passing stopped loops to LoopRunner (and therefore Client/Cluster) #6680

deprecate passing stopped loops to LoopRunner (and therefore Client/Cluster) #6680

graingert commented Jul 6, 2022

github-actions bot commented Jul 6, 2022 •

edited

Loading

graingert commented Jul 7, 2022

graingert commented Jul 27, 2022

graingert commented Jul 27, 2022

gjoseph92 left a comment

graingert commented Aug 1, 2022

hendrikmakait left a comment

deprecate passing stopped loops to LoopRunner (and therefore Client/Cluster) #6680

deprecate passing stopped loops to LoopRunner (and therefore Client/Cluster) #6680

Conversation

graingert commented Jul 6, 2022

github-actions bot commented Jul 6, 2022 • edited Loading

Unit Test Results

graingert commented Jul 7, 2022

graingert commented Jul 27, 2022

graingert commented Jul 27, 2022

gjoseph92 left a comment

Choose a reason for hiding this comment

graingert commented Aug 1, 2022

hendrikmakait left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 6, 2022 •

edited

Loading