Make all LocalCluster tests async

[pentschev]

We should make all LocalCluster tests async as it's generally simpler to test (e.g., to have timeouts) and generally more in line with real use-cases. Parts of that were addressed in #1084 and #1086 .

Some of the tests are a bit challenging though, for instance attempting to move

dask-cuda/dask_cuda/tests/test_explicit_comms.py

Lines 333 to 339 in 52dd850

	with LocalCluster(
	protocol="tcp",
	dashboard_address=None,
	n_workers=4,
	threads_per_worker=5,
	processes=True,
	) as cluster:

to async had processes failing to connect, I'm not sure why.

Also most of the tests in https://github.com/rapidsai/dask-cuda/blob/branch-23.02/dask_cuda/tests/test_explicit_comms.py are executed in a child process but a quick glance makes me think that's not necessary. Could you remind me why that is that case @madsbk ?

[madsbk]

Also most of the tests in https://github.com/rapidsai/dask-cuda/blob/branch-23.02/dask_cuda/tests/test_explicit_comms.py are executed in a child process but a quick glance makes me think that's not necessary. Could you remind me why that is that case @madsbk ?

We did it to make the tests handle UCX segfaulting, something that happen a lot in the early days :)
But I think you are right, we hopefully doesn't need it anymore :)