[dask] fix teardown issues in Dask tests (fixes #3829)

[jameslamb]

See #3829 (comment) for an explanation of this pull request.

The tests results below should builds I used to try to establish how often this teardown timeout issue happens. See #3829 (comment) for a summary.

This PR proposes just increasing the timeout on the client.close() calls used in Dask tests, to 60 seconds. It also adds client.close() calls to two tests that were missing them.

---

Test runs with no changes to anything in python-package/ or tests/

sdist (5 Linux, 5 Linux_latest per build, Python=3.8)

build 1: ✔️ 10, ❌ 0
build 2: ✔️ 10, ❌ 0
build 3: ✔️ 10, ❌ 0
build 4: ✔️ 10, ❌ 0
build 5: ✔️ 10, ❌ 0

bdist (5 Linux, 5 Linux_latest per build, Python=3.8)

build 1: ✔️ 10, ❌ 0
build 2: ✔️ 9, ❌ 1

Linux bdist timeout failure (1)

• https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=8804&view=logs&j=d831b008-d707-5ae6-9e58-ffbfc237c21b&t=8ad88a41-8c9e-503c-488a-9cfbe17b63cd

build 3: ✔️ 10, ❌ 0
build 4: ✔️ 10, ❌ 0
build 5: ✔️ 10, ❌ 0

regular (5 Linux, 5 Linux_latest per build, Python=3.8)

build 1: ✔️ 10, ❌ 0
build 2: ✔️ 8, ❌ 2

Linux regular timeout failure (2)

• https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=8825&view=logs&j=c28dceab-947a-5848-c21f-eef3695e5f11&t=fa158246-17e2-53d4-5936-86070edbaacf
• https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=8825&view=logs&j=1ddbb484-dc4c-5a38-d907-cdcebe381ed8&t=ffac9979-7cb1-5d54-e6ca-05cb966029aa

build 3: ✔️ 10, ❌ 0
build 4: ✔️ 10, ❌ 0
build 5: ✔️ 10, ❌ 0

bdist (5 Linux, 5 Linux_latest per build, Python=3.7)

build 1: ✔️ 10, ❌ 0
build 2: ✔️ 10, ❌ 0
build 3: ✔️ 10, ❌ 0
build 4: ✔️ 9, ❌ 1

Linux bdist timeout failure (1)

• https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=8814&view=logs&j=0a34017b-57e8-5d4e-23c1-ac8583a4537b&t=8fbfafb9-b3a7-5aa8-b468-58ede22c65ed

build 5: ✔️ 10, ❌ 0

regular (5 Linux, 5 Linux_latest per build, Python=3.7)

build 1: ✔️ 10, ❌ 0
build 2: ✔️ 10, ❌ 0
build 3: ✔️ 10, ❌ 0
build 4: ✔️ 10, ❌ 0
build 5: ✔️ 10, ❌ 0

[jameslamb]

Based on #3829 (comment), I believe this is ready to review.

[StrikerRUS]

Maybe we can wrap client as a resource-like fixture?

@pytest.fixture()
def resource():
    print("setup")
    yield "resource"
    print("teardown")

https://stackoverflow.com/a/39401087

It will reduce code duplication and help to avoid cases where we forget to add this code line.
#3829 (comment)

[jameslamb]

The client we import from distributed is already itself a pytest fixture: https://github.com/dask/distributed/blob/b5c36b587f0e3295fe59db330603d5c78b39331f/distributed/utils_test.py#L543-L547

I'm really nervous about adding a second layer of @pytest.fixture() on top of this, when this PR's goal is specifically to fix an error caused by instability in teardowns. I'm worried that will expose us to new types of problems related to having fixtures-of-fixtures.

[StrikerRUS]

Thanks a lot for all your hard work!

[jameslamb]

Ok thanks for the review and conversation in #3829 ! I'm going to merge this, let's see if it helps.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.