Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shutdown code might not be working as expected. [BUG] #158

Closed
ncclementi opened this issue Jun 2, 2022 · 3 comments
Closed

Shutdown code might not be working as expected. [BUG] #158

ncclementi opened this issue Jun 2, 2022 · 3 comments

Comments

@ncclementi
Copy link
Contributor

In the coiled dashboard, I'm seeing these clusters that have been up for a while not doing any work and the idle code is not closing them, even though the CI looks like passed with no problem. (Closing the clusters manually)

Screen Shot 2022-06-02 at 3 57 13 PM

https://github.com/coiled/coiled-runtime/runs/6714230098?check_suite_focus=true
any idea what could it be?

Details pages:
https://cloud.coiled.io/dask-engineering/clusters/31051/details
https://cloud.coiled.io/dask-engineering/clusters/31122/details

@jrbourbeau
Copy link
Member

I suspect they're running into the preload issue that dask/distributed#6458 fixes. Updating to the latest distributed release (xref #149) should fix this

@ncclementi
Copy link
Contributor Author

I see, well maybe in the meantime we should be checking for hanging clusters to avoid extra costs for no reason.

@jrbourbeau
Copy link
Member

Closing as the 0.0.4 release is out and includes dask/distributed#6458

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants