Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Leaky tests hang on Windows #656

Closed
chipsenkbeil opened this issue Nov 17, 2022 · 5 comments · Fixed by #668
Closed

Leaky tests hang on Windows #656

chipsenkbeil opened this issue Nov 17, 2022 · 5 comments · Fixed by #668

Comments

@chipsenkbeil
Copy link

chipsenkbeil commented Nov 17, 2022

This seems to only happen on Windows, but nextest hangs after all tests have run. I have a lot of leaky tests because I'm spawning a cli that runs for the entire test run and I cannot kill it until I do so manually as part of my ci.

image

Is there something I can do to determine why it's hanging only on Windows? Anything I can do to prevent the hanging?


.config/nextest.toml

[profile.ci]
fail-fast = false
retries = 2
slow-timeout = { period = "60s", terminate-after = 3 }
status-level = "fail"
final-status-level = "fail"
@sunshowers
Copy link
Member

sunshowers commented Nov 17, 2022

Hi there, thanks for the report!

Honestly I'm not sure what's going on there. Based on your screenshot it seems like all the tests have completed running and the test run has ended, yet nextest hasn't quit yet. This seems to suggest that one of the futures that was spawned hasn't exited yet (but I don't know which one).

Do you have a minimal example you can share, that I can try and debug? This will also help me see what sort of logging I can add in the future.

@chipsenkbeil
Copy link
Author

chipsenkbeil commented Nov 19, 2022

@sunshowers I finally got an example working. I had to set up a Windows pipe server that receives connections from a client.

The example repository is https://github.com/chipsenkbeil/nextest-hang-example

Run using cargo nextest run --profile ci --release --all-features.

The test stands up a singleton server that persists, and regardless of success or failure the nextest runner hangs at the end. I've also verified that even when I create an instance of a server per test that should be killed when dropped, if the test fails and leaves the process around, nextest still hangs. So my guess is that this is something about the Window pipe logic that tokio has implemented.

A video illustrating hanging with nextest is this:

nextest-hanging-2.mp4

@sunshowers
Copy link
Member

Thanks! Going to look at this soon (trying out tokio-console to debug this: #665).

@sunshowers
Copy link
Member

OK, figured it out. Will try and publish a fix in the next couple of days.

sunshowers added a commit to sunshowers/nextest that referenced this issue Nov 23, 2022
On Windows, spawned processes can leave the Tokio runtime hanging.
Ensure that the runtime is shut down immediately rather than waiting for
it to exit -- it is better for nextest to leak resources than to hang.

Fixes nextest-rs#656.
sunshowers added a commit to sunshowers/nextest that referenced this issue Nov 23, 2022
On Windows, spawned processes can leave the Tokio runtime hanging.
Ensure that the runtime is shut down immediately rather than waiting for
it to exit -- it is better for nextest to leak resources than to hang.

Fixes nextest-rs#656.
sunshowers added a commit that referenced this issue Nov 23, 2022
On Windows, spawned processes can leave the Tokio runtime hanging.
Ensure that the runtime is shut down immediately rather than waiting for
it to exit -- it is better for nextest to leak resources than to hang.

Fixes #656.
@sunshowers
Copy link
Member

This is now out as part of cargo-nextest 0.9.44. Thanks for reporting it and for the fantastic minimal example!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants