Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(iroh-blobs): comment out ignored test (that is not a flaky test) #2559

Merged
merged 2 commits into from
Aug 1, 2024

Conversation

matheus23
Copy link
Contributor

Description

There's this test that was introduced in the local_pool PR.
It was ignored via #[ignore = "todo"]. Notably, it's not flaky, it always fails.
Our flaky tests are run with cargo nextest run --run-ignored all [...]. We can't be more specific with the ignored tests. The only options are default, ignored-only and all.
This kind of test is really hard to write. IIUC, #[should_panic] can only test for the panic happening in the thread that the test is initiated in, it doesn't detect panics that are thrown in threads spawned from the test. I assume this is the reason writing this test was abandoned.

Keeping this test with the #[ignore = "todo"] on it means we're always running it in our flaky test suite, which is confusing. We thought this test was flaky, but it's not. IMO it's better to comment it out/remove it than to pollute our flaky test results.

Breaking Changes

None

Notes & open questions

In this PR I'm commenting this test. Should we remove it instead?
Or do people have ideas on how to make this test work?
Do we have an idea what we're expecting of our LocalPool implementation? Should a panic on one of the threads cause a panic in the finish() function?

Change checklist

  • Self-review.
  • [ ] Documentation updates following the style guide, if relevant.
  • [ ] Tests if relevant.
  • All breaking changes documented.

@matheus23 matheus23 self-assigned this Jul 29, 2024
Copy link

github-actions bot commented Jul 29, 2024

Documentation for this PR has been generated and is available at: https://n0-computer.github.io/iroh/pr/2559/docs/iroh/

Last updated: 2024-08-01T07:10:47Z

Copy link
Contributor

@divagant-martian divagant-martian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

personally agree.

Having a test that always fails makes it harder to identify real flakes vs outright broken stuff failing 100% of the time. As @matheus23 said: alarm fatigue

@matheus23 matheus23 added this pull request to the merge queue Aug 1, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 1, 2024
@matheus23
Copy link
Contributor Author

The irony of a new flaky test keeping this PR from being merged 🙃

@matheus23 matheus23 added this pull request to the merge queue Aug 1, 2024
Merged via the queue into main with commit 15f36b3 Aug 1, 2024
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants