Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add stale issue workflow for flaky test reports #43547

Merged
merged 4 commits into from
Aug 24, 2022

Conversation

Mamaduka
Copy link
Member

@Mamaduka Mamaduka commented Aug 24, 2022

Fixes #37968

What?

PR adds a new workflow to check issues with the "[Type] Flaky Test" label and no activity in the past 30 days. Detected issues will be labeled "Stale" and closed in one day without activity.

Why?

I think it's safe to assume that the flaky tests report with no activity in the past 30 days might be resolved or was a false report due to some specific conditions.

Testing Instructions

The GitHub workflows are a bit hard to test, so verify settings based on the actions/stale docs - https://github.com/actions/stale/tree/v3.0.17#readme.

@Mamaduka Mamaduka self-assigned this Aug 24, 2022
@Mamaduka Mamaduka added the [Type] Project Management Meta-issues related to project management of Gutenberg label Aug 24, 2022
@gziolo gziolo requested a review from noisysocks August 24, 2022 06:32
Copy link
Contributor

@talldan talldan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together - I've marked the PR as closing issue #37968.

There was some discussion before about not doing this because some flakey tests are skipped and still need to be fixed, so the issue will become stale and closed but the test will never be fixed.

But I think the volume is so much now that it's worth doing this anyway.

@Mamaduka
Copy link
Member Author

Thanks, @talldan. Somehow I missed that issue.

There was some discussion before about not doing this because some flakey tests are skipped and still need to be fixed, so the issue will become stale and closed but the test will never be fixed.

Maybe we can prevent this by using the exempt-issue-labels option. I'm unsure if we already have a label that can be used for it or need to introduce a new one.

@talldan
Copy link
Contributor

talldan commented Aug 24, 2022

Maybe we can prevent this by using the exempt-issue-labels option. I'm unsure if we already have a label that can be used for it or need to introduce a new one.

Sounds like a good idea, it might be tricky to get contributors to understand the workflow, but worth trying. Maybe the stale or flaky messages can explain it.

@Mamaduka
Copy link
Member Author

Also, considering that skipped tests require manual action, this might not be a big problem.

I will merge workflow in the current state but keep an eye out for improvements.

@Mamaduka Mamaduka merged commit 7762948 into trunk Aug 24, 2022
@Mamaduka Mamaduka deleted the try/stale-issue-flaky-test branch August 24, 2022 18:21
@github-actions github-actions bot added this to the Gutenberg 14.1 milestone Aug 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Type] Project Management Meta-issues related to project management of Gutenberg
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Flaky tests - consider auto-closing stale flaky test reports
2 participants