Trialing an Automated Issue Cleanup Experiment

[eiriktsarpalis]

Creating this issue to track the issue cleanup experiment we're rolling out in dotnet/runtime. Please refer to the original issue in microsoft/contributor-community-experiments for more background.

Hypotheses

Our issue backlog contains a substantial number of issues that have received no activity in over a year, typically occupying the Future milestone. For certain areas, I would claim that the backlog size is so large that taking it into account during annual planning is effectively not possible.

From an area owner's perspective, a stagnated issues backlog can be partitioned into roughly two categories:

1. Low priority/wontfix issues, duplicates, obsolete/out of date issues, low quality/vague feature requests, or a combination of the above.
2. Verified bug reports, approved APIs or feature requests that are compatible with the product roadmap.

While this is more of a spectrum rather than a binary classification, I claim that the majority of Future issues fall into category 1, which makes planning and prioritization for category 2 issues increasingly challenging as the backlog size gets inflated over time.

This is a proposal to roll out a localized automated issue cleanup experiment (initially only affecting areas owned by @eiriktsarpalis). The experiment is based on a couple of hypotheses:

• Reducing backlog sizes allows for more efficient bottom-up planning and makes continuous planning feasible.
• Using automation to close long-inactive issues is an effective way of reducing backlog sizes.

Several major repos including non-Microsoft repos have an issue cleanup/issue bankruptcy policy. We have received positive feedback from partner teams that successfully applied such policies to their backlogs.

Experiment

There are a number of parameters to be considered for the experiment:

Selection Criteria

• Focus issues that have not been updated in over 5 years.
• Exclude popular issues, e.g. issues that contain a large number of comments or reactions. [Not currently possible due to FB scheduled search restriction, can explore this avenue in the future].
• Exclude issues where the ball is clearly in our court (although that might be difficult to identify in the large).
• Incorporate a manual opt-out mechanism (e.g. by applying a label).

Execution

1. FabricBot applies the no recent activity label to selected issues and writes a post explaining that the issue has been marked for closure, linking to this issue and encouraging users to submit their feedback on the process.
2. If no response has been provided within 14 days, the issue will be closed automatically by FabricBot. Otherwise, FabricBot will remove the no recent activity label and the issue will remain open.

The policy would initially cleanup issues in bulk, but will remain in place so that any other issues going over the 15 month threshold are subsequently marked for closure.

Success criteria

• Reduction in the number of open issues in the Future milestone.
• More clearly identify issues that are important but have stagnated.
• Get up-to-date customer feedback on stagnated issues.

Risks

• Closing "legitimate" feature requests. While this is expected to happen to an extent, it is only possible if neither issue contributors nor area owners object to the closure going ahead. This would suggest that it's a low priority issue and closing due to low popularity and lack of activity is perfectly acceptable.
• Initial bulk cleanup might trigger a flurry of activity, substantially increasing workload for area owners. We do not believe this to be a concern: based on reports from the VS Feedback team only 5% of their closed issues prompted some form of customer reaction.
• Automated closure of issues can be controversial with the community. We hope that a carefully worded message pointing out this is an experiment and linking back to this issue should serve to mitigate any such concerns.

cc @danmoseley @layomia @krwq