Rework spec issue management

[jack-berg]

As of the writing of this (1/10/24), this repository has 646 open issues. The process of submitting an issue, triaging, and making a contribution to resolve leaves a lot to be desired for both authors of issues and TC members who are responsible for maintaining the spec.

I've done some analysis on the process we use today and want to propose a series of changes to rework it.

Problems

• Conflicting information on how issues are managed
  • We have a issue management document which was last updated 5 years ago. It mentions labels and roles that don't exist, and a process we definitely don't follow.
  • There's a section on proposing a change, which mentions that a "yes" response indicates the issue is accepted. This is not part of any workflow I've observed.
  • We have an issue triaging section, which we do not follow. It states the automatically assigned spec approver has 3 business days to provide a first response, or assigns to a more appropriate person. It mentions labels that don't exist, like "backlog".
• Inconsistent / undocumented label strategy
  • The area:* labels seems to capture cross cutting concerns (i.e. api, sdk) where spec:* seems to identify the pillar, but these are incomplete and inconsistent.
  • The intent / usage of priority labels isn't clear.
  • There are release:* labels which don't seem to make sense anymore.
  • There are semconv:* labels which don't make sense anymore.
  • There are variety of miscellaneous labels which are used inconsistently, and some not at all.
• Issues linger which should be closed
  • Unidentified duplicates: the same issue was already raised but not identified as a duplicate because lack of awareness, different framing, or only partial overlap.
  • The idea is pretty far out there but nobody wants to make the call that we don't plan on doing it.
  • The idea is outdated - we went in a different direction and the proposal no longer makes sense.
• More issues than can be addressed
  • There are 646 issues and a relatively small number of people who are willing / able to drive spec level issues to completion.
  • We have no mechanism for soliciting priority information.
  • We don't have a good mechanism for growing the set of people able to drive spec level issues.

Proposal

• Document label strategy
  • Create a labels document, describing which labels we use and what they mean. See new triaging workflow below for proposed labels.
  • Delete all labels that aren't part of a well-defined workflow.
• Rework the organization, responsibilities, and membership of spec approvers
  • The spec approvers are underutilized and overly siloed.
  • Get rid of spec approver silos (i.e. Trace Approvers, Metrics Approvers, Logs Approvers). Instead, have a single set of approvers and list any topics members emphasize.
  • Expand the list to include more of the folks who regularly write and engage with spec issues and PRs.
  • Define new concept of a Spec issue sponsor. These are people who are trusted to sponsor non-trivial spec issues as discussed in the proposed new triaging workflow below. A sponsor may directly contribute PRs to address the issue, or work with an outside collaborator to be the point person on coaching and reviewing any PRs associated with the issue. The set of spec issue sponsors would consist of the TC and spec approvers.
• Adopt a more liberal stance on closing issues
  • There are more issues than can be reasonably reviewed / discussed in a group, let alone resolved.
  • Empower all spec approvers and TC members to independently close issues, being more liberal with issues that are clearly out of date.
  • When closing an issue, apply the appropriate triage:rejected:* labels defined below.
  • When closing an issue, inform the author of the spec issue workflow (described below), and that the author can present additional arguments and request reconsideration.
• Introduce new triaging workflow, which encourages community feedback to help prioritize
  • Delete all old process issue management references and replace with new process defined below.
  • Outcome of triaging is to assign a series of labels:
    • area:* - Describes the cross-cutting concern(s) of the spec involved in the issue. E.g. api, sdk, exporter, data model, configuration, etc.
    • spec:* - Describes the vertical(s) of the spec involved in the issue. E.g. trace, metrics, logs, protocol, propagation, etc.
    • triage:accepted:* or triage:rejected:* or triage:deciding:community-feedback - See process below for more info.
  • Process (see diagram below for visualization of flow):
    • New issue is arrives and is automatically assigned triage:deciding:new-issue.
      • Do not auto-assign the issue to a TC member, which is meaningless today. See below for new meaning of issue assignee.
    • The spec triagers (TC and spec approvers) review the issue, engage with author to collect more information, and assign correct area:* and spec:* labels.
      • Triagers are expected to meet regularly to stay on top of new issues. Applying this process to old issues is a best effort.
    • After reviewing information, triagers may reject the issue:
      • If the author is unresponsive to triagers for > 2 weeks, close and assign label triage:rejected:insufficient-info.
      • If the issue is a duplicate (perfect match not required, rule of thumb is 80% overlap), close and assign label triage:rejected:duplicate.
      • If the issue is out of project scope, close and assign label triage:rejected:out-of-scope.
      • If the issue is too large of scope, close, assign label triage:rejected:scope-too-large, and direct the user to create an otep.
      • If the triagers don't believe the issue is aligned with project goals / strategy, close and assign label triage:rejected:declined.
      • If an author disagrees with a decision to reject an issue, they can present additional arguments and request reconsideration.
    • Or triagers may determine that more input is needed from the community:
      • If the issue may have merit, but it is unclear that there is sufficient benefit or demand relative to other project strategy and goals or if proposals are needed to explain how it would fit in, assign the label triage:deciding:community-feedback.
        • This label indicates that more information is needed from the community to understand the severity of the problem and design proposals to resolve it.
        • If / when the triagers decide the issue and any associated design proposals are aligned with project strategy and goals, change the label to trage:accepted:needs-sponsor (see details below).
    • Or triagers may accept the issue:
      • If the issue is small in scope (i.e. typos, formatting, grammatical corrections) and can be worked on immediately by anyone, assign label triage:accepted:ready. The may be good first issue candidates.
      • If the triagers decide that the issue and any associated design proposals are aligned with project strategy and goals, assign label triage:accepted:needs-sponsor.
        • This indicates that the issue is awaiting a volunteer spec issue sponsor (defined above).
        • PRs submitted to address the issue prior to a sponsor volunteering should be rejected.
        • When a sponsor volunteers, change the label to triage:accepted:ready-with-sponsor and assign the sponsor to the issue.
        • If the sponsor withdraws, change the label to triage:accepted:needs-sponsor and remove the assigned sponsor.
        • If PRs associated with the accepted design proposal are ultimately rejected, change the label to triage:deciding:community-feedback (see details above) and remove the assigned sponsor.
  • Solicit community feedback to help prioritize effort.
    • There are more issues than can be solved so we call on the community to help prioritize.
    • Issues labeled triage:deciding:community-feedback need more information about the severity and design proposals to fix. Issues labeled triage:accepted:needs-sponsor have an accepted design proposal, but lack required sponsorship. Users are encouraged to comment with their use cases, vote with 👍, and contribute design proposals. Users are encouraged to share issues to solicit feedback from a wider audience.
    • The spec triagers should review the feedback of issues with labels triage:deciding:community-feedback and triage:accepted:needs-sponsor to help inform priority. Spec issue sponsors are encouraged to sponsor issues that the community feedback mechanism indicates have high priority.
    • The GC is encouraged to play an active role in providing feedback directly and soliciting the feedback mechanism amongst the community and users.

Outcomes

I hope that by adopting this proposal, we can:

• Provide more transparency and realistic expectations to users.
• Obtain a sustainable workflow for TC members and spec approvers.
• Eliminate cruft and old process that we no longer follow.
• Grow the group of people that address spec issues.
• Provide a mechanism to solicit feedback from the community allowing high-value issues to bubble up and get attention.

[yurishkuro]

One other aspect that I think is missing is the expiration / timeout of issues. If we have no expiration, then issues with deciding:community-feedback might sit around forever, creating the same backlog that we're trying to avoid. Ditto for accepted:needs-sponsor. I would suggest:

• an issue can stay inactive in accepted:needs-sponsor for at most 3 months, after which it automatically reverts to deciding:community-feedback
• an issue can stay inactive in deciding:community-feedback for at most 3 months, after which it is automatically closed (or marked stale and then closed later).

If there are more "waiting" states they should all have predefined timeout periods and a downgrade path.

[jack-berg]

I agree with that. The counter I've heard in the past is that closing issues just hides issues. But having an intractable number of open issues doesn't provide any additional visibility over closing them. I would advocate for adopting a culture where we close issues which are stale and are comfortable re-opening them when they are relevant again.

[carlosalberto]

Overall good, only a minor non-blocking comment:

PRs submitted to address the issue prior to a sponsor volunteering should be rejected.

I've seen this a lot for small corrections and clarifications, and probably we should me more lenient at first? Holding them from merging or blocking while issue & triaging take shape, etc.

[danielgblanco]

I think it's a great approach and very clear workflow. Minor question, I assume a single approver can mark an issue as triage:rejected:declined and close it, as they understand it doesn't align with the project goals/strategy. If this decision is appealed by the author and re-opened, do we expect a different approver to pick it up, or at least get involved? Trying to avoid personal back and forth between two individuals.

[joaopgrassi]

I really liked the proposal, and whatever we come up here, will probably make sense to also use in the semconv repo. I feel a lot of people that contribute to the spec are also contributing to semconv so it would be nice that the user experience across repos is the same.

[joaopgrassi]

The collector also has some nice process: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#triage-process

[austinlparker]

We've been doing the new spec issue management process for a few months now; How's it going? Are we good to call this complete?