Request for Proposals: Changelog and release note process

[joshuarrrr]

This is a request for proposals for processes and tools that make it easier for the OpenSearch community to understand changes to our repositories, both at release time, and in between releases. The objectives, tenants, and goals provided here will be used as our criteria for evaluating the different proposals.

Objective

The objective is to solve the entire collection of issues around generating both ongoing CHANGELOGs, and release notes during GA of the product, for all OpenSearch project repositories.

Tenets

• Changelogs are for users [1], not project contributors nor machines. Not every commit is relevant to a user so not all commits are in the changelog.
• Changelog entries are peer reviewed and approved like any other part of a contribution.
• Changelogs list all significant unreleased changes.
• Changelog entries are editable after initial commit.
• Changelogs minimize maintainer work, are adopted
• Changelogs are trustworthy and up-to-date.

[1] Users are defined as operators/administrators, developers integrating with SDKs, and end-users interacting with UIs or REST APIs.

Goals

• Preparing release notes from any maintained branch at any time should be trivial and quick (< 1 hour).
• The content of the changelog at the time of release is exactly what goes into the component level release notes. This doesn’t include a “release highlights” or similar summary that may be included in any particular release.
• The changelog is transparent to the versioning / branching strategy, i.e. if a commit that introduces a changelog entry is merged into a branch for a particular release, then the changelog entry goes to the right place with minimal or no additional effort.
• Existing backporting automation should be unaffected.
• The contributor experience should be improved, not degraded.

Scope

• Tenets and goals apply to all OpenSearch project repositories
  • We’d like a generalized solution to CHANGELOGs and release notes that can be applied to any repository in the organization.
  • Individual repository owners are free to experiment, and may choose a different solution or approach, as long as it satisfies the goals above.
• Working outside of GitHub/Jenkins is not required

Proposal Process

1. Submit proposals via the proposal issue template or as a PR in this repository by April 13, 2023
2. Link your proposal issue or PR to this issue via a comment here. This discussion thread here will be for clarification questions and updates on the overall proposal progress.
3. Review linked proposals and comment on them directly.

[ashwin-pc]

I had a few questions around the goals here:

1. Does the changelog have to be accurate all the time or can there be a lag. e.g. Is it okay to collect the changes at the end of the day into a single changelog entry?
2. Does the Changelog entry require a link to the pull request?

[dblock]

I feel pretty strongly that CHANGELOG should be updated with the changes being made and have a link to the PR that contains the change. I think it's OK if it's automated enough that it happens "quickly enough", e.g. same day, but it must always happen.

[ashwin-pc]

@dblock and what about changes that go into multiple releases? e.g. 1.3 and 2.x, do we have multiple entries in the changelog for the change?

[dblock]

@dblock and what about changes that go into multiple releases? e.g. 1.3 and 2.x, do we have multiple entries in the changelog for the change?

CHANGELOGs are diffs with their previous release. So for diverging branches such as 1.3.10 and 2.7.0 right now we effectively ship a fix in both for the first time. On each of these two branches we would keep an unreleased section in the CHANGELOG, and when we backport a fix into that branch, the unreleased changelog gets it. When we release, you should be reading in release notes of 1.3.10 and in the release notes of 2.7.0 about the fix. It won't appear again in any of the subsequent versions such as 1.3.11 (it was already in 1.3.10) or 2.8.0 (it was already in 2.7.0) or 3.0 (it was already in a 2.7.0 release).

[joshuarrrr]

[PROPOSAL] Introduce in automatically generated release notes from @ZilongX

[joshuarrrr]

[Proposal] Automate versioning, releasing and changelog generating with Changesets from @ruanyl

[joshuarrrr]

Status quo: [PROPOSAL] Author CHANGELOG in each PR instead of collecting them in the last days before a release from @dblock

[seraphjiang]

There is no data to prove other repo facing the challenge OS and OSD repo has today on changelog. We should not introduce this to other repo to add additional engineering overhead. It is suggested to change the scope to limit to OS/OSD repo only.

It looks more like look for brainstorm about solution other than provide options. Maybe i missed or didn't get it.

Proposal Process
Submit proposals via the proposal issue template or as a PR in this repository by April 13, 2023
Link your proposal issue or PR to this issue via a comment here. This discussion thread here will be for clarification questions and updates on the overall proposal progress.
Review linked proposals and comment on them directly.

could we list actionable options and provide data to prove like before after how process/productivity has been improve?

[ruanyl]

I have setup a simple example to demonstrate the idea of changesets, the tool may not exactly fit for OpenSearch development, but the philosophy of CHANGELOG management it introduced could be insightful.

The workflow has two steps:

STEP 1: Add a changeset, this is normally added in a PR when the changes are still fresh in the author's mind
A changeset is normally a markdown file which indicates which package is changed? And what are the changes?

|___.changeset/
|       |___UNIQUE_ID_A.md
|       |___UNIQUE_ID_B.md

An example PR can be found here: https://github.com/ruanyl/OpenSearch-Dashboards/pull/2/files#diff-ec60e1e3b0c0317311062aed2aa6885e9cadd255da88d212eb4bcb9882465e50

It's a markdown file which is part of the PR and it can be reviewed and commented by the reviewer just like any other source code files.

STEP 2: The base branch may contain multiple changeset entries merged from different PRs during the development. They will be consumed at the time when making a release. All changeset entries will be merged into the CHANGELOG and then deleted.

See example: https://github.com/ruanyl/OpenSearch-Dashboards/pull/3/files#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4ed

Since changeset entries are just individual files with unique name, back-port PRs will not result in conflicts.

The tool(Changesets) that I'm referring is more than a tool of CHANGELOG management, it includes versioning and publishing, and especially works well under a monorepo. It also comes with a GitHub Action that automate the release and a GitHub Bot to verify if a PRs contains changesets.

But again, changesets is more like a generic tool, it may not fit for OpenSearch workflow. We can adopt its concepts and come up with tools that works for OpenSearch projects.

[ashwin-pc]

There is no data to prove other repo facing the challenge OS and OSD repo has today on changelog. We should not introduce this to other repo to add additional engineering overhead. It is suggested to change the scope to limit to OS/OSD repo only.

@seraphjiang valid point, i dont know if any other repo who is facing this problem, but i also dont have any idea if they dont or how successful they are at meeting the goals of the changelog working group. The goal of this process is not only to improve the process but also ensure accuracy, something thats a lot harder to pin down with our current backport process. Given that we havent settled on a proposal yet, i wouldnt start the discussion on whether it needs to be opt in or mandatory just yet. IMO, as long as each repo is able to successfully meet the changelog goals, it does not matter what process they have set in place. But what i'm not so sure of just yet if if they actually meet it (especially the accuracy bit).

It looks more like look for brainstorm about solution other than provide options. Maybe i missed or didn't get it.

Not sure what you mean by the but the issue asks for proposals for the changelog process that meet all the goals defined in the issue that are within scope. These are the goals agreed upon by everyone present in the working group after weeks of discussion.

[joshuarrrr]

After reviewing all the proposals listed here, in the working group meeting on 5/4/2023, we unanimously voted to move forward with #156 (the "2.2.2 Automation" variation specifically). Next steps will be tracked on that issue.