Proposal: Improvements to the release channel strategy

[awentzel]

This is an initial stab at improving our release process with the goal to improve our process based on our existing experiences and feedback over the past couple of months. Please take the time to understand and contribute to the process so we can maintain the best product quality possible while improving developer and operational efficiencies as well.

Release Challenges (Packages)

It's cumbersome and disruptive to partner commitments, collaborator and contributor support, operations, and engaging in growth opportunities.
 
As a solution, we're proposing a consistent release cycle on which our community may depend on.

Release milestones are difficult to determine what has been committed and must be completed in each release.
 

Process 

Publishing Packages

Publishing of NPM packages to be done on nightly/weekly cadence enabled through GH Actions running on a cron schedule and/or merge to master. These packages will be released into different channels for which engineering picks up each morning to eliminate blocking productivity. These packages will then be deployed during the release process at the time of deployment to coincide with the documentation release. Additionally, documentation on pre-releases is still available on NPM or on staging sites which will no longer be secured by Azure Active Directory.

Beachball looks promising on better supporting our requirements.

Publish to NPM

Links for inspiration.

• Advice on Nightly Builds beachball#429
• https://github.com/danillouz/til-npm-release-channels

Release channels

Milestone 2021-3 Releases

• Daily pull_request trigger will perform cross-platform validation without publishing
• Nightly automated publishing to NPM

Milestone changes (future improvements)
Subscoping for tooling packages, Jane?

• Nightly automated dev

• Weekly automated beta

• Monthly as desired manually stable

• • Releasing Features
• • Releasing Major Features

• dev or canary: released nightly and/or on merge to master by cron in ci-nightly.

• next or beta: released on the same cadence of dev and updated tag to reflect the version of dev by cron in ci-weekly. For example, [major.minor.patch-suffice+yyyymmdd] (Beachball feature enhancement required for +yyyymmdd semver 2 metadata option. Future feature addition.

• latest or release: released on merge to master of major package bumps.

npm publish --tag `dev`

For details https://docs.npmjs.com/adding-dist-tags-to-packages#:~:text=Publishing%20a%20package%20with%20a%20dist-tag%20%C2%A7%20By,navigate%20to%20the%20root%20directory%20of%20your%20package

This PR is related to #3889 #3543 #3660

[chrisdholt]

Working my way through, but I think a monthly release is way too much time between things going in and being released. Just from an internal partner standpoint, we're already stretched at every other week / every two weeks.

From previous conversations, we've discussed regular, automated nightly releases, and I would wager that would be preferred to a manual release by a majority of folks. If we had continual release which included v-diffing and unit tests as part of our build process would that be preferred to the extensive manual testing throughout the month above?

The only other thing I'm trying to reconcile is how the schedule works with other commitments we have for the product teams some of us support. Looking at this in isolation for FAST, it seems like a reasonable schedule. But when I consider that folks on my team internally have other commitments during a given week or month, and I include the potential for external contributors and their workflow - this seems difficult to stick to.

[EisenbergEffect]

Just had a quick convo with @awentzel about introducing different release channels to help with this. Aaron will circle back with you @chrisdholt to discuss and then update the issue with more details based on that.

[awentzel]

This proposal does not address the publishing of NPM packages. This schedule is intended for site releases and the schedule is distributed over an entire month to minimize work disruptions. It ensures uninterrupted work and dedicates time for TDD to improve quality while also allowing for stabilization after release.

[awentzel]

I've added multiple sections to further outline NPM packaging process, the use of Containers with Docker, and the use of scripts to spin up and down temporary app deployments for each PR. Beachball, could be used for NPM publishing on nightly cadence. @chrisdholt @EisenbergEffect @nicholasrice @janechu @radium-v for review and feedback, anyone else is also free to provide feedback.

[awentzel]

@microsoft/fast-collaborators please review prior to release 12. Thanks

[EisenbergEffect]

The highlights for me that I like are:

• Moving to Docker
• Nightly dev channel releases of all packages
• Stage docs for pre-release packages open to the public
• Code freeze via a dedicated release branch merge
  • locks the code for release while allowing continued dev work on master
• Tagging of the release branch to mark package publish point in time

The only real hesitation I have is over week 3 in the cycle. I think that's likely to remain a dev-focused week based on sheer quantity of work and inability to actually make use of everyone testing. If we freeze the release first thing on Monday of week 4, then we could use all day Monday for testing and maybe even Tuesday if needed, with a release happening either Tuesday or Wednesday depending on how that goes. Then sites pushing out on Thursday.

[janechu]

For me the site release should coincide with the package release, ideally it would be same day or shortly after package publish because we intend on putting the published package version number up on the site as well. I think we also eventually will need a published changelog on the site. Correct me if I'm misinterpreting because it seems like it's being suggested that NPM release is every 2 weeks and sites are monthly.

I agree with @EisenbergEffect the move to docker is going to make the process a lot better as well as using nightly releases. I also don't mind the dedicated release branch method.

[chrisdholt]

I agree with @EisenbergEffect on the timelines here:

The only real hesitation I have is over week 3 in the cycle. I think that's likely to remain a dev-focused week based on sheer quantity of work and inability to actually make use of everyone testing. If we freeze the release first thing on Monday of week 4, then we could use all day Monday for testing and maybe even Tuesday if needed, with a release happening either Tuesday or Wednesday depending on how that goes. Then sites pushing out on Thursday.

As I mentioned above, I don't think part of the team will be able to join and freeze commitments for a dedicated testing sprint. I think some UI automation or vdiff tests for the components AND the sites which run on a regular cadence would make testing more of a part of the development cadence. Again, I don't want to play down the importance of testing, but the current plan to set aside a couple days to do just that - it's not feasible for part of the team due to internal commitments.

I'd also agree with @janechu on having the package release coincide with the npm publish. If the proposed timeline is still once a month, I'm unsure of a monthly cycle as opposed to our current cycle. I think a nightly will help relieve the pressure we've felt recently to "not miss", which seems to be at least some of the reason (along with the need for more testing) to extend this release to once a month.

[EisenbergEffect]

Agree on releasing docs along side packages. 💯

[janechu]

Just a note, we could put the version number on the production site for:

• The latest production release

Once staging is opened up, the nightly releases would accumulate and we could include both:

• The latest production release
• The latest nightly release

I think we could also include a changelog for these releases. This way consumers of our nightly releases can see exactly what they're getting.

It would also be nice to include nightly releases on the prod site, but I'm not sure how we can do that without an automatic deployment and a separate compilation to grab the information for nightly releases. Which in the short term might be a hassle.

[awentzel]

Feedback has been re-incorporated into the top detail section. Noting that Testing and Releasing "weeks" does not infer must spend the entire week doing the activity, only that the hour of testing must be performed. This work should result in no issues or new issues to be triaged and prioritized for this release or future release. This "week" then affords time for completing the commitment. The following week similarity for releasing. The hour or two of releasing should be performed assuming testing is 100% complete and any follow up work for that week has been completed. The release will be performed and completed that Monday. If anything comes up the remainder of the week we have time to resolve. This time also allows for stability and monitoring to ensure no issues come about the release.

[awentzel]

@janechu nightly releases for sites would be a long term goal as require major infrastructure changes and based on full automation which I'm in favor of. Staging will continue to deploy every time a PR is merged into master. There are just a lot of moving parts to get this in so I'd rather focus on improving the current process to solve current challenges. Vdiff etc are major feature improvements that would include many details in themselves and out of scope for this process strategy.

[awentzel]

Added "Publish to NPM and GitHub Packages" and "Release channels" section in the issue details.

[awentzel]

I've updated the schedule, deleted sections not currently in-scope. Please provide opinions on NPM publishing cadence, issues, and requirements.

[awentzel]

Two new requirements:

1. build should fail if no /changes directory has been generated, which is the output from running beachball change
2. beachball should support metadata creation. Request submitted here. It does not currently. I recommend simply using a single tag for pre-releases using x.y.z-next and then when we get additional functionality we can add the +yyyymmss and next tag.

[awentzel]

I understand this can be tested with beachball canary --tag dev --registry http://localhost:4873 though still need to try and it and adding it here for documentation when I pick this task up again soon.

[EisenbergEffect]

@kenotron Would you take a PR to add the metadata features that @awentzel needs? If you can point me to the relevant locations in the codebase, I may be able to squeeze out some time to put that together. We're anxious to get this going for our project and happy to make necessary contributions back to Beachball to enable that.

[awentzel]

@EisenbergEffect from previous conversations with @kenotron you can find more details on its issue here.

[awentzel]

I believe everyone is in agreement to how this could be done. However, beachball requires another feature to fully implement around support for semver metadata. Until this is implemented we could not fully implement this release strategy.

[EisenbergEffect]

Let's not close this. We do intend to implement this as soon as we can get the Beachball PR in.

[awentzel]

The hardest part of this work is completed. I've started a new issue #4626 for the remainder of the future when the dependency is completed by Beachball. All important remaining details from this have been moved to the new issue. The rest is now considered complete. Closing this issue.