Improve Adopter Experience for Breaking Changes

[tsmaeder]

We have had reports from the community that updating to new releases generates too much work for them. The problem is less the pace of change, but that we are not good enough in guiding adopters on how to adapt to those changes. In particular

• Sometimes we're not calling out breaking changes in the change log
• When we do mention the changes, we're often not telling adopters what they need to do in order to adapt their products

This concerns both code changes (example: the recent move from "QuickOpen" to "QuickPick" API's) and procedure changes (like the recent move from Webpack 4 to Webpack 5). Procedural changes are worse, because the breakage is often not obvious. With API changes, the compiler can guide you through the migration.

One of the unique selling points of Theia is that a lot of code is considered API and can be customized. But a lot of API also means that in order to move the project forward, we will have relatively frequent breaking changes. Some approaches to mitigating the problem have been proposed:

• Improve breaking change documentation
  • Clarify the expectations we have for documentation of breaking changes in the review checklist. PR's must conform to the documentation requirements before being approved
  • The amount and form of needed documentation depends on the nature of the change: it can range from a simple "the API on Class X has changed" to a complete migration guide. The goal should be that adopters can update to a new release without doing their own research on how to fix their products [@tsmaeder: Personally, I don't believe we should overformalize the requirements here. Good judgment and accountability will serve us better.]
• Pool experiences
  There will be cases where updating to a new release is not as straight-forward as we hoped. In that case, we could at least share the burden of researching the update path:
  • Early adopters are encouraged to share their experiences when adapting to (breaking) changes. I believe opening a bug report ("Update to build 1.x.y") would be the simplest way to do so.
  • If we want to share the burden of finding out how to update to a new release, someone needs to go first and share their findings. For this we need volunteers: I suggest that orgs interested in participating in some kind of rotation tell the project leads who can organize a system. As for the organization, we could announce releases to the theia-dev mailing list and whoever "goes first" replies "we're adopting this release", followed be "no problem" or "problems found, here's the issue we filed: https....
• Improve testing to detect breakage
  In order to detect when changes are breaking products, we should regularly build and test a system that closely resembles an adopter's product, for example:
  • Build Blueprint nightly against latest/master
  • Build Yeoman-generated app against latest/master
    Once we detect breakage using those tests, we need a process of handling the breakage. The tests need to be adapted and the change path needs to be documented. If the tests are fast and we can run them as a PR check, the handling happens within the PR. If the tests can only be run as nightlies, for example, we need have a process to assign responsibility for handling the failures.

[tsmaeder]

Todo [@tsmaeder ] propose changes to review checklist, etc. to implement point 1. "Improve breaking change documentation"

[tsmaeder]

Ideas:

• announce releases to the mailing list: gives us a forum to talk about the release.
• Add a section to the general Readme.md explaining about migration guides, etc.

[tsmaeder]

We build our "che-theia" app against the master branch of Theia in our build. I think our build would serve as a good canary test for build breakages.

[JonasHelming]

@adamretter

[adamretter]

@tsmaeder @JonasHelming Thank you very much for this. Also FYI @ccheraa @duncdrum and @marmoure

In terms of documentation for a Migration Guide, how do you see this working in terms of collaboration:

1. A GitHub Wiki page
2. Part of the Release Notes
3. A .md file in the Git repo?

[tsmaeder]

We were thinking an *.md file: I think non-committers cannot edit the wiki and the process with treating the wiki as a git repo seems too arcane to me.

[adamretter]

@tsmaeder Good stuff. Perhaps we can also link to the .md file from the Release Notes.

[JonasHelming]

@tsmaeder Good stuff. Perhaps we can also link to the .md file from the Release Notes.

Yes, we discussed that, too, it is possible

[tsmaeder]

#9817 is now merged.

[tsmaeder]

migration guide is now linked in the releases.

[tsmaeder]

I think with the regular Theia IDE release, we're also covering the "testing" part of this issue.