How to better QA UI release? - so we can avoid cutting several patches while releasing the instance

[hanbyul-here]

Context

Even though VEDA UI holds most of the core features of the instance, the VEDA-UI release is not getting proper QA before the release. As a result, some of the bugs were found only when GHG instance was trying to use the release. We had to cut several hot-fix patch releases as a result of it: https://github.com/NASA-IMPACT/veda-ui/releases (5.1.2, 5.1.1, 5.0.1 were hotfixes ).

Assuming we cannot get as many resources for UI release QA as for GHG release, how can we improve UI QA? Here are some ideas.

Ideas

Please feel free to edit, or add comments.

Dev side

• Make datasets we use for dev as close as possible to the real datasets: We haven't updated our mocked datasets for a while. We can copy the dataset configuration files from the instances.
• Trigger the preview from the instance side from UI release: I am not sure how feasible it is, but it might be helpful to make an automatic PR with the latest UI version from the instance side so that we can test the UI version from the instance immediately.

Product side

• Make several scenarios that we should test. We can start with a simpler scenario like @j08lue listed out in this comment: Deploy v0.16.0 veda-config#411 (review) and add more cases that have required more of our attention before. (A side note that this can be a valuable resource for more robust e2e testing @dzole0311 suggested)

Acceptance Criteria

• Given when there are action items for better QA for UI release

Related Tickets

[If applicable, link any tickets that are related]

[j08lue]

Thanks for calling this, @hanbyul-here, I thought the same after the latest release.

Suggestion: Release candidate testing

I think a simple process change could already improve things:

1. Develop changes to VEDA UI
2. Publish changes to VEDA UI under a pre-release tag (e.g. using a release candidate version like 5.2.0-rc.1)
3. Create instance previews (primarily GHG Center) and share with instance-level testers
4. When instance preview testing was successful:
   1. Create a VEDA UI release
   2. Update the instance release candidate preview to the release preview
   3. Merge to staging on the instance
   4. Test staging on the instance to be extra sure

Rationale

Some issues only show up with real content and the instance-level testers have the sharpest eyes to spot them, since they are familiar with the content. We should make use of that testing capacity we have.

I was astonished that the GHG Center upgrade of VEDA UI for this release was merged to staging before extensive review. Was that how we expect things to go?

• Bump version to v5.1.0 US-GHG-Center/veda-config-ghg#411

I think we should test before we merge to staging. Having bugs on staging is dangerous and an obstacle to by-passing blocking bugs, if necessary.

[dzole0311]

Make several scenarios that we should test. We can start with a simpler scenario like @j08lue listed out in this comment: NASA-IMPACT/veda-config#411 (review) and add more cases that have required more of our attention before

I copied the comment from @j08lue and started a doc with simple scenarios for QA; let me know if we already have that documented somewhere (cc @aboydnw). I think we should also try out how Cypress integrates within the veda-ui for automating some of those scenarios following this proposal. I created an issue for that.

[hanbyul-here]

@j08lue I like the RC idea! I think we can take advantage of GitHub actions to reduce the dev's efforts to make the whole process work. For example, it would save our time if we could create the PR on the instance side whenever there is an RC release on the UI side. This PR will have a preview url to test. It also can have a pre-filled scenario for QA, and e2e test running in the future. I will take a look at how much we can take advantage of GitHub actions.

[slesaad]

We created a QA checklist for the GHG instance - https://docs.google.com/document/d/1k8jQh5cuadwdWS5FEILo28cpkmhyKHLCNWXw2ETPilo/edit if it is helpful

There's also a simple testing suite PR here, which could be expanded.

[hanbyul-here]

Thanks a lot for your engagements in RFC: https://docs.google.com/document/d/1MYwF2chy8LcRTRyja---x7sIs4Wod44jXFPYLUQbta4/edit#heading=h.hpihcughr9w

To summarize the comments I could receive: RC seems to be a good idea, as long as we do not have to spend too much extra developers' extra time and effort. While we can mitigate the problem by automating the process, we must establish a better protocol for rc - official release first. Then, we can automate the process more gradually, starting with the process we already nailed down (such as the official release process). So, action items from the RFC can be categorized into two categories. (These two don't need to be executed in a sequential order.)

1. Establish bi-weekly RC release and official process sequence.
   This will include answering some questions such as

• how many testing days we should give ourselves. I suggested a week, which would include any patch release needed for the rc.
  Are we feature-freezing after RC? This means that the official release won't have any additional features on top of RC (other than patch fixes). If so, how should we organize our git flow? (If the new feature is merged to the main before the patch fix is merged in, we can't just target the main for the release.)

2. Find where we can automate the process for the official release.
   The candidates that can be automated are like below. (Not all of these need to be implemented, but these are ideas.)

• Make a draft release with generated messages (should be reviewed by a developer)
• Use GitHub Action to fill up the version number
• Slack alarm for release review, testing
• Deploy previews from the instances

I'd like to add a note that I intentionally did not include automated testing in this action item since it deserves its own ticket (and should not depend on this ticket).

Since the dashboard team is working for R3, this might not be a great time to introduce a new protocol. I will create the action item tickets in the backlog that can be picked up after R3.