[Retrospective] Release 2.6.0

[gaiksaya]

How to use this issue?

Please add comments to this issue, they can be small or large in scope. Honest feedback is important to improve our processes, suggestions are also welcomed but not required.

What will happen to this issue post release?

There will be a discussion(s) about how the release went and how the next release can be improved. Then this ticket will be updated with the notes of that discussion along side action items.

Release issue: #3081

[SuZhou-Joe]

Patch pkg.branch when build in CI flow opensearch-project/OpenSearch-Dashboards#3490

[gaiksaya]

Various components had to manually sign off on the failing integration tests. Listing them down here:

• security analytics dashboards
• notificationsDashboards
• dashboards-search-relevance
• IM Dashboards
• Reporting
• ml-dashboards
• k-NN with security
• IM: (flaky test)
• SQL (script incompatibility)
• Asynchronous search

Looks like code in opensearch-functional-test repo is not exercised regularly leading to last min findings.

[bbarani]

Tests in OSD functional tests repo are not sync with individual plugin repo causing last minute issues.

[gaiksaya]

Below are the action items that need to be taken care of if we want our releases to go smoother and increase the quality of the product.

• Issue: Tests in OSD functional tests repo are not in sync with individual plugin repo and are not exercised regularly causing last minute issues.
  AI:
  • Keep tests between open search dashboards plugins’ repositories and functional test repository in sync
  • Exercise the code in the CI
  • Run regular testing on nightlies using the same.
• Issue : Last minute changes in the OS / OSD core caused snowball effect across all plugins
  AI: Come up with better communication mechanism between core and dependent components
• Issue: There is still some confusion on experimental plugins mainly around the code / usage / testing perspective ( What flag needs to be added, how to add it, how can it be tested, what level of testing is required etc..)
  AI: Add documentation for onboarding experimental plugins to the distribution.
• Issue: K-NN team found a glaring issue (that would make K-NN not usable when security is enabled) only after we provided the test results of the 2nd release candidate
  AI: How to better test plugin set up with security enabled cluster in the repository’s CI. We recommended using security plugin zips: https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-security/ available on maven now. Other components please let us know how you guys are testing it today. Add documentation regarding the same that can be adapted by other components as well
  Issue: Manually signed off components need to create GitHub issue tracking the fix for next coming release.
• Enable testing as soon as possible for each version irrespective of the release.