Integrate OpenSearch Dashboards release functional tests into CI process

[tianleh]

Is your feature request related to a problem? Please describe

We want to integrate OSD release test https://github.com/opensearch-project/opensearch-dashboards-functional-test into the infra CI. We have already verified that it can run in Github Action env.

(See this sample run https://github.com/opensearch-project/opensearch-dashboards-functional-test/runs/3705023275?check_suite_focus=true)

Describe the solution you'd like

The overall process would be:

For security enabled cluster,

checkout opensearch-dashboards-functional-test from Github
run npx cypress run --spec "cypress/integration/security-enabled/*.js"

For security disabled cluster,

checkout opensearch-dashboards-functional-test from Github
run npx cypress run --spec "cypress/integration/security-disabled/*.js"

(One call out is that opensearch-dashboards-functional-test imports another library repo https://github.com/opensearch-project/opensearch-dashboards-test-library through NPM registry.)

Describe alternatives you've considered

We could just use the Github Actions https://github.com/opensearch-project/opensearch-dashboards-functional-test However, it is isolated outside of the org workflow.

Task breakdown

• Add dashboards functional test repo into 1.2.0 manifest #834
• Move process related logic under system/ #852
• Refactor LocalTestCluster into LocalTestClusterOpenSearch and LocalTestClusterOpenSearchDashboards #837
  • Extract OpenSearch service related logic outside of LocalTestCluster #1079
  • Create ServiceOpenSearchDashboards class to run Dashboards #1080
  • Distinguish OpenSearch vs OpenSearchDashboards manifest file during build and assemble workflow #1134
  • Create LocalTestClusterOpenSearchDashboards to create test cluster for Dashboards #1081
• Instantiate LocalTestClusterOpenSearch vs. LocalTestClusterOpenSearchDashboards depending on the product in the manifest #838
  • Update name and location of test manifest file #1161
  • Configure logging dest for OpenSearch Dashboards service #1170
  • Add functionalTestDashboards to OpenSearch Dashboards test manifest #1171
  • Update Test Suite to start Dashboards test cluster #1177
  • Require test manifest path for running tests #1228
  • Run either OpenSearch or OpenSearch Dashboards integ test from test runner/shell #1211
• Jenkins integration
  • Run Dashboards integ test from public endpoint #1259
  • [Bug]: The locations inside Bundle Manifest are missing the product in the path #1264
  • Add new attribute filename to build manifest class property #1269
  • Run Dashboards integ test against static endpoint in Jenkins job #1322
  • Add test manifests to previous version #1323
  • Add notification post OpenSearch Dashboards integ test execution #1371
  • Trigger OpenSearch Dashboards integ test post bundle build #1372
  • Indicate the execution status of Dashboards integ test Jenkins job #1373
  • Support arm64 when running Dashboards integ test in Jenkins #1381
  • Unify the structure for release url and regular distribution url for running Dashboards integ test #1482
  • Update artifacts promotion Jenkins to support OpenSearch Dashboards #1489
• Update functional test repo
  • Modify Cypress version to match infra Docker image opensearch-dashboards-functional-test#44
  • Add "-v" support to integ test shell opensearch-dashboards-functional-test#39
  • Move test results into a single root dir opensearch-dashboards-functional-test#49

Fast Followups

• Save Cypress test artifacts from Dashboards functional tests to logs #1190
• Save OpenSearch logs for OpenSearch Dashboards integ test #1191
• Make OSD integ test result human readable #1707
• [Bug]: Cypress unexpectedly exited when running OSD integ test #1708
• [Bug]: Some dashboards plugin test cases fail when security is enabled when running OSD integ test #1709
• Add links to test results in the OSD integ test notification #1728

[peterzhuamazon]

Since test-workflow haven't officially support dashboards/cypress yet, we will try to run this on the old Jenkins Workflow next week with repository integtest.sh.
We will migrate it to test-workflow once it is ready.

Thanks.

[seraphjiang]

what's the gap to run cypress? is there an open issue to track, want to know more

[peterzhuamazon]

We have successfully test the changes on Jenkins with both Security and No-Security tests on x64 docker container.
Due to arm64 cypress not having available binary, we have not test on arm64 yet.

[seraphjiang]

Is there an issue to track we need two parameters need to run test?

[tianleh]

Is there an issue to track we need two parameters need to run test?

Created a Github issue for the tight coupling of build and bundle in current integ test.

#761

[tianleh]

My ongoing POC work is pushed to this dev branch https://github.com/tianleh/opensearch-build/tree/add-test-support

Still in progress to address build and bundle related params

[peterzhuamazon]

Since @tianleh has mentioned in chat before, you need to set these up in your docker host to ensure cypress wont crash:

https://github.com/opensearch-project/opensearch-build/blob/main/scripts/README.md#requirements

[peternied]

[Triage] @tianleh Could you add individual tasks that would align to the PRs that you are creating where this task is broken into more digestible pieces?

[peternied]

[Triage] @peterzhuamazon is tracking this item, and the costing reflects this. If it makes sense to break this apart into more than one item please do what works best for you.

[tianleh]

Refer to the description section of this issue (top)

[tianleh]

My POC work https://github.com/tianleh/opensearch-build/tree/add-test-support is working. Focusing on moving the POC code to official repo

[tianleh]

To track PR/Issue related to release integ test, I created a new label ci-test-automation

To track those PRs which have been approved/merged with comments, I created a new label comment-to-followup
(Once remaining comments are resolved in follow up PRs, I will remove the label)

[tianleh]

Added a few items to track our work with infra team about upgrading to Chromium to

1. stabilize the test
2. support Reporting plugin

[bbarani]

@tianleh Whats the status of this issue?

[bbarani]

@tianleh @peterzhuamazon @kavilla Can you please update the current status of this issue?

[gaiksaya]

Closing this issue as we are now using opensearch-functional-test repo to run tests for all dashboards plugin. No action items pending from infra side.

Please re-open if you think any items are pending.

Thanks!