[IntegTest] Onboard all plugins into automated integration tests for the distribution

[bbarani]

Remaining items:

• [FEATURE] Run integration tests as part of the OpenSearch-Dashboards distribution notifications#433
• [FEATURE] Run integration tests as part of the OpenSearch distribution cross-cluster-replication#383
• [FEATURE] Run integration tests as part of the OpenSearch distribution performance-analyzer#198
• [Testing] Security needs integration tests that run with the distribution suite. security#1761
• [FEATURE] Support integTestRemote with security enabled endpoint reporting#348
• Add security integration tests notifications#258
• [FEATURE] Support integTestRemote with security enabled endpoint observability#676
• [BUG] Add integration tests for Job Scheduler job-scheduler#146
• [META] After tests are complete there should be a test-run manifest that allows for finding all related test artifacts #1274
• [FEATURE] Run integration tests as part of the OpenSearch distribution geospatial#115
• [FEATURE] Run integration tests as part of the OpenSearch-Dashboards distribution dashboards-maps#26
• [FEATURE] Run integration tests as part of the OpenSearch distribution security-analytics#64
• [FEATURE] Run integration tests as part of the OpenSearch-Dashboards distribution security-analytics-dashboards-plugin#17
• [FEATURE] Run integration tests as part of the OpenSearch distribution neural-search#24
• [FEATURE] Run integration tests as part of the OpenSearch-Dashboards distribution dashboards-search-relevance#8

---

Run the Integration Test cases in the OpenSearch distribution building workflow for all of the plugins
Please make sure your plugin is a part of this manifest file inorder to run integration test via build workflow

⏳ Total Status:

---

🔎 OpenSearch Plugins:

Plugin Name	X64	ARM64	Infra CICD Integrated
AD Plugin	With security without security	With security without security
SQL Plugin	With security without security	With security without security
Alerting Plugin	With security without security	With security without security
PA/RCA Plugin opensearch-project/performance-analyzer#198	With security without security	With security without security
Security Plugin (this plugin relies on multiple different cluster setup, thus not bundled with Infra IntegTest pipeline #58 (comment))	With security	With security
IM Plugin	With security without security	With security without security
KNN Plugin	With security without security	With security without security
Reports Scheduler(with-security test not exist yet track opensearch-project/reporting#348)	With security without security	With security without security
Async-search	With security without security	With security without security
Notebooks(removed plugin)	- [ ] With security - [x] without security	- [ ] With security - [x] without security	- [x]
Job Scheduler(No need to test as other plugin use it already testing it)	- [ ] With security - [ ] without security	- [ ] With security - [ ] without security	- [ ]
CCR opensearch-project/cross-cluster-replication#383	With security without security	With security without security
Notifications (opensearch-project/notifications#258)	With security without security	With security without security
ML-commons	With security without security	With security without security
Observability (they are ok to have w/o security only test @vamsi-amazon for now, will add later opensearch-project/observability#676)	With security without security	With security without security
Geospatial opensearch-project/geospatial#115	With security without security	With security without security
Security Analytics	With security without security	With security without security
Neural Search	With security without security	With security without security

---

📊 OpenSearch Dashboards Plugins:

Plugin Name	X64	ARM64	Infra CICD Integrated
AD Dashboards Plugin	With security without security	With security without security
SQL (Workbench) Dashboards Plugin	With security without security	With security without security
Alerting Dashboards Plugin	With security without security	With security without security
IM Dashboards Plugin	With security without security	With security without security
Security Dashboards Plugin	With security	With security
Notebooks Dashboards Plugin (removed plugin combined with trace analytics since 1.2.0 later as observability)	- [x] With security - [x] without security	- [x] With security - [x] without security	- [x]
Reports Dashboards Plugin	With security without security	With security without security
Gantt Chart Dashboards Plugin	With security without security	With security without security
Observability Dashboards	With security without security	With security without security
Notifications Dashboards opensearch-project/notifications#433	With security without security	With security without security
Dashboards-maps opensearch-project/dashboards-maps#26	With security without security	With security without security
Security-Analytics-Dashboards-Plugin	With security without security	With security without security
Search-Relevance-Plugin	With security without security	With security without security

[MMakaria]

I Think in a glance these are missing: Perftop/ Job Schedule / Performance Analyzer/Common Utils and Trace Analytics

[sean-zheng-amazon]

kNN with security test cases have been added for both x64 and arm, please update.

[peterzhuamazon]

kNN with security test cases have been added for both x64 and arm, please update.

Updated. Thanks.

[peterzhuamazon]

So @bowenlan-amzn has informed me that IM opensearch has Security Enabled since this PR:
opensearch-project/index-management#93

He also mentions that IM Dashboards tests are now available, but flaky:

yarn run cypress run
yarn run cypress run --env security_enabled=true

[lezzago]

I have tested Alerting dashboards on OS arm64 manually.
I was able to successfully create a chime destination and a monitor.
I was able to have a trigger in the monitor get triggered and send chime messages to the chime destination.

[bbarani]

@lezzago Please make sure to update your automated testing to cover the above (manual testing) scenarios as well.

[dblock]

What’s the criteria of testing with security vs. without or both?

[peterzhuamazon]

Seems like Async Search WITH SEC is up. Need confirmation.

[peterzhuamazon]

Seems like SQL Workbench WITH SEC is up. Need confirmation.

[peterzhuamazon]

As of 2021/11/22, SQL Workbench has SEC, Notebook dashboards 1.1 and before has SEC, Reports dashboards has SEC, Gantt Charts has NOSEC + SEC.

So all the plugins in dashboards now have NOSEC + SEC tests.

As of now, everything can use 1.2 branch for 1.2.0 tests, only observability is using main for the test cases (backport later).

Dashboards notebooks, trace analytics and observability all merged into trace analytics plugin and repo, will rename to observability after 1.2.0 release.

[peternied]

@bbarani is there a campaign associated with driving this work? Checking in here for security as we realized this is a gap we'd like to address. Created issue in our repo for tracking opensearch-project/security#1761

CC @davidlago

[cliu123]

Security plugin will provide manual sign-off: opensearch-project/security#1499 (comment).

[peterzhuamazon]

Added notificationsDashboards now.

[lezzago]

Please note that this issue is slated to be resolved in 2.1 as this is an enhancement to the current integration tests that have already onboarded for the notification plugin.

It might be better to remove this notification issue from this github issue as this current issue revolves around onboarding automated integration tests for the distribution, which Notification plugin already has. The notification github issue mentioned here is just a further enhancement of the existing integration tests.