[META] RPM, Deb and Windows integration testing lacks jenkins automation

[gaiksaya]

Describe the bug

Currently, we are running integration tests manually on windows and other distributions since the process is not automated yet.
Looks like the test workflow already has the required support for them but needs to be automated in terms of CI or Jenkins.

To reproduce

Need to create a VM and then run IT on that for all components. It is much more complicated for OSD

Expected behavior

The integration testing should be automated to save human hours.

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

No response

Relevant log output

No response

Sub tasks

• Create integration test job per distribution type
  1. Currently we have automated integration test job for TAR only
• Support —component parameter to OpenSearch Dashboards functional tests execution
  1. This is required to execute tests in parallel + Execute tests for specific plugin(s)
• Report, surface test failures along with logs ( including cluster logs ) via appropriate channel (GitHub, slack etc..)
• Resolving RPM/DEB IntegTest having config modify issues. #3156 @peterzhuamazon
• Adding RPM/DEB supported systemd docker images for integTest on Jenkins #3190 @peterzhuamazon
• [Enhancement] Add RPM/DEB integTest to Jenkins Pipelines #3198 @peterzhuamazon
• [Enhancement] Add config editing and experimental flag enabler for OSD integ-test workflow #3243 @peterzhuamazon
• [Bug]: Fix OS integration tests for Windows platform #3196 @peterzhuamazon
• [Bug]: Incorrect URL of OS build manifest passed into OSD integ test parameter  #3432 @peterzhuamazon
• [BUG] NPM not found when running integTest OSD through su <> opensearch-build-libraries#197 @peterzhuamazon
• Updated integ-tests implementation to run all the OSD plugins.  #3316 (comment) @Divyaasm
  • Add an optional parameter -o to integtest.sh  opensearch-dashboards-functional-test#644
  • Notify the test failures for OSD Integtest build with Github Issues #3476
• [Bug]: OpenSearch integration test deploys stages for components absent from test manifest #3461
• Execute integration tests at nightly cadence ( Along from release level tests )
• Add necessary tests for all Jenkins files

Acceptance Criteria

• Integration tests should run via test_workflow (python framework) with jenkins as a wrapper
• Integration tests should run on all supported platforms (eg: tar, rpm and deb running on linux)
• The workflow should run independently on any workflow. Distribution workflow should not be blocked just because integration testing is taking a long time.
• Integration test failures should create GH issues on respective GH repos similar to build failure issue

[peterzhuamazon]

Related to

• Automate integ test using RPM bundle. #1951
• [WIN M3] WIN (ZIP/MSI) Validation #2308

[peterzhuamazon]

I would rename this issue to Jenkins automation, as they both have the Python code automation completed, tho with bugs.

Thanks.

[gaiksaya]

Sure. Do you know the bugs @peterzhuamazon ?

[bbarani]

@gaiksaya @peterzhuamazon Lets make sure we close the gap for Debian distribution as well.

[peterzhuamazon]

• Resolving RPM/DEB IntegTest having config modify issues. #3156

[bbarani]

@peterzhuamazon Referencing #2144 to execute parallel integ tests for OpenSearch Dashboards

[peterzhuamazon]

New issue:

• Adding RPM/DEB supported systemd docker images for integTest on Jenkins #3190
• Add RPM systemd images with non-root sudo support #3191

[peterzhuamazon]

• [Enhancement] Add RPM/DEB integTest to Jenkins Pipelines #3198

[peterzhuamazon]

Hi,

More issues:

1. windows integTest automation: [Bug]: Fix OS integration tests for Windows platform #3196
2. distribution osd send wrong link to integTest: [Bug]: Incorrect URL of OS build manifest passed into OSD integ test parameter  #3432
3. osd integTest on deb/rpm have path errors which exit the test: [BUG] NPM not found when running integTest OSD through su <> opensearch-build-libraries#197
4. osd integTest parallel issues due to functional test repo script have TEST_TYPE=default vs manifest: Updated integ-tests implementation to run all the OSD plugins.  #3316 (comment) (Need to add entry in test_finder.sh and integtest.sh to make TEST_TYPE=default for Jenkins run so that components can be called individually)

• Add an optional parameter -o to integtest.sh  opensearch-dashboards-functional-test#644

Thanks.

[bbarani]

@zelinh Can you provide an update here?

[peterzhuamazon]

Windows integTest is completed with windows container here:

• [Bug]: Windows CI builds failing to find docker (Update to Run Windows inside docker containers) opensearch-ci#281

[peterzhuamazon]

This means RPM/DEB/Windows all have integration Test today.

[peterzhuamazon]

Close this issue now as it will be handled in the above issue for Windows.

[peterzhuamazon]

• [Windows Container] Add Windows IntegTest support on Jenkins Libs opensearch-build-libraries#308
• [Windows Container] Make sure WindowsZip/LinuxTar remove correct dir when uninstall #4023
• [Windows Container] Refactor integTest java version handling logic for all distributions opensearch-build-libraries#314
• [Windows Container] Add Windows IntegTest support to Jenkins with Windows Docker Container #4027

[gaiksaya]

I think this issue was more for automation related part. This PR should add it now.#4368