Stablize Infrastructure on Jenkins IntegTest regarding setup test env before running the test

[eirsep]

Describe the bug

Security Analytics plugin is seeing integ test failures in 2.10 RC build on jenkins
opensearch-project/security-analytics#594

We are unable to reproduce the client timeout exception which is seen in above ci failure. We suspect this may be due to resource crunch on instance which is simultaneously running different containers with each container running builds for different plugins

To reproduce

opensearch-project/security-analytics#594

Expected behavior

Reserve right amount of hardware for each container to ensure there is no test failure due to hardware settings or CPU/memory crunch.

Screenshots

opensearch-project/security-analytics#594

Host / Environment

opensearch-project/security-analytics#594

Additional context

opensearch-project/security-analytics#594

Relevant log output

https://github.com/opensearch-project/security-analytics/issues/594

[peterzhuamazon]

PRs:

• Provision integTest container with resource locking #4037
• Make all integTest having resource locking #4039

[peterzhuamazon]

Those two plugins are very consistent now after the changes in #4037.
I will propagate this change to all the integTest and build docker now.
https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/6012/pipeline
https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/6013/pipeline
https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/6014/pipeline
https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/6015/pipeline
https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/6017/pipeline
https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/6016/pipeline
https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/6020/pipeline
https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/6021/pipeline

Thanks.

[gaiksaya]

Hi @eirsep @praveensameneni

We reverted above resource constraint and looks like tests till went through. Since tests were ignored in recent RC I believe issue is on plugin side. No issue found on infra side.

[peterzhuamazon]

Need more investigation.

Can try this new combination just for integTest:

1. m5 2xlarge runner with 2 executors, each 4cpu/16g.
2. Add pigz to speed up the stack/unstack process.
3. Windows still need some observation at this point.

Thanks.

[praveensameneni]

Hi @eirsep @praveensameneni

We reverted above resource constraint and looks like tests till went through. Since tests were ignored in recent RC I believe issue is on plugin side. No issue found on infra side.

@gaiksaya , can we re-revert the change and run the tests until the release.

[bbarani]

@gaiksaya @peterzhuamazon Do we have anything pending on this issue or can we close it?

[gaiksaya]

We were unable to conclude if infra was actually affecting the tests since tests were ignored before they can be tested. Would suggest to not ignore the tests and keep monitoring for 2.11.0.

[Divyaasm]

Closing the issue . Please refer to the related meta issue #3966 for more information