Add support for multiple clusters for integration tests (CCR)

[dblock]

Is your feature request related to a problem? Please describe

Coming from opensearch-project/cross-cluster-replication#383 (comment), CCR needs a multi-node cluster for replication tests. Because of the different scenarios under test with security, CCR ITs create local clusters with specific configurations for each test case. An analysis put that number of different configurations at 7 clusters. This approach was discussed on this PR where I updated the language around recommendations/practices for testing in OpenSearch.

Describe the solution you'd like

Need to find a path forward to test at least basic CCR replication functionality for the complete distribution. This could be a "good enough" multi-node cluster available to all plugins to run integration tests.

Describe alternatives you've considered

No response

Additional context

#1499
#1428

[prudhvigodithi]

[Triage] Adding @ankitkala, any reason why we cant re-use the performance testing setup that is out there for integration testing?
@peterzhuamazon @gaiksaya

[ankitkala]

I think both LocalTestCluster and PerfTestCluster can be used. Primary issue is that we only create one test cluster for running the integration tests during the distro build.
Cross cluster replication requires at least 2 clusters to run the performance tests.

[bbarani]

@peterzhuamazon @gaiksaya What would it take to support multiple clusters for CCR integration test? I think we can specifically spin up multiple clusters just for CCR tests as we are running parallely now.

[peterzhuamazon]

There are several big changes needs to be done in test framework.
As of now it is assuming only one cluster so discovery is not needed to be configured.
This is not a small task and would require some code diving before we can tell the efforts. Thanks.

[dblock]

Practically speaking I would introduce test configuration settings that define the cluster configuration, e.g.:

  - name: cross-cluster-replication
    integ-test:
      topology: multi-cluster
      test-configs:
        - with-security
        - without-security

or something more elaborate

  - name: cross-cluster-replication
    integ-test:
      topology: 
        - name: cluster1
           nodes: 3
        - name: cluster2
           ...
      test-configs:
        - with-security
        - without-security
        - 

This is a good opportunity to revisit whether with-security and without-security was the right approach, and whether we can reuse blocks to avoid the boilerplate, e.g.

  - multi-cluster: &multi-cluster
     nodes: 3
  - name: cross-cluster-replication
    integ-test:
      - name: with-security
         security: true
         topology: &multi-cluster

[bbarani]

@ankitkala Would you be able to create a PR with the above changes? We currently do not have bandwidth to support customized testing process and would really appreciate if you can help implement this functionality.

[ankitkala]

I'll be OOO till 17/6. Let me see if I can pick this up after I'm back.

[ankitkala]

Needs prioritisation @manishdev-amzn

[bbarani]

@manishdev-amzn can you please prioritize this issue to automat the integration test for CCR?

[ankitkala]

@monu-aws would be working on this.

[monusingh-1]

Hi @dblock @ankitkala
Currently LocalTestCluster does not support creating multi-node clusters, also the tests to verify CCR does not depend on the clusters having multiple nodes. I am thinking of keeping the structure of the yml file to be like below

- name: cross-cluster-replication
  integ-test:
  topology: multi-cluster
    test-configs:
      - with-security
      - without-security

OR

- name: cross-cluster-replication
  integ-test:
  topology: single-cluster
    test-configs:
      - with-security
      - without-security

This can be implemented by creating the two clusters by using LocalTestCluster to create two clusters and pass the endpoints to the integtest.sh script to run the tests.
Is there any other better way to structure the manifest file ?

[ankitkala]

In that case, instead of topology should we just have a attribute like number_of_clusters: 2?

[bbarani]

@ankitkala @monu-aws Whats the current status of this issue?

[monusingh-1]

@bbarani, changes are already merged in both CCR and Build repo.
We are tracking this change for 2.6, and will enable the tests for 2.6 immediately after the 2.5 release.

[gaiksaya]

Closing as duplicate of #3448