[BUG] CrossClusterSearchIT fails when run against a remote cluster with security plugin turned on

[acarbonetto]

What is the bug?
CrossClusterSearchIT fails when the security plugin is enabled. We should update the cross cluster search IT tests so that they work when the security plugin is working on the cluster.

How can one reproduce the bug?
Create a multicluster with the security plugin enabled.
CrossClusterSearchIT will be unable to hit the client due to SSL

  2> java.lang.IllegalStateException: Failed to perform request
        at __randomizedtesting.SeedInfo.seed([2B9C4ECD5AA96CAA:1BF8BCC4ED529521]:0)
        at org.opensearch.sql.legacy.TestUtils.isIndexExist(TestUtils.java:89)
        at org.opensearch.sql.legacy.SQLIntegTestCase.loadIndex(SQLIntegTestCase.java:201)
        at org.opensearch.sql.ppl.CrossClusterSearchIT.init(CrossClusterSearchIT.java:38)

        Caused by:
        java.io.IOException: Unrecognized SSL message, plaintext connection?
            at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:958)
            at org.opensearch.client.RestClient.performRequest(RestClient.java:332)
            at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
            at org.opensearch.sql.legacy.TestUtils.isIndexExist(TestUtils.java:86)
            ... 2 more

            Caused by:
            javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
                at java.base/sun.security.ssl.SSLEngineInputRecord.bytesInCompletePacket(SSLEngineInputRecord.java:145)
                at java.base/sun.security.ssl.SSLEngineInputRecord.bytesInCompletePacket(SSLEngineInputRecord.java:64)
                at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:612)
                at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)
                at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)
                at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
                at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:278)
                at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:332)
                at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:547)
                at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
                at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
                at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
                at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
                at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
                at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
                at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
                at java.base/java.lang.Thread.run(Thread.java:833)

REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests "org.opensearch.sql.ppl.CrossClusterSearchIT" -Dtests.seed=2B9C4ECD5AA96CAA -Dtests.security.manager=false -Dtests.locale=en-IN -Dtests.timezone=America/El_Salvador -Druntime.java=17

Related:

• AUTOCUT ticket: [AUTOCUT] Integration Test failed for sql: 2.8.0 tar distribution #1700
• AUTOCUT release logs pass without security plugin: stdout.txt
• AUTOCUT release logs fail with security plugin: stdout.txt
• Note: failures not seen in 2.8 release: https://github.com/opensearch-project/sql/actions/runs/5126820875/jobs/9222179630

Do you have any screenshots?
N/A

Do you have any additional context?

• Maybe follow how AbstractMultiClustersTestCase sets up their test cases and how their clients deal with the security plugin.
• Cross-cluster replication has these tests running with the security plugin enabled. See: https://github.com/opensearch-project/cross-cluster-replication/blob/main/build.gradle#L376

[Yury-Fridlyand]

I think we also need to run all ITs with all plugins and with Security on all release branches (x.y) on every push/PR.

[acarbonetto]

I think we also need to run all ITs with all plugins and with Security on all release branches (x.y) on every push/PR.

related: #1713

This ticket is to clean up the cross cluster search test engine so that it works with security plugin turned on