Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add backwards compatibility tests for automation #1417

Closed
VachaShah opened this issue Aug 23, 2021 · 16 comments
Closed

Add backwards compatibility tests for automation #1417

VachaShah opened this issue Aug 23, 2021 · 16 comments
Assignees
@cliu123
Labels
backwards-compatibility

Comments

@VachaShah
Copy link
Contributor

Add backwards compatibility tests for the plugin to automate end-to-end upgrade paths for faster releases. Developer documentation for implementing bwc tests and hooking them to CI: https://github.com/opensearch-project/opensearch-plugins/blob/main/TESTING.md#backwards-compatibility-testing
@VachaShah VachaShah mentioned this issue Aug 23, 2021
Closed
16 tasks
@peternied peternied self-assigned this Mar 1, 2022
This was referenced Mar 2, 2022
Merged
Closed
@cliu123
Copy link
Member

cliu123 commented Mar 8, 2022

@opensearch-project/opensearch-core Security plugin BWC tests require to run some scripts to install some demo certificates before start clusters. This needs some help from OpenSearch core team. Anyone can help with this? Cc: @CEHENKLE @saratvemulapalli @VachaShah

@saratvemulapalli
Copy link
Member

@cliu123 does it have to be manual scripts? What exactly are you trying to install?

@cliu123
Copy link
Member

cliu123 commented Mar 8, 2022 •
edited
Loading

@saratvemulapalli Thanks for jumping on this! Security plugin needs to run this script to generate demo certificates and to update the configurations in elasticsearch.yml/opensearch.yml before starting cluster. The command to run in ES/OS cluster is sh plugins/opensearch-security/tools/install_demo_configuration.sh

@saratvemulapalli
Copy link
Member

saratvemulapalli commented Mar 8, 2022 •
edited
Loading

I know @VachaShah is taking a deeper look. But see if this works for you:
You should be able to install certificates for these test clusters.
Take a look at: Index Management plugin.

You can generate them on the fly, put it in your test resources and feed it to the test clusters.

@VachaShah
Copy link
Contributor Author

Thank you @saratvemulapalli for pointing this out! @cliu123 Can you try this? All the settings that can be added to a test cluster node are: https://github.com/opensearch-project/OpenSearch/blob/main/buildSrc%2Fsrc%2Fmain%2Fjava%2Forg%2Fopensearch%2Fgradle%2Ftestclusters%2FOpenSearchNode.java

@cliu123
Copy link
Member

cliu123 commented Mar 8, 2022

@saratvemulapalli @VachaShah Thanks for the suggestion! Let me try that.

@cliu123
Copy link
Member

cliu123 commented Mar 9, 2022 •
edited
Loading

The changes that @saratvemulapalli pointed to seem have resolved the certificates and configuration issue. And not nodes are not connecting to each other.
@saratvemulapalli @VachaShah Have you seen this?

=== Standard output of node `node{::securityBwcCluster-2}` ===

»    ↓ errors and warnings from /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/logs/es.stdout.log ↓
» WARN ][c.a.o.s.OpenDistroSecurityPlugin] [securityBwcCluster-2] Directory /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config has insecure file permissions (should be 0700)
» WARN ][c.a.o.s.OpenDistroSecurityPlugin] [securityBwcCluster-2] File /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config/root-ca.pem has insecure file permissions (should be 0600)
» WARN ][c.a.o.s.OpenDistroSecurityPlugin] [securityBwcCluster-2] File /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config/kirk-key.pem has insecure file permissions (should be 0600)
» WARN ][c.a.o.s.OpenDistroSecurityPlugin] [securityBwcCluster-2] File /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config/kirk.pem has insecure file permissions (should be 0600)
» WARN ][c.a.o.s.OpenDistroSecurityPlugin] [securityBwcCluster-2] File /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config/elasticsearch.yml has insecure file permissions (should be 0600)
» WARN ][c.a.o.s.OpenDistroSecurityPlugin] [securityBwcCluster-2] File /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config/esnode-key.pem has insecure file permissions (should be 0600)
» WARN ][c.a.o.s.OpenDistroSecurityPlugin] [securityBwcCluster-2] File /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config/esnode.pem has insecure file permissions (should be 0600)
» WARN ][c.a.o.s.OpenDistroSecurityPlugin] [securityBwcCluster-2] Directory /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config/jvm.options.d has insecure file permissions (should be 0700)
» WARN ][c.a.o.s.c.Salt           ] [securityBwcCluster-2] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
» WARN ][o.e.g.DanglingIndicesState] [securityBwcCluster-2] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
» WARN ][o.e.d.FileBasedSeedHostsProvider] [securityBwcCluster-2] expected, but did not find, a dynamic hosts list at [/local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config/unicast_hosts.txt]
» WARN ][stderr                   ] [securityBwcCluster-2] java.lang.reflect.InvocationTargetException
» WARN ][stderr                   ] [securityBwcCluster-2]      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
» WARN ][stderr                   ] [securityBwcCluster-2]      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
» WARN ][stderr                   ] [securityBwcCluster-2]      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
» WARN ][stderr                   ] [securityBwcCluster-2]      at java.base/java.lang.reflect.Method.invoke(Method.java:564)
» WARN ][stderr                   ] [securityBwcCluster-2]      at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
» WARN ][stderr                   ] [securityBwcCluster-2]      at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535)
» WARN ][stderr                   ] [securityBwcCluster-2] Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.jdk.internal.org.objectweb.asm")
» WARN ][stderr                   ] [securityBwcCluster-2]      at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
» WARN ][stderr                   ] [securityBwcCluster-2]      at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
» WARN ][stderr                   ] [securityBwcCluster-2]      at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
» WARN ][stderr                   ] [securityBwcCluster-2]      at java.base/java.lang.SecurityManager.checkPackageAccess(SecurityManager.java:1324)
» WARN ][stderr                   ] [securityBwcCluster-2]      at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:164)
» WARN ][stderr                   ] [securityBwcCluster-2]      at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
» WARN ][stderr                   ] [securityBwcCluster-2]      at Log4jHotPatch.asmVersion(Log4jHotPatch.java:71)
» WARN ][stderr                   ] [securityBwcCluster-2]      at Log4jHotPatch.agentmain(Log4jHotPatch.java:93)
» WARN ][stderr                   ] [securityBwcCluster-2]      ... 6 more
» WARN ][c.a.o.s.a.r.AuditMessageRouter] [securityBwcCluster-2] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
» WARN ][o.e.c.NodeConnectionsService] [securityBwcCluster-2] failed to connect to {securityBwcCluster-0}{cdALqi-wR9i1DlK4oEXSow}{oXIPF_J1TlWUhswWp1MfwQ}{127.0.0.1}{127.0.0.1:43471}{dimr}{testattr=test} (tried [1] times)
»  org.elasticsearch.transport.ConnectTransportException: [securityBwcCluster-0][127.0.0.1:43471] connect_exception
»       at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:978) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:198) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.10.2.jar:7.10.2]
»       at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
»       at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883) ~[?:?]
»       at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2315) ~[?:?]
»       at org.elasticsearch.common.concurrent.CompletableContext.addListener(CompletableContext.java:45) ~[elasticsearch-core-7.10.2.jar:7.10.2]
»       at org.elasticsearch.transport.netty4.Netty4TcpChannel.addConnectListener(Netty4TcpChannel.java:121) ~[?:?]
»       at org.elasticsearch.transport.TcpTransport.initiateConnection(TcpTransport.java:324) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:291) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.transport.ClusterConnectionManager.internalOpenConnection(ClusterConnectionManager.java:254) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.transport.ClusterConnectionManager.connectToNode(ClusterConnectionManager.java:139) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:380) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:364) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.NodeConnectionsService$ConnectionTarget$1.doRun(NodeConnectionsService.java:314) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]
»       at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
»       at org.elasticsearch.cluster.NodeConnectionsService.connectToNodes(NodeConnectionsService.java:137) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.ClusterApplierService.connectToNodesAndWait(ClusterApplierService.java:490) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:458) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:418) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.ClusterApplierService.access$000(ClusterApplierService.java:68) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:162) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
»       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
»       at java.lang.Thread.run(Thread.java:832) [?:?]
»  Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 127.0.0.1/127.0.0.1:43471
»  Caused by: java.net.ConnectException: Connection refused
»       at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
»       at sun.nio.ch.Net.pollConnectNow(Net.java:660) ~[?:?]
»       at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:875) ~[?:?]
»       at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]
»       at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
»       at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[?:?]
»       at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
»       at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
»       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
»       at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
»       at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
»       ... 1 more
» WARN ][o.e.c.r.a.AllocationService] [securityBwcCluster-2] [security-auditlog-2022.03.09][0] marking unavailable shards as stale: [AV6qIRB_QauLztzRdaimiQ]
»   ↓ last 40 non error or warning messages from /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/logs/es.stdout.log ↓
» [2022-03-09T00:54:30,981][DEBUG][o.e.c.c.PreVoteCollector ] [securityBwcCluster-2] PreVotingRound{preVotesReceived={{securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}=PreVoteResponse{currentTerm=2, lastAcceptedTerm=2, lastAcceptedVersion=22}}, electionStarted=false, preVoteRequest=PreVoteRequest{sourceNode={securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}, currentTerm=2}, isClosed=false} added PreVoteResponse{currentTerm=2, lastAcceptedTerm=2, lastAcceptedVersion=22} from {securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}, no quorum yet
» [2022-03-09T00:54:30,982][DEBUG][o.e.c.c.PreVoteCollector ] [securityBwcCluster-2] PreVotingRound{preVotesReceived={{securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}=PreVoteResponse{currentTerm=2, lastAcceptedTerm=2, lastAcceptedVersion=22}, {securityBwcCluster-1}{QpMkz_inQwWBYHP1DKF7AQ}{2Jg1OUeOSl2yroiltJPbOQ}{127.0.0.1}{127.0.0.1:33435}{dimr}{testattr=test}=PreVoteResponse{currentTerm=2, lastAcceptedTerm=2, lastAcceptedVersion=22}}, electionStarted=true, preVoteRequest=PreVoteRequest{sourceNode={securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}, currentTerm=2}, isClosed=false} added PreVoteResponse{currentTerm=2, lastAcceptedTerm=2, lastAcceptedVersion=22} from {securityBwcCluster-1}{QpMkz_inQwWBYHP1DKF7AQ}{2Jg1OUeOSl2yroiltJPbOQ}{127.0.0.1}{127.0.0.1:33435}{dimr}{testattr=test}, starting election
» [2022-03-09T00:54:30,983][DEBUG][o.e.c.c.Coordinator      ] [securityBwcCluster-2] starting election with StartJoinRequest{term=3,node={securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}}
» [2022-03-09T00:54:30,984][DEBUG][o.e.c.c.Coordinator      ] [securityBwcCluster-2] joinLeaderInTerm: for [{securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}] with term 3
» [2022-03-09T00:54:30,984][DEBUG][o.e.c.c.CoordinationState] [securityBwcCluster-2] handleStartJoin: leaving term [2] due to StartJoinRequest{term=3,node={securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}}
» [2022-03-09T00:54:30,991][DEBUG][o.e.c.c.JoinHelper       ] [securityBwcCluster-2] attempting to join {securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test} with JoinRequest{sourceNode={securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}, minimumTerm=2, optionalJoin=Optional[Join{term=3, lastAcceptedTerm=2, lastAcceptedVersion=22, sourceNode={securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}, targetNode={securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}}]}
» [2022-03-09T00:54:30,992][DEBUG][o.e.c.c.JoinHelper       ] [securityBwcCluster-2] successful response to StartJoinRequest{term=3,node={securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}} from {securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}
» [2022-03-09T00:54:30,993][DEBUG][o.e.c.c.CoordinationState] [securityBwcCluster-2] handleJoin: added join Join{term=3, lastAcceptedTerm=2, lastAcceptedVersion=22, sourceNode={securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}, targetNode={securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}} from [{securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}] for election, electionWon=false lastAcceptedTerm=2 lastAcceptedVersion=22
» [2022-03-09T00:54:30,993][DEBUG][o.e.c.c.JoinHelper       ] [securityBwcCluster-2] successful response to StartJoinRequest{term=3,node={securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}} from {securityBwcCluster-1}{QpMkz_inQwWBYHP1DKF7AQ}{2Jg1OUeOSl2yroiltJPbOQ}{127.0.0.1}{127.0.0.1:33435}{dimr}{testattr=test}
» [2022-03-09T00:54:30,995][DEBUG][o.e.c.c.CoordinationState] [securityBwcCluster-2] handleJoin: added join Join{term=3, lastAcceptedTerm=2, lastAcceptedVersion=22, sourceNode={securityBwcCluster-1}{QpMkz_inQwWBYHP1DKF7AQ}{2Jg1OUeOSl2yroiltJPbOQ}{127.0.0.1}{127.0.0.1:33435}{dimr}{testattr=test}, targetNode={securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}} from [{securityBwcCluster-1}{QpMkz_inQwWBYHP1DKF7AQ}{2Jg1OUeOSl2yroiltJPbOQ}{127.0.0.1}{127.0.0.1:33435}{dimr}{testattr=test}] for election, electionWon=true lastAcceptedTerm=2 lastAcceptedVersion=22
» [2022-03-09T00:54:30,995][DEBUG][o.e.c.c.CoordinationState] [securityBwcCluster-2] handleJoin: election won in term [3] with VoteCollection{votes=[QpMkz_inQwWBYHP1DKF7AQ, IOsUBrCnQVyziiLa5cMe2w], joins=[Join{term=3, lastAcceptedTerm=2, lastAcceptedVersion=22, sourceNode={securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}, targetNode={securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}}, Join{term=3, lastAcceptedTerm=2, lastAcceptedVersion=22, sourceNode={securityBwcCluster-1}{QpMkz_inQwWBYHP1DKF7AQ}{2Jg1OUeOSl2yroiltJPbOQ}{127.0.0.1}{127.0.0.1:33435}{dimr}{testattr=test}, targetNode={securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}}]}
» [2022-03-09T00:54:30,995][DEBUG][o.e.c.c.Coordinator      ] [securityBwcCluster-2] handleJoinRequest: coordinator becoming LEADER in term 3 (was CANDIDATE, lastKnownLeader was [Optional[{securityBwcCluster-0}{cdALqi-wR9i1DlK4oEXSow}{oXIPF_J1TlWUhswWp1MfwQ}{127.0.0.1}{127.0.0.1:43471}{dimr}{testattr=test}]])
» [2022-03-09T00:54:31,006][DEBUG][o.e.c.c.JoinHelper       ] [securityBwcCluster-2] received a join request for an existing node [{securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}]
» [2022-03-09T00:54:31,007][DEBUG][o.e.c.c.JoinHelper       ] [securityBwcCluster-2] received a join request for an existing node [{securityBwcCluster-1}{QpMkz_inQwWBYHP1DKF7AQ}{2Jg1OUeOSl2yroiltJPbOQ}{127.0.0.1}{127.0.0.1:33435}{dimr}{testattr=test}]
» [2022-03-09T00:54:31,010][INFO ][o.e.c.s.MasterService    ] [securityBwcCluster-2] elected-as-master ([2] nodes joined)[{securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test} elect leader, {securityBwcCluster-1}{QpMkz_inQwWBYHP1DKF7AQ}{2Jg1OUeOSl2yroiltJPbOQ}{127.0.0.1}{127.0.0.1:33435}{dimr}{testattr=test} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 3, version: 23, delta: master node changed {previous [], current [{securityBwcCluster-2}{IOsUBrCnQVyziiLa5cMe2w}{28LfMiQoQq2vTeNhZ9X-wQ}{127.0.0.1}{127.0.0.1:41881}{dimr}{testattr=test}]}
» [2022-03-09T00:54:31,020][DEBUG][o.e.c.c.FollowersChecker ] [securityBwcCluster-2] FollowerChecker{discoveryNode={securityBwcCluster-0}{cdALqi-wR9i1DlK4oEXSow}{oXIPF_J1TlWUhswWp1MfwQ}{127.0.0.1}{127.0.0.1:43471}{dimr}{testattr=test}, failureCountSinceLastSuccess=1, [cluster.fault_detection.follower_check.retry_count]=3} disconnected
»  org.elasticsearch.transport.NodeNotConnectedException: [securityBwcCluster-0][127.0.0.1:43471] Node not connected
»       at org.elasticsearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:189) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:682) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:602) [elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:317) [elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.coordination.FollowersChecker$FollowerChecker.start(FollowersChecker.java:295) [elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.coordination.FollowersChecker.lambda$setCurrentNodes$3(FollowersChecker.java:150) [elasticsearch-7.10.2.jar:7.10.2]
»       at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) [?:?]
»       at java.util.Iterator.forEachRemaining(Iterator.java:133) [?:?]
»       at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) [?:?]
»       at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) [?:?]
»       at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) [?:?]
»       at java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312) [?:?]
»       at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) [?:?]
»       at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) [?:?]
»       at org.elasticsearch.cluster.coordination.FollowersChecker.setCurrentNodes(FollowersChecker.java:143) [elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1113) [elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) [elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) [elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) [elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.10.2.jar:7.10.2]
»       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
»       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
»       at java.lang.Thread.run(Thread.java:832) [?:?]

@VachaShah
Copy link
Contributor Author

@cliu123 It looks to be some issue with the SSL record:

» ERROR][c.a.o.s.s.h.n.OpenDistroSecuritySSLNettyHttpServerTransport] [securityBwcCluster-0] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f5f636c75737465722f6865616c74683f776169745f666f725f6e6f6465733d3e3d3126776169745f666f725f7374617475733d79656c6c6f7720485454502f312e310d0a557365722d4167656e743a204a6176612f31340d0a486f73743a205b3a3a315d3a33363537310d0a4163636570743a20746578742f68746d6c2c20696d6167652f6769662c20696d6167652f6a7065672c202a3b20713d2e322c202a2f2a3b20713d2e320d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a0d0a

The first node has this issue, so once security is setup, the other nodes are not able to join since the first node is the primary node.

@cliu123
Copy link
Member

cliu123 commented Mar 9, 2022 •
edited
Loading

@VachaShah But isn't this supposed to install security plugin on each of the nodes? Why security plugin is only installed on the first node, but not installed on other nodes?
BTW, the first node does not start either:

=== Standard error of node `node{::securityBwcCluster-1}` ===
»   ↓ last 40 non error or warning messages from /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-1/logs/es.stderr.log ↓
» Exception in thread "Attach Listener" Agent failed to start!

=== Standard output of node `node{::securityBwcCluster-2}` ===

»    ↓ errors and warnings from /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/logs/es.stdout.log ↓
» WARN ][c.a.o.s.OpenDistroSecurityPlugin] [securityBwcCluster-2] Directory /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config has insecure file permissions (should be 0700)
» WARN ][c.a.o.s.OpenDistroSecurityPlugin] [securityBwcCluster-2] File /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config/root-ca.pem has insecure file permissions (should be 0600)
» WARN ][c.a.o.s.OpenDistroSecurityPlugin] [securityBwcCluster-2] File /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config/kirk-key.pem has insecure file permissions (should be 0600)
» WARN ][c.a.o.s.OpenDistroSecurityPlugin] [securityBwcCluster-2] File /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config/kirk.pem has insecure file permissions (should be 0600)
» WARN ][c.a.o.s.OpenDistroSecurityPlugin] [securityBwcCluster-2] File /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config/elasticsearch.yml has insecure file permissions (should be 0600)
» WARN ][c.a.o.s.OpenDistroSecurityPlugin] [securityBwcCluster-2] File /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config/esnode-key.pem has insecure file permissions (should be 0600)

Looks like the nodes were initialized, but they couldn't join the cluster for some reason.

[2022-03-09T07:06:37,897][INFO ][c.a.o.s.c.ConfigurationRepository] [securityBwcCluster1-1] Node 'securityBwcCluster1-1' initialized

[2022-03-09T07:06:30,428][INFO ][c.a.o.s.c.ConfigurationRepository] [securityBwcCluster1-2] Node 'securityBwcCluster1-2' initialized

» [2022-03-09T07:07:08,252][INFO ][o.e.c.s.ClusterApplierService] [securityBwcCluster1-2] removed {{securityBwcCluster1-0}{baO3-MsHQUeiqqhkWZP86w}{OVRuH7tWTQuYVIw_sYmPlQ}{127.0.0.1}{127.0.0.1:42343}{dimr}{testattr=test}}, term: 1, version: 24, reason: ApplyCommitRequest{term=1, version=24, sourceNode={securityBwcCluster1-1}{xXWkXAIaQHCjk066Z3nQvg}{x1E1-1OURCy0vRvwtGHe9A}{127.0.0.1}{127.0.0.1:45889}{dimr}{testattr=test}}
» [2022-03-09T07:07:08,275][DEBUG][o.e.c.c.PublicationTransportHandler] [securityBwcCluster1-2] received diff cluster state version [25] with uuid [Cgobw-FJTWuCJd5Acs1wBg], diff size [377]
» [2022-03-09T07:07:08,587][DEBUG][o.e.c.c.LeaderChecker    ] [securityBwcCluster1-2] leader [{securityBwcCluster1-1}{xXWkXAIaQHCjk066Z3nQvg}{x1E1-1OURCy0vRvwtGHe9A}{127.0.0.1}{127.0.0.1:45889}{dimr}{testattr=test}] disconnected
» [2022-03-09T07:07:08,588][INFO ][o.e.c.c.Coordinator      ] [securityBwcCluster1-2] master node [{securityBwcCluster1-1}{xXWkXAIaQHCjk066Z3nQvg}{x1E1-1OURCy0vRvwtGHe9A}{127.0.0.1}{127.0.0.1:45889}{dimr}{testattr=test}] failed, restarting discovery
»  org.elasticsearch.transport.NodeDisconnectedException: [securityBwcCluster1-1][127.0.0.1:45889][disconnected] disconnected
» [2022-03-09T07:07:08,590][DEBUG][o.e.c.c.Coordinator      ] [securityBwcCluster1-2] onLeaderFailure: coordinator becoming CANDIDATE in term 1 (was FOLLOWER, lastKnownLeader was [Optional[{securityBwcCluster1-1}{xXWkXAIaQHCjk066Z3nQvg}{x1E1-1OURCy0vRvwtGHe9A}{127.0.0.1}{127.0.0.1:45889}{dimr}{testattr=test}]])
» [2022-03-09T07:07:08.590Z] [BUILD] Stopping node

=== Standard error of node `node{::securityBwcCluster1-2}` ===
»   ↓ last 40 non error or warning messages from /local/home/git/security/bwc-test/build/testclusters/securityBwcCluster1-2/logs/es.stderr.log ↓
» Exception in thread "Attach Listener" Agent failed to start!

@VachaShah
Copy link
Contributor Author

Security plugin looks to be getting installed on all nodes when the nodes are created, from the logs:

-> Installed opendistro_security
Starting process 'command './bin/elasticsearch-keystore''. Working directory: /home/ubuntu/security/bwc-test/build/testclusters/securityBwcCluster-0/distro/7.10.2-ARCHIVE Command: ./bin/elasticsearch-keystore -v create

-> Installed opendistro_security
Starting process 'command './bin/elasticsearch-keystore''. Working directory: /home/ubuntu/security/bwc-test/build/testclusters/securityBwcCluster-1/distro/7.10.2-ARCHIVE Command: ./bin/elasticsearch-keystore -v create

-> Installed opendistro_security
Starting process 'command './bin/elasticsearch-keystore''. Working directory: /home/ubuntu/security/bwc-test/build/testclusters/securityBwcCluster-2/distro/7.10.2-ARCHIVE Command: ./bin/elasticsearch-keystore -v create

@cliu123
Copy link
Member

cliu123 commented Mar 9, 2022 •
edited
Loading

@VachaShah Thanks! Yes, security plugin can be installed on all nodes, but cluster formation fails.

Discussed with @dbbaughe on the cluster formation issue, and confirmed that ISM security-enabled tests never worked on internal cuslter formed by ES/OS test framework. The internal groovy code that was constructing the internal cluster in ES core had hardcoded branching logic that was specific to their security implementation, so OpenSearch test framework(forked from ES test framework) cannot form a cluster with security enabled. This needs to fix the buildSrc in core where the internal cluster is being formed. buildSrc has logic in it that was specific to Elastics implementation of security which wasn't something that we could easily hook into and set the values we needed to set up the security plugin. With the current issue, running security-enabled tests with ./gradlew does not work. ISM has to point their security-enabled integTests to external cluster to run those tests.

@opensearch-project/engineering-effectiveness Would you please point me to where the process to create an external test cluster with security enabled and to point tests to the cluster?
Cc: @bbarani @peterzhuamazon

2022-03-09T19:45:25.336+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong:
2022-03-09T19:45:25.336+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Execution failed for task ':securityBwcCluster#oldVersionClusterTask1'.
2022-03-09T19:45:25.336+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > `cluster{::securityBwcCluster1}` failed to wait for cluster health yellow after 40 SECONDS
2022-03-09T19:45:25.336+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]     IO error while waiting cluster
2022-03-09T19:45:25.336+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]     Unexpected end of file from server

2022-03-09T22:00:46.357+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][o.e.c.NodeConnectionsService] [securityBwcCluster1-1] failed to connect to {securityBwcCluster1-0}{YipzTfYVT5uJgdIptIv5jg}{VhgGjEmZQICgpd1hW5YS6Q}{127.0.0.1}{127.0.0.1:40729}{dimr}{testattr=test} (tried [1] times)
»  org.elasticsearch.transport.ConnectTransportException: [securityBwcCluster1-0][127.0.0.1:40729] connect_exception
»       at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:978) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:198) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.10.2.jar:7.10.2]
»       at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
»       at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883) ~[?:?]
»       at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2315) ~[?:?]
»       at org.elasticsearch.common.concurrent.CompletableContext.addListener(CompletableContext.java:45) ~[elasticsearch-core-7.10.2.jar:7.10.2]
»       at org.elasticsearch.transport.netty4.Netty4TcpChannel.addConnectListener(Netty4TcpChannel.java:121) ~[?:?]
»       at org.elasticsearch.transport.TcpTransport.initiateConnection(TcpTransport.java:324) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:291) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.transport.ClusterConnectionManager.internalOpenConnection(ClusterConnectionManager.java:254) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.transport.ClusterConnectionManager.connectToNode(ClusterConnectionManager.java:139) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:380) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:364) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.NodeConnectionsService$ConnectionTarget$1.doRun(NodeConnectionsService.java:314) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]
»       at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
»       at org.elasticsearch.cluster.NodeConnectionsService.connectToNodes(NodeConnectionsService.java:137) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.ClusterApplierService.connectToNodesAndWait(ClusterApplierService.java:490) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:458) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:418) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.ClusterApplierService.access$000(ClusterApplierService.java:68) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:162) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-7.10.2.jar:7.10.2]
»       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
»       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
»       at java.lang.Thread.run(Thread.java:832) [?:?]
»  Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 127.0.0.1/127.0.0.1:40729
»  Caused by: java.net.ConnectException: Connection refused
»       at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
»       at sun.nio.ch.Net.pollConnectNow(Net.java:660) ~[?:?]
»       at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:875) ~[?:?]
»       at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]
»       at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
»       at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[?:?]
»       at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
»       at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
»       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
»       at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
»       at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
»       ... 1 more

@gaiksaya
Copy link
Member

gaiksaya commented Mar 9, 2022

https://github.com/orgs/opensearch-project/teams/engineering-effectiveness Would you please point me to where the process to create an external test cluster with security enabled and to point tests to the cluster?
Cc: @bbarani @peterzhuamazon

Hi @cliu123 You can check our code base for running integration on external (localhost:9200) local clusters here https://github.com/opensearch-project/opensearch-build/blob/main/src/test_workflow/integ_test/integ_test_suite_opensearch.py.
We deploy the tarball on the host, making opensearch cluster available at localhost:9200 and run test against it.
I hope that answers your query.

@cliu123 cliu123 removed the v1.3.0 label Mar 9, 2022
@cliu123
Copy link
Member

cliu123 commented Mar 9, 2022 •
edited
Loading

@gaiksaya Thank you!
Looks like BWC tests do not run against external cluster. Only integTests do.

@opensearch-project/opensearch-core Would you please help look into this cluster formation issue?
Cc: @VachaShah @saratvemulapalli

More logs:

2022-03-09T23:42:28.070+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » [2022-03-09T23:41:57,528][INFO ][c.a.o.s.c.ConfigurationRepository] [securityBwcCluster-2] Node 'securityBwcCluster-2' initialized
2022-03-09T23:42:28.070+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » [2022-03-09T23:42:27,735][DEBUG][o.e.c.c.FollowersChecker ] [securityBwcCluster-2] FollowerChecker{discoveryNode={securityBwcCluster-0}{EbPGUjl1QtOX2wHqiePqfw}{1DlbeQnaR_G-DJa5oCoJiw}{127.0.0.1}{127.0.0.1:36227}{dimr}{testattr=test}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty
2022-03-09T23:42:28.070+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » [2022-03-09T23:42:27,737][INFO ][o.e.c.r.a.AllocationService] [securityBwcCluster-2] updating number_of_replicas to [1] for indices [.opendistro_security]
2022-03-09T23:42:28.070+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » [2022-03-09T23:42:27,739][INFO ][o.e.c.s.MasterService    ] [securityBwcCluster-2] node-left[{securityBwcCluster-0}{EbPGUjl1QtOX2wHqiePqfw}{1DlbeQnaR_G-DJa5oCoJiw}{127.0.0.1}{127.0.0.1:36227}{dimr}{testattr=test} reason: disconnected], term: 2, version: 24, delta: removed {{securityBwcCluster-0}{EbPGUjl1QtOX2wHqiePqfw}{1DlbeQnaR_G-DJa5oCoJiw}{127.0.0.1}{127.0.0.1:36227}{dimr}{testattr=test}}
2022-03-09T23:42:28.070+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » [2022-03-09T23:42:27,740][DEBUG][o.e.c.c.PublicationTransportHandler] [securityBwcCluster-2] received diff cluster state version [24] with uuid [Kktvng-2RaKdlMqX93klfA], diff size [712]
2022-03-09T23:42:28.070+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » [2022-03-09T23:42:27,774][INFO ][o.e.c.s.ClusterApplierService] [securityBwcCluster-2] removed {{securityBwcCluster-0}{EbPGUjl1QtOX2wHqiePqfw}{1DlbeQnaR_G-DJa5oCoJiw}{127.0.0.1}{127.0.0.1:36227}{dimr}{testattr=test}}, term: 2, version: 24, reason: Publication{term=2, version=24}

2022-03-09T23:42:28.069+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][c.a.o.s.OpenDistroSecurityPlugin] [securityBwcCluster-2] Directory /local/home/cgliu/git/security/bwc-test/build/testclusters/securityBwcCluster-2/config/jvm.options.d has insecure file permissions (should be 0700)
2022-03-09T23:42:28.069+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][stderr                   ] [securityBwcCluster-2] java.lang.reflect.InvocationTargetException
2022-03-09T23:42:28.069+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][stderr                   ] [securityBwcCluster-2]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2022-03-09T23:42:28.069+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][stderr                   ] [securityBwcCluster-2]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
2022-03-09T23:42:28.069+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][stderr                   ] [securityBwcCluster-2]         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2022-03-09T23:42:28.069+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][stderr                   ] [securityBwcCluster-2]         at java.base/java.lang.reflect.Method.invoke(Method.java:564)
2022-03-09T23:42:28.069+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][stderr                   ] [securityBwcCluster-2]         at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
2022-03-09T23:42:28.069+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][stderr                   ] [securityBwcCluster-2]         at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535)
2022-03-09T23:42:28.069+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][stderr                   ] [securityBwcCluster-2] Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.jdk.internal.org.objectweb.asm")
2022-03-09T23:42:28.069+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][stderr                   ] [securityBwcCluster-2]         at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
2022-03-09T23:42:28.070+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][stderr                   ] [securityBwcCluster-2]         at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
2022-03-09T23:42:28.070+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][stderr                   ] [securityBwcCluster-2]         at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
2022-03-09T23:42:28.070+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][stderr                   ] [securityBwcCluster-2]         at java.base/java.lang.SecurityManager.checkPackageAccess(SecurityManager.java:1324)
2022-03-09T23:42:28.070+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][stderr                   ] [securityBwcCluster-2]         at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:164)
2022-03-09T23:42:28.070+0000 [LIFECYCLE] [org.opensearch.gradle.testclusters.OpenSearchNode] » WARN ][stderr                   ] [securityBwcCluster-2]         at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)

@cliu123
Copy link
Member

cliu123 commented Mar 15, 2022

ODFE/ES cluster formation is not successfuly.

@peternied
Copy link
Member

@cliu123 could you provide the steps to see this problem? It might make sense to make a separate issue to track this.

@cliu123 cliu123 mentioned this issue Mar 16, 2022
Merged
3 tasks
@cliu123 cliu123 assigned cliu123 and unassigned peternied Mar 16, 2022
@cliu123
Copy link
Member

cliu123 commented Mar 16, 2022

@cliu123 could you provide the steps to see this problem? It might make sense to make a separate issue to track this.

There is no manual steps. The BWC tests don't run against ODFE/ES versions.

@cliu123 cliu123 closed this as completed Mar 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cliu123 cliu123

Labels
backwards-compatibility
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@peternied @davidlago @VachaShah @cliu123 @saratvemulapalli @gaiksaya