[CI][master] DocsClientYamlTestSuiteIT put-auto-follow-pattern #53430

mayya-sharipova · 2020-03-11T19:01:04Z

Log : https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=centos-8&&immutable/627/console
Build Scans: https://gradle-enterprise.elastic.co/s/t7pibrqbzjvsm

REPRODUCE WITH: ./gradlew ':docs:integTestRunner' --tests "org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT.test {yaml=reference/ccr/apis/auto-follow/put-auto-follow-pattern/line_15}"
-Dtests.seed=1FB43FBC4EBFC245
-Dtests.security.manager=true
-Dtests.locale=da
-Dtests.timezone=America/Cuiaba
-Dcompiler.java=13

Doesn't reproduce for me

Caused by: java.lang.AssertionError: expected [2xx] status code but api [raw[method=PUT path=_ccr/auto_follow/auto_follow_pattern_name]] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"license info is missing for cluster [remote_cluster]","stack_trace":"org.elasticsearch.ResourceNotFoundException: license info is missing for cluster [remote_cluster]\n\tat org.elasticsearch.license.RemoteClusterLicenseChecker$1.onResponse(RemoteClusterLicenseChecker.java:168)\n\tat org.elasticsearch.license.RemoteClusterLicenseChecker$1.onResponse(RemoteClusterLicenseChecker.java:162)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:54)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1057)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1057)\n\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:220)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:177)\n\tat org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:212)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:138)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:102)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:667)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"}],"type":"status_exception","reason":"can not fetch remote cluster state as the license state of the remote cluster [remote_cluster] could not be determined","caused_by":{"type":"resource_not_found_exception","reason":"license info is missing for cluster [remote_cluster]","stack_trace":"org.elasticsearch.ResourceNotFoundException: license info is missing for cluster [remote_cluster]\n\tat org.elasticsearch.license.RemoteClusterLicenseChecker$1.onResponse(RemoteClusterLicenseChecker.java:168)\n\tat org.elasticsearch.license.RemoteClusterLicenseChecker$1.onResponse(RemoteClusterLicenseChecker.java:162)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:54)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1057)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1057)\n\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:220)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:177)\n\tat org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:212)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:138)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:102)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:667)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"},"stack_trace":"org.elasticsearch.ElasticsearchStatusException: can not fetch remote cluster state as the license state of the remote cluster [remote_cluster] could not be determined\n\tat org.elasticsearch.xpack.ccr.CcrLicenseChecker.clusterStateUnknownRemoteLicense(CcrLicenseChecker.java:448)\n\tat org.elasticsearch.xpack.ccr.CcrLicenseChecker.lambda$checkRemoteClusterLicenseAndFetchClusterState$5(CcrLicenseChecker.java:180)\n\tat org.elasticsearch.xpack.ccr.CcrLicenseChecker$1.onFailure(CcrLicenseChecker.java:231)\n\tat org.elasticsearch.license.RemoteClusterLicenseChecker$1.onResponse(RemoteClusterLicenseChecker.java:168)\n\tat org.elasticsearch.license.RemoteClusterLicenseChecker$1.onResponse(RemoteClusterLicenseChecker.java:162)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:54)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1057)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1057)\n\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:220)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:177)\n\tat org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:212)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:138)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:102)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:667)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\nCaused by: org.elasticsearch.ResourceNotFoundException: license info is missing for cluster [remote_cluster]\n\t... 38 more\n"},"status":400}]Open stacktrace
[2020-03-11T14:07:09,441][INFO ][o.e.s.DocsClientYamlTestSuiteIT] [test] [yaml=reference/ccr/apis/auto-follow/put-auto-follow-pattern/line_15] before test
[2020-03-11T14:07:09,674][INFO ][o.e.s.DocsClientYamlTestSuiteIT] [test] initializing REST clients against [http://[::1]:34583, http://127.0.0.1:36961]
[2020-03-11T14:07:10,741][INFO ][o.e.s.DocsClientYamlTestSuiteIT] [test] initializing client, minimum es version [8.0.0], master version, [8.0.0], hosts [http://[::1]:34583, http://127.0.0.1:36961]
[2020-03-11T14:07:11,405][INFO ][o.e.s.DocsClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  "stash" : {
    "host" : "127.0.0.1:36961",
    "body" : {
      "error" : {
        "root_cause" : [
          {
            "type" : "resource_not_found_exception",
            "reason" : "license info is missing for cluster [remote_cluster]",
            "stack_trace" : "org.elasticsearch.ResourceNotFoundException: license info is missing for cluster [remote_cluster]
	at org.elasticsearch.license.RemoteClusterLicenseChecker$1.onResponse(RemoteClusterLicenseChecker.java:168)
	at org.elasticsearch.license.RemoteClusterLicenseChecker$1.onResponse(RemoteClusterLicenseChecker.java:162)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)

Possibly related to : #36815

The text was updated successfully, but these errors were encountered:

elasticmachine · 2020-03-11T19:01:06Z

Pinging @elastic/es-distributed (:Distributed/CCR)

davidkyle · 2020-03-16T13:13:00Z

And a failure on 7.x

[2020-03-16T10:10:24,787][INFO ][o.e.s.DocsClientYamlTestSuiteIT] [test] [yaml=reference/ccr/apis/auto-follow/delete-auto-follow-pattern/line_66] before test
[2020-03-16T10:10:24,993][INFO ][o.e.s.DocsClientYamlTestSuiteIT] [test] initializing REST clients against [http://[::1]:45619, http://127.0.0.1:39081]
[2020-03-16T10:10:25,947][INFO ][o.e.s.DocsClientYamlTestSuiteIT] [test] initializing client, minimum es version [7.7.0], master version, [7.7.0], hosts [http://[::1]:45619, http://127.0.0.1:39081]
[2020-03-16T10:10:26,448][INFO ][o.e.s.DocsClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  "stash" : {
    "host" : "127.0.0.1:39081",
    "body" : {
      "error" : {
        "root_cause" : [
          {
            "type" : "resource_not_found_exception",
            "reason" : "license info is missing for cluster [remote_cluster]",
            "stack_trace" : "ResourceNotFoundException[license info is missing for cluster [remote_cluster]]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-unix-compatibility/os=debian-9&&immutable/630/console
https://gradle-enterprise.elastic.co/s/ofxsmtx4lwxow

DocsClientYamlTestSuiteIT sometimes fails for CCR related tests because tests are started before the license is fully applied and active within the cluster. The first tests to be executed then fails with the error noticed in #53430. This can be easily reproduced locally by only running CCR docs tests. This commit adds some @before logic in DocsClientYamlTestSuiteIT so that it waits for the license to be active before running CCR tests. Closes #53430

mayya-sharipova added >test-failure Triaged test failures from CI :Distributed Indexing/CCR Issues around the Cross Cluster State Replication features labels Mar 11, 2020

tlrx self-assigned this Mar 18, 2020

tlrx mentioned this issue Mar 23, 2020

Wait for license to be applied and active before running CCR API documentation tests #53966

Merged

tlrx closed this as completed in #53966 Mar 24, 2020

tvernum mentioned this issue Jul 14, 2020

[CI] CcrRestIT unable to fetch remote license #59486

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI][master] DocsClientYamlTestSuiteIT put-auto-follow-pattern #53430

[CI][master] DocsClientYamlTestSuiteIT put-auto-follow-pattern #53430

mayya-sharipova commented Mar 11, 2020

elasticmachine commented Mar 11, 2020

davidkyle commented Mar 16, 2020

[CI][master] DocsClientYamlTestSuiteIT put-auto-follow-pattern #53430

[CI][master] DocsClientYamlTestSuiteIT put-auto-follow-pattern #53430

Comments

mayya-sharipova commented Mar 11, 2020

elasticmachine commented Mar 11, 2020

davidkyle commented Mar 16, 2020