Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LocalIndexFollowingIT#testRemoveRemoteConnection failure #38695

Closed
martijnvg opened this issue Feb 11, 2019 · 3 comments
Closed

LocalIndexFollowingIT#testRemoveRemoteConnection failure #38695

martijnvg opened this issue Feb 11, 2019 · 3 comments
Labels
:Distributed Indexing/CCR Issues around the Cross Cluster State Replication features >test-failure Triaged test failures from CI

Comments

@martijnvg
Copy link
Member

martijnvg commented Feb 11, 2019 •
edited
Loading 

ERROR   11.6s J0 | LocalIndexFollowingIT.testRemoveRemoteConnection <<< FAILURES!
   > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=59, name=elasticsearch[node_s_0][ccr][T#20], state=RUNNABLE, group=TGRP-LocalIndexFollowingIT]
   >    at __randomizedtesting.SeedInfo.seed([6DB8A6A33D5435DA:D29C737D60446C43]:0)
   > Caused by: NoSuchRemoteClusterException[no such remote cluster: [local]]
   >    at __randomizedtesting.SeedInfo.seed([6DB8A6A33D5435DA]:0)
   >    at org.elasticsearch.transport.RemoteClusterService.getRemoteClusterClient(RemoteClusterService.java:460)
   >    at org.elasticsearch.client.node.NodeClient.getRemoteClusterClient(NodeClient.java:123)
   >    at org.elasticsearch.xpack.ccr.action.ShardFollowTasksExecutor.remoteClient(ShardFollowTasksExecutor.java:248)
   >    at org.elasticsearch.xpack.ccr.action.ShardFollowTasksExecutor.access$000(ShardFollowTasksExecutor.java:66)
   >    at org.elasticsearch.xpack.ccr.action.ShardFollowTasksExecutor$1.innerUpdateMapping(ShardFollowTasksExecutor.java:116)
   >    at org.elasticsearch.xpack.ccr.action.ShardFollowNodeTask.updateMapping(ShardFollowNodeTask.java:431)
   >    at org.elasticsearch.xpack.ccr.action.ShardFollowNodeTask.lambda$updateMapping$13(ShardFollowNodeTask.java:432)
   >    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
   >    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   >    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   >    at java.lang.Thread.run(Thread.java:748)

Build url: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+internalClusterTest/940/console
@martijnvg martijnvg added >test-failure Triaged test failures from CI :Distributed Indexing/CCR Issues around the Cross Cluster State Replication features labels Feb 11, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Feb 11, 2019
@martijnvg
Mute test.
b1d1819 
Relates to elastic#38695
@martijnvg martijnvg mentioned this issue Feb 11, 2019
Merged
martijnvg added a commit that referenced this issue Feb 11, 2019
@martijnvg
Mute test.
e213ad3 
Relates to #38695
martijnvg added a commit that referenced this issue Feb 11, 2019
@martijnvg
Mute test. (#38696)
8ed6915 
Relates to #38695
martijnvg added a commit that referenced this issue Feb 11, 2019
@martijnvg
Mute test.
9ad2eb7 
Relates to #38695
martijnvg added a commit that referenced this issue Feb 11, 2019
@martijnvg
Mute test.
3cff608 
Relates to #38695
@martijnvg
Copy link
Member Author

Another variant of this test failing is:

ERROR   10.7s J0 | LocalIndexFollowingIT.testRemoveRemoteConnection <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: 
   > Expected: <1L>
   >      but: was <-1L>
   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >    at org.elasticsearch.xpack.ccr.LocalIndexFollowingIT.lambda$testRemoveRemoteConnection$5(LocalIndexFollowingIT.java:139)
  2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-unix-compatibility/os/centos-7/x-pack/plugin/ccr/build/testrun/internalClusterTest/J0/temp/org.elasticsearch.xpack.ccr.LocalIndexFollowingIT_AA7194AF0C2ECD87-001
  2> NOTE: test params are: codec=Asserting(Lucene80), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@28dc4c88), locale=ga, timezone=Etc/GMT+1
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:858)
  2> NOTE: Linux 3.10.0-957.5.1.el7.x86_64 amd64/Oracle Corporation 1.8.0_202 (64-bit)/cpus=16,threads=1,free=441734008,total=513802240
  2> NOTE: All tests run in this JVM: [LocalIndexFollowingIT]
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:832)
   >    at org.elasticsearch.xpack.ccr.LocalIndexFollowingIT.testRemoveRemoteConnection(LocalIndexFollowingIT.java:129)
   >    at java.lang.Thread.run(Thread.java:748)

With a warning happening earlier:

1> [2019-02-11T00:19:40,166][WARN ][o.e.x.c.a.AutoFollowCoordinator] [node_s_0] failure occurred while fetching cluster state for auto follow pattern [my_pattern]
  1> org.elasticsearch.ElasticsearchStatusException: can not fetch remote cluster state as the license state of the remote cluster [local] could not be determined
  1>    at org.elasticsearch.xpack.ccr.CcrLicenseChecker.clusterStateUnknownRemoteLicense(CcrLicenseChecker.java:434) ~[main/:?]
  1>    at org.elasticsearch.xpack.ccr.CcrLicenseChecker.lambda$checkRemoteClusterLicenseAndFetchClusterState$5(CcrLicenseChecker.java:175) ~[main/:?]
  1>    at org.elasticsearch.xpack.ccr.CcrLicenseChecker$1.onFailure(CcrLicenseChecker.java:226) [main/:?]
  1>    at org.elasticsearch.license.RemoteClusterLicenseChecker$1.onFailure(RemoteClusterLicenseChecker.java:188) [x-pack-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:50) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.transport.TransportService$8.run(TransportService.java:960) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]
  1>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]
  1>    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]

I suspect that this warning is the reason why this assertion fails.

martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Feb 11, 2019
@martijnvg
Fix LocalIndexFollowingIT#testRemoveRemoteConnection() test
6bbd719 
* During fetching remote mapping if remote client is missing then
`NoSuchRemoteClusterException` was not handled.
* When adding remote connection, check that it is really connected
before continue-ing to run the tests.

Relates to elastic#38695
@martijnvg martijnvg mentioned this issue Feb 11, 2019
Merged
martijnvg added a commit that referenced this issue Feb 14, 2019
@martijnvg
Fix LocalIndexFollowingIT#testRemoveRemoteConnection() test (#38709)
a68c275 
* During fetching remote mapping if remote client is missing then
`NoSuchRemoteClusterException` was not handled.
* When adding remote connection, check that it is really connected
before continue-ing to run the tests.

Relates to #38695
martijnvg added a commit that referenced this issue Feb 18, 2019
@martijnvg
Fix LocalIndexFollowingIT#testRemoveRemoteConnection() test (#38709)
ed08bc3 
* During fetching remote mapping if remote client is missing then
`NoSuchRemoteClusterException` was not handled.
* When adding remote connection, check that it is really connected
before continue-ing to run the tests.

Relates to #38695
martijnvg added a commit that referenced this issue Feb 18, 2019
@martijnvg
Fix LocalIndexFollowingIT#testRemoveRemoteConnection() test (#38709)
e72c721 
* During fetching remote mapping if remote client is missing then
`NoSuchRemoteClusterException` was not handled.
* When adding remote connection, check that it is really connected
before continue-ing to run the tests.

Relates to #38695
martijnvg added a commit that referenced this issue Feb 18, 2019
@martijnvg
Fix LocalIndexFollowingIT#testRemoveRemoteConnection() test (#38709)
2220bed 
* During fetching remote mapping if remote client is missing then
`NoSuchRemoteClusterException` was not handled.
* When adding remote connection, check that it is really connected
before continue-ing to run the tests.

Relates to #38695
@martijnvg
Copy link
Member Author

Looks like this test failure has been fixed by the above commits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Distributed Indexing/CCR Issues around the Cross Cluster State Replication features >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@martijnvg @elasticmachine