Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] WatchBackwardsCompatibilityIT failures #40841

Closed
martijnvg opened this issue Apr 4, 2019 · 12 comments
Closed

[CI] WatchBackwardsCompatibilityIT failures #40841

martijnvg opened this issue Apr 4, 2019 · 12 comments
Assignees
@martijnvg
Labels
:Data Management/Watcher :Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. >test-failure Triaged test failures from CI

Comments

@martijnvg
Copy link
Member

Several bwc tests fail, because of connection timeouts. In the es logs of the upgraded node the following fatal error can be found:

[2019-04-03T22:02:01,777][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [upgraded-node-0] fatal error in thread [elasticsearch[upgraded-node-0][write][T#4]], exi
ting
java.lang.AssertionError: setIfMatch [3], currentDocTem [1]
        at org.elasticsearch.action.index.IndexRequest.writeTo(IndexRequest.java:657) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.action.DocWriteRequest.writeDocumentRequest(DocWriteRequest.java:244) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.action.bulk.BulkItemRequest.writeTo(BulkItemRequest.java:110) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.action.bulk.BulkShardRequest.writeTo(BulkShardRequest.java:73) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$ConcreteShardRequest.writeTo(TransportReplicationAction.java:1276) ~[elasticsearch-6
.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$ConcreteReplicaRequest.writeTo(TransportReplicationAction.java:1332) ~[elasticsearch
-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:70) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:53) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:107) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:93) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:140) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.OutboundHandler.internalSendMessage(OutboundHandler.java:78) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:70) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.TcpTransport.sendRequestToChannel(TcpTransport.java:690) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.TcpTransport.sendRequestToChannel(TcpTransport.java:679) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.TcpTransport.access$300(TcpTransport.java:100) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:272) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:626) ~[elasticsearch-6.7.1-SNAPSHOT.jar:6.7.1-SNAPSHOT]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:155) ~[?:?]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$400(SecurityServerTransportInterceptor.java:52) ~[?:?]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.lambda$sendRequest$2(SecurityServerTransportInterceptor.java:129) ~[?:?]
        at org.elasticsearch.xpack.core.security.SecurityContext.executeAfterRewritingAuthentication(SecurityContext.java:111) ~[?:?]

I suspect that this assertion error is caused by watcher trying to update a watch status, because the watcher tests was the first one to fail with connection timeout. Also Watcher's execution service does not take into account nodes before 6.6.0 that don't support if_seq_no and if_primary_term parameters on write requests.

build url: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+bwc-tests/226/console
@martijnvg martijnvg added >test-failure Triaged test failures from CI :Data Management/Watcher labels Apr 4, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

@martijnvg martijnvg self-assigned this Apr 4, 2019
martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Apr 5, 2019
@martijnvg
Only use UpdateRequest#setIfSeqNo(...) and UpdateRequest#setIfPrimary…
f2241b2 
…Term(...) for updating watch status if all nodes are at least on 6.7.0.

Otherwise fallback using UpdateRequest#version(...)

Closes elastic#40841
@martijnvg martijnvg mentioned this issue Apr 5, 2019
Merged
martijnvg added a commit that referenced this issue Apr 11, 2019
@martijnvg
Use setIfSeqNo(...) and setIfPrimaryTerm(...) for updating watch stat…
b366fb9 
…us if all nodes are at least on 6.7.0 (#40888)

Only use UpdateRequest#setIfSeqNo(...) and UpdateRequest#setIfPrimaryTerm(...) for updating watch status if all nodes are at least on 6.7.0.
Otherwise fallback using UpdateRequest#version(...)

Closes #40841
@martijnvg
Copy link
Member Author

This should be fixed by #40888

@davidkyle
Copy link
Member

I have seen a similar error in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+bwc-tests/364/console

[2019-05-08T23:06:37,431][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [upgraded-node-0] fatal error in thread [elasticsearch[upgraded-node-0][write][T#1]], exiting
java.lang.AssertionError: setIfMatch [630], currentDocTem [1]
	at org.elasticsearch.action.index.IndexRequest.writeTo(IndexRequest.java:657) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.DocWriteRequest.writeDocumentRequest(DocWriteRequest.java:244) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.bulk.BulkItemRequest.writeTo(BulkItemRequest.java:110) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.bulk.BulkShardRequest.writeTo(BulkShardRequest.java:73) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ConcreteShardRequest.writeTo(TransportReplicationAction.java:1276) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]

The error is on the one third upgraded node upgrading from 6.1.0 to 6.7.3, I grabbed relevant logs

rolling-upgrade-logs.zip

@davidkyle davidkyle reopened this May 9, 2019
@davidkyle
Copy link
Member

I suspect https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+bwc-tests/366/console and https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+bwc-tests/27/console are the same root problem. In both cases a node seems to have died unexpectedly

@droberts195
Copy link
Contributor

There have been more failures of this today (and I suspect every day last week too as nobody was doing test triage during the all hands event).

These all manifest themselves in the test log as a failure to kill a process, for example:

Standard error:
kill: (812909): No such process

Then in the one third upgraded cluster node log is the error from the original issue description:

[2019-05-20T13:20:13,714][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [upgraded-node-0] fatal error in thread [elasticsearch[upgraded-node-0][write][T#13]], exiting
java.lang.AssertionError: setIfMatch [648], currentDocTem [1]
	at org.elasticsearch.action.index.IndexRequest.writeTo(IndexRequest.java:657) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.DocWriteRequest.writeDocumentRequest(DocWriteRequest.java:244) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.bulk.BulkItemRequest.writeTo(BulkItemRequest.java:110) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.bulk.BulkShardRequest.writeTo(BulkShardRequest.java:73) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ConcreteShardRequest.writeTo(TransportReplicationAction.java:1276) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ConcreteReplicaRequest.writeTo(TransportReplicationAction.java:1332) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:70) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:53) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:107) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:93) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:140) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.OutboundHandler.internalSendMessage(OutboundHandler.java:78) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:70) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport.sendRequestToChannel(TcpTransport.java:690) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport.sendRequestToChannel(TcpTransport.java:679) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport.access$300(TcpTransport.java:100) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:272) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:632) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:155) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$400(SecurityServerTransportInterceptor.java:52) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.lambda$sendRequest$2(SecurityServerTransportInterceptor.java:129) ~[?:?]
	at org.elasticsearch.xpack.core.security.SecurityContext.executeAfterRewritingAuthentication(SecurityContext.java:111) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:129) ~[?:?]
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:542) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:530) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.sendReplicaRequest(TransportReplicationAction.java:1213) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1175) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplica(ReplicationOperation.java:165) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplicas(ReplicationOperation.java:152) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:126) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:424) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:370) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2561) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:987) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:369) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:324) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:311) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:299) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:344) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1540) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1540) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1540) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1540) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$notifyListener$1(ListenableFuture.java:97) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1540) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:143) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:109) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.StepListener.onResponse(StepListener.java:62) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:345) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:322) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:263) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:604) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:579) ~[?:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:488) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:281) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:550) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:251) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:312) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:247) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:250) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:273) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:297) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:270) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:261) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:113) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:246) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:119) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:107) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195) ~[?:?]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:150) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:245) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:305) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:316) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:243) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:195) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:138) ~[?:?]
	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:133) ~[?:?]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:306) ~[?:?]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]

@droberts195
Copy link
Contributor

All 6.7 and 6.8 BWC builds still in the history (~11 days) have failed with this problem:

@henningandersen
Copy link
Contributor

One more here from today:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+bwc-tests/62/

@jakelandis
Copy link
Contributor

I can reproduce this locally with

./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v6.0.1#bwcTest -Dtests.class=org.elasticsearch.upgrades.WatcherUpgradeIT

but not

./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v6.6.0#bwcTest -Dtests.class=org.elasticsearch.upgrades.WatcherUpgradeIT

I am pretty confident that this error:

Standard error:
kill: (812909): No such process

is not related to anything with Watcher. Still looking, but this could be an indication of more general rolling upgrade from early version of 6.x issues.

@jakelandis jakelandis mentioned this issue May 22, 2019
Closed
@ywelsch ywelsch mentioned this issue May 24, 2019
Closed
@matriv
Copy link
Contributor

matriv commented May 24, 2019

Another failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+bwc-tests/72/console
Full logs can be found here

@jakelandis jakelandis mentioned this issue May 26, 2019
Closed
@jakelandis
Copy link
Contributor

Got to the root cause of this failure and logged a new issue: #42561

This isn't really a Watcher issue, it shows up in the Watcher tests since Watcher is continually issuing partial document updates (to .watches) while in a mixed cluster. Real world reproduction results in an IllegalStateException but since the tests are run with the assertion, the assertion crashes the running JVM.

This only impacts the 6.x branches and I am bit hesitant to simply mute all Watcher BWC tests (which would be required) since it discovered a real issue. We could simply remove (or comment out) the assertion and I tests pass just fine without the assertion (though there are extra errors in the logs).

I would prefer that we comment out the assertion with a link to the issue. This would allow the tests to pass without having to disable all of Watcher's BWC tests for reasons Watcher does not control. If there are other BWC tests that get caught up on this, this would allow them to pass too.

Thoughts on commenting out the assertion in 6.8 ? (as opposed to muting the test)

@dnhatn dnhatn added the :Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. label May 27, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@dnhatn
Copy link
Member

dnhatn commented May 27, 2019

Great find. Thanks @jakelandis. This is fixed by #42596.

@dnhatn dnhatn closed this as completed May 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@martijnvg martijnvg

Labels
:Data Management/Watcher :Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@martijnvg @jakelandis @davidkyle @matriv @droberts195 @dnhatn @elasticmachine @henningandersen