Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PointInTimeIT#testMultipleIndices failed (Unexpected ShardFailures) #61627

Closed
DaveCTurner opened this issue Aug 27, 2020 · 1 comment · Fixed by #61658
Closed

PointInTimeIT#testMultipleIndices failed (Unexpected ShardFailures) #61627

DaveCTurner opened this issue Aug 27, 2020 · 1 comment · Fixed by #61658
Assignees
@dnhatn
Labels
:Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team >test-failure Triaged test failures from CI

Comments

@DaveCTurner
Copy link
Contributor

Build scan: https://gradle-enterprise.elastic.co/s/alsc5n2wskfyc/console-log?task=:x-pack:plugin:core:internalClusterTest

Repro line:

REPRODUCE WITH: ./gradlew ':x-pack:plugin:core:internalClusterTest' --tests "org.elasticsearch.xpack.core.search.PointInTimeIT.testMultipleIndices" -Dtests.seed=9D7169CD5CBEFE35 -Dtests.security.manager=true -Dtests.locale=en-MT -Dtests.timezone=Etc/GMT-11 -Druntime.java=11

Reproduces locally?: Yes (after <5 iterations, doesn't fail reliably)

Applicable branches: master

Failure history: One previous similar-looking failure, in a PR branch: https://gradle-enterprise.elastic.co/s/3fzowvfm3t34i

Failure excerpt:

org.elasticsearch.xpack.core.search.PointInTimeIT > testMultipleIndices FAILED
    java.lang.AssertionError: Unexpected ShardFailures: [shard [[K3_-IIK9Q_m_2Q3INmxriA][index-2][7]], reason [org.elasticsearch.transport.RemoteTransportException: [node_s1][127.0.0.1:36963][indices:data/read/search[phase/dfs]]
    Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [7]
    	at org.elasticsearch.search.SearchService.findReaderContext(SearchService.java:648)
    	at org.elasticsearch.search.SearchService.getShard(SearchService.java:445)
    	at org.elasticsearch.search.SearchService.executeDfsPhase(SearchService.java:331)
    	at org.elasticsearch.action.search.SearchTransportService.lambda$registerRequestHandler$5(SearchTransportService.java:311)
    	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72)
    	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:192)
    	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:99)
    	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:81)
    	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:680)
    	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142)
    	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117)
    	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82)
    	at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:297)
    	at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:228)
    	at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:40)
    	at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:139)
    	at org.elasticsearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:151)
    	at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:420)
    	at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246)
    	at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:174)
    	at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:131)
    	at java.lang.Thread.run(Thread.java:834)
    ], cause [org.elasticsearch.search.SearchContextMissingException: No search context found for id [7]
    	at org.elasticsearch.search.SearchService.findReaderContext(SearchService.java:648)
    	at org.elasticsearch.search.SearchService.getShard(SearchService.java:445)
    	at org.elasticsearch.search.SearchService.executeDfsPhase(SearchService.java:331)
    	at org.elasticsearch.action.search.SearchTransportService.lambda$registerRequestHandler$5(SearchTransportService.java:311)
    	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72)
    	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:192)
    	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:99)
    	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:81)
    	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:680)
    	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142)
    	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117)
    	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82)
    	at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:297)
    	at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:228)
    	at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:40)
    	at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:139)
    	at org.elasticsearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:151)
    	at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:420)
    	at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:246)
    	at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:174)
    	at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:131)
    	at java.lang.Thread.run(Thread.java:834)
    ]]
    Expected: <0>
         but: was <1>
        at __randomizedtesting.SeedInfo.seed([9D7169CD5CBEFE35:845FE8F49266CD52]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertNoFailures(ElasticsearchAssertions.java:305)
        at org.elasticsearch.xpack.core.search.PointInTimeIT.testMultipleIndices(PointInTimeIT.java:137)
REPRODUCE WITH: ./gradlew ':x-pack:plugin:core:internalClusterTest' --tests "org.elasticsearch.xpack.core.search.PointInTimeIT.testMultipleIndices" -Dtests.seed=9D7169CD5CBEFE35 -Dtests.security.manager=true -Dtests.locale=en-MT -Dtests.timezone=Etc/GMT-11 -Druntime.java=11
@DaveCTurner DaveCTurner added :Search/Search Search-related issues that do not fall into other categories >test-failure Triaged test failures from CI labels Aug 27, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (:Search/Search)

@elasticmachine elasticmachine added the Team:Search Meta label for search team label Aug 27, 2020
@dnhatn dnhatn self-assigned this Aug 27, 2020
@dnhatn dnhatn mentioned this issue Aug 27, 2020
Merged
dnhatn added a commit that referenced this issue Sep 1, 2020
@dnhatn
Always include the matching node when resolving point in time (#61658)
294e9cc 
If shards are relocated to new nodes, then searches with a point in time 
will fail, although a pit keeps search contexts open. This commit solves
this problem by reducing info used by SearchShardIterator and always
including the matching nodes when resolving a point in time.

Closes #61627
dnhatn added a commit to dnhatn/elasticsearch that referenced this issue Sep 10, 2020
@dnhatn
Always include the matching node when resolving point in time (elasti…
7c678e8 
…c#61658)

If shards are relocated to new nodes, then searches with a point in time
will fail, although a pit keeps search contexts open. This commit solves
this problem by reducing info used by SearchShardIterator and always
including the matching nodes when resolving a point in time.

Closes elastic#61627
dnhatn added a commit that referenced this issue Sep 10, 2020
@dnhatn
Always include the matching node when resolving point in time (#61658)
808c868 
If shards are relocated to new nodes, then searches with a point in time
will fail, although a pit keeps search contexts open. This commit solves
this problem by reducing info used by SearchShardIterator and always
including the matching nodes when resolving a point in time.

Closes #61627
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
:Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Always include the matching node when resolving point in time dnhatn/elasticsearch
3 participants
@DaveCTurner @dnhatn @elasticmachine