Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open PIT API incorrectly suggests using allow_partial_search_results if a shard is unavailable #103165

Closed
rseldner opened this issue Dec 7, 2023 · 3 comments
Closed

Open PIT API incorrectly suggests using allow_partial_search_results if a shard is unavailable #103165

rseldner opened this issue Dec 7, 2023 · 3 comments
Labels
>bug priority:normal A label for assessing bug priority to be used by ES engineers :Search Foundations/Search Catch all for Search Foundations Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch

Comments

@rseldner
Copy link
Contributor

rseldner commented Dec 7, 2023

Elasticsearch Version

8.11.1

Installed Plugins

No response

Java Version

bundled

OS Version

Ubuntu 20.04.6 LTS (Elastic Cloud)

Problem Description

Open PIT requires for all shards involved to be available.
If a shard is unavailable, the error response will suggest using allow_partial_search_results to bypass the error.

Request: POST kibana_sample_data_flights*/_pit?keep_alive=1m
Response:

 "caused_by": {
  "type": "search_phase_execution_exception",
  "reason": "Search rejected due to missing shards [[kibana_sample_data_flights_2][0]]. Consider using `allow_partial_search_results` setting to bypass this error.",
   "phase": "open_search_context",

This is misleading as it is not currently possible/supported to use allow_partial_search_results when opening a PIT.

Related: #96775

Steps to Reproduce

  1. Cause a primary shard to become unavailable. Example on a 2+ data node cluster:
    • Create an index with 2 primaries shards and set total shards per node to 1:
    PUT <index>
{
  "settings": {
    "number_of_shards": 2,
    "routing.allocation.total_shards_per_node": 1
  }
}
    • Shutdown one of the data nodes
  2. Open a PIT POST <index>/_pit?keep_alive=1m

Logs (if relevant)

Full Errors+Stack Traces:

8.7.1 
{
  "error": {
    "root_cause": [],
    "type": "search_phase_execution_exception",
    "reason": "",
    "phase": "open_search_context",
    "grouped": true,
    "failed_shards": [],
    "caused_by": {
      "type": "search_phase_execution_exception",
      "reason": "Search rejected due to missing shards [[kibana_sample_data_flights_2][0]]. Consider using `allow_partial_search_results` setting to bypass this error.",
      "phase": "open_search_context",
      "grouped": true,
      "failed_shards": [],
      "stack_trace": """Failed to execute phase [open_search_context], Search rejected due to missing shards [[kibana_sample_data_flights_2][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:245)
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:469)
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:217)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1041)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$5(TransportSearchAction.java:459)
	at [email protected]/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.maybeFinish(TransportSearchAction.java:1380)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.onResponse(TransportSearchAction.java:1345)
	at [email protected]/org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43)
	at [email protected]/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1367)
	at [email protected]/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:400)
	at [email protected]/org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:349)
	at [email protected]/org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:144)
	at [email protected]/org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:97)
	at [email protected]/org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:829)
	at [email protected]/org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:150)
	at [email protected]/org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121)
	at [email protected]/org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86)
	at [email protected]/org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:63)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at [email protected]/io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at [email protected]/io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)
	at [email protected]/io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
	at [email protected]/io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
	at [email protected]/io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
	at [email protected]/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
	at [email protected]/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at [email protected]/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at [email protected]/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at [email protected]/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at [email protected]/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at [email protected]/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)
	at [email protected]/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)
	at [email protected]/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at [email protected]/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at [email protected]/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:1623)
"""
    },
    "stack_trace": """Failed to execute phase [open_search_context], 
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:728)
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:474)
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:217)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1041)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$5(TransportSearchAction.java:459)
	at [email protected]/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.maybeFinish(TransportSearchAction.java:1380)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.onResponse(TransportSearchAction.java:1345)
	at [email protected]/org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43)
	at [email protected]/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1367)
	at [email protected]/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:400)
	at [email protected]/org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:349)
	at [email protected]/org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:144)
	at [email protected]/org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:97)
	at [email protected]/org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:829)
	at [email protected]/org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:150)
	at [email protected]/org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121)
	at [email protected]/org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86)
	at [email protected]/org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:63)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at [email protected]/io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at [email protected]/io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)
	at [email protected]/io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
	at [email protected]/io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
	at [email protected]/io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
	at [email protected]/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
	at [email protected]/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at [email protected]/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at [email protected]/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at [email protected]/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at [email protected]/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at [email protected]/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)
	at [email protected]/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)
	at [email protected]/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at [email protected]/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at [email protected]/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:1623)
Caused by: Failed to execute phase [open_search_context], Search rejected due to missing shards [[kibana_sample_data_flights_2][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:245)
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:469)
	... 49 more
"""
  },
  "status": 503
}
8.11.1 
{
  "error": {
    "root_cause": [],
    "type": "search_phase_execution_exception",
    "reason": "",
    "phase": "indices:data/read/open_point_in_time",
    "grouped": true,
    "failed_shards": [],
    "caused_by": {
      "type": "search_phase_execution_exception",
      "reason": "Search rejected due to missing shards [[kibana_sample_data_flights_2][0]]. Consider using `allow_partial_search_results` setting to bypass this error.",
      "phase": "indices:data/read/open_point_in_time",
      "grouped": true,
      "failed_shards": [],
      "stack_trace": """Failed to execute phase [indices:data/read/open_point_in_time], Search rejected due to missing shards [[kibana_sample_data_flights_2][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:249)
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:473)
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:221)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1123)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$8(TransportSearchAction.java:415)
	at [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.maybeFinish(TransportSearchAction.java:1489)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.onFailure(TransportSearchAction.java:1475)
	at [email protected]/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:54)
	at [email protected]/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1421)
	at [email protected]/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1421)
	at [email protected]/org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:1305)
	at [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
"""
    },
    "stack_trace": """Failed to execute phase [indices:data/read/open_point_in_time], 
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:731)
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:478)
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:221)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1123)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$8(TransportSearchAction.java:415)
	at [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.maybeFinish(TransportSearchAction.java:1489)
	at [email protected]/org.elasticsearch.action.search.TransportSearchAction$CCSActionListener.onFailure(TransportSearchAction.java:1475)
	at [email protected]/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:54)
	at [email protected]/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1421)
	at [email protected]/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1421)
	at [email protected]/org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:1305)
	at [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: Failed to execute phase [indices:data/read/open_point_in_time], Search rejected due to missing shards [[kibana_sample_data_flights_2][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:249)
	at [email protected]/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:473)
	... 15 more
"""
  },
  "status": 503
}
@rseldner rseldner added >bug :Search/Search Search-related issues that do not fall into other categories needs:triage Requires assignment of a team area label v8.7.1 v8.11.1 labels Dec 7, 2023
@elasticsearchmachine elasticsearchmachine added Team:Search Meta label for search team and removed needs:triage Requires assignment of a team area label labels Dec 7, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@javanna javanna mentioned this issue Dec 8, 2023
Open
8 tasks
@benwtrent benwtrent added the priority:normal A label for assessing bug priority to be used by ES engineers label Jul 9, 2024
@javanna javanna added :Search Foundations/Search Catch all for Search Foundations and removed :Search/Search Search-related issues that do not fall into other categories labels Jul 17, 2024
@elasticsearchmachine elasticsearchmachine added the Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch label Jul 17, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search-foundations (Team:Search Foundations)

@elasticsearchmachine elasticsearchmachine removed the Team:Search Meta label for search team label Jul 17, 2024
@benwtrent benwtrent mentioned this issue Jul 17, 2024
Merged
@pmpailis pmpailis mentioned this issue Jul 26, 2024
Merged
@javanna
Copy link
Member

javanna commented Nov 11, 2024 •
edited
Loading

Closed by #111516

@javanna javanna closed this as completed Nov 11, 2024
@javanna javanna mentioned this issue Nov 11, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>bug priority:normal A label for assessing bug priority to be used by ES engineers :Search Foundations/Search Catch all for Search Foundations Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@javanna @benwtrent @rseldner @elasticsearchmachine