Count shards skipped on the coordinating node

[nik9000]

Description

#85162 is adding support for skipping TSDB indices with disjoint time ranges on the coordinating node. And we can skip them for read only indices already. It'd be useful for debugging performance issues and for testing if we got a count of the number of shards that we were able to skip on the coordinating node.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[martijnvg]

Perhaps we can report more on can match in search stats? Today we report on query and fetch level requests. Maybe we can report on the number of can match request and/or can match requests performed on the coordinating node.

[javanna]

@nik9000 did you mean that we should add it in each response, or in the search stats like @martijnvg suggested? Also, we already have a count of skipped shards in the search response, isn't that enough? I would like to discuss this with the team, but first, how did you envision this additional count to fit in the existing info?

[nik9000]

I'm interested in being able to assert on it during tests and in looking at it in SDHs.

Also, we already have a count of skipped shards in the search response, isn't that enough?

I'm having a little trouble following things, but it looks like we count any skipped shards there, but I was quite interested in shards we skipped on the coordinating node.

[javanna]

We discussed this with the team and some concerns were raised about adding a new count in the search response. We already have the skipped shards that include the overall count of the skipped shards whether they were skipped directly on the coordinating node or when executing can_match. While it used to be quite important that queries avoid going to the shards when the coordinating node can support skipping, as can match was executed per shard, it is now executed per node which makes it possibly less important to make a distinction about where the decision of skipping the shard was made.

Also, work on #84369 is being done which could help the supportability / debugging aspect, and we are planning to work on improved query performance analysis that should help inspect which shards were skipped at what level. With this, I would opt for closing this issue for now, but feel free to add feedback if you feel this is important and we can always reopen.

[nik9000]

We talked about this in the search team and didn't seem super keen on doing it. We don't like that it exposes an internal detail (that we skipped on the coordinating node) but.... oh, @javanna just replied with better words than I was typing. I'm glad github sent me the update.

[martijnvg]

I think in #85162 we need to find another to verify that the shard skipping happened on the coordinating node.
But I don't know whether that is possible.

[dnhatn]

@martijnvg You can intercept CanMatchNodeRequest (via MockTransportService) and tracks the list of shards that aren't skipped on the coordinating node.