Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Group shard request per node in the field capabilities API #74648

Closed
Tracked by #77466
jimczi opened this issue Jun 28, 2021 · 4 comments · Fixed by #79212
Closed
Tracked by #77466

Group shard request per node in the field capabilities API #74648

jimczi opened this issue Jun 28, 2021 · 4 comments · Fixed by #79212
Assignees
Labels
>enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch

Comments

@jimczi
Copy link
Contributor

jimczi commented Jun 28, 2021

The field capabilities splits the original request on a per-index basis. If 5 indices are requested, the API will send 5 independent requests, one per index. These individual requests have a cost:

  • They serialize the same request multiple times.
  • Occupy one network thread to perform the local operation and send the response back.
  • They need to resolve the index permissions multiple times.

In order to reduce this cost, it might be beneficial to switch to a per-node request model where indices/shards that appear on the same node are grouped into a single request. Reducing the number of requests could also help to move these operations into a dedicated thread pool like we did for the new value suggester.

@jimczi jimczi added >enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types labels Jun 28, 2021
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Jun 28, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@jtibshirani
Copy link
Contributor

I looked into this and think it could be cleaner to implement if we changed the execution strategy. Currently, we select one shard from an index to send a request. If it doesn't satisfy the index_filter or fails, we retry on the next one, working through all shard copies.

I wonder if we could just select an active copy for each shard, and not attempt to retry on other copies. This would let us perform only one round trip to the nodes to gather all the information needed. If the request had an index_filter we would include all shards in the index, instead of just one. This execution strategy would be very similar to the one for _terms_enum requests.

@jtibshirani
Copy link
Contributor

We decided to split up the work into several PRs. I created a short-lived feature branch group-field-caps where we can merge intermediate PRs. Using the feature branch ensures we avoid changing the transport API several times on a mainline branch, which can make BWC complex.

@jimczi
Copy link
Contributor Author

jimczi commented Oct 20, 2021

Implemented in #79212

@jimczi jimczi closed this as completed Oct 20, 2021
@javanna javanna added Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch and removed Team:Search Meta label for search team labels Jul 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants