Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge field caps responses on each node? #82879

Closed
Tracked by #77466
jtibshirani opened this issue Jan 20, 2022 · 2 comments · Fixed by #83494
Closed
Tracked by #77466

Merge field caps responses on each node? #82879

jtibshirani opened this issue Jan 20, 2022 · 2 comments · Fixed by #83494
Assignees
@dnhatn
Labels
>enhancement :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team

Comments

@jtibshirani
Copy link
Contributor

We'd like to look into merging field caps responses on each node, to prevent the coordinator from doing all merging. This will help reduce latency and memory when the field caps request includes many shards.

As part of this work, we can see if we could detect when two indices share the same mappings, and in that case skip over the shards with mappings we've already collected. This can help in the case of time-based indices, where most indices may have the same mappings.

This is a spin-off from #76509 (comment) to capture unresolved work.
@jtibshirani jtibshirani added >enhancement :Search/Search Search-related issues that do not fall into other categories labels Jan 20, 2022
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Jan 20, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

This was referenced Jan 20, 2022
Closed
Open
@jtibshirani
Copy link
Contributor Author

A note from @original-brownbear about detecting duplicate mappings:

We track the sha256 of each mapping (hashed via a relatively stable algorithm) and deduplicate MappingMetadata instances in the clusterstate. See Metadata 's org.elasticsearch.cluster.metadata.Metadata#getMappingsByHash for example. This makes detecting if two indices share the same mapping as cheap as comparing two strings.

@dnhatn dnhatn self-assigned this Feb 3, 2022
@dnhatn dnhatn mentioned this issue Feb 10, 2022
Merged
dnhatn added a commit that referenced this issue Feb 17, 2022
@dnhatn
Group field caps response by index mapping hash (#83494)
69e898d 
This commit utilizes the index mapping hash to share the fields-caps for 
indices with the same index mapping to reduce the memory usage and the
size of transport messages.

Closes #78665
Closes #82879
probakowski pushed a commit to probakowski/elasticsearch that referenced this issue Feb 23, 2022
@dnhatn @probakowski
Group field caps response by index mapping hash (elastic#83494)
31ead1f 
This commit utilizes the index mapping hash to share the fields-caps for 
indices with the same index mapping to reduce the memory usage and the
size of transport messages.

Closes elastic#78665
Closes elastic#82879
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
>enhancement :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Group field caps response by index mapping hash dnhatn/elasticsearch
3 participants
@jtibshirani @dnhatn @elasticmachine