Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The formula for calculating the shard which a document is routed in reference is wrong. #76072

Closed
ViggoC opened this issue Aug 4, 2021 · 6 comments · Fixed by #76203
Closed
Assignees
Labels
>docs General docs changes :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Docs Meta label for docs team Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch

Comments

@ViggoC
Copy link
Contributor

ViggoC commented Aug 4, 2021

The formula for calculating the shard which a document is routed in reference is wrong. It does not match the actual calculation process in the code.

Formula in reference

shard_num = hash(_routing) % num_primary_shards

Code

indexMetaData.routingFactor = routingNumShards / numberOfShards;
Math.floorMod(hash, indexMetaData.getRoutingNumShards()) / indexMetaData.getRoutingFactor();
@not-napoleon not-napoleon added the >docs General docs changes label Aug 4, 2021
@elasticmachine elasticmachine added the Team:Docs Meta label for docs team label Aug 4, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-docs (Team:Docs)

@jrodewig jrodewig self-assigned this Aug 5, 2021
@jrodewig jrodewig added the :Search Foundations/Mapping Index mappings, including merging and defining field types label Aug 5, 2021
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Aug 5, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@jrodewig
Copy link
Contributor

jrodewig commented Aug 6, 2021

Thanks for raising this @ViggoC. I'm not sure we need the formulae here at all. It's probably enough to say that we use a hash of the document _id to route documents by default and that this evenly distributes documents among an index's shards.

@ViggoC
Copy link
Contributor Author

ViggoC commented Aug 7, 2021

@jrodewig Thank you for your response, I agree with your opinion that it's enough for some users. But I believe that a link to more detail about how the shard num is calculated will be helpful for senior user and developer. Maybe the information can be appended to where index.number_of_routing_shards is mentioned.

@jrodewig
Copy link
Contributor

jrodewig commented Aug 8, 2021

Thanks for the feedback @ViggoC. I'll revise #76203 to update the formulas and cross-reference the settings involved. Thanks again for raising this.

@ViggoC
Copy link
Contributor Author

ViggoC commented Aug 9, 2021

👍

@ViggoC ViggoC closed this as completed Aug 9, 2021
jrodewig added a commit that referenced this issue Aug 9, 2021
The `_routing` metadata field docs currently include formulas for how
Elasticsearch routes documents to shards. However, these formulas were not
updated for #18699.  This updates the routing formulas and adds xrefs for
related settings.

Closes #76072
elasticsearchmachine pushed a commit to elasticsearchmachine/elasticsearch that referenced this issue Aug 9, 2021
The `_routing` metadata field docs currently include formulas for how
Elasticsearch routes documents to shards. However, these formulas were not
updated for elastic#18699.  This updates the routing formulas and adds xrefs for
related settings.

Closes elastic#76072
elasticsearchmachine pushed a commit to elasticsearchmachine/elasticsearch that referenced this issue Aug 9, 2021
The `_routing` metadata field docs currently include formulas for how
Elasticsearch routes documents to shards. However, these formulas were not
updated for elastic#18699.  This updates the routing formulas and adds xrefs for
related settings.

Closes elastic#76072
elasticsearchmachine added a commit that referenced this issue Aug 9, 2021
The `_routing` metadata field docs currently include formulas for how
Elasticsearch routes documents to shards. However, these formulas were not
updated for #18699.  This updates the routing formulas and adds xrefs for
related settings.

Closes #76072

Co-authored-by: James Rodewig <[email protected]>
elasticsearchmachine added a commit that referenced this issue Aug 9, 2021
The `_routing` metadata field docs currently include formulas for how
Elasticsearch routes documents to shards. However, these formulas were not
updated for #18699.  This updates the routing formulas and adds xrefs for
related settings.

Closes #76072

Co-authored-by: James Rodewig <[email protected]>
@javanna javanna added Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch and removed Team:Search Meta label for search team labels Jul 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>docs General docs changes :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Docs Meta label for docs team Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants