Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.4] Refine size-your-shards wording (#89081) #89161

Merged
merged 1 commit into from
Aug 8, 2022
Merged

[8.4] Refine size-your-shards wording (#89081) #89161

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 22 additions & 7 deletions docs/reference/how-to/size-your-shards.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,17 +175,25 @@ index prirep shard store

[discrete]
[[shard-count-recommendation]]
==== Aim for 3000 indices or fewer per GB of heap memory on each master node
==== Master-eligible nodes should have at least 1GB of heap per 3000 indices

The number of indices a master node can manage is proportional to its heap
size. The exact amount of heap memory needed for each index depends on various
factors such as the size of the mapping and the number of shards per index.

As a general rule of thumb, you should aim for 3000 indices or fewer per GB of
heap on master nodes. For example, if your cluster contains 12000 indices then
each dedicated master node should have at least 4GB of heap. For non-dedicated
master nodes, the same rule holds and should be added to the heap requirements
of the other roles of each node.
As a general rule of thumb, you should have fewer than 3000 indices per GB of
heap on master nodes. For example, if your cluster has dedicated master nodes
with 4GB of heap each then you should have fewer than 12000 indices. If your
master nodes are not dedicated master nodes then the same sizing guidance
applies: you should reserve at least 1GB of heap on each master-eligible node
for every 3000 indices in your cluster.

Note that this rule defines the absolute maximum number of indices that a
master node can manage, but does not guarantee the performance of searches or
indexing involving this many indices. You must also ensure that your data nodes
have adequate resources for your workload and that your overall sharding
strategy meets all your performance requirements. See also
<<single-thread-per-shard>> and <<each-shard-has-overhead>>.

To check the configured size of each node's heap, use the <<cat-nodes,cat nodes
API>>.
Expand All @@ -207,7 +215,7 @@ GET _cat/shards?v=true

[discrete]
[[field-count-recommendation]]
==== Allow 1kB of heap per field per index on data nodes, plus overheads
==== Data nodes should have at least 1kB of heap per field per index, plus overheads

The exact resource usage of each mapped field depends on its type, but a rule
of thumb is to allow for approximately 1kB of heap overhead per mapped field
Expand All @@ -222,6 +230,13 @@ For example, if a data node holds shards from 1000 indices, each containing
of heap for the fields and another 0.5GB of heap for its workload and other
overheads, and therefore this node will need a heap size of at least 4.5GB.

Note that this rule defines the absolute maximum number of indices that a data
node can manage, but does not guarantee the performance of searches or indexing
involving this many indices. You must also ensure that your data nodes have
adequate resources for your workload and that your overall sharding strategy
meets all your performance requirements. See also <<single-thread-per-shard>>
and <<each-shard-has-overhead>>.

[discrete]
[[avoid-node-hotspots]]
==== Avoid node hotspots
Expand Down