-
Notifications
You must be signed in to change notification settings - Fork 24.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add cluster-wide shard limit warnings #34021
Changes from 3 commits
1e7d778
e8a762a
1f33086
92dec5d
ff0e4a6
9aed58a
499a987
87d35e3
51605b1
fe02b7b
0d22857
14034d7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -22,6 +22,47 @@ user with access to the <<cluster-update-settings,cluster-update-settings>> | |
API can make the cluster read-write again. | ||
|
||
|
||
[[cluster-shard-limit]] | ||
|
||
==== Cluster Shard Limit | ||
|
||
In a Elasticsearch 7.0 and later, there will be a soft limit on the number of | ||
shards in a cluster, based on the number of nodes in the cluster. This is | ||
intended to prevent operations which may unintentionally destabilize the | ||
cluster. Until 7.0, actions which would result in the cluster going over the | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
limit will issue a deprecation warning. | ||
|
||
NOTE: You can set the system property `es.enforce.shard_limit` to `true` to opt | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think we need to namespace this under |
||
in to strict enforcement of the shard limit. If this system property is set, | ||
actions which would result in the cluster going over the limit will result in an | ||
error, rather than a deprecation warning. This property will be removed in | ||
Elasticsearch 7.0, as strict enforcement of the limit will be the default and | ||
only behavior. | ||
|
||
If an operation, such as creating a new index, restoring a snapshot of an index, | ||
or opening a closed index would lead to the number of shards in the cluster | ||
going over this limit, the operation will issue a deprecation warning. | ||
|
||
If the cluster is already over the limit, due to changes in node membership or | ||
setting changes, all operations that create or open indices will issue warnings | ||
until either the limit is increased as described below, or some indices | ||
are closed or deleted to bring the number of shards below the limit. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Would you link to the sections of the documentation relevant to closing, and separately deleting an index? |
||
|
||
Replicas count towards this limit, but closed indexes do not. An index with 5 | ||
primary shards and 2 replicas will be counted as 15 shards. Any closed index | ||
is counted as 0, no matter how many shards and replicas it contains. | ||
|
||
The limit defaults to 1,000 shards per node, and be dynamically adjusted using | ||
the following property: | ||
|
||
`cluster.shards.max_per_node`:: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am doubting whether this needs to be in a |
||
|
||
Controls the number of shards allowed in the cluster per node. | ||
|
||
For example, a 3-node cluster with the default setting would allow 3,000 shards | ||
total, across all open indexes. If the above setting is changed to 1,500, then | ||
the cluster would allow 4,500 shards total. | ||
|
||
[[user-defined-data]] | ||
==== User Defined Cluster Metadata | ||
|
||
|
@@ -103,4 +144,4 @@ Enable or disable allocation for persistent tasks: | |
This setting does not affect the persistent tasks that are already being executed. | ||
Only newly created persistent tasks, or tasks that must be reassigned (after a node | ||
left the cluster, for example), are impacted by this setting. | ||
-- | ||
-- |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -22,7 +22,6 @@ | |
import com.carrotsearch.hppc.ObjectHashSet; | ||
import com.carrotsearch.hppc.cursors.ObjectCursor; | ||
import com.carrotsearch.hppc.cursors.ObjectObjectCursor; | ||
|
||
import org.apache.logging.log4j.Logger; | ||
import org.apache.lucene.util.CollectionUtil; | ||
import org.elasticsearch.action.AliasesRequest; | ||
|
@@ -124,9 +123,11 @@ public enum XContentContext { | |
public interface Custom extends NamedDiffable<Custom>, ToXContentFragment, ClusterState.FeatureAware { | ||
|
||
EnumSet<XContentContext> context(); | ||
|
||
} | ||
|
||
public static final Setting<Integer> SETTING_CLUSTER_MAX_SHARDS_PER_NODE = | ||
Setting.intSetting("cluster.shards.max_per_node", 1000, 1, Property.Dynamic, Property.NodeScope); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
||
public static final Setting<Boolean> SETTING_READ_ONLY_SETTING = | ||
Setting.boolSetting("cluster.blocks.read_only", false, Property.Dynamic, Property.NodeScope); | ||
|
||
|
@@ -162,6 +163,7 @@ public interface Custom extends NamedDiffable<Custom>, ToXContentFragment, Clust | |
private final ImmutableOpenMap<String, Custom> customs; | ||
|
||
private final transient int totalNumberOfShards; // Transient ? not serializable anyway? | ||
private final int totalOpenIndexShards; | ||
private final int numberOfShards; | ||
|
||
private final String[] allIndices; | ||
|
@@ -183,12 +185,17 @@ public interface Custom extends NamedDiffable<Custom>, ToXContentFragment, Clust | |
this.customs = customs; | ||
this.templates = templates; | ||
int totalNumberOfShards = 0; | ||
int totalOpenIndexShards = 0; | ||
int numberOfShards = 0; | ||
for (ObjectCursor<IndexMetaData> cursor : indices.values()) { | ||
totalNumberOfShards += cursor.value.getTotalNumberOfShards(); | ||
numberOfShards += cursor.value.getNumberOfShards(); | ||
if (IndexMetaData.State.OPEN.equals(cursor.value.getState())) { | ||
totalOpenIndexShards += cursor.value.getTotalNumberOfShards(); | ||
} | ||
} | ||
this.totalNumberOfShards = totalNumberOfShards; | ||
this.totalOpenIndexShards = totalOpenIndexShards; | ||
this.numberOfShards = numberOfShards; | ||
|
||
this.allIndices = allIndices; | ||
|
@@ -676,10 +683,29 @@ public <T extends Custom> T custom(String type) { | |
} | ||
|
||
|
||
/** | ||
* Gets the total number of shards from all indices, including replicas and | ||
* closed indices. | ||
* @return The total number shards from all indices. | ||
*/ | ||
public int getTotalNumberOfShards() { | ||
return this.totalNumberOfShards; | ||
} | ||
|
||
/** | ||
* Gets the total number of active shards from all indices. Includes | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not all open indices are active. Let us be precise here and say |
||
* replicas, but does not include shards that are part of closed indices. | ||
* @return The total number of active shards from all indices. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
*/ | ||
public int getTotalOpenIndexShards() { | ||
return this.totalOpenIndexShards; | ||
} | ||
|
||
/** | ||
* Gets the number of primary shards from all indices, not including | ||
* replicas. | ||
* @return The number of primary shards from all indices. | ||
*/ | ||
public int getNumberOfShards() { | ||
return this.numberOfShards; | ||
} | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -53,6 +53,7 @@ | |
import org.elasticsearch.common.component.AbstractComponent; | ||
import org.elasticsearch.common.compress.CompressedXContent; | ||
import org.elasticsearch.common.io.PathUtils; | ||
import org.elasticsearch.common.logging.DeprecationLogger; | ||
import org.elasticsearch.common.settings.IndexScopedSettings; | ||
import org.elasticsearch.common.settings.Setting; | ||
import org.elasticsearch.common.settings.Settings; | ||
|
@@ -82,6 +83,7 @@ | |
import java.util.List; | ||
import java.util.Locale; | ||
import java.util.Map; | ||
import java.util.Optional; | ||
import java.util.Set; | ||
import java.util.concurrent.atomic.AtomicInteger; | ||
import java.util.function.BiFunction; | ||
|
@@ -587,19 +589,29 @@ public void onFailure(String source, Exception e) { | |
|
||
private void validate(CreateIndexClusterStateUpdateRequest request, ClusterState state) { | ||
validateIndexName(request.index(), state); | ||
validateIndexSettings(request.index(), request.settings(), forbidPrivateIndexSettings); | ||
validateIndexSettings(request.index(), request.settings(), state, forbidPrivateIndexSettings); | ||
} | ||
|
||
public void validateIndexSettings( | ||
final String indexName, final Settings settings, final boolean forbidPrivateIndexSettings) throws IndexCreationException { | ||
public void validateIndexSettings(String indexName, final Settings settings, ClusterState clusterState, final boolean forbidPrivateIndexSettings) throws IndexCreationException { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
List<String> validationErrors = getIndexSettingsValidationErrors(settings, forbidPrivateIndexSettings); | ||
|
||
Optional<String> shardAllocation = checkShardLimit(settings, clusterState, deprecationLogger); | ||
shardAllocation.ifPresent(validationErrors::add); | ||
|
||
if (validationErrors.isEmpty() == false) { | ||
ValidationException validationException = new ValidationException(); | ||
validationException.addValidationErrors(validationErrors); | ||
throw new IndexCreationException(indexName, validationException); | ||
} | ||
} | ||
|
||
static Optional<String> checkShardLimit(Settings settings, ClusterState clusterState, DeprecationLogger deprecationLogger) { | ||
gwbrown marked this conversation as resolved.
Show resolved
Hide resolved
|
||
int shardsToCreate = IndexMetaData.INDEX_NUMBER_OF_SHARDS_SETTING.get(settings) | ||
* (1 + IndexMetaData.INDEX_NUMBER_OF_REPLICAS_SETTING.get(settings)); | ||
|
||
return IndicesService.checkShardLimit(shardsToCreate, clusterState, deprecationLogger); | ||
} | ||
|
||
List<String> getIndexSettingsValidationErrors(final Settings settings, final boolean forbidPrivateIndexSettings) { | ||
String customPath = IndexMetaData.INDEX_DATA_PATH_SETTING.get(settings); | ||
List<String> validationErrors = new ArrayList<>(); | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -36,8 +36,10 @@ | |
import org.elasticsearch.cluster.routing.allocation.AllocationService; | ||
import org.elasticsearch.cluster.service.ClusterService; | ||
import org.elasticsearch.common.Priority; | ||
import org.elasticsearch.common.ValidationException; | ||
import org.elasticsearch.common.component.AbstractComponent; | ||
import org.elasticsearch.common.inject.Inject; | ||
import org.elasticsearch.common.logging.DeprecationLogger; | ||
import org.elasticsearch.common.settings.Settings; | ||
import org.elasticsearch.index.Index; | ||
import org.elasticsearch.indices.IndicesService; | ||
|
@@ -50,6 +52,7 @@ | |
import java.util.Arrays; | ||
import java.util.HashSet; | ||
import java.util.List; | ||
import java.util.Optional; | ||
import java.util.Set; | ||
|
||
/** | ||
|
@@ -175,6 +178,8 @@ public ClusterState execute(ClusterState currentState) { | |
} | ||
} | ||
|
||
validateShardLimit(currentState, request.indices(), deprecationLogger); | ||
|
||
if (indicesToOpen.isEmpty()) { | ||
return currentState; | ||
} | ||
|
@@ -217,4 +222,25 @@ public ClusterState execute(ClusterState currentState) { | |
}); | ||
} | ||
|
||
static void validateShardLimit(ClusterState currentState, Index[] indices, DeprecationLogger deprecationLogger) { | ||
gwbrown marked this conversation as resolved.
Show resolved
Hide resolved
|
||
int shardsToOpen = Arrays.stream(indices) | ||
.filter(index -> currentState.metaData().index(index).getState().equals(IndexMetaData.State.CLOSE)) | ||
.mapToInt(index -> getTotalShardCount(currentState, index)) | ||
.sum(); | ||
|
||
Optional<String> error = IndicesService.checkShardLimit(shardsToOpen, currentState, deprecationLogger); | ||
if (error.isPresent()) { | ||
ValidationException ex = new ValidationException(); | ||
ex.addValidationError(error.get()); | ||
throw ex; | ||
} | ||
|
||
} | ||
|
||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There is an extraneous newline here. |
||
private static int getTotalShardCount(ClusterState state, Index index) { | ||
IndexMetaData indexMetaData = state.metaData().index(index); | ||
return indexMetaData.getNumberOfShards() * (1 + indexMetaData.getNumberOfReplicas()); | ||
} | ||
|
||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -38,6 +38,7 @@ | |
import org.elasticsearch.cluster.ClusterState; | ||
import org.elasticsearch.cluster.metadata.IndexMetaData; | ||
import org.elasticsearch.cluster.metadata.IndexNameExpressionResolver; | ||
import org.elasticsearch.cluster.metadata.MetaData; | ||
import org.elasticsearch.cluster.routing.RecoverySource; | ||
import org.elasticsearch.cluster.routing.ShardRouting; | ||
import org.elasticsearch.common.CheckedFunction; | ||
|
@@ -52,6 +53,7 @@ | |
import org.elasticsearch.common.io.stream.StreamInput; | ||
import org.elasticsearch.common.io.stream.StreamOutput; | ||
import org.elasticsearch.common.lease.Releasable; | ||
import org.elasticsearch.common.logging.DeprecationLogger; | ||
import org.elasticsearch.common.settings.IndexScopedSettings; | ||
import org.elasticsearch.common.settings.Setting; | ||
import org.elasticsearch.common.settings.Setting.Property; | ||
|
@@ -156,6 +158,20 @@ public class IndicesService extends AbstractLifecycleComponent | |
public static final String INDICES_SHARDS_CLOSED_TIMEOUT = "indices.shards_closed_timeout"; | ||
public static final Setting<TimeValue> INDICES_CACHE_CLEAN_INTERVAL_SETTING = | ||
Setting.positiveTimeSetting("indices.cache.cleanup_interval", TimeValue.timeValueMinutes(1), Property.NodeScope); | ||
private static final boolean ENFORCE_SHARD_LIMIT; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
static { | ||
gwbrown marked this conversation as resolved.
Show resolved
Hide resolved
|
||
final String ENFORCE_SHARD_LIMIT_KEY = "es.enforce.shard_limit"; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
final String enforceShardLimitSetting = System.getProperty(ENFORCE_SHARD_LIMIT_KEY); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
if (enforceShardLimitSetting == null) { | ||
ENFORCE_SHARD_LIMIT = false; | ||
} else if ("true".equals(enforceShardLimitSetting)) { | ||
ENFORCE_SHARD_LIMIT = true; | ||
} else { | ||
throw new IllegalArgumentException(ENFORCE_SHARD_LIMIT_KEY + " may only be unset or set to [true] but was [" + | ||
enforceShardLimitSetting + "]"); | ||
} | ||
} | ||
|
||
private final PluginsService pluginsService; | ||
private final NodeEnvironment nodeEnv; | ||
private final NamedXContentRegistry xContentRegistry; | ||
|
@@ -1347,4 +1363,42 @@ public Function<String, Predicate<String>> getFieldFilter() { | |
public boolean isMetaDataField(String field) { | ||
return mapperRegistry.isMetaDataField(field); | ||
} | ||
|
||
/** | ||
* Checks to see if an operation can be performed without taking the cluster | ||
* over the cluster-wide shard limit. Adds a deprecation warning or returns | ||
* an error message as appropriate | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You don't have to wrap these so narrowly, we can use the full 140-column line length here. |
||
* | ||
* @param newShards The number of shards to be added by this operation | ||
* @param state The current cluster state | ||
* @param deprecationLogger The logger to use for deprecation warnings | ||
* @return If present, an error message to be given as the reason for failing | ||
* an operation. If empty, a sign that the operation is valid. | ||
*/ | ||
public static Optional<String> checkShardLimit(int newShards, ClusterState state, DeprecationLogger deprecationLogger) { | ||
Settings theseSettings = state.metaData().settings(); | ||
int nodeCount = state.getNodes().getDataNodes().size(); | ||
|
||
// Only enforce the shard limit if we have at least one data node, so that we don't block | ||
// index creation during cluster setup | ||
if (nodeCount == 0 || newShards < 0) { | ||
return Optional.empty(); | ||
} | ||
int maxShardsPerNode = MetaData.SETTING_CLUSTER_MAX_SHARDS_PER_NODE.get(theseSettings); | ||
int maxShardsInCluster = maxShardsPerNode * nodeCount; | ||
int currentOpenShards = state.getMetaData().getTotalOpenIndexShards(); | ||
|
||
if ((currentOpenShards + newShards) > maxShardsInCluster) { | ||
String errorMessage = "this action would add [" + newShards + "] total shards, but this cluster currently has [" + | ||
currentOpenShards + "]/[" + maxShardsInCluster + "] maximum shards open"; | ||
if (ENFORCE_SHARD_LIMIT) { | ||
return Optional.of(errorMessage); | ||
} else { | ||
deprecationLogger.deprecated("In a future major version, this request will fail because {}. Before upgrading, " + | ||
"reduce the number of shards in your cluster or adjust the cluster setting [{}].", | ||
errorMessage, MetaData.SETTING_CLUSTER_MAX_SHARDS_PER_NODE.getKey()); | ||
} | ||
} | ||
return Optional.empty(); | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about
cluster.shards.max_per_node
-> cluster.max_shards_per_node`.