From 106efb47eaac93fbb6175a039f26079bb8c22754 Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Tue, 13 Nov 2018 23:23:08 +0100 Subject: [PATCH 01/38] Add a `_freeze` / `_unfreeze` API This commit adds a rest endpoint for freezing and unfreezing an index. Among other cleanups mainly fixing an issue accessing package private APIs from a plugin that got caught by integration tests this change also adds documentation for frozen indices. Note: frozen indices are marked as `beta` and available as a basic feature. Relates to #34352 --- docs/reference/frozen-indices.asciidoc | 82 +++++++++ docs/reference/modules/threadpool.asciidoc | 4 + .../OpenIndexClusterStateUpdateRequest.java | 2 +- .../admin/indices/open/OpenIndexResponse.java | 4 +- .../metadata/MetaDataIndexStateService.java | 72 ++++---- .../index/engine/ReadOnlyEngine.java | 7 +- .../index/shard/IndexShardTestCase.java | 14 +- .../index/engine/FrozenEngine.java | 2 +- .../elasticsearch/xpack/core/XPackClient.java | 24 ++- .../xpack/core/XPackClientPlugin.java | 5 +- .../elasticsearch/xpack/core/XPackPlugin.java | 2 + .../action/TransportFreezeIndexAction.java | 128 +++++++++++++-- .../rest/action/RestFreezeIndexAction.java | 46 ++++++ .../engine/FrozenIndexRecoveryTests.java | 43 +++++ .../index/engine/FrozenIndexTests.java | 155 ++++++++++++------ .../rest-api-spec/api/indices.freeze.json | 48 ++++++ .../rest-api-spec/api/indices.unfreeze.json | 46 ++++++ .../test/indices.freeze/10_basic.yml | 128 +++++++++++++++ 18 files changed, 699 insertions(+), 113 deletions(-) create mode 100644 docs/reference/frozen-indices.asciidoc create mode 100644 x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rest/action/RestFreezeIndexAction.java create mode 100644 x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexRecoveryTests.java create mode 100644 x-pack/plugin/src/test/resources/rest-api-spec/api/indices.freeze.json create mode 100644 x-pack/plugin/src/test/resources/rest-api-spec/api/indices.unfreeze.json create mode 100644 x-pack/plugin/src/test/resources/rest-api-spec/test/indices.freeze/10_basic.yml diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc new file mode 100644 index 0000000000000..40e8303c3ba1d --- /dev/null +++ b/docs/reference/frozen-indices.asciidoc @@ -0,0 +1,82 @@ +[role="xpack"] +[testenv="basic"] +[[frozen-indices]] += Frozen Indices + +Elasticsearch indices can require a significant amount of memory available in order to be open and searchable. Yet, not all indices need +to be writable at the same time and have different access patters over time. For example indices in the time-series or logging use-case +are unlikely to be queried once they age out but still need to be kept around for retention policy purposes. + +In order to keep indices around for a longer time period but at the same time reducing their hardware requirements they can be transitioned +into a frozen state. Once an index is frozen, all it's transient shard memory aside of mappings and and necessary in-memory structures for +analysis is moved to persistent storage. This allows for a much higher disk to heap storage ratio on individual nodes. Once an index is +frozen their are read-only and require their data-structures required for query execution to be loaded on demand. A search request that hits +one or more shards from frozen indices will execute it's search through a throttling component that ensures that we never search more than +`N` (`1` by default) searches concurrently. This protects nodes from exceeding the available memory due to incoming search requests. + +In contrast to ordinary open indices, frozen indices are expected to execute slow and are not designed for high query load. Parallelism is +gained only on a per-node level and loading data-structures on demand is expected to be one or more orders of a magnitude slower than query +execution on a per shard level. Depending on the index and it's data executing a search on a frozen index is expected to be in the seconds +or even minutes range compared to milliseconds on a non-frozen index. + +== Best Practices + +Since frozen indices provide a much higher disk to heap ratio for the expense of latency it is recommended to allocate frozen indices on +dedicated nodes to prevent searches on frozen indices influencing traffic on low latency nodes. There is a significant overhead in loading +data-structures on demand which can cause page-faults and garbage collections further slowing down query execution. + +Since indices that are eligible for freezing are likely to not change in the future disk space can be optimized. Many strategies are +outlined in <>. + +== Freezing and unfreezing an index + +The freeze and unfreeze index APIs allow to freeze an index, and later on +unfreeze it. A frozen index has almost no overhead on the cluster (except +for maintaining its metadata in memory), and is blocked for write operations. +A frozen index can be unfrozen which will then go through the normal recovery process. + +The REST endpoint is `/{index}/_freeze` and `/{index}/_unfreeze`. For +example: + +[source,js] +-------------------------------------------------- +POST /my_index/_freeze + +POST /my_index/_unfreeze +-------------------------------------------------- +// CONSOLE +// TEST[s/^/PUT my_index\n/] + + +[IMPORTANT] +================================ + Freezing an index will close the index and reopen it within the same API call. This causes primaries to not be allocated for a short + amount of time and causes the cluster to go red until the primaries are allocated again. This limitation might be removed in the future +================================ + +== Searching a frozen index + +Frozen indices are throttled in order limit memory consumptions per node. The number of concurrently loaded frozen indices per node is +limited by the number of threads in the `search_throttled` <> which is `1` by default. At the same time, +search requests hitting may indices on a cluster due to concrete lists, expanded wildcards or a date pattern exclude frozen indices by +default to prevent accidental slowdowns when a frozen index is hit. To include frozen indices a search request must be executed with +`ignore_throttled=false`. + +[source,js] +-------------------------------------------------- +GET /twitter/_search?q=user:kimchy&ignore_throttled=false +-------------------------------------------------- +// CONSOLE +// TEST[setup:twitter] + +[IMPORTANT] +================================ +While frozen indices are slow to search, they offer an efficient pre-filter phase. The request parameter `pre_filter_shard_size` specifies +a threshold that enforces a round-trip to filter search shards based on query rewriting if the number of shards the search +request expands to exceeds the threshold. This filter phase can limit the number of shards significantly if for instance a shard can not +match any documents based on it's rewrite method ie. if date filters are mandatory to match but the shard bounds and the query are disjoint. +The default value for `pre_filter_shard_size` is `128` while for searching frozen indices it's recommended to set it to `1`. There is no +significant overhead associated with this pre-filter phase. +================================ + + diff --git a/docs/reference/modules/threadpool.asciidoc b/docs/reference/modules/threadpool.asciidoc index 515959e4ea580..d0492e66aa29d 100644 --- a/docs/reference/modules/threadpool.asciidoc +++ b/docs/reference/modules/threadpool.asciidoc @@ -19,6 +19,10 @@ There are several thread pools, but the important ones include: `int((# of available_processors * 3) / 2) + 1`, and initial queue_size of `1000`. +`search_throttled`:: + For count/search/suggest/get operations on `search_throttled indices`. Thread pool type is + `fixed_auto_queue_size` with a size of `1`, and initial queue_size of `100`. + `get`:: For get operations. Thread pool type is `fixed` with a size of `# of available processors`, diff --git a/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexClusterStateUpdateRequest.java b/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexClusterStateUpdateRequest.java index ea3abe5e21a54..4062393167b79 100644 --- a/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexClusterStateUpdateRequest.java +++ b/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexClusterStateUpdateRequest.java @@ -28,7 +28,7 @@ public class OpenIndexClusterStateUpdateRequest extends IndicesClusterStateUpdat private ActiveShardCount waitForActiveShards = ActiveShardCount.DEFAULT; - OpenIndexClusterStateUpdateRequest() { + public OpenIndexClusterStateUpdateRequest() { } diff --git a/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexResponse.java b/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexResponse.java index 97db91f8973f4..6ff03cb5291ea 100644 --- a/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexResponse.java +++ b/server/src/main/java/org/elasticsearch/action/admin/indices/open/OpenIndexResponse.java @@ -40,10 +40,10 @@ public class OpenIndexResponse extends ShardsAcknowledgedResponse { declareAcknowledgedAndShardsAcknowledgedFields(PARSER); } - OpenIndexResponse() { + public OpenIndexResponse() { } - OpenIndexResponse(boolean acknowledged, boolean shardsAcknowledged) { + public OpenIndexResponse(boolean acknowledged, boolean shardsAcknowledged) { super(acknowledged, shardsAcknowledged); } diff --git a/server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexStateService.java b/server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexStateService.java index 38d83b398856e..b6bb1fd058064 100644 --- a/server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexStateService.java +++ b/server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexStateService.java @@ -100,46 +100,50 @@ protected ClusterStateUpdateResponse newResponse(boolean acknowledged) { @Override public ClusterState execute(ClusterState currentState) { - Set indicesToClose = new HashSet<>(); - for (Index index : request.indices()) { - final IndexMetaData indexMetaData = currentState.metaData().getIndexSafe(index); - if (indexMetaData.getState() != IndexMetaData.State.CLOSE) { - indicesToClose.add(indexMetaData); - } - } + return closeIndex(currentState, request.indices(), indicesAsString); + } + }); + } - if (indicesToClose.isEmpty()) { - return currentState; - } + public ClusterState closeIndex(ClusterState currentState, final Index[] indices, String indicesAsString) { + Set indicesToClose = new HashSet<>(); + for (Index index : indices) { + final IndexMetaData indexMetaData = currentState.metaData().getIndexSafe(index); + if (indexMetaData.getState() != IndexMetaData.State.CLOSE) { + indicesToClose.add(indexMetaData); + } + } - // Check if index closing conflicts with any running restores - RestoreService.checkIndexClosing(currentState, indicesToClose); - // Check if index closing conflicts with any running snapshots - SnapshotsService.checkIndexClosing(currentState, indicesToClose); - logger.info("closing indices [{}]", indicesAsString); + if (indicesToClose.isEmpty()) { + return currentState; + } - MetaData.Builder mdBuilder = MetaData.builder(currentState.metaData()); - ClusterBlocks.Builder blocksBuilder = ClusterBlocks.builder() - .blocks(currentState.blocks()); - for (IndexMetaData openIndexMetadata : indicesToClose) { - final String indexName = openIndexMetadata.getIndex().getName(); - mdBuilder.put(IndexMetaData.builder(openIndexMetadata).state(IndexMetaData.State.CLOSE)); - blocksBuilder.addIndexBlock(indexName, INDEX_CLOSED_BLOCK); - } + // Check if index closing conflicts with any running restores + RestoreService.checkIndexClosing(currentState, indicesToClose); + // Check if index closing conflicts with any running snapshots + SnapshotsService.checkIndexClosing(currentState, indicesToClose); + logger.info("closing indices [{}]", indicesAsString); + + MetaData.Builder mdBuilder = MetaData.builder(currentState.metaData()); + ClusterBlocks.Builder blocksBuilder = ClusterBlocks.builder() + .blocks(currentState.blocks()); + for (IndexMetaData openIndexMetadata : indicesToClose) { + final String indexName = openIndexMetadata.getIndex().getName(); + mdBuilder.put(IndexMetaData.builder(openIndexMetadata).state(IndexMetaData.State.CLOSE)); + blocksBuilder.addIndexBlock(indexName, INDEX_CLOSED_BLOCK); + } - ClusterState updatedState = ClusterState.builder(currentState).metaData(mdBuilder).blocks(blocksBuilder).build(); + ClusterState updatedState = ClusterState.builder(currentState).metaData(mdBuilder).blocks(blocksBuilder).build(); - RoutingTable.Builder rtBuilder = RoutingTable.builder(currentState.routingTable()); - for (IndexMetaData index : indicesToClose) { - rtBuilder.remove(index.getIndex().getName()); - } + RoutingTable.Builder rtBuilder = RoutingTable.builder(currentState.routingTable()); + for (IndexMetaData index : indicesToClose) { + rtBuilder.remove(index.getIndex().getName()); + } - //no explicit wait for other nodes needed as we use AckedClusterStateUpdateTask - return allocationService.reroute( - ClusterState.builder(updatedState).routingTable(rtBuilder.build()).build(), - "indices closed [" + indicesAsString + "]"); - } - }); + //no explicit wait for other nodes needed as we use AckedClusterStateUpdateTask + return allocationService.reroute( + ClusterState.builder(updatedState).routingTable(rtBuilder.build()).build(), + "indices closed [" + indicesAsString + "]"); } public void openIndex(final OpenIndexClusterStateUpdateRequest request, diff --git a/server/src/main/java/org/elasticsearch/index/engine/ReadOnlyEngine.java b/server/src/main/java/org/elasticsearch/index/engine/ReadOnlyEngine.java index fc4b0632c8076..0ebd848859cf3 100644 --- a/server/src/main/java/org/elasticsearch/index/engine/ReadOnlyEngine.java +++ b/server/src/main/java/org/elasticsearch/index/engine/ReadOnlyEngine.java @@ -20,6 +20,7 @@ import org.apache.lucene.index.DirectoryReader; import org.apache.lucene.index.IndexCommit; +import org.apache.lucene.index.IndexReader; import org.apache.lucene.index.IndexWriter; import org.apache.lucene.index.SegmentCommitInfo; import org.apache.lucene.index.SegmentInfos; @@ -66,7 +67,7 @@ public class ReadOnlyEngine extends Engine { private final IndexCommit indexCommit; private final Lock indexWriterLock; private final DocsStats docsStats; - protected final RamAccountingSearcherFactory searcherFactory; + private final RamAccountingSearcherFactory searcherFactory; /** * Creates a new ReadOnlyEngine. This ctor can also be used to open a read-only engine on top of an already opened @@ -414,4 +415,8 @@ public void updateMaxUnsafeAutoIdTimestamp(long newTimestamp) { public void initializeMaxSeqNoOfUpdatesOrDeletes() { advanceMaxSeqNoOfUpdatesOrDeletes(seqNoStats.getMaxSeqNo()); } + + protected void processReaders(IndexReader reader, IndexReader previousReader) { + searcherFactory.processReaders(reader, previousReader); + } } diff --git a/test/framework/src/main/java/org/elasticsearch/index/shard/IndexShardTestCase.java b/test/framework/src/main/java/org/elasticsearch/index/shard/IndexShardTestCase.java index c9ef79720a29e..4199d414bf345 100644 --- a/test/framework/src/main/java/org/elasticsearch/index/shard/IndexShardTestCase.java +++ b/test/framework/src/main/java/org/elasticsearch/index/shard/IndexShardTestCase.java @@ -389,6 +389,18 @@ protected IndexShard reinitShard(IndexShard current, IndexingOperationListener.. * @param listeners new listerns to use for the newly created shard */ protected IndexShard reinitShard(IndexShard current, ShardRouting routing, IndexingOperationListener... listeners) throws IOException { + return reinitShard(current, routing, current.engineFactory, listeners); + } + + /** + * Takes an existing shard, closes it and starts a new initialing shard at the same location + * + * @param routing the shard routing to use for the newly created shard. + * @param listeners new listerns to use for the newly created shard + * @param engineFactory the engine factory for the new shard + */ + protected IndexShard reinitShard(IndexShard current, ShardRouting routing, EngineFactory engineFactory, + IndexingOperationListener... listeners) throws IOException { closeShards(current); return newShard( routing, @@ -396,7 +408,7 @@ protected IndexShard reinitShard(IndexShard current, ShardRouting routing, Index current.indexSettings().getIndexMetaData(), null, null, - current.engineFactory, + engineFactory, current.getGlobalCheckpointSyncer(), EMPTY_EVENT_LISTENER, listeners); } diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java index ee1135ab53fe4..5a496a2979a5b 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/index/engine/FrozenEngine.java @@ -158,7 +158,7 @@ private synchronized DirectoryReader getOrOpenReader() throws IOException { listeners.beforeRefresh(); } reader = DirectoryReader.open(engineConfig.getStore().directory()); - searcherFactory.processReaders(reader, null); + processReaders(reader, null); reader = lastOpenedReader = wrapReader(reader, Function.identity()); reader.getReaderCacheHelper().addClosedListener(this::onReaderClosed); for (ReferenceManager.RefreshListener listeners : config ().getInternalRefreshListener()) { diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClient.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClient.java index 75b5fa05edbc7..e6cd2ed176c9c 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClient.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClient.java @@ -6,13 +6,15 @@ package org.elasticsearch.xpack.core; import org.elasticsearch.action.ActionListener; -import org.elasticsearch.action.support.master.AcknowledgedResponse; +import org.elasticsearch.action.support.PlainActionFuture; import org.elasticsearch.client.Client; import org.elasticsearch.common.settings.SecureString; import org.elasticsearch.license.LicensingClient; import org.elasticsearch.protocol.xpack.XPackInfoRequest; import org.elasticsearch.protocol.xpack.XPackInfoResponse; -import org.elasticsearch.xpack.core.action.TransportFreezeIndexAction; +import org.elasticsearch.xpack.core.action.TransportFreezeIndexAction.FreezeIndexAction; +import org.elasticsearch.xpack.core.action.TransportFreezeIndexAction.FreezeRequest; +import org.elasticsearch.xpack.core.action.TransportFreezeIndexAction.FreezeResponse; import org.elasticsearch.xpack.core.action.XPackInfoAction; import org.elasticsearch.xpack.core.action.XPackInfoRequestBuilder; import org.elasticsearch.xpack.core.ccr.client.CcrClient; @@ -25,6 +27,7 @@ import java.util.Collections; import java.util.Map; import java.util.Objects; +import java.util.concurrent.ExecutionException; import static org.elasticsearch.xpack.core.security.authc.support.UsernamePasswordToken.BASIC_AUTH_HEADER; import static org.elasticsearch.xpack.core.security.authc.support.UsernamePasswordToken.basicAuthHeaderValue; @@ -106,7 +109,20 @@ public void info(XPackInfoRequest request, ActionListener lis client.execute(XPackInfoAction.INSTANCE, request, listener); } - public void freeze(TransportFreezeIndexAction.FreezeRequest request, ActionListener listener) { - client.execute(TransportFreezeIndexAction.FreezeIndexAction.INSTANCE, request, listener); + /** + * Freezes or unfreeze one or more indices + */ + public void freeze(FreezeRequest request, ActionListener listener) { + client.execute(FreezeIndexAction.INSTANCE, request, listener); + } + + /** + * Freeze or unfreeze one or more indices + */ + public FreezeResponse freeze(FreezeRequest request) + throws ExecutionException, InterruptedException { + PlainActionFuture future = new PlainActionFuture<>(); + freeze(request, future); + return future.get(); } } diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClientPlugin.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClientPlugin.java index b774b2990e096..283b059e030f9 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClientPlugin.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClientPlugin.java @@ -37,6 +37,7 @@ import org.elasticsearch.tasks.Task; import org.elasticsearch.threadpool.ThreadPool; import org.elasticsearch.transport.Transport; +import org.elasticsearch.xpack.core.action.TransportFreezeIndexAction; import org.elasticsearch.xpack.core.action.XPackInfoAction; import org.elasticsearch.xpack.core.action.XPackUsageAction; import org.elasticsearch.xpack.core.beats.BeatsFeatureSetUsage; @@ -344,7 +345,9 @@ public List> getClientActions() { ExplainLifecycleAction.INSTANCE, RemoveIndexLifecyclePolicyAction.INSTANCE, MoveToStepAction.INSTANCE, - RetryAction.INSTANCE + RetryAction.INSTANCE, + // Frozen indices + TransportFreezeIndexAction.FreezeIndexAction.INSTANCE ); } diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackPlugin.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackPlugin.java index ae68252a941a0..bc861b3904f96 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackPlugin.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackPlugin.java @@ -63,6 +63,7 @@ import org.elasticsearch.xpack.core.action.XPackInfoAction; import org.elasticsearch.xpack.core.action.XPackUsageAction; import org.elasticsearch.xpack.core.ml.MlMetadata; +import org.elasticsearch.xpack.core.rest.action.RestFreezeIndexAction; import org.elasticsearch.xpack.core.rest.action.RestXPackInfoAction; import org.elasticsearch.xpack.core.rest.action.RestXPackUsageAction; import org.elasticsearch.xpack.core.security.authc.TokenMetaData; @@ -297,6 +298,7 @@ public List getRestHandlers(Settings settings, RestController restC List handlers = new ArrayList<>(); handlers.add(new RestXPackInfoAction(settings, restController)); handlers.add(new RestXPackUsageAction(settings, restController)); + handlers.add(new RestFreezeIndexAction(settings, restController)); handlers.addAll(licensing.getRestHandlers(settings, restController, clusterSettings, indexScopedSettings, settingsFilter, indexNameExpressionResolver, nodesInCluster)); return handlers; diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java index 693c2b4395f02..05ab20800a9e1 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java @@ -10,7 +10,10 @@ import org.elasticsearch.action.ActionListener; import org.elasticsearch.action.ActionRequestValidationException; import org.elasticsearch.action.IndicesRequest; +import org.elasticsearch.action.admin.indices.open.OpenIndexClusterStateUpdateRequest; +import org.elasticsearch.action.admin.indices.open.OpenIndexResponse; import org.elasticsearch.action.support.ActionFilters; +import org.elasticsearch.action.support.ActiveShardCount; import org.elasticsearch.action.support.DestructiveOperations; import org.elasticsearch.action.support.IndicesOptions; import org.elasticsearch.action.support.master.AcknowledgedRequest; @@ -18,12 +21,14 @@ import org.elasticsearch.action.support.master.TransportMasterNodeAction; import org.elasticsearch.cluster.AckedClusterStateUpdateTask; import org.elasticsearch.cluster.ClusterState; +import org.elasticsearch.cluster.ack.OpenIndexClusterStateUpdateResponse; import org.elasticsearch.cluster.block.ClusterBlockException; import org.elasticsearch.cluster.block.ClusterBlockLevel; import org.elasticsearch.cluster.block.ClusterBlocks; import org.elasticsearch.cluster.metadata.IndexMetaData; import org.elasticsearch.cluster.metadata.IndexNameExpressionResolver; import org.elasticsearch.cluster.metadata.MetaData; +import org.elasticsearch.cluster.metadata.MetaDataIndexStateService; import org.elasticsearch.cluster.service.ClusterService; import org.elasticsearch.common.Priority; import org.elasticsearch.common.inject.Inject; @@ -39,22 +44,27 @@ import org.elasticsearch.transport.TransportService; import java.io.IOException; +import java.util.ArrayList; +import java.util.List; import static org.elasticsearch.action.ValidateActions.addValidationError; public final class TransportFreezeIndexAction extends - TransportMasterNodeAction { + TransportMasterNodeAction { private final DestructiveOperations destructiveOperations; + private final MetaDataIndexStateService indexStateService; @Inject - public TransportFreezeIndexAction(TransportService transportService, ClusterService clusterService, + public TransportFreezeIndexAction(MetaDataIndexStateService indexStateService, TransportService transportService, + ClusterService clusterService, ThreadPool threadPool, ActionFilters actionFilters, IndexNameExpressionResolver indexNameExpressionResolver, DestructiveOperations destructiveOperations) { super(FreezeIndexAction.NAME, transportService, clusterService, threadPool, actionFilters, indexNameExpressionResolver, FreezeRequest::new); this.destructiveOperations = destructiveOperations; + this.indexStateService = indexStateService; } @Override protected String executor() { @@ -62,27 +72,74 @@ protected String executor() { } @Override - protected void doExecute(Task task, FreezeRequest request, ActionListener listener) { + protected void doExecute(Task task, FreezeRequest request, ActionListener listener) { destructiveOperations.failDestructive(request.indices()); super.doExecute(task, request, listener); } @Override - protected AcknowledgedResponse newResponse() { - return new AcknowledgedResponse(); + protected FreezeResponse newResponse() { + return new FreezeResponse(); + } + + private Index[] resolveIndices(FreezeRequest request, ClusterState state) { + List indices = new ArrayList<>(); + for (Index index : indexNameExpressionResolver.concreteIndices(state, request)) { + IndexMetaData metaData = state.metaData().index(index); + Settings settings = metaData.getSettings(); + // only unfreeze if we are frozen and only freeze if we are not frozen already. + // this prevents all indices that are already frozen that match a pattern to + // go through the cycles again. + if ((request.freeze() && FrozenEngine.INDEX_FROZEN.get(settings) == false) || + (request.freeze() == false && FrozenEngine.INDEX_FROZEN.get(settings))) { + indices.add(index); + } + } + return indices.toArray(new Index[0]); } @Override - protected void masterOperation(FreezeRequest request, ClusterState state, ActionListener listener) { - final Index[] concreteIndices = indexNameExpressionResolver.concreteIndices(state, request); + protected void masterOperation(FreezeRequest request, ClusterState state, ActionListener listener) { + final Index[] concreteIndices = resolveIndices(request, state); if (concreteIndices == null || concreteIndices.length == 0) { - throw new ResourceNotFoundException("index not found"); + throw new ResourceNotFoundException("no index found to " + (request.freeze() ? "freeze" : "unfreeze")); } - clusterService.submitStateUpdateTask("toggle-frozen-settings", - new AckedClusterStateUpdateTask(Priority.URGENT, request, listener) { + new AckedClusterStateUpdateTask(Priority.URGENT, request, new ActionListener() { + @Override + public void onResponse(AcknowledgedResponse acknowledgedResponse) { + OpenIndexClusterStateUpdateRequest updateRequest = new OpenIndexClusterStateUpdateRequest() + .ackTimeout(request.timeout()).masterNodeTimeout(request.masterNodeTimeout()) + .indices(concreteIndices).waitForActiveShards(request.waitForActiveShards()); + indexStateService.openIndex(updateRequest, new ActionListener() { + @Override + public void onResponse(OpenIndexClusterStateUpdateResponse openIndexClusterStateUpdateResponse) { + listener.onResponse(new FreezeResponse(openIndexClusterStateUpdateResponse.isAcknowledged(), + openIndexClusterStateUpdateResponse.isShardsAcknowledged())); + } + + @Override + public void onFailure(Exception e) { + listener.onFailure(e); + } + }); + } + + @Override + public void onFailure(Exception e) { + listener.onFailure(e); + } + }) { @Override - public ClusterState execute(final ClusterState currentState) { + public ClusterState execute(ClusterState currentState) { + List toClose = new ArrayList<>(); + for (Index index : concreteIndices) { + IndexMetaData metaData = currentState.metaData().index(index); + if (metaData.getState() != IndexMetaData.State.CLOSE) { + toClose.add(index); + } + } + currentState = indexStateService.closeIndex(currentState, toClose.toArray(new Index[0]), toClose.toString()); final MetaData.Builder builder = MetaData.builder(currentState.metaData()); ClusterBlocks.Builder blocks = ClusterBlocks.builder().blocks(currentState.blocks()); for (Index index : concreteIndices) { @@ -94,7 +151,6 @@ public ClusterState execute(final ClusterState currentState) { final Settings.Builder settingsBuilder = Settings.builder() .put(currentState.metaData().index(index).getSettings()) - .put("index.blocks.write", request.freeze()) .put(FrozenEngine.INDEX_FROZEN.getKey(), request.freeze()) .put(IndexSettings.INDEX_SEARCH_THROTTLED.getKey(), request.freeze()); if (request.freeze()) { @@ -110,6 +166,7 @@ public ClusterState execute(final ClusterState currentState) { @Override protected AcknowledgedResponse newResponse(boolean acknowledged) { + return new AcknowledgedResponse(acknowledged); } }); @@ -121,7 +178,17 @@ protected ClusterBlockException checkBlock(FreezeRequest request, ClusterState s indexNameExpressionResolver.concreteIndexNames(state, request)); } - public static class FreezeIndexAction extends Action { + public static class FreezeResponse extends OpenIndexResponse { + public FreezeResponse() { + super(); + } + + public FreezeResponse(boolean acknowledged, boolean shardsAcknowledged) { + super(acknowledged, shardsAcknowledged); + } + } + + public static class FreezeIndexAction extends Action { public static final FreezeIndexAction INSTANCE = new FreezeIndexAction(); public static final String NAME = "indices:admin/freeze"; @@ -131,8 +198,8 @@ private FreezeIndexAction() { } @Override - public AcknowledgedResponse newResponse() { - return new AcknowledgedResponse(); + public FreezeResponse newResponse() { + return new FreezeResponse(); } } @@ -140,7 +207,8 @@ public static class FreezeRequest extends AcknowledgedRequest implements IndicesRequest.Replaceable { private String[] indices; private boolean freeze = true; - private IndicesOptions indicesOptions = IndicesOptions.fromOptions(false, false, false, true); + private IndicesOptions indicesOptions = IndicesOptions.strictExpandOpen(); + private ActiveShardCount waitForActiveShards = ActiveShardCount.DEFAULT; public FreezeRequest(String... indices) { this.indices = indices; @@ -155,8 +223,9 @@ public ActionRequestValidationException validate() { return validationException; } - public void setFreeze(boolean freeze) { + public FreezeRequest setFreeze(boolean freeze) { this.freeze = freeze; + return this; } public boolean freeze() { @@ -169,6 +238,7 @@ public void readFrom(StreamInput in) throws IOException { indicesOptions = IndicesOptions.readIndicesOptions(in); indices = in.readStringArray(); freeze = in.readBoolean(); + waitForActiveShards = ActiveShardCount.readFrom(in); } @Override @@ -177,6 +247,7 @@ public void writeTo(StreamOutput out) throws IOException { indicesOptions.writeIndicesOptions(out); out.writeStringArray(indices); out.writeBoolean(freeze); + waitForActiveShards.writeTo(out); } /** @@ -215,5 +286,28 @@ public IndicesRequest indices(String... indices) { this.indices = indices; return this; } + + public ActiveShardCount waitForActiveShards() { + return waitForActiveShards; + } + + /** + * Sets the number of shard copies that should be active for indices opening to return. + * Defaults to {@link ActiveShardCount#DEFAULT}, which will wait for one shard copy + * (the primary) to become active. Set this value to {@link ActiveShardCount#ALL} to + * wait for all shards (primary and all replicas) to be active before returning. + * Otherwise, use {@link ActiveShardCount#from(int)} to set this value to any + * non-negative integer, up to the number of copies per shard (number of replicas + 1), + * to wait for the desired amount of shard copies to become active before returning. + * Indices opening will only wait up until the timeout value for the number of shard copies + * to be active before returning. Check {@link OpenIndexResponse#isShardsAcknowledged()} to + * determine if the requisite shard copies were all started before returning or timing out. + * + * @param waitForActiveShards number of active shard copies to wait on + */ + public FreezeRequest waitForActiveShards(ActiveShardCount waitForActiveShards) { + this.waitForActiveShards = waitForActiveShards; + return this; + } } } diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rest/action/RestFreezeIndexAction.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rest/action/RestFreezeIndexAction.java new file mode 100644 index 0000000000000..9604cdd8b3183 --- /dev/null +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rest/action/RestFreezeIndexAction.java @@ -0,0 +1,46 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +package org.elasticsearch.xpack.core.rest.action; + +import org.elasticsearch.action.support.ActiveShardCount; +import org.elasticsearch.action.support.IndicesOptions; +import org.elasticsearch.common.Strings; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.rest.RestController; +import org.elasticsearch.rest.RestRequest; +import org.elasticsearch.rest.action.RestToXContentListener; +import org.elasticsearch.xpack.core.XPackClient; +import org.elasticsearch.xpack.core.action.TransportFreezeIndexAction; +import org.elasticsearch.xpack.core.rest.XPackRestHandler; + +public final class RestFreezeIndexAction extends XPackRestHandler { + public RestFreezeIndexAction(Settings settings, RestController controller) { + super(settings); + controller.registerHandler(RestRequest.Method.POST, "/{index}/_freeze", this); + controller.registerHandler(RestRequest.Method.POST, "/{index}/_unfreeze", this); + } + + @Override + protected RestChannelConsumer doPrepareRequest(RestRequest request, XPackClient client) { + boolean freeze = request.path().endsWith("/_freeze"); + TransportFreezeIndexAction.FreezeRequest freezeRequest = + new TransportFreezeIndexAction.FreezeRequest(Strings.splitStringByCommaToArray(request.param("index"))); + freezeRequest.timeout(request.paramAsTime("timeout", freezeRequest.timeout())); + freezeRequest.masterNodeTimeout(request.paramAsTime("master_timeout", freezeRequest.masterNodeTimeout())); + freezeRequest.indicesOptions(IndicesOptions.fromRequest(request, freezeRequest.indicesOptions())); + String waitForActiveShards = request.param("wait_for_active_shards"); + if (waitForActiveShards != null) { + freezeRequest.waitForActiveShards(ActiveShardCount.parseString(waitForActiveShards)); + } + freezeRequest.setFreeze(freeze); + return channel -> client.freeze(freezeRequest, new RestToXContentListener<>(channel)); + } + + @Override + public String getName() { + return "freeze_index"; + } +} diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexRecoveryTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexRecoveryTests.java new file mode 100644 index 0000000000000..22c646383776c --- /dev/null +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexRecoveryTests.java @@ -0,0 +1,43 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +package org.elasticsearch.index.engine; + +import org.elasticsearch.cluster.routing.RecoverySource; +import org.elasticsearch.cluster.routing.ShardRouting; +import org.elasticsearch.cluster.routing.ShardRoutingHelper; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase; +import org.elasticsearch.index.shard.IndexShard; + +import java.io.IOException; + +import static org.hamcrest.Matchers.equalTo; + +public class FrozenIndexRecoveryTests extends ESIndexLevelReplicationTestCase { + + /** + * Make sure we can recover from a frozen engine + */ + public void testRecoverFromFrozenPrimary() throws IOException { + IndexShard indexShard = newStartedShard(true); + indexDoc(indexShard, "_doc", "1"); + indexDoc(indexShard, "_doc", "2"); + indexDoc(indexShard, "_doc", "3"); + indexShard.close("test", true); + final ShardRouting shardRouting = indexShard.routingEntry(); + IndexShard frozenShard = reinitShard(indexShard, ShardRoutingHelper.initWithSameId(shardRouting, + shardRouting.primary() ? RecoverySource.ExistingStoreRecoverySource.INSTANCE : RecoverySource.PeerRecoverySource.INSTANCE + ), FrozenEngine::new); + recoverShardFromStore(frozenShard); + assertThat(frozenShard.getMaxSeqNoOfUpdatesOrDeletes(), equalTo(frozenShard.seqNoStats().getMaxSeqNo())); + assertDocCount(frozenShard, 3); + + IndexShard replica = newShard(false, Settings.EMPTY, FrozenEngine::new); + recoverReplica(replica, frozenShard, true); + assertDocCount(replica, 3); + closeShards(frozenShard, replica); + } +} diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexTests.java index 5ef1bbd4fa553..4e98f7f3afe24 100644 --- a/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexTests.java +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexTests.java @@ -6,14 +6,13 @@ package org.elasticsearch.index.engine; import org.elasticsearch.action.ActionListener; +import org.elasticsearch.action.admin.cluster.state.ClusterStateResponse; import org.elasticsearch.action.admin.indices.stats.IndicesStatsResponse; import org.elasticsearch.action.search.SearchResponse; import org.elasticsearch.action.search.SearchType; -import org.elasticsearch.action.support.ActiveShardCount; import org.elasticsearch.action.support.IndicesOptions; -import org.elasticsearch.action.support.PlainActionFuture; -import org.elasticsearch.action.support.master.AcknowledgedResponse; import org.elasticsearch.cluster.block.ClusterBlockException; +import org.elasticsearch.cluster.metadata.IndexMetaData; import org.elasticsearch.common.Strings; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.unit.TimeValue; @@ -58,14 +57,8 @@ public void testCloseFreezeAndOpen() throws ExecutionException, InterruptedExcep client().prepareIndex("index", "_doc", "1").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); client().prepareIndex("index", "_doc", "2").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); client().prepareIndex("index", "_doc", "3").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); - - client().admin().indices().prepareFlush("index").get(); - client().admin().indices().prepareClose("index").get(); XPackClient xPackClient = new XPackClient(client()); - PlainActionFuture future = new PlainActionFuture<>(); - xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("index"), future); - assertAcked(future.get()); - assertAcked(client().admin().indices().prepareOpen("index")); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("index"))); expectThrows(ClusterBlockException.class, () -> client().prepareIndex("index", "_doc", "4").setSource("field", "value") .setRefreshPolicy(IMMEDIATE).get()); IndicesService indexServices = getInstanceFromNode(IndicesService.class); @@ -101,7 +94,7 @@ public void testCloseFreezeAndOpen() throws ExecutionException, InterruptedExcep } while (searchResponse.getHits().getHits().length > 0); } - public void testSearchAndGetAPIsAreThrottled() throws ExecutionException, InterruptedException, IOException { + public void testSearchAndGetAPIsAreThrottled() throws InterruptedException, IOException, ExecutionException { XContentBuilder mapping = XContentFactory.jsonBuilder().startObject().startObject("_doc") .startObject("properties").startObject("field").field("type", "text").field("term_vector", "with_positions_offsets_payloads") .endObject().endObject() @@ -110,15 +103,8 @@ public void testSearchAndGetAPIsAreThrottled() throws ExecutionException, Interr for (int i = 0; i < 10; i++) { client().prepareIndex("index", "_doc", "" + i).setSource("field", "foo bar baz").get(); } - client().admin().indices().prepareFlush("index").get(); - client().admin().indices().prepareClose("index").get(); XPackClient xPackClient = new XPackClient(client()); - PlainActionFuture future = new PlainActionFuture<>(); - TransportFreezeIndexAction.FreezeRequest request = - new TransportFreezeIndexAction.FreezeRequest("index"); - xPackClient.freeze(request, future); - assertAcked(future.get()); - assertAcked(client().admin().indices().prepareOpen("index")); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("index"))); int numRequests = randomIntBetween(20, 50); CountDownLatch latch = new CountDownLatch(numRequests); ActionListener listener = ActionListener.wrap(latch::countDown); @@ -152,21 +138,17 @@ public void testSearchAndGetAPIsAreThrottled() throws ExecutionException, Interr assertEquals(numRefreshes, index.getTotal().refresh.getTotal()); } - public void testFreezeAndUnfreeze() throws ExecutionException, InterruptedException { + public void testFreezeAndUnfreeze() throws InterruptedException, ExecutionException { createIndex("index", Settings.builder().put("index.number_of_shards", 2).build()); client().prepareIndex("index", "_doc", "1").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); client().prepareIndex("index", "_doc", "2").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); client().prepareIndex("index", "_doc", "3").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); - - client().admin().indices().prepareFlush("index").get(); - client().admin().indices().prepareClose("index").get(); + if (randomBoolean()) { + // sometimes close it + assertAcked(client().admin().indices().prepareClose("index").get()); + } XPackClient xPackClient = new XPackClient(client()); - PlainActionFuture future = new PlainActionFuture<>(); - TransportFreezeIndexAction.FreezeRequest request = - new TransportFreezeIndexAction.FreezeRequest("index"); - xPackClient.freeze(request, future); - assertAcked(future.get()); - assertAcked(client().admin().indices().prepareOpen("index")); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("index"))); { IndicesService indexServices = getInstanceFromNode(IndicesService.class); Index index = resolveIndex("index"); @@ -175,12 +157,7 @@ public void testFreezeAndUnfreeze() throws ExecutionException, InterruptedExcept IndexShard shard = indexService.getShard(0); assertEquals(0, shard.refreshStats().getTotal()); } - client().admin().indices().prepareClose("index").get(); - request.setFreeze(false); - PlainActionFuture future1= new PlainActionFuture<>(); - xPackClient.freeze(request, future1); - assertAcked(future1.get()); - assertAcked(client().admin().indices().prepareOpen("index")); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("index").setFreeze(false))); { IndicesService indexServices = getInstanceFromNode(IndicesService.class); Index index = resolveIndex("index"); @@ -193,16 +170,61 @@ public void testFreezeAndUnfreeze() throws ExecutionException, InterruptedExcept client().prepareIndex("index", "_doc", "4").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); } - public void testIndexMustBeClosed() { + private void assertIndexFrozen(String idx) { + IndicesService indexServices = getInstanceFromNode(IndicesService.class); + Index index = resolveIndex(idx); + IndexService indexService = indexServices.indexServiceSafe(index); + assertTrue(indexService.getIndexSettings().isSearchThrottled()); + assertTrue(FrozenEngine.INDEX_FROZEN.get(indexService.getIndexSettings().getSettings())); + } + + public void testDoubleFreeze() throws ExecutionException, InterruptedException { createIndex("test-idx", Settings.builder().put("index.number_of_shards", 2).build()); XPackClient xPackClient = new XPackClient(client()); - PlainActionFuture future = new PlainActionFuture<>(); - TransportFreezeIndexAction.FreezeRequest request = - new TransportFreezeIndexAction.FreezeRequest("test-idx"); - xPackClient.freeze(request, future); - ExecutionException executionException = expectThrows(ExecutionException.class, () -> future.get()); - assertThat(executionException.getCause(), Matchers.instanceOf(IllegalStateException.class)); - assertEquals("index [test-idx] is not closed", executionException.getCause().getMessage()); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("test-idx"))); + ExecutionException executionException = expectThrows(ExecutionException.class, + () -> xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("test-idx"))); + assertEquals("no index found to freeze", executionException.getCause().getMessage()); + } + + public void testUnfreezeClosedIndices() throws ExecutionException, InterruptedException { + createIndex("idx", Settings.builder().put("index.number_of_shards", 1).build()); + client().prepareIndex("idx", "_doc", "1").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); + createIndex("idx-closed", Settings.builder().put("index.number_of_shards", 1).build()); + client().prepareIndex("idx-closed", "_doc", "1").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); + XPackClient xPackClient = new XPackClient(client()); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("idx"))); + assertAcked(client().admin().indices().prepareClose("idx-closed").get()); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("idx*").setFreeze(false) + .indicesOptions(IndicesOptions.strictExpand()))); + ClusterStateResponse stateResponse = client().admin().cluster().prepareState().get(); + assertEquals(IndexMetaData.State.CLOSE, stateResponse.getState().getMetaData().index("idx-closed").getState()); + assertEquals(IndexMetaData.State.OPEN, stateResponse.getState().getMetaData().index("idx").getState()); + assertHitCount(client().prepareSearch().get(), 1L); + } + + public void testFreezePattern() throws ExecutionException, InterruptedException { + createIndex("test-idx", Settings.builder().put("index.number_of_shards", 1).build()); + client().prepareIndex("test-idx", "_doc", "1").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); + createIndex("test-idx-1", Settings.builder().put("index.number_of_shards", 1).build()); + client().prepareIndex("test-idx-1", "_doc", "1").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); + XPackClient xPackClient = new XPackClient(client()); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("test-idx"))); + assertIndexFrozen("test-idx"); + + IndicesStatsResponse index = client().admin().indices().prepareStats("test-idx").clear().setRefresh(true).get(); + assertEquals(0, index.getTotal().refresh.getTotal()); + assertHitCount(client().prepareSearch("test-idx").setIndicesOptions(IndicesOptions.STRICT_EXPAND_OPEN_FORBID_CLOSED).get(), 1); + index = client().admin().indices().prepareStats("test-idx").clear().setRefresh(true).get(); + assertEquals(1, index.getTotal().refresh.getTotal()); + + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("test*"))); + assertIndexFrozen("test-idx"); + assertIndexFrozen("test-idx-1"); + index = client().admin().indices().prepareStats("test-idx").clear().setRefresh(true).get(); + assertEquals(1, index.getTotal().refresh.getTotal()); + index = client().admin().indices().prepareStats("test-idx-1").clear().setRefresh(true).get(); + assertEquals(0, index.getTotal().refresh.getTotal()); } public void testCanMatch() throws ExecutionException, InterruptedException, IOException { @@ -232,15 +254,9 @@ public void testCanMatch() throws ExecutionException, InterruptedException, IOEx Strings.EMPTY_ARRAY, false, new AliasFilter(null, Strings.EMPTY_ARRAY), 1f, true, null, null))); } - client().admin().indices().prepareFlush("index").get(); - client().admin().indices().prepareClose("index").get(); + XPackClient xPackClient = new XPackClient(client()); - PlainActionFuture future = new PlainActionFuture<>(); - TransportFreezeIndexAction.FreezeRequest request = - new TransportFreezeIndexAction.FreezeRequest("index"); - xPackClient.freeze(request, future); - assertAcked(future.get()); - assertAcked(client().admin().indices().prepareOpen("index").setWaitForActiveShards(ActiveShardCount.DEFAULT)); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("index"))); { IndicesService indexServices = getInstanceFromNode(IndicesService.class); @@ -266,4 +282,41 @@ public void testCanMatch() throws ExecutionException, InterruptedException, IOEx assertEquals(0, response.getTotal().refresh.getTotal()); // never opened a reader } } + + public void testWriteToFrozenIndex() throws ExecutionException, InterruptedException { + createIndex("idx", Settings.builder().put("index.number_of_shards", 1).build()); + client().prepareIndex("idx", "_doc", "1").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get(); + XPackClient xPackClient = new XPackClient(client()); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("idx"))); + assertIndexFrozen("idx"); + expectThrows(ClusterBlockException.class, () -> + client().prepareIndex("idx", "_doc", "2").setSource("field", "value").setRefreshPolicy(IMMEDIATE).get()); + } + + public void testIgnoreUnavailable() throws ExecutionException, InterruptedException { + createIndex("idx", Settings.builder().put("index.number_of_shards", 1).build()); + createIndex("idx-close", Settings.builder().put("index.number_of_shards", 1).build()); + assertAcked(client().admin().indices().prepareClose("idx-close")); + XPackClient xPackClient = new XPackClient(client()); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("idx*", "not_available") + .indicesOptions(IndicesOptions.fromParameters(null, "true", null, null, IndicesOptions.strictExpandOpen())))); + assertIndexFrozen("idx"); + assertEquals(IndexMetaData.State.CLOSE, + client().admin().cluster().prepareState().get().getState().metaData().index("idx-close").getState()); + } + + public void testUnfreezeClosedIndex() throws ExecutionException, InterruptedException { + createIndex("idx", Settings.builder().put("index.number_of_shards", 1).build()); + XPackClient xPackClient = new XPackClient(client()); + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("idx"))); + assertAcked(client().admin().indices().prepareClose("idx")); + assertEquals(IndexMetaData.State.CLOSE, + client().admin().cluster().prepareState().get().getState().metaData().index("idx").getState()); + expectThrows(ExecutionException.class, + () -> xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("id*").setFreeze(false))); + // we don't resolve to closed indices + assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("idx").setFreeze(false))); + assertEquals(IndexMetaData.State.OPEN, + client().admin().cluster().prepareState().get().getState().metaData().index("idx").getState()); + } } diff --git a/x-pack/plugin/src/test/resources/rest-api-spec/api/indices.freeze.json b/x-pack/plugin/src/test/resources/rest-api-spec/api/indices.freeze.json new file mode 100644 index 0000000000000..a602f341e1524 --- /dev/null +++ b/x-pack/plugin/src/test/resources/rest-api-spec/api/indices.freeze.json @@ -0,0 +1,48 @@ +{ + "indices.freeze": { + "documentation": "https://www.elastic.co/guide/en/elasticsearch/reference/current/frozen.html", + "methods": [ "POST" ], + "url": { + "path": "/{index}/_freeze", + "paths": [ + "/{index}/_freeze" + ], + "parts": { + "index": { + "type": "string", + "required": true, + "description": "The name of the index to freeze" + } + }, + "params": { + "timeout": { + "type" : "time", + "description" : "Explicit operation timeout" + }, + "master_timeout": { + "type" : "time", + "description" : "Specify timeout for connection to master" + }, + "ignore_unavailable": { + "type" : "boolean", + "description" : "Whether specified concrete indices should be ignored when unavailable (missing or closed)" + }, + "allow_no_indices": { + "type" : "boolean", + "description" : "Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes `_all` string or when no indices have been specified)" + }, + "expand_wildcards": { + "type" : "enum", + "options" : ["open","closed","none","all"], + "default" : "closed", + "description" : "Whether to expand wildcard expression to concrete indices that are open, closed or both." + }, + "wait_for_active_shards": { + "type" : "string", + "description" : "Sets the number of active shards to wait for before the operation returns." + } + } + }, + "body": null + } +} diff --git a/x-pack/plugin/src/test/resources/rest-api-spec/api/indices.unfreeze.json b/x-pack/plugin/src/test/resources/rest-api-spec/api/indices.unfreeze.json new file mode 100644 index 0000000000000..b10e869a95758 --- /dev/null +++ b/x-pack/plugin/src/test/resources/rest-api-spec/api/indices.unfreeze.json @@ -0,0 +1,46 @@ +{ + "indices.unfreeze": { + "documentation": "https://www.elastic.co/guide/en/elasticsearch/reference/current/frozen.html", + "methods": [ "POST" ], + "url": { + "path": "/{index}/_unfreeze", + "paths": [ "/{index}/_unfreeze" ], + "parts": { + "index": { + "type": "string", + "required": true, + "description": "The name of the index to unfreeze" + } + }, + "params": { + "timeout": { + "type" : "time", + "description" : "Explicit operation timeout" + }, + "master_timeout": { + "type" : "time", + "description" : "Specify timeout for connection to master" + }, + "ignore_unavailable": { + "type" : "boolean", + "description" : "Whether specified concrete indices should be ignored when unavailable (missing or closed)" + }, + "allow_no_indices": { + "type" : "boolean", + "description" : "Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes `_all` string or when no indices have been specified)" + }, + "expand_wildcards": { + "type" : "enum", + "options" : ["open","closed","none","all"], + "default" : "closed", + "description" : "Whether to expand wildcard expression to concrete indices that are open, closed or both." + }, + "wait_for_active_shards": { + "type" : "string", + "description" : "Sets the number of active shards to wait for before the operation returns." + } + } + }, + "body": null + } +} diff --git a/x-pack/plugin/src/test/resources/rest-api-spec/test/indices.freeze/10_basic.yml b/x-pack/plugin/src/test/resources/rest-api-spec/test/indices.freeze/10_basic.yml new file mode 100644 index 0000000000000..8f79225ddc37d --- /dev/null +++ b/x-pack/plugin/src/test/resources/rest-api-spec/test/indices.freeze/10_basic.yml @@ -0,0 +1,128 @@ +--- +"Basic": + +- skip: + version: " - 6.99.99" + reason: types are required in requests before 7.0.0 + +- do: + index: + index: test + id: "1" + body: { "foo": "Hello: 1" } +- do: + index: + index: test + id: "2" + body: { "foo": "Hello: 2" } + +- do: + indices.freeze: + index: test + +- do: + search: + index: test + ignore_throttled: false + body: + query: + match: + foo: hello + +- match: {hits.total: 2} + +# unfreeze +- do: + indices.unfreeze: + index: test + +- do: + search: + index: _all + body: + query: + match: + foo: hello + +- match: {hits.total: 2} + +- do: + index: + index: test-01 + id: "1" + body: { "foo": "Hello: 01" } + + +- do: + indices.freeze: + index: test* + +- do: + search: + index: _all + ignore_throttled: false + body: + query: + match: + foo: hello + +- match: {hits.total: 3} + +- do: + search: + index: _all + body: + query: + match: + foo: hello + +- match: {hits.total: 0} + +--- +"Test index options": + +- skip: + version: " - 6.99.99" + reason: types are required in requests before 7.0.0 + +- do: + index: + index: test + id: "1" + body: { "foo": "Hello: 1" } + +- do: + index: + index: test-close + id: "1" + body: { "foo": "Hello: 1" } + +- do: + indices.close: + index: test-close + +- do: + indices.freeze: + index: test*,not_available + ignore_unavailable: true + +- do: + search: + index: _all + body: + query: + match: + foo: hello + +- match: {hits.total: 0} + +- do: + search: + index: _all + ignore_throttled: false + body: + query: + match: + foo: hello + +- match: {hits.total: 1} From 6579645a06a19d4851c2f5a23d8c1027bbc725e1 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:21:49 +0100 Subject: [PATCH 02/38] Update docs/reference/modules/threadpool.asciidoc Co-Authored-By: s1monw --- docs/reference/modules/threadpool.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/modules/threadpool.asciidoc b/docs/reference/modules/threadpool.asciidoc index d0492e66aa29d..38f8d40e67f23 100644 --- a/docs/reference/modules/threadpool.asciidoc +++ b/docs/reference/modules/threadpool.asciidoc @@ -19,7 +19,7 @@ There are several thread pools, but the important ones include: `int((# of available_processors * 3) / 2) + 1`, and initial queue_size of `1000`. -`search_throttled`:: +[[search-throttled]]`search_throttled`:: For count/search/suggest/get operations on `search_throttled indices`. Thread pool type is `fixed_auto_queue_size` with a size of `1`, and initial queue_size of `100`. From da1a7e1c238fa97b0fa4318da86881bfe9fccff7 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:22:01 +0100 Subject: [PATCH 03/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 40e8303c3ba1d..0b0851b3dfbed 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -4,7 +4,7 @@ = Frozen Indices Elasticsearch indices can require a significant amount of memory available in order to be open and searchable. Yet, not all indices need -to be writable at the same time and have different access patters over time. For example indices in the time-series or logging use-case +to be writable at the same time and have different access patterns over time. For example indices in the time-series or logging use-case are unlikely to be queried once they age out but still need to be kept around for retention policy purposes. In order to keep indices around for a longer time period but at the same time reducing their hardware requirements they can be transitioned From 98784772b18bb120bc70fcb5a5c0897f2907e6d0 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:22:10 +0100 Subject: [PATCH 04/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 0b0851b3dfbed..e4f0ff1e9bf70 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -7,7 +7,7 @@ Elasticsearch indices can require a significant amount of memory available in or to be writable at the same time and have different access patterns over time. For example indices in the time-series or logging use-case are unlikely to be queried once they age out but still need to be kept around for retention policy purposes. -In order to keep indices around for a longer time period but at the same time reducing their hardware requirements they can be transitioned +In order to keep indices available and queryable for a longer period but at the same time reduce their hardware requirements they can be transitioned into a frozen state. Once an index is frozen, all it's transient shard memory aside of mappings and and necessary in-memory structures for analysis is moved to persistent storage. This allows for a much higher disk to heap storage ratio on individual nodes. Once an index is frozen their are read-only and require their data-structures required for query execution to be loaded on demand. A search request that hits From 7e284c7255fb564f8fabf13a1441b48b56aee4f2 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:22:18 +0100 Subject: [PATCH 05/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index e4f0ff1e9bf70..5c79d12be180e 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -8,7 +8,7 @@ to be writable at the same time and have different access patterns over time. Fo are unlikely to be queried once they age out but still need to be kept around for retention policy purposes. In order to keep indices available and queryable for a longer period but at the same time reduce their hardware requirements they can be transitioned -into a frozen state. Once an index is frozen, all it's transient shard memory aside of mappings and and necessary in-memory structures for +into a frozen state. Once an index is frozen, all of it's transient shard memory (aside from mappings and analyzers) analysis is moved to persistent storage. This allows for a much higher disk to heap storage ratio on individual nodes. Once an index is frozen their are read-only and require their data-structures required for query execution to be loaded on demand. A search request that hits one or more shards from frozen indices will execute it's search through a throttling component that ensures that we never search more than From 72f6dbfa803e5d3a6b9ad630474c1cd05097d195 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:23:11 +0100 Subject: [PATCH 06/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 5c79d12be180e..d9dc92030e8df 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -9,7 +9,7 @@ are unlikely to be queried once they age out but still need to be kept around fo In order to keep indices available and queryable for a longer period but at the same time reduce their hardware requirements they can be transitioned into a frozen state. Once an index is frozen, all of it's transient shard memory (aside from mappings and analyzers) -analysis is moved to persistent storage. This allows for a much higher disk to heap storage ratio on individual nodes. Once an index is +is moved to persistent storage. This allows for a much higher disk to heap storage ratio on individual nodes. Once an index is frozen their are read-only and require their data-structures required for query execution to be loaded on demand. A search request that hits one or more shards from frozen indices will execute it's search through a throttling component that ensures that we never search more than `N` (`1` by default) searches concurrently. This protects nodes from exceeding the available memory due to incoming search requests. From 095dc8f39ad21dc7ab0d5a96de3df1a81422ba0d Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:23:21 +0100 Subject: [PATCH 07/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index d9dc92030e8df..d6fde30ac8466 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -71,7 +71,7 @@ GET /twitter/_search?q=user:kimchy&ignore_throttled=false [IMPORTANT] ================================ -While frozen indices are slow to search, they offer an efficient pre-filter phase. The request parameter `pre_filter_shard_size` specifies +While frozen indices are slow to search, they can be pre-filtered efficiently. The request parameter `pre_filter_shard_size` specifies a threshold that enforces a round-trip to filter search shards based on query rewriting if the number of shards the search request expands to exceeds the threshold. This filter phase can limit the number of shards significantly if for instance a shard can not match any documents based on it's rewrite method ie. if date filters are mandatory to match but the shard bounds and the query are disjoint. From 2ac89f0211811e4f253eeda4c0df01812be5d43d Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:23:29 +0100 Subject: [PATCH 08/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index d6fde30ac8466..26870e7d3d0c1 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -72,7 +72,7 @@ GET /twitter/_search?q=user:kimchy&ignore_throttled=false [IMPORTANT] ================================ While frozen indices are slow to search, they can be pre-filtered efficiently. The request parameter `pre_filter_shard_size` specifies -a threshold that enforces a round-trip to filter search shards based on query rewriting if the number of shards the search +a threshold that, when exceeded, will enforce a round-trip to pre-filter search shards that cannot possibly match. request expands to exceeds the threshold. This filter phase can limit the number of shards significantly if for instance a shard can not match any documents based on it's rewrite method ie. if date filters are mandatory to match but the shard bounds and the query are disjoint. The default value for `pre_filter_shard_size` is `128` while for searching frozen indices it's recommended to set it to `1`. There is no From 27f465c2eb02319f360511b986f798fdaef86dc5 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:25:00 +0100 Subject: [PATCH 09/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 26870e7d3d0c1..0e555df695315 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -75,7 +75,7 @@ While frozen indices are slow to search, they can be pre-filtered efficiently. T a threshold that, when exceeded, will enforce a round-trip to pre-filter search shards that cannot possibly match. request expands to exceeds the threshold. This filter phase can limit the number of shards significantly if for instance a shard can not match any documents based on it's rewrite method ie. if date filters are mandatory to match but the shard bounds and the query are disjoint. -The default value for `pre_filter_shard_size` is `128` while for searching frozen indices it's recommended to set it to `1`. There is no +The default value for `pre_filter_shard_size` is `128` but it's recommended to set it to `1` when searching frozen indices. There is no significant overhead associated with this pre-filter phase. ================================ From 0a52d499625bd37443d410c224ca8ac577a57283 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:25:08 +0100 Subject: [PATCH 10/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 0e555df695315..40e738cdf5667 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -74,7 +74,6 @@ GET /twitter/_search?q=user:kimchy&ignore_throttled=false While frozen indices are slow to search, they can be pre-filtered efficiently. The request parameter `pre_filter_shard_size` specifies a threshold that, when exceeded, will enforce a round-trip to pre-filter search shards that cannot possibly match. request expands to exceeds the threshold. This filter phase can limit the number of shards significantly if for instance a shard can not -match any documents based on it's rewrite method ie. if date filters are mandatory to match but the shard bounds and the query are disjoint. The default value for `pre_filter_shard_size` is `128` but it's recommended to set it to `1` when searching frozen indices. There is no significant overhead associated with this pre-filter phase. ================================ From 433d407ba7dafebea85cfdbd269e04d08f4acc8c Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:25:16 +0100 Subject: [PATCH 11/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 40e738cdf5667..0d5b4af88124f 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -73,7 +73,7 @@ GET /twitter/_search?q=user:kimchy&ignore_throttled=false ================================ While frozen indices are slow to search, they can be pre-filtered efficiently. The request parameter `pre_filter_shard_size` specifies a threshold that, when exceeded, will enforce a round-trip to pre-filter search shards that cannot possibly match. -request expands to exceeds the threshold. This filter phase can limit the number of shards significantly if for instance a shard can not +This filter phase can limit the number of shards significantly. For instance, if a date range filter is applied, then all indices (frozen or unfrozen) that do not contain documents within the date range can be skipped efficiently. The default value for `pre_filter_shard_size` is `128` but it's recommended to set it to `1` when searching frozen indices. There is no significant overhead associated with this pre-filter phase. ================================ From 59db168d90d3478c6cec9e5f9a5f2a18cbb8ac06 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:25:26 +0100 Subject: [PATCH 12/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 0d5b4af88124f..c6df2f40c2d0d 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -10,7 +10,7 @@ are unlikely to be queried once they age out but still need to be kept around fo In order to keep indices available and queryable for a longer period but at the same time reduce their hardware requirements they can be transitioned into a frozen state. Once an index is frozen, all of it's transient shard memory (aside from mappings and analyzers) is moved to persistent storage. This allows for a much higher disk to heap storage ratio on individual nodes. Once an index is -frozen their are read-only and require their data-structures required for query execution to be loaded on demand. A search request that hits +frozen, it is made read-only and drops its transient data structures from memory. These will need to be reloaded on demand (and subsequently dropped) for each search request that targets the frozen index. A search request that hits one or more shards from frozen indices will execute it's search through a throttling component that ensures that we never search more than `N` (`1` by default) searches concurrently. This protects nodes from exceeding the available memory due to incoming search requests. From 2e33e30b17f85cb508a9bec7cd470035ec2ad3cd Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:25:54 +0100 Subject: [PATCH 13/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index c6df2f40c2d0d..472d7e3b0d62b 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -11,7 +11,7 @@ In order to keep indices available and queryable for a longer period but at the into a frozen state. Once an index is frozen, all of it's transient shard memory (aside from mappings and analyzers) is moved to persistent storage. This allows for a much higher disk to heap storage ratio on individual nodes. Once an index is frozen, it is made read-only and drops its transient data structures from memory. These will need to be reloaded on demand (and subsequently dropped) for each search request that targets the frozen index. A search request that hits -one or more shards from frozen indices will execute it's search through a throttling component that ensures that we never search more than +one or more frozen shards will be executed on a throttled threadpool that ensures that we never search more than `N` (`1` by default) searches concurrently. This protects nodes from exceeding the available memory due to incoming search requests. In contrast to ordinary open indices, frozen indices are expected to execute slow and are not designed for high query load. Parallelism is From 311884cf985822609af96def29876b329952fc76 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:26:08 +0100 Subject: [PATCH 14/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 472d7e3b0d62b..b5a9a3d58b083 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -12,7 +12,7 @@ into a frozen state. Once an index is frozen, all of it's transient shard memory is moved to persistent storage. This allows for a much higher disk to heap storage ratio on individual nodes. Once an index is frozen, it is made read-only and drops its transient data structures from memory. These will need to be reloaded on demand (and subsequently dropped) for each search request that targets the frozen index. A search request that hits one or more frozen shards will be executed on a throttled threadpool that ensures that we never search more than -`N` (`1` by default) searches concurrently. This protects nodes from exceeding the available memory due to incoming search requests. +`N` (`1` by default) searches concurrently (see <>). This protects nodes from exceeding the available memory due to incoming search requests. In contrast to ordinary open indices, frozen indices are expected to execute slow and are not designed for high query load. Parallelism is gained only on a per-node level and loading data-structures on demand is expected to be one or more orders of a magnitude slower than query From 71f61922edc9c5a0e8349bcbccfd10e5f18f6c22 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:26:22 +0100 Subject: [PATCH 15/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index b5a9a3d58b083..8e66c68026344 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -60,7 +60,7 @@ Frozen indices are throttled in order limit memory consumptions per node. The nu limited by the number of threads in the `search_throttled` <> which is `1` by default. At the same time, search requests hitting may indices on a cluster due to concrete lists, expanded wildcards or a date pattern exclude frozen indices by default to prevent accidental slowdowns when a frozen index is hit. To include frozen indices a search request must be executed with -`ignore_throttled=false`. +the query parameter `ignore_throttled=false`. [source,js] -------------------------------------------------- From a4932e1dbe5b0f84375e48ccc0c945cdaef45c5d Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:26:34 +0100 Subject: [PATCH 16/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 8e66c68026344..2653ce921926c 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -14,7 +14,7 @@ frozen, it is made read-only and drops its transient data structures from memory one or more frozen shards will be executed on a throttled threadpool that ensures that we never search more than `N` (`1` by default) searches concurrently (see <>). This protects nodes from exceeding the available memory due to incoming search requests. -In contrast to ordinary open indices, frozen indices are expected to execute slow and are not designed for high query load. Parallelism is +In contrast to ordinary open indices, frozen indices are expected to execute slowly and are not designed for high query load. Parallelism is gained only on a per-node level and loading data-structures on demand is expected to be one or more orders of a magnitude slower than query execution on a per shard level. Depending on the index and it's data executing a search on a frozen index is expected to be in the seconds or even minutes range compared to milliseconds on a non-frozen index. From 3b4eef72910abebc7ab1a6f93dfe856fc286ff3a Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:26:51 +0100 Subject: [PATCH 17/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 2653ce921926c..5421237c1ee6e 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -16,7 +16,7 @@ one or more frozen shards will be executed on a throttled threadpool that ensure In contrast to ordinary open indices, frozen indices are expected to execute slowly and are not designed for high query load. Parallelism is gained only on a per-node level and loading data-structures on demand is expected to be one or more orders of a magnitude slower than query -execution on a per shard level. Depending on the index and it's data executing a search on a frozen index is expected to be in the seconds +execution on a per shard level. Depending on the data in an index, a frozen index may execute searches in the seconds to minutes range, when the same index in an unfrozen state may execute the same search request in milliseconds. or even minutes range compared to milliseconds on a non-frozen index. == Best Practices From 5364a72dc9cc3c53bae221f945c691fb89ad6b70 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:27:04 +0100 Subject: [PATCH 18/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 5421237c1ee6e..e95b3637c3e8a 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -17,7 +17,6 @@ one or more frozen shards will be executed on a throttled threadpool that ensure In contrast to ordinary open indices, frozen indices are expected to execute slowly and are not designed for high query load. Parallelism is gained only on a per-node level and loading data-structures on demand is expected to be one or more orders of a magnitude slower than query execution on a per shard level. Depending on the data in an index, a frozen index may execute searches in the seconds to minutes range, when the same index in an unfrozen state may execute the same search request in milliseconds. -or even minutes range compared to milliseconds on a non-frozen index. == Best Practices From 0e5664fe43b5dffc7bc58811654caa8ec44700df Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:27:34 +0100 Subject: [PATCH 19/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index e95b3637c3e8a..151bf9558dcbe 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -20,7 +20,7 @@ execution on a per shard level. Depending on the data in an index, a frozen inde == Best Practices -Since frozen indices provide a much higher disk to heap ratio for the expense of latency it is recommended to allocate frozen indices on +Since frozen indices provide a much higher disk to heap ratio at the expense of search latency, it is advisable to allocate frozen indices to dedicated nodes to prevent searches on frozen indices influencing traffic on low latency nodes. There is a significant overhead in loading data-structures on demand which can cause page-faults and garbage collections further slowing down query execution. From b04e56ce6c116d394e3c21c5a0c1af770de1561f Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:27:51 +0100 Subject: [PATCH 20/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 151bf9558dcbe..b45fecd014c14 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -21,7 +21,7 @@ execution on a per shard level. Depending on the data in an index, a frozen inde == Best Practices Since frozen indices provide a much higher disk to heap ratio at the expense of search latency, it is advisable to allocate frozen indices to -dedicated nodes to prevent searches on frozen indices influencing traffic on low latency nodes. There is a significant overhead in loading +dedicated nodes to prevent searches on frozen indices influencing traffic on low latency nodes. There is significant overhead in loading data-structures on demand which can cause page-faults and garbage collections further slowing down query execution. Since indices that are eligible for freezing are likely to not change in the future disk space can be optimized. Many strategies are From 774369d674f352d92e2fbd53a4a10d99ea410b83 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:28:32 +0100 Subject: [PATCH 21/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index b45fecd014c14..d9f7253534ed8 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -22,7 +22,7 @@ execution on a per shard level. Depending on the data in an index, a frozen inde Since frozen indices provide a much higher disk to heap ratio at the expense of search latency, it is advisable to allocate frozen indices to dedicated nodes to prevent searches on frozen indices influencing traffic on low latency nodes. There is significant overhead in loading -data-structures on demand which can cause page-faults and garbage collections further slowing down query execution. +data-structures on demand which can cause page-faults and garbage collections, which further slow down query execution. Since indices that are eligible for freezing are likely to not change in the future disk space can be optimized. Many strategies are outlined in <>. From af5e748fbc1f9dc79ba34ff478487415190ff6f0 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:33:27 +0100 Subject: [PATCH 22/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index d9f7253534ed8..e2b9d8392c1bf 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -24,7 +24,7 @@ Since frozen indices provide a much higher disk to heap ratio at the expense of dedicated nodes to prevent searches on frozen indices influencing traffic on low latency nodes. There is significant overhead in loading data-structures on demand which can cause page-faults and garbage collections, which further slow down query execution. -Since indices that are eligible for freezing are likely to not change in the future disk space can be optimized. Many strategies are +Since indices that are eligible for freezing are unlikely to change in the future, disk space can be optimized as described in <>. outlined in <>. == Freezing and unfreezing an index From 521e7851fb73372a49d4a8f7f75c1b8161535c0c Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:33:40 +0100 Subject: [PATCH 23/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index e2b9d8392c1bf..86e73e4e97662 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -25,7 +25,6 @@ dedicated nodes to prevent searches on frozen indices influencing traffic on low data-structures on demand which can cause page-faults and garbage collections, which further slow down query execution. Since indices that are eligible for freezing are unlikely to change in the future, disk space can be optimized as described in <>. -outlined in <>. == Freezing and unfreezing an index From 0f915c0001b9b59194b797509304a6349c9d725d Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:33:54 +0100 Subject: [PATCH 24/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 86e73e4e97662..878f14a9187bf 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -28,7 +28,7 @@ Since indices that are eligible for freezing are unlikely to change in the futur == Freezing and unfreezing an index -The freeze and unfreeze index APIs allow to freeze an index, and later on +The freeze and unfreeze index APIs are used to freeze and unfreeze an index. unfreeze it. A frozen index has almost no overhead on the cluster (except for maintaining its metadata in memory), and is blocked for write operations. A frozen index can be unfrozen which will then go through the normal recovery process. From 30be388feb6ca5f124fd6bc664dc53b46afe17f8 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:34:05 +0100 Subject: [PATCH 25/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 878f14a9187bf..5b298e97d04c5 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -29,7 +29,7 @@ Since indices that are eligible for freezing are unlikely to change in the futur == Freezing and unfreezing an index The freeze and unfreeze index APIs are used to freeze and unfreeze an index. -unfreeze it. A frozen index has almost no overhead on the cluster (except +A frozen index has almost no overhead on the cluster (except for maintaining its metadata in memory), and is blocked for write operations. A frozen index can be unfrozen which will then go through the normal recovery process. From 471798b29630587168b358263021eb047243e96b Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:34:26 +0100 Subject: [PATCH 26/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 5b298e97d04c5..9e487abdd9fdb 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -31,7 +31,7 @@ Since indices that are eligible for freezing are unlikely to change in the futur The freeze and unfreeze index APIs are used to freeze and unfreeze an index. A frozen index has almost no overhead on the cluster (except for maintaining its metadata in memory), and is blocked for write operations. -A frozen index can be unfrozen which will then go through the normal recovery process. +When a frozen index is unfrozen, then index will go through the normal recovery process and become writeable again. The REST endpoint is `/{index}/_freeze` and `/{index}/_unfreeze`. For example: From 060f5a2127a5dc35a87be80f929379122ef11380 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:34:45 +0100 Subject: [PATCH 27/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 9e487abdd9fdb..16ec1be4bce8a 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -49,7 +49,7 @@ POST /my_index/_unfreeze [IMPORTANT] ================================ Freezing an index will close the index and reopen it within the same API call. This causes primaries to not be allocated for a short - amount of time and causes the cluster to go red until the primaries are allocated again. This limitation might be removed in the future + amount of time and causes the cluster to go red until the primaries are allocated again. This limitation might be removed in the future. ================================ == Searching a frozen index From 4e04574f8c5a1a56f0e8a7a5aeff26ae4a7a7c49 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:34:56 +0100 Subject: [PATCH 28/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 16ec1be4bce8a..97bb518bc8724 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -55,7 +55,7 @@ POST /my_index/_unfreeze == Searching a frozen index Frozen indices are throttled in order limit memory consumptions per node. The number of concurrently loaded frozen indices per node is -limited by the number of threads in the `search_throttled` <> which is `1` by default. At the same time, +limited by the number of threads in the <> threadpool, which is `1` by default. search requests hitting may indices on a cluster due to concrete lists, expanded wildcards or a date pattern exclude frozen indices by default to prevent accidental slowdowns when a frozen index is hit. To include frozen indices a search request must be executed with the query parameter `ignore_throttled=false`. From 6a144e1c812963cbede81837d9c52e3a037a8f40 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:35:09 +0100 Subject: [PATCH 29/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 97bb518bc8724..af6a5d9151eff 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -56,7 +56,7 @@ POST /my_index/_unfreeze Frozen indices are throttled in order limit memory consumptions per node. The number of concurrently loaded frozen indices per node is limited by the number of threads in the <> threadpool, which is `1` by default. -search requests hitting may indices on a cluster due to concrete lists, expanded wildcards or a date pattern exclude frozen indices by +Search requests will not be executed against frozen indices by default, even if a frozen index is named explicitly. Th is is ``` default to prevent accidental slowdowns when a frozen index is hit. To include frozen indices a search request must be executed with the query parameter `ignore_throttled=false`. From e29c978e44efd2b766d64732e5ba75c14342ad49 Mon Sep 17 00:00:00 2001 From: Clinton Gormley Date: Thu, 15 Nov 2018 16:35:21 +0100 Subject: [PATCH 30/38] Update docs/reference/frozen-indices.asciidoc Co-Authored-By: s1monw --- docs/reference/frozen-indices.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index af6a5d9151eff..a2daf0c61af53 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -57,7 +57,7 @@ POST /my_index/_unfreeze Frozen indices are throttled in order limit memory consumptions per node. The number of concurrently loaded frozen indices per node is limited by the number of threads in the <> threadpool, which is `1` by default. Search requests will not be executed against frozen indices by default, even if a frozen index is named explicitly. Th is is ``` -default to prevent accidental slowdowns when a frozen index is hit. To include frozen indices a search request must be executed with +to prevent accidental slowdowns by targeting a frozen index by mistake.. To include frozen indices a search request must be executed with the query parameter `ignore_throttled=false`. [source,js] From 84806f62d73c1032b41bf70f9255616f835f4ed9 Mon Sep 17 00:00:00 2001 From: lcawl Date: Thu, 15 Nov 2018 16:06:05 -0800 Subject: [PATCH 31/38] [DOCS] Adds partintro and small edits --- docs/reference/frozen-indices.asciidoc | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index a2daf0c61af53..d15223e429e97 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -3,26 +3,29 @@ [[frozen-indices]] = Frozen Indices +[partintro] +-- Elasticsearch indices can require a significant amount of memory available in order to be open and searchable. Yet, not all indices need -to be writable at the same time and have different access patterns over time. For example indices in the time-series or logging use-case +to be writable at the same time and have different access patterns over time. For example, indices in the time series or logging use cases are unlikely to be queried once they age out but still need to be kept around for retention policy purposes. In order to keep indices available and queryable for a longer period but at the same time reduce their hardware requirements they can be transitioned -into a frozen state. Once an index is frozen, all of it's transient shard memory (aside from mappings and analyzers) +into a frozen state. Once an index is frozen, all of its transient shard memory (aside from mappings and analyzers) is moved to persistent storage. This allows for a much higher disk to heap storage ratio on individual nodes. Once an index is -frozen, it is made read-only and drops its transient data structures from memory. These will need to be reloaded on demand (and subsequently dropped) for each search request that targets the frozen index. A search request that hits +frozen, it is made read-only and drops its transient data structures from memory. These data structures will need to be reloaded on demand (and subsequently dropped) for each search request that targets the frozen index. A search request that hits one or more frozen shards will be executed on a throttled threadpool that ensures that we never search more than `N` (`1` by default) searches concurrently (see <>). This protects nodes from exceeding the available memory due to incoming search requests. In contrast to ordinary open indices, frozen indices are expected to execute slowly and are not designed for high query load. Parallelism is gained only on a per-node level and loading data-structures on demand is expected to be one or more orders of a magnitude slower than query execution on a per shard level. Depending on the data in an index, a frozen index may execute searches in the seconds to minutes range, when the same index in an unfrozen state may execute the same search request in milliseconds. +-- == Best Practices -Since frozen indices provide a much higher disk to heap ratio at the expense of search latency, it is advisable to allocate frozen indices to +Since frozen indices provide a much higher disk to heap ratio at the expense of search latency, it is advisable to allocate frozen indices to dedicated nodes to prevent searches on frozen indices influencing traffic on low latency nodes. There is significant overhead in loading -data-structures on demand which can cause page-faults and garbage collections, which further slow down query execution. +data structures on demand which can cause page faults and garbage collections, which further slow down query execution. Since indices that are eligible for freezing are unlikely to change in the future, disk space can be optimized as described in <>. @@ -56,8 +59,8 @@ POST /my_index/_unfreeze Frozen indices are throttled in order limit memory consumptions per node. The number of concurrently loaded frozen indices per node is limited by the number of threads in the <> threadpool, which is `1` by default. -Search requests will not be executed against frozen indices by default, even if a frozen index is named explicitly. Th is is ``` -to prevent accidental slowdowns by targeting a frozen index by mistake.. To include frozen indices a search request must be executed with +Search requests will not be executed against frozen indices by default, even if a frozen index is named explicitly. This is +to prevent accidental slowdowns by targeting a frozen index by mistake. To include frozen indices a search request must be executed with the query parameter `ignore_throttled=false`. [source,js] From 0cc2613bbe9ac2507ca45b305e6fe859fb6085ec Mon Sep 17 00:00:00 2001 From: lcawl Date: Thu, 15 Nov 2018 16:07:19 -0800 Subject: [PATCH 32/38] [DOCS] Adds frozen indices page to Elasticsearch Reference --- docs/reference/index.asciidoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference/index.asciidoc b/docs/reference/index.asciidoc index 48cb7700a52ea..208829481514c 100644 --- a/docs/reference/index.asciidoc +++ b/docs/reference/index.asciidoc @@ -63,6 +63,8 @@ include::monitoring/index.asciidoc[] include::rollup/index.asciidoc[] +include::frozen-indices.asciidoc[] + include::rest-api/index.asciidoc[] include::commands/index.asciidoc[] From 3c1935cd712d52997da5c7e92eff61dd3f0807e6 Mon Sep 17 00:00:00 2001 From: lcawl Date: Thu, 15 Nov 2018 16:34:20 -0800 Subject: [PATCH 33/38] [DOCS] Adds API reference pages for freeze and unfreeze --- docs/reference/indices/apis/freeze.asciidoc | 44 +++++++++++++++++++ docs/reference/indices/apis/unfreeze.asciidoc | 44 +++++++++++++++++++ docs/reference/rest-api/index.asciidoc | 3 ++ 3 files changed, 91 insertions(+) create mode 100644 docs/reference/indices/apis/freeze.asciidoc create mode 100644 docs/reference/indices/apis/unfreeze.asciidoc diff --git a/docs/reference/indices/apis/freeze.asciidoc b/docs/reference/indices/apis/freeze.asciidoc new file mode 100644 index 0000000000000..a0f9bc7961a17 --- /dev/null +++ b/docs/reference/indices/apis/freeze.asciidoc @@ -0,0 +1,44 @@ +[role="xpack"] +[testenv="basic"] +[[freeze-index-api]] +== Freeze Index API +++++ +Freeze Index +++++ + +Freezes an index. + +[float] +=== Request + +`POST //_freeze` + +[float] +=== Description + +A frozen index has almost no overhead on the cluster (except +for maintaining its metadata in memory), and is blocked for write operations. +See <> and <>. + +[float] +=== Path Parameters + +`index` (required):: +(string) Identifier for the index + +//=== Query Parameters + +//=== Authorization + +[float] +=== Examples + +The following example freezes and unfreezes an index: + +[source,js] +-------------------------------------------------- +POST /my_index/_freeze + POST /my_index/_unfreeze +-------------------------------------------------- +// CONSOLE +// TEST[s/^/PUT my_index\n/] \ No newline at end of file diff --git a/docs/reference/indices/apis/unfreeze.asciidoc b/docs/reference/indices/apis/unfreeze.asciidoc new file mode 100644 index 0000000000000..bf75b4d475cde --- /dev/null +++ b/docs/reference/indices/apis/unfreeze.asciidoc @@ -0,0 +1,44 @@ +[role="xpack"] +[testenv="basic"] +[[unfreeze-index-api]] +== Unfreeze Index API +++++ +Unfreeze Index +++++ + +Unfreezes an index. + +[float] +=== Request + +`POST //_unfreeze` + +[float] +=== Description + +When a frozen index is unfrozen, the index goes through the normal recovery +process and becomes writeable again. See <> and <>. + +[float] +=== Path Parameters + +`index` (required):: +(string) Identifier for the index + + +//=== Query Parameters + +//=== Authorization + +[float] +=== Examples + +The following example freezes and unfreezes an index: + +[source,js] +-------------------------------------------------- +POST /my_index/_freeze + POST /my_index/_unfreeze +-------------------------------------------------- +// CONSOLE +// TEST[s/^/PUT my_index\n/] \ No newline at end of file diff --git a/docs/reference/rest-api/index.asciidoc b/docs/reference/rest-api/index.asciidoc index eedc2dfa1f51f..e834249724aa3 100644 --- a/docs/reference/rest-api/index.asciidoc +++ b/docs/reference/rest-api/index.asciidoc @@ -10,6 +10,7 @@ directly to configure and access {xpack} features. * <> * <> * <> +* <>, <> * <> * <> * <> @@ -23,11 +24,13 @@ directly to configure and access {xpack} features. include::info.asciidoc[] include::{es-repo-dir}/ccr/apis/ccr-apis.asciidoc[] include::{es-repo-dir}/graph/explore.asciidoc[] +include::{es-repo-dir}/indices/apis/freeze.asciidoc[] include::{es-repo-dir}/ilm/apis/ilm-api.asciidoc[] include::{es-repo-dir}/licensing/index.asciidoc[] include::{es-repo-dir}/migration/migration.asciidoc[] include::{es-repo-dir}/ml/apis/ml-api.asciidoc[] include::{es-repo-dir}/rollup/rollup-api.asciidoc[] include::{xes-repo-dir}/rest-api/security.asciidoc[] +include::{es-repo-dir}/indices/apis/unfreeze.asciidoc[] include::{xes-repo-dir}/rest-api/watcher.asciidoc[] include::defs.asciidoc[] From 2d0667fe6cc2713aa213d79569b338ad28bb7b9c Mon Sep 17 00:00:00 2001 From: lcawl Date: Thu, 15 Nov 2018 17:15:59 -0800 Subject: [PATCH 34/38] [DOCS] Fixes gradle errors --- docs/reference/indices/apis/freeze.asciidoc | 2 +- docs/reference/indices/apis/unfreeze.asciidoc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/indices/apis/freeze.asciidoc b/docs/reference/indices/apis/freeze.asciidoc index a0f9bc7961a17..5b63cb824f15e 100644 --- a/docs/reference/indices/apis/freeze.asciidoc +++ b/docs/reference/indices/apis/freeze.asciidoc @@ -38,7 +38,7 @@ The following example freezes and unfreezes an index: [source,js] -------------------------------------------------- POST /my_index/_freeze - POST /my_index/_unfreeze +POST /my_index/_unfreeze -------------------------------------------------- // CONSOLE // TEST[s/^/PUT my_index\n/] \ No newline at end of file diff --git a/docs/reference/indices/apis/unfreeze.asciidoc b/docs/reference/indices/apis/unfreeze.asciidoc index bf75b4d475cde..2dee7b6276ef1 100644 --- a/docs/reference/indices/apis/unfreeze.asciidoc +++ b/docs/reference/indices/apis/unfreeze.asciidoc @@ -38,7 +38,7 @@ The following example freezes and unfreezes an index: [source,js] -------------------------------------------------- POST /my_index/_freeze - POST /my_index/_unfreeze +POST /my_index/_unfreeze -------------------------------------------------- // CONSOLE // TEST[s/^/PUT my_index\n/] \ No newline at end of file From 801c462d7332ecc526fa5bf5519efaca2b7b43e7 Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Mon, 19 Nov 2018 10:15:37 +0100 Subject: [PATCH 35/38] move warning --- docs/reference/frozen-indices.asciidoc | 26 ------------------- docs/reference/indices/apis/freeze.asciidoc | 8 +++++- docs/reference/indices/apis/unfreeze.asciidoc | 8 +++++- 3 files changed, 14 insertions(+), 28 deletions(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index d15223e429e97..4ab78961cc6e5 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -29,32 +29,6 @@ data structures on demand which can cause page faults and garbage collections, w Since indices that are eligible for freezing are unlikely to change in the future, disk space can be optimized as described in <>. -== Freezing and unfreezing an index - -The freeze and unfreeze index APIs are used to freeze and unfreeze an index. -A frozen index has almost no overhead on the cluster (except -for maintaining its metadata in memory), and is blocked for write operations. -When a frozen index is unfrozen, then index will go through the normal recovery process and become writeable again. - -The REST endpoint is `/{index}/_freeze` and `/{index}/_unfreeze`. For -example: - -[source,js] --------------------------------------------------- -POST /my_index/_freeze - -POST /my_index/_unfreeze --------------------------------------------------- -// CONSOLE -// TEST[s/^/PUT my_index\n/] - - -[IMPORTANT] -================================ - Freezing an index will close the index and reopen it within the same API call. This causes primaries to not be allocated for a short - amount of time and causes the cluster to go red until the primaries are allocated again. This limitation might be removed in the future. -================================ - == Searching a frozen index Frozen indices are throttled in order limit memory consumptions per node. The number of concurrently loaded frozen indices per node is diff --git a/docs/reference/indices/apis/freeze.asciidoc b/docs/reference/indices/apis/freeze.asciidoc index 5b63cb824f15e..5ca9ecbc6b801 100644 --- a/docs/reference/indices/apis/freeze.asciidoc +++ b/docs/reference/indices/apis/freeze.asciidoc @@ -41,4 +41,10 @@ POST /my_index/_freeze POST /my_index/_unfreeze -------------------------------------------------- // CONSOLE -// TEST[s/^/PUT my_index\n/] \ No newline at end of file +// TEST[s/^/PUT my_index\n/] + +[IMPORTANT] +================================ + Freezing an index will close the index and reopen it within the same API call. This causes primaries to not be allocated for a short + amount of time and causes the cluster to go red until the primaries are allocated again. This limitation might be removed in the future. +================================ diff --git a/docs/reference/indices/apis/unfreeze.asciidoc b/docs/reference/indices/apis/unfreeze.asciidoc index 2dee7b6276ef1..4a01813463516 100644 --- a/docs/reference/indices/apis/unfreeze.asciidoc +++ b/docs/reference/indices/apis/unfreeze.asciidoc @@ -41,4 +41,10 @@ POST /my_index/_freeze POST /my_index/_unfreeze -------------------------------------------------- // CONSOLE -// TEST[s/^/PUT my_index\n/] \ No newline at end of file +// TEST[s/^/PUT my_index\n/] + +[IMPORTANT] +================================ + Freezing an index will close the index and reopen it within the same API call. This causes primaries to not be allocated for a short + amount of time and causes the cluster to go red until the primaries are allocated again. This limitation might be removed in the future. +================================ From 304adc67d511f28e45cb6fedf3dd0cdb0f667db1 Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Mon, 19 Nov 2018 10:44:21 +0100 Subject: [PATCH 36/38] address comments --- docs/reference/frozen-indices.asciidoc | 2 +- .../xpack/core/action/TransportFreezeIndexAction.java | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/reference/frozen-indices.asciidoc b/docs/reference/frozen-indices.asciidoc index 4ab78961cc6e5..28264547575b7 100644 --- a/docs/reference/frozen-indices.asciidoc +++ b/docs/reference/frozen-indices.asciidoc @@ -31,7 +31,7 @@ Since indices that are eligible for freezing are unlikely to change in the futur == Searching a frozen index -Frozen indices are throttled in order limit memory consumptions per node. The number of concurrently loaded frozen indices per node is +Frozen indices are throttled in order to limit memory consumptions per node. The number of concurrently loaded frozen indices per node is limited by the number of threads in the <> threadpool, which is `1` by default. Search requests will not be executed against frozen indices by default, even if a frozen index is named explicitly. This is to prevent accidental slowdowns by targeting a frozen index by mistake. To include frozen indices a search request must be executed with diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java index 05ab20800a9e1..78f9b36fce322 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java @@ -95,7 +95,7 @@ private Index[] resolveIndices(FreezeRequest request, ClusterState state) { indices.add(index); } } - return indices.toArray(new Index[0]); + return indices.toArray(Index.EMPTY_ARRAY); } @Override @@ -166,7 +166,6 @@ public ClusterState execute(ClusterState currentState) { @Override protected AcknowledgedResponse newResponse(boolean acknowledged) { - return new AcknowledgedResponse(acknowledged); } }); From f38f14513aa68d0d2d981e49d14b1219940a58fe Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Mon, 19 Nov 2018 15:42:19 +0100 Subject: [PATCH 37/38] address feedback from @bleskes --- .../admin/indices/close/TransportCloseIndexAction.java | 2 +- .../cluster/metadata/MetaDataIndexStateService.java | 6 +++--- .../elasticsearch/xpack/core/XPackClientPlugin.java | 1 - .../xpack/core/action/TransportFreezeIndexAction.java | 10 +++++++--- .../index/engine/FrozenIndexRecoveryTests.java | 4 ++-- .../elasticsearch/index/engine/FrozenIndexTests.java | 9 +++++++-- 6 files changed, 20 insertions(+), 12 deletions(-) diff --git a/server/src/main/java/org/elasticsearch/action/admin/indices/close/TransportCloseIndexAction.java b/server/src/main/java/org/elasticsearch/action/admin/indices/close/TransportCloseIndexAction.java index 5124ec25525d9..fb5fdf536a248 100644 --- a/server/src/main/java/org/elasticsearch/action/admin/indices/close/TransportCloseIndexAction.java +++ b/server/src/main/java/org/elasticsearch/action/admin/indices/close/TransportCloseIndexAction.java @@ -109,7 +109,7 @@ protected void masterOperation(final CloseIndexRequest request, final ClusterSta .ackTimeout(request.timeout()).masterNodeTimeout(request.masterNodeTimeout()) .indices(concreteIndices); - indexStateService.closeIndex(updateRequest, new ActionListener() { + indexStateService.closeIndices(updateRequest, new ActionListener() { @Override public void onResponse(ClusterStateUpdateResponse response) { diff --git a/server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexStateService.java b/server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexStateService.java index 457b0f9b7d714..3d3258c5ae301 100644 --- a/server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexStateService.java +++ b/server/src/main/java/org/elasticsearch/cluster/metadata/MetaDataIndexStateService.java @@ -84,7 +84,7 @@ public MetaDataIndexStateService(ClusterService clusterService, AllocationServic this.activeShardsObserver = new ActiveShardsObserver(clusterService, threadPool); } - public void closeIndex(final CloseIndexClusterStateUpdateRequest request, final ActionListener listener) { + public void closeIndices(final CloseIndexClusterStateUpdateRequest request, final ActionListener listener) { if (request.indices() == null || request.indices().length == 0) { throw new IllegalArgumentException("Index name is required"); } @@ -99,12 +99,12 @@ protected ClusterStateUpdateResponse newResponse(boolean acknowledged) { @Override public ClusterState execute(ClusterState currentState) { - return closeIndex(currentState, request.indices(), indicesAsString); + return closeIndices(currentState, request.indices(), indicesAsString); } }); } - public ClusterState closeIndex(ClusterState currentState, final Index[] indices, String indicesAsString) { + public ClusterState closeIndices(ClusterState currentState, final Index[] indices, String indicesAsString) { Set indicesToClose = new HashSet<>(); for (Index index : indices) { final IndexMetaData indexMetaData = currentState.metaData().getIndexSafe(index); diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClientPlugin.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClientPlugin.java index 283b059e030f9..a8167187bb2e6 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClientPlugin.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClientPlugin.java @@ -346,7 +346,6 @@ public List> getClientActions() { RemoveIndexLifecyclePolicyAction.INSTANCE, MoveToStepAction.INSTANCE, RetryAction.INSTANCE, - // Frozen indices TransportFreezeIndexAction.FreezeIndexAction.INSTANCE ); } diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java index 78f9b36fce322..312742e2d7a4a 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java @@ -95,14 +95,18 @@ private Index[] resolveIndices(FreezeRequest request, ClusterState state) { indices.add(index); } } + if (indices.isEmpty() && request.indicesOptions().allowNoIndices() == false) { + throw new ResourceNotFoundException("no index found to " + (request.freeze() ? "freeze" : "unfreeze")); + } return indices.toArray(Index.EMPTY_ARRAY); } @Override protected void masterOperation(FreezeRequest request, ClusterState state, ActionListener listener) { final Index[] concreteIndices = resolveIndices(request, state); - if (concreteIndices == null || concreteIndices.length == 0) { - throw new ResourceNotFoundException("no index found to " + (request.freeze() ? "freeze" : "unfreeze")); + if (concreteIndices.length == 0) { + listener.onResponse(new FreezeResponse(true, true)); + return; } clusterService.submitStateUpdateTask("toggle-frozen-settings", new AckedClusterStateUpdateTask(Priority.URGENT, request, new ActionListener() { @@ -139,7 +143,7 @@ public ClusterState execute(ClusterState currentState) { toClose.add(index); } } - currentState = indexStateService.closeIndex(currentState, toClose.toArray(new Index[0]), toClose.toString()); + currentState = indexStateService.closeIndices(currentState, toClose.toArray(new Index[0]), toClose.toString()); final MetaData.Builder builder = MetaData.builder(currentState.metaData()); ClusterBlocks.Builder blocks = ClusterBlocks.builder().blocks(currentState.blocks()); for (Index index : concreteIndices) { diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexRecoveryTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexRecoveryTests.java index 22c646383776c..b67258dd9b3d0 100644 --- a/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexRecoveryTests.java +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexRecoveryTests.java @@ -9,14 +9,14 @@ import org.elasticsearch.cluster.routing.ShardRouting; import org.elasticsearch.cluster.routing.ShardRoutingHelper; import org.elasticsearch.common.settings.Settings; -import org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase; import org.elasticsearch.index.shard.IndexShard; +import org.elasticsearch.index.shard.IndexShardTestCase; import java.io.IOException; import static org.hamcrest.Matchers.equalTo; -public class FrozenIndexRecoveryTests extends ESIndexLevelReplicationTestCase { +public class FrozenIndexRecoveryTests extends IndexShardTestCase { /** * Make sure we can recover from a frozen engine diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexTests.java index 4e98f7f3afe24..884dafdcd395b 100644 --- a/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexTests.java +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/index/engine/FrozenIndexTests.java @@ -38,6 +38,7 @@ import java.io.IOException; import java.util.Collection; +import java.util.EnumSet; import java.util.concurrent.CountDownLatch; import java.util.concurrent.ExecutionException; @@ -183,7 +184,9 @@ public void testDoubleFreeze() throws ExecutionException, InterruptedException { XPackClient xPackClient = new XPackClient(client()); assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("test-idx"))); ExecutionException executionException = expectThrows(ExecutionException.class, - () -> xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("test-idx"))); + () -> xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("test-idx") + .indicesOptions(new IndicesOptions(EnumSet.noneOf(IndicesOptions.Option.class), + EnumSet.of(IndicesOptions.WildcardStates.OPEN))))); assertEquals("no index found to freeze", executionException.getCause().getMessage()); } @@ -313,7 +316,9 @@ public void testUnfreezeClosedIndex() throws ExecutionException, InterruptedExce assertEquals(IndexMetaData.State.CLOSE, client().admin().cluster().prepareState().get().getState().metaData().index("idx").getState()); expectThrows(ExecutionException.class, - () -> xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("id*").setFreeze(false))); + () -> xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("id*").setFreeze(false) + .indicesOptions(new IndicesOptions(EnumSet.noneOf(IndicesOptions.Option.class), + EnumSet.of(IndicesOptions.WildcardStates.OPEN))))); // we don't resolve to closed indices assertAcked(xPackClient.freeze(new TransportFreezeIndexAction.FreezeRequest("idx").setFreeze(false))); assertEquals(IndexMetaData.State.OPEN, From 9648d7172e8a4af2abd95334ddeffafd4a6d4b5c Mon Sep 17 00:00:00 2001 From: Simon Willnauer Date: Mon, 19 Nov 2018 17:16:20 +0100 Subject: [PATCH 38/38] make sure blocks survive cluster restarts --- .../xpack/core/action/TransportFreezeIndexAction.java | 2 ++ 1 file changed, 2 insertions(+) diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java index 312742e2d7a4a..1feb84d9b9539 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/action/TransportFreezeIndexAction.java @@ -158,8 +158,10 @@ public ClusterState execute(ClusterState currentState) { .put(FrozenEngine.INDEX_FROZEN.getKey(), request.freeze()) .put(IndexSettings.INDEX_SEARCH_THROTTLED.getKey(), request.freeze()); if (request.freeze()) { + settingsBuilder.put("index.blocks.write", true); blocks.addIndexBlock(index.getName(), IndexMetaData.INDEX_WRITE_BLOCK); } else { + settingsBuilder.remove("index.blocks.write"); blocks.removeIndexBlock(index.getName(), IndexMetaData.INDEX_WRITE_BLOCK); } imdBuilder.settings(settingsBuilder);