Skip shard refreshes if shard is `search idle` #27500

s1monw · 2017-11-22T21:36:23Z

Today we refresh automatically in the backgroud by default very second.
This default behavior has a significant impact on indexing performance
if the refreshes are not needed.
This change introduces a notion of a shard being search idle which a
shard transitions to after (default) 30s without any access to an
external searcher. Once a shard is search idle all scheduled refreshes
will be skipped unless there are any refresh listeners registered.
If a search happens on a serach idle shard the search request park
on a refresh listener and will be executed once the next scheduled refresh
occurs. This will also turn the shard into the non-idle state immediately.

This behavior is only applied if there is no explicit refresh interval set.

Today we refresh automatically in the backgroud by default very second. This default behavior has a significant impact on indexing performance if the refreshes are not needed. This change introduces a notion of a shard being `search idle` which a shard transitions to after (default) `30s` without any access to an external searcher. Once a shard is search idle all scheduled refreshes will be skipped unless there are any refresh listeners registered. If a search happens on a `serach idle` shard the search request _park_ on a refresh listener and will be executed once the next scheduled refresh occurs. This will also turn the shard into the `non-idle` state immediately. This behavior is only applied if there is no explicit refresh interval set.

jpountz

Not very familiar with this part of the code base but what I understand makes sense to me. You will probably want to add docs as well.

jpountz · 2017-11-23T08:38:57Z

core/src/main/java/org/elasticsearch/index/shard/IndexShard.java

+        } while (pendingRefreshLocation.compareAndSet(location, lastWriteLocation) == false);
+    }
+
+    public void awaitPendingRefresh(Consumer<Boolean> listener) {


Please add javadocs.

tlrx · 2017-11-23T08:55:42Z

core/src/main/java/org/elasticsearch/index/shard/IndexShard.java

+            if (refreshListeners.refreshNeeded() == false // if we have a listener that is waiting for a refresh we need to force it
+                && isSearchIdle() && indexSettings.isExplicitRefresh() == false) {
+                // lets skip this refresh since we are search idle and
+                // don't necessarily need to refresh. the next search execute cause a


nit: truncated comment

…he backgroud

bleskes

I can't fully comment to the integration with the search system but the IndexShard approach is nice. I left some comments for discussion.

bleskes · 2017-11-27T10:12:04Z

core/src/main/java/org/elasticsearch/action/get/TransportGetAction.java

+    protected void shardOperation(GetRequest request, ShardId shardId, ActionListener<GetResponse> listener) throws IOException {
+        IndexService indexService = indicesService.indexServiceSafe(shardId.getIndex());
+        IndexShard indexShard = indexService.getShard(shardId.id());
+        indexShard.awaitPendingRefresh(b -> {


I'm on the fence as to whether we should only do this on non-realtime get. Real time gets don't really relate to refresh cycles (they force a refresh if needed). They are already "efficient" in the sense that they only refresh if they need to (i.e., there's a pending doc change in the version map).

bleskes · 2017-11-27T10:14:06Z

.../src/main/java/org/elasticsearch/action/support/single/shard/TransportSingleShardAction.java

@@ -97,6 +100,19 @@ protected void doExecute(Request request, ActionListener<Response> listener) {

    protected abstract Response shardOperation(Request request, ShardId shardId) throws IOException;

+    protected void shardOperation(Request request, ShardId shardId, ActionListener<Response> listener) throws IOException {


maybe call this asyncShardOperation to avoid confusion? also, can you please java doc the fact that this is still called on the networking thread?

bleskes · 2017-11-27T10:18:46Z

core/src/main/java/org/elasticsearch/action/termvectors/TransportTermVectorsAction.java

+    protected void shardOperation(TermVectorsRequest request, ShardId shardId, ActionListener<TermVectorsResponse> listener) throws IOException {
+        IndexService indexService = indicesService.indexServiceSafe(shardId.getIndex());
+        IndexShard indexShard = indexService.getShard(shardId.id());
+        indexShard.awaitPendingRefresh(b -> {


same comment here w.r.t real time requests.

bleskes · 2017-11-27T10:26:25Z

core/src/main/java/org/elasticsearch/index/IndexService.java

+                // once we change the refresh interval we schedule yet another refresh
+                // to ensure we are in a clean and predictable state.
+                // it doesn't matter if we move from or to <code>-1</code>  in both cases we want
+                // docs to become visible immediately


I guess the important part here is the we need to flush any pending search requests which are waiting for the next refresh. If so, can you add it to the comment? It's not trivial to figure out.

Also, if that's the case, maybe we're better off doing it in IndexShard#onSettingsChanged. Then all the corresponding code is contained in the same class.
PS I wonder if we do the right thing with RefreshListeners - we people stop refreshing we should do a refresh and release all pending listeners (and refuse to add new ones). This is, of course, not relating to your PR.

yeah so I wonder if we should rather build it into the refresh listener that is forces a refresh if refreshes are disabled

I'm not sure people setting wait_for will want a immediately executed because refreshes are set to -1. I'm +1 to solving it in the refresh listeners - out of scope for this PR of course.

bleskes · 2017-11-27T10:32:02Z

core/src/main/java/org/elasticsearch/index/shard/IndexShard.java

+    private void setRefreshPending() {
+        Engine engine = getEngine();
+        if (isSearchIdle()) {
+            acquireSearcher("setRefreshPending").close(); // move the shard into non-search idle


why is this needed?

it's a bug I removed it and added a test

bleskes · 2017-11-27T12:48:15Z

core/src/test/java/org/elasticsearch/index/shard/IndexShardIT.java

+        IndexShard shard = indexService.getShard(0);
+        boolean hasRefreshed = shard.scheduledRefresh();
+        if (randomTimeValue == TimeValue.ZERO) {
+            // with ZERO we are guaranteed to see the doc since we will wait for a refresh in the background


I'm not sure what you mean by see the doc? I guess triggering a refresh due to it?

bleskes · 2017-11-27T12:49:13Z

core/src/test/java/org/elasticsearch/index/shard/IndexShardIT.java

+            assertFalse(shard.isSearchIdle());
+        }
+        assertHitCount(client().prepareSearch().get(), 1);
+        for (int i = 1; i < numDocs; i++) {


what does this extra indexing buy us?

that's just how the test works?

we do nothing after the indexing is done nor during the indexing?

bleskes · 2017-11-27T12:51:23Z

core/src/test/java/org/elasticsearch/index/shard/IndexShardIT.java

+        assertFalse(shard.scheduledRefresh());
+        assertTrue(shard.isSearchIdle());
+        CountDownLatch refreshLatch = new CountDownLatch(1);
+        client().admin().indices().prepareRefresh().execute(ActionListener.wrap(refreshLatch::countDown)); // async on purpose


this is just to speed up the test, right? if so, can you add a comment? if not please tell me what I miss ;)

just to not span a thread, it happens concurrently?

bleskes · 2017-11-27T12:54:31Z

core/src/test/java/org/elasticsearch/index/shard/IndexShardIT.java

+        t.join();
+    }
+
+    public void testPendingRefreshWithIntervalChange() throws InterruptedException {


bleskes · 2017-11-27T12:55:19Z

core/src/test/java/org/elasticsearch/index/shard/IndexShardTests.java

+        settings = Settings.builder().put(settings).put(IndexSettings.INDEX_SEARCH_IDLE_AFTER.getKey(), TimeValue.timeValueMillis(10))
+            .build();
+        scopedSettings.applySettings(settings);
+        while (primary.isSearchIdle() == false) {


lol. assert busy?

s1monw · 2017-11-27T14:30:10Z

@bleskes I pushed changes..

I also run benchmarks and the results are promising:

|                        Metric |         Task |   Baseline |   Contender |     Diff |   Unit |
|------------------------------:|-------------:|-----------:|------------:|---------:|-------:|
|                 Indexing time |              |     25.308 |     23.4078 | -1.90015 |    min |
|                    Merge time |              |    5.72372 |     3.62805 | -2.09567 |    min |
|                  Refresh time |              |    1.84012 |     0.30375 | -1.53637 |    min |
|                    Flush time |              |   0.425417 |    0.456483 |  0.03107 |    min |
|           Merge throttle time |              |    0.93475 |    0.714817 | -0.21993 |    min |
|            Total Young Gen GC |              |      9.917 |       9.457 |    -0.46 |      s |
|              Total Old Gen GC |              |       0.34 |       0.602 |    0.262 |      s |
|                    Index size |              |    5.83426 |     5.90199 |  0.06773 |     GB |
|               Totally written |              |    16.4158 |     12.9643 | -3.45156 |     GB |
|        Heap used for segments |              |    20.6814 |      3.5807 | -17.1007 |     MB |
|      Heap used for doc values |              |  0.0629349 |   0.0243225 | -0.03861 |     MB |
|           Heap used for terms |              |    19.4335 |     3.35073 | -16.0828 |     MB |
|           Heap used for norms |              |    0.11969 |   0.0499878 |  -0.0697 |     MB |
|          Heap used for points |              |   0.277458 |   0.0367823 | -0.24068 |     MB |
|   Heap used for stored fields |              |   0.787804 |    0.118881 | -0.66892 |     MB |
|                 Segment count |              |        154 |          64 |      -90 |        |
|                Min Throughput | index-append |    71529.7 |       75340 |   3810.3 | docs/s |
|             Median Throughput | index-append |    72673.4 |     75821.7 |  3148.25 | docs/s |
|                Max Throughput | index-append |    73106.8 |     76198.4 |  3091.59 | docs/s |
|       50th percentile latency | index-append |     466.36 |     346.672 | -119.689 |     ms |
|       90th percentile latency | index-append |    771.469 |     704.963 | -66.5056 |     ms |
|       99th percentile latency | index-append |    1006.82 |     1755.92 |  749.099 |     ms |
|      100th percentile latency | index-append |    1081.72 |     1904.62 |    822.9 |     ms |
|  50th percentile service time | index-append |     466.36 |     346.672 | -119.689 |     ms |
|  90th percentile service time | index-append |    771.469 |     704.963 | -66.5056 |     ms |
|  99th percentile service time | index-append |    1006.82 |     1755.92 |  749.099 |     ms |
| 100th percentile service time | index-append |    1081.72 |     1904.62 |    822.9 |     ms |
|                    error rate | index-append |          0 |           0 |        0 |      % |

especially since it seems like the indices that are created by this are much more efficient (better compressed and less merges etc.)

jpountz · 2017-11-27T16:40:55Z

I like the reduced refresh time, total bytes written and segments memory usage!

bleskes

LGTM. Thanks @s1monw .

PS - did you purposely skip over #27500 (comment) ? I'm fine is so, but want to make sure it was not a mistake.

s1monw · 2017-11-27T17:09:40Z

PS - did you purposely skip over #27500 (comment) ? I'm fine is so, but want to make sure it was not a mistake.

I did, I think it's not easy to implement since I don' have a history in that method W.R.T the refresh interval.

* master: Skip shard refreshes if shard is `search idle` (#27500) Remove workaround in translog rest test (#27530) inner_hits: Return an empty _source for nested inner hit when filtering on a field that doesn't exist. percolator: Avoid TooManyClauses exception if number of terms / ranges is exactly equal to 1024 Dedup translog operations by reading in reverse (#27268) Ensure logging is configured for CLI commands Ensure `doc_stats` are changing even if refresh is disabled (#27505) Fix classes that can exit Revert "Adjust CombinedDeletionPolicy for multiple commits (#27456)" Transpose expected and actual, and remove duplicate info from message. (#27515) [DOCS] Fixed broken link in breaking changes

Once a shard goes inactive we want the shard to be refreshed if the refresh interval is default since we might hold on to unnecessary segments and in the inactive case we stopped indexing and can release old segments. Relates to elastic#27500

Once a shard goes inactive we want the shard to be refreshed if the refresh interval is default since we might hold on to unnecessary segments and in the inactive case we stopped indexing and can release old segments. Relates to #27500

…read The change in elastic#27500 introduces this regression that causes `_get` and `_term_vector` actions to run on the network thread if the realtime flag is set. This fixes the issue by delegating to the super method forking on the corresponding threadpool.

…read (#33814) The change in #27500 introduces this regression that causes `_get` and `_term_vector` actions to run on the network thread if the realtime flag is set. This fixes the issue by delegating to the super method forking on the corresponding threadpool.

With this change, we will always return true for can_match requests on idle search shards; otherwise, some shards will never get refreshed if all search requests perform the can_match phase (i.e., total shards > pre_filter_shard_size). Relates #27500 Relates #50043

With this change, we will always return true for can_match requests on idle search shards; otherwise, some shards will never get refreshed if all search requests perform the can_match phase (i.e., total shards > pre_filter_shard_size). Relates #27500 Relates #50043 Co-authored-by: Nhat Nguyen <[email protected]>

s1monw added :Core/Infra/Core Core issues without another label release highlight v7.0.0 labels Nov 22, 2017

s1monw requested review from jpountz and bleskes November 22, 2017 21:36

s1monw added 2 commits November 22, 2017 23:20

Merge branch 'master' into introduce_auto_refresh

a50cd19

remove unnecessary javadoc link

6e6edf2

jpountz approved these changes Nov 23, 2017

View reviewed changes

tlrx reviewed Nov 23, 2017

View reviewed changes

clintongormley added the >feature label Nov 23, 2017

s1monw added 16 commits November 23, 2017 14:21

Merge branch 'master' into introduce_auto_refresh

67eea42

add javadocs

52fc286

add integration test that checks that we are actually refreshing in t…

47dd97f

…he backgroud

ensure we refresh again after we switch the refresh interval

046c6be

remove unnecessary long markers

69583a0

add docs

c0e4176

fix imports

ca92314

improve docs

b2f77a4

more docs

20f5c21

Merge branch 'master' into introduce_auto_refresh

9a18fd2

bring back listeners check

bdf9a7e

Merge branch 'master' into introduce_auto_refresh

d8266d3

Merge branch 'master' into introduce_auto_refresh

e99a136

Merge branch 'master' into introduce_auto_refresh

9b84786

Merge branch 'master' into introduce_auto_refresh

dc52ff5

fix import after merge with master

88a8ec0

bleskes suggested changes Nov 27, 2017

View reviewed changes

s1monw added 2 commits November 27, 2017 14:46

ensure we set shard non idle on awaitPendingRefresh

1e1d739

apply feedback from @bleskes

dd97797

fix naming and more shard active out of acquireSearcher

cba052b

bleskes approved these changes Nov 27, 2017

View reviewed changes

s1monw merged commit f23ed61 into elastic:master Nov 27, 2017

s1monw mentioned this pull request Nov 28, 2017

Ensure shard is refreshed once it's inactive #27559

Merged

s1monw mentioned this pull request Sep 18, 2018

Ensure realtime _get and _termvectors don't run on the network thread #33814

Merged

jimczi added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

This was referenced Jun 28, 2019

[Docs] refresh_interval change made it unclear #43724

Closed

[DOC] Clarify refresh_interval new behavior #43726

Merged

jasontedor mentioned this pull request Sep 3, 2019

Multi-get requests should wait for search active #46283

Merged

dnhatn mentioned this pull request Apr 18, 2020

Return true for can_match on idle search shards #55428

Merged

nknize mentioned this pull request Nov 11, 2021

[Performance] Adaptive Replica Selection + Shard Idle introduces inconsistent search performance opensearch-project/OpenSearch#1536

Open

msfroh mentioned this pull request Sep 1, 2023

Increase index.search.idle.after setting default from 30s to 10 minutes (or more) opensearch-project/OpenSearch#9707

Open

arteam mentioned this pull request Jul 10, 2024

[test] Fix IndexShardTests#testScheduledRefresh #110312

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skip shard refreshes if shard is `search idle` #27500

Skip shard refreshes if shard is `search idle` #27500

s1monw commented Nov 22, 2017

jpountz left a comment

jpountz Nov 23, 2017

tlrx Nov 23, 2017

bleskes left a comment

bleskes Nov 27, 2017

bleskes Nov 27, 2017

bleskes Nov 27, 2017

bleskes Nov 27, 2017

bleskes Nov 27, 2017

s1monw Nov 27, 2017

bleskes Nov 27, 2017

bleskes Nov 27, 2017

s1monw Nov 27, 2017

bleskes Nov 27, 2017

bleskes Nov 27, 2017

s1monw Nov 27, 2017

bleskes Nov 27, 2017

bleskes Nov 27, 2017

s1monw Nov 27, 2017

bleskes Nov 27, 2017

bleskes Nov 27, 2017

s1monw commented Nov 27, 2017

jpountz commented Nov 27, 2017

bleskes left a comment

s1monw commented Nov 27, 2017

		@@ -97,6 +100,19 @@ protected void doExecute(Request request, ActionListener<Response> listener) {

		protected abstract Response shardOperation(Request request, ShardId shardId) throws IOException;

		protected void shardOperation(Request request, ShardId shardId, ActionListener<Response> listener) throws IOException {

Skip shard refreshes if shard is search idle #27500

Skip shard refreshes if shard is search idle #27500

Conversation

s1monw commented Nov 22, 2017

jpountz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bleskes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

s1monw commented Nov 27, 2017

jpountz commented Nov 27, 2017

bleskes left a comment

Choose a reason for hiding this comment

s1monw commented Nov 27, 2017

Skip shard refreshes if shard is `search idle` #27500

Skip shard refreshes if shard is `search idle` #27500