[test] Fix IndexShardTests#testScheduledRefresh #110312

arteam · 2024-06-30T18:06:48Z

After we flushed the shard, we only make sure that the refresh call is propagated to the shard engine, but we can't be sure that the call is actually ends up in a shard refresh. The call in InternalEngine#refresh can return false if we couldn't acquire the lock on ElasticsearchDirectoryReader, because it's already being refreshed.

We can wrap the call in assertBusy to retry it in order to make sure that the shard eventually gets refreshed.

Resolves #101008

After we flushed the shard, we only make sure that the refresh call is propagated to the shard engine, but we can't be sure that the call is actually ends up in a shard refresh. The call in `InternalEngine#refresh` can return `false` if we couldn't acquire the lock on `ElasticsearchDirectoryReader`, because it's already being refreshed. We can wrap the call in `assertBusy` to retry it in order to make sure that the shard eventually gets refreshed. Resolves #101008

…fresh

elasticsearchmachine · 2024-07-01T06:20:16Z

Pinging @elastic/es-distributed (Team:Distributed)

fcofdez · 2024-07-02T15:39:50Z

The call in InternalEngine#refresh can return false if we couldn't acquire the lock on ElasticsearchDirectoryReader, because it's already being refreshed.

But this test concerns the IndexShard alone, meaning that there shouldn't be any background threads trying to refresh the shard (i.e. scheduled refreshes). Could you elaborate on which thread is calling refresh concurrently?

server/src/test/java/org/elasticsearch/index/shard/IndexShardTests.java

…fresh

server/src/test/java/org/elasticsearch/index/shard/IndexShardTests.java

server/src/main/java/org/elasticsearch/index/shard/IndexShard.java

This reverts commit a37a174.

server/src/test/java/org/elasticsearch/index/shard/IndexShardTests.java

kingherc · 2024-07-03T13:26:31Z

server/src/test/java/org/elasticsearch/index/shard/IndexShardTests.java

        logger.info("--> scheduledRefresh(future5)");
+        ensureNoPendingScheduledRefresh();


If there's some spurious refresh being run, we cannot be sure it is at this exact spot.

Maybe a better approach would be to get the number of external refreshes before the future5, and asserting that after the scheduled refresh, it's incremented by 1.

@kingherc That's a great idea. I've replaced the hack with blocking the refresh thread pool with checking the refresh stats.

I would like to understand why the test was failing, in the logs of the test failures you can see that the flush is executed after the assertion trips, so I'm not convinced about the flush being the issue here

server/src/test/java/org/elasticsearch/index/shard/IndexShardTests.java

arteam · 2024-07-09T09:38:07Z

@elasticmachine update branch

arteam · 2024-07-16T07:30:50Z

@elasticmachine update branch

fcofdez

LGTM

kingherc

LGTM. Please handle the two comments I mentioned before merging.

kingherc · 2024-07-16T08:06:54Z

server/src/test/java/org/elasticsearch/index/shard/IndexShardTests.java

-        primary.scheduledRefresh(future5);
-        assertTrue(future5.actionGet()); // make sure we refresh once the shard is inactive
+        primary.scheduledRefresh(ActionListener.noop());
+        // We can't check whether scheduledRefresh returns true because it races with a potential


Since @fcofdez approved, I am also fine with the current state and we see if in the future there's any issues with any other concurrent refreshes going on.

I believe this comment may not be up to date now. Since above we assertBusy that the flush has happened, probably the scheduled refresh here will also be true. I'd just remove the comment to avoid confusion.

kingherc · 2024-07-16T08:07:29Z

server/src/test/java/org/elasticsearch/index/shard/IndexShardTests.java

@@ -3925,11 +3924,16 @@ public void testScheduledRefresh() throws Exception {
        logger.info("--> ensure search idle");
        assertTrue(primary.isSearchIdle());
        assertTrue(primary.searchIdleTime() >= TimeValue.ZERO.millis());
+        long periodicFlushesBefore = primary.flushStats().getPeriodic();


Also can you change the comment above
while shard is search active and ensure scheduleRefresh(...) makes documen visible:
?
because the shard was search idle and that's why scheduled refresh is false there.

arteam · 2024-07-16T09:01:29Z

@kingherc Sorry about the auto merge, I will address your comments in a follow up PR!

* First scheduledRefresh returns false because search is idle * Remove the comment about the inability to control the result of scheduleRefresh Follow-up for elastic#110312

* First scheduledRefresh returns false because search is idle * Remove the comment about the inability to control the result of scheduleRefresh Follow-up for #110312

* First scheduledRefresh returns false because search is idle * Remove the comment about the inability to control the result of scheduleRefresh Follow-up for elastic#110312

arteam added :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. >test Issues or PRs that are addressing/adding tests labels Jun 30, 2024

elasticsearchmachine added the v8.15.0 label Jun 30, 2024

arteam added 2 commits June 30, 2024 22:24

Merge remote-tracking branch 'origin/main' into fix-test-scheduled-re…

9ed2634

…fresh

Run checkstyle

45e9ddb

arteam marked this pull request as ready for review July 1, 2024 06:19

arteam requested a review from fcofdez July 1, 2024 06:20

elasticsearchmachine added the Team:Distributed Meta label for distributed team (obsolete) label Jul 1, 2024

arteam requested review from kingherc and tlrx July 1, 2024 11:02

kingherc reviewed Jul 2, 2024

View reviewed changes

server/src/test/java/org/elasticsearch/index/shard/IndexShardTests.java Show resolved Hide resolved

arteam added 2 commits July 3, 2024 11:05

Merge remote-tracking branch 'origin/main' into fix-test-scheduled-re…

d3781ab

…fresh

Add ability to distinguish whether we called refresh

a37a174

kingherc reviewed Jul 3, 2024

View reviewed changes

server/src/test/java/org/elasticsearch/index/shard/IndexShardTests.java Outdated Show resolved Hide resolved

server/src/main/java/org/elasticsearch/index/shard/IndexShard.java Outdated Show resolved Hide resolved

Revert "Add ability to distinguish whether we called refresh"

c57ce1a

This reverts commit a37a174.

arteam requested a review from kingherc July 3, 2024 10:25

arteam added 2 commits July 3, 2024 14:28

Make sure the flush completes and there are no pending refresh

5afadce

Use Phaser

26be0a1

arteam requested review from kingherc and removed request for kingherc July 3, 2024 12:34

Run spotless

1921630

kingherc reviewed Jul 3, 2024

View reviewed changes

arteam added 2 commits July 4, 2024 08:59

Use CyclingBarrier instead of Phaser

0720dee

Check refreshStats that one of the refhreses wins

759c80a

arteam requested a review from kingherc July 4, 2024 07:20

elasticsearchmachine added v8.16.0 and removed v8.15.0 labels Jul 4, 2024

kingherc reviewed Jul 4, 2024

View reviewed changes

server/src/test/java/org/elasticsearch/index/shard/IndexShardTests.java Show resolved Hide resolved

Merge branch 'main' into fix-test-scheduled-refresh

8793296

arteam requested review from fcofdez and kingherc and removed request for fcofdez July 16, 2024 07:30

Merge branch 'main' into fix-test-scheduled-refresh

8829cba

fcofdez approved these changes Jul 16, 2024

View reviewed changes

arteam added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Jul 16, 2024

kingherc approved these changes Jul 16, 2024

View reviewed changes

elasticsearchmachine merged commit 199910c into main Jul 16, 2024
16 checks passed

elasticsearchmachine deleted the fix-test-scheduled-refresh branch July 16, 2024 08:25

arteam mentioned this pull request Jul 16, 2024

Update comments in IndexShardTests#testScheduledRefresh #110943

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[test] Fix IndexShardTests#testScheduledRefresh #110312

[test] Fix IndexShardTests#testScheduledRefresh #110312

arteam commented Jun 30, 2024 •

edited

Loading

elasticsearchmachine commented Jul 1, 2024

fcofdez commented Jul 2, 2024

kingherc Jul 3, 2024

arteam Jul 4, 2024

fcofdez Jul 4, 2024

arteam commented Jul 9, 2024

arteam commented Jul 16, 2024

fcofdez left a comment

kingherc left a comment

kingherc Jul 16, 2024

kingherc Jul 16, 2024

arteam commented Jul 16, 2024

		logger.info("--> scheduledRefresh(future5)");
		ensureNoPendingScheduledRefresh();

[test] Fix IndexShardTests#testScheduledRefresh #110312

[test] Fix IndexShardTests#testScheduledRefresh #110312

Conversation

arteam commented Jun 30, 2024 • edited Loading

elasticsearchmachine commented Jul 1, 2024

fcofdez commented Jul 2, 2024

kingherc Jul 3, 2024

Choose a reason for hiding this comment

arteam Jul 4, 2024

Choose a reason for hiding this comment

fcofdez Jul 4, 2024

Choose a reason for hiding this comment

arteam commented Jul 9, 2024

arteam commented Jul 16, 2024

fcofdez left a comment

Choose a reason for hiding this comment

kingherc left a comment

Choose a reason for hiding this comment

kingherc Jul 16, 2024

Choose a reason for hiding this comment

kingherc Jul 16, 2024

Choose a reason for hiding this comment

arteam commented Jul 16, 2024

arteam commented Jun 30, 2024 •

edited

Loading