Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IndexRecoveryIT.testRecoverLocallyUpToGlobalCheckpoint fails on master #75451

Closed
przemekwitek opened this issue Jul 19, 2021 · 1 comment · Fixed by #75898
Closed

IndexRecoveryIT.testRecoverLocallyUpToGlobalCheckpoint fails on master #75451

przemekwitek opened this issue Jul 19, 2021 · 1 comment · Fixed by #75898
Assignees
@fcofdez
Labels
:Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI v8.0.0-alpha1

Comments

@przemekwitek
Copy link
Contributor

Build scan:

[7.15.0] https://gradle-enterprise.elastic.co/s/ae6xz7qxyhfze
https://gradle-enterprise.elastic.co/s/vmpplhnu2xmyg

Repro line:

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.indices.recovery.IndexRecoveryIT.testRecoverLocallyUpToGlobalCheckpoint" \
  -Dtests.seed=70409EED32B9833E \
  -Dtests.locale=sv-SE \
  -Dtests.timezone=Africa/Mogadishu \
  -Druntime.java=16

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.indices.recovery.IndexRecoveryIT.testRecoverLocallyUpToGlobalCheckpoint" \
  -Dtests.seed=70409EED32B9833E \
  -Dtests.locale=sv-SE \
  -Dtests.timezone=Africa/Mogadishu \
  -Druntime.java=16

Reproduces locally?:

No

Applicable branches:

master

Failure history:

7th failure this year (~1 failure per month)

Failure excerpt:

java.lang.AssertionError: 
Expected: <16L>
     but: was <22L>
	at __randomizedtesting.SeedInfo.seed([70409EED32B9833E:8A1B13A925BC2776]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.indices.recovery.IndexRecoveryIT.testRecoverLocallyUpToGlobalCheckpoint(IndexRecoveryIT.java:1336)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
	at java.base/java.lang.Thread.run(Thread.java:831)
@przemekwitek przemekwitek added >test-failure Triaged test failures from CI v8.0.0 :Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. labels Jul 19, 2021
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Jul 19, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@fcofdez fcofdez self-assigned this Jul 21, 2021
fcofdez added a commit to fcofdez/elasticsearch that referenced this issue Jul 30, 2021
@fcofdez
Fix IndexRecoveryIT.testRecoverLocallyUpToGlobalCheckpoint
3f47852 
In certain scenarios the synced global checkpoint can fall behind
due to a slow disk. The test made the assumption that the global
checkpoint was stable when the global checkpoint was fetched,
this commit adds a new method that waits until the global checkpoint
is stable.

Closes elastic#75451
@fcofdez fcofdez mentioned this issue Jul 30, 2021
Merged
fcofdez added a commit that referenced this issue Aug 2, 2021
@fcofdez
Fix IndexRecoveryIT.testRecoverLocallyUpToGlobalCheckpoint (#75898)
6099d68 
In certain scenarios the synced global checkpoint can fall behind
due to a slow disk. The test made the assumption that the global
checkpoint was stable when the global checkpoint was fetched,
this commit adds a new method that waits until the global checkpoint
is stable.

Closes #75451
fcofdez added a commit to fcofdez/elasticsearch that referenced this issue Aug 2, 2021
@fcofdez
Fix IndexRecoveryIT.testRecoverLocallyUpToGlobalCheckpoint
6b8bf22 
In certain scenarios the synced global checkpoint can fall behind
due to a slow disk. The test made the assumption that the global
checkpoint was stable when the global checkpoint was fetched,
this commit adds a new method that waits until the global checkpoint
is stable.

Closes elastic#75451
Backport of elastic#75898
@fcofdez fcofdez mentioned this issue Aug 2, 2021
Merged
fcofdez added a commit that referenced this issue Aug 2, 2021
@fcofdez
Fix IndexRecoveryIT.testRecoverLocallyUpToGlobalCheckpoint (#75921)
24c36f0 
In certain scenarios the synced global checkpoint can fall behind
due to a slow disk. The test made the assumption that the global
checkpoint was stable when the global checkpoint was fetched,
this commit adds a new method that waits until the global checkpoint
is stable.

Closes #75451
Backport of #75898
@bpintea bpintea mentioned this issue Oct 27, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fcofdez fcofdez

Labels
:Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI v8.0.0-alpha1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix IndexRecoveryIT.testRecoverLocallyUpToGlobalCheckpoint fcofdez/elasticsearch
4 participants
@fcofdez @mark-vieira @elasticmachine @przemekwitek