[BUG] org.opensearch.remotestore.RemoteStoreClusterStateRestoreIT.testFullClusterRestoreGlobalMetadata is flaky #10749

reta · 2023-10-19T16:51:32Z

Describe the bug
The test case org.opensearch.remotestore.RemoteStoreClusterStateRestoreIT.testFullClusterRestoreGlobalMetadata is flaky:

java.lang.AssertionError: timed out waiting for green state

java.lang.AssertionError: timed out waiting for green state
	at org.junit.Assert.fail(Assert.java:89)
	at org.opensearch.test.OpenSearchIntegTestCase.ensureColor(OpenSearchIntegTestCase.java:1012)
	at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:943)
	at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:932)
	at org.opensearch.remotestore.BaseRemoteStoreRestoreIT.verifyRestoredData(BaseRemoteStoreRestoreIT.java:63)
	at org.opensearch.remotestore.RemoteStoreClusterStateRestoreIT.testFullClusterRestoreGlobalMetadata(RemoteStoreClusterStateRestoreIT.java:249)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1623)

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.RemoteStoreClusterStateRestoreIT.testFullClusterRestoreGlobalMetadata" -Dtests.seed=4FA936F087F8BF6B

Expected behavior
The test must always pass

Plugins
Standard

Screenshots
N/A

Host/Environment (please complete the following information):

CI

Additional context

https://build.ci.opensearch.org/job/gradle-check/28446/testReport/junit/org.opensearch.remotestore/RemoteStoreClusterStateRestoreIT/testFullClusterRestoreGlobalMetadata/

The text was updated successfully, but these errors were encountered:

linuxpi · 2023-10-30T08:49:59Z

We have fixed all flaky tests in org.opensearch.remotestore.RemoteStoreClusterStateRestoreIT. Closing, please reopen if you see any failures

ketanv3 · 2023-11-01T05:40:54Z

@linuxpi Slightly different occurrence: https://build.ci.opensearch.org/job/gradle-check/29421/testReport/junit/org.opensearch.remotestore/RemoteStoreClusterStateRestoreIT/testFullClusterRestoreGlobalMetadata/

reta · 2023-11-06T16:25:10Z

And again here https://build.ci.opensearch.org/job/gradle-check/29586/testReport/junit/org.opensearch.remotestore/RemoteStoreClusterStateRestoreIT/testFullClusterRestoreGlobalMetadata/

java.lang.IllegalStateException: Error while downloading cluster metadata - manifest__9223372036854775806__9223372036854775776__P__9223370337568370205__1

mch2 · 2023-11-06T18:19:04Z

Got a failure in this test class on [testFullClusterRestoreMultipleIndices](https://build.ci.opensearch.org/job/gradle-check/29562/testReport/junit/org.opensearch.remotestore/RemoteStoreClusterStateRestoreIT/testFullClusterRestoreMultipleIndices/) - https://build.ci.opensearch.org/job/gradle-check/29562/

peternied · 2023-11-07T23:22:22Z

Another failure, impacted #11102, logs.

bowenlan-amzn · 2024-02-10T01:13:46Z

saw this test failed again https://build.ci.opensearch.org/job/gradle-check/33624/testReport/
@linuxpi could you maybe take a look

notice your fix is not on 2.12, backporting #12282

peternied · 2024-02-10T17:46:10Z

@msfroh could you close when this has been backported to 2.12/2.x?

bowenlan-amzn · 2024-02-12T01:36:24Z

Close as the backport 2.12 PR has been merged

reta added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run and removed untriaged labels Oct 19, 2023

github-actions bot added the untriaged label Oct 19, 2023

reta mentioned this issue Oct 19, 2023

[Streaming Indexing] Introduce new experimental HTTP transport based on Netty 4 and Project Reactor (Reactor Netty) #9672

Merged

6 tasks

reta added bug Something isn't working and removed bug Something isn't working untriaged labels Oct 19, 2023

reta mentioned this issue Oct 19, 2023

Update Github pull request template to have a task for inspecting failing checks #10751

Merged

8 tasks

ticheng-aws mentioned this issue Oct 19, 2023

Introduce ConcurrentQueryProfiler to profile query using concurrent segment search path and support concurrency during rewrite and create weight #10352

Merged

7 tasks

ashking94 mentioned this issue Oct 20, 2023

[Backport 2.x] [Remote Store] Fix relocation failure due to transport receive timeout #10788

Merged

reta mentioned this issue Oct 20, 2023

[Backport 2.x] Update Github pull request template to have a task for inspecting failing checks #10792

Merged

RamakrishnaChilaka assigned linuxpi Oct 24, 2023

linuxpi closed this as completed Oct 30, 2023

reta reopened this Nov 6, 2023

github-actions bot added the untriaged label Nov 6, 2023

reta mentioned this issue Nov 6, 2023

[AUTOCUT] Gradle Check Failure on push to main #11107

Closed

mch2 mentioned this issue Nov 6, 2023

[Backport 2.x] Fix flaky pit/scroll tests in SegmentReplicationIT #11098

Merged

ticheng-aws mentioned this issue Nov 7, 2023

[AUTOCUT] Gradle Check Failure on push to main #11124

Closed

peternied mentioned this issue Nov 7, 2023

[AUTOCUT] Gradle Check Failure on push to 2.x #11102

Closed

reta mentioned this issue Nov 16, 2023

[AUTOCUT] Gradle Check Failure on push to main #11232

Closed

gaobinlong mentioned this issue Nov 20, 2023

Disallow removing some metadata fields by remove ingest processor #10895

Merged

8 tasks

peternied removed the untriaged label Nov 30, 2023

peternied mentioned this issue Dec 15, 2023

[AUTOCUT] Gradle Check Failure on push to 2.x #11624

Closed

reta mentioned this issue Dec 18, 2023

Bump com.squareup.okio:okio from 3.6.0 to 3.7.0 in /test/fixtures/hdfs-fixture #11632

Merged

8 tasks

This was referenced Jan 2, 2024

Bump com.avast.gradle:gradle-docker-compose-plugin from 0.17.5 to 0.17.6 #11692

Merged

[AUTOCUT] Gradle Check Failure on push to 2.x #11709

Closed

mch2 added Cluster Manager Storage:Remote labels Jan 10, 2024

mch2 mentioned this issue Jan 10, 2024

Bump 2.12.0 to Lucene 9.9.1 in Version.java #11837

Merged

8 tasks

reta mentioned this issue Jan 15, 2024

Extract cluster management for integration tests into JUnit test rule out of OpenSearchIntegTestCase #11877

Merged

8 tasks

linuxpi mentioned this issue Jan 23, 2024

fix flaky test - handle IllegalStateException and throw AssertionError to use assertbusy #11981

Merged

8 tasks

sachinpkale closed this as completed in #11981 Jan 31, 2024

bowenlan-amzn mentioned this issue Feb 10, 2024

[Backport 2.12] [Backport 2.x] Apply the fast filter optimization to composite aggregation #12233

Closed

peternied reopened this Feb 10, 2024

github-actions bot added the untriaged label Feb 10, 2024

bowenlan-amzn closed this as completed Feb 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] org.opensearch.remotestore.RemoteStoreClusterStateRestoreIT.testFullClusterRestoreGlobalMetadata is flaky #10749

[BUG] org.opensearch.remotestore.RemoteStoreClusterStateRestoreIT.testFullClusterRestoreGlobalMetadata is flaky #10749

reta commented Oct 19, 2023

linuxpi commented Oct 30, 2023

ketanv3 commented Nov 1, 2023

reta commented Nov 6, 2023

mch2 commented Nov 6, 2023

peternied commented Nov 7, 2023

bowenlan-amzn commented Feb 10, 2024 •

edited

Loading

peternied commented Feb 10, 2024

bowenlan-amzn commented Feb 12, 2024

[BUG] org.opensearch.remotestore.RemoteStoreClusterStateRestoreIT.testFullClusterRestoreGlobalMetadata is flaky #10749

[BUG] org.opensearch.remotestore.RemoteStoreClusterStateRestoreIT.testFullClusterRestoreGlobalMetadata is flaky #10749

Comments

reta commented Oct 19, 2023

linuxpi commented Oct 30, 2023

ketanv3 commented Nov 1, 2023

reta commented Nov 6, 2023

mch2 commented Nov 6, 2023

peternied commented Nov 7, 2023

bowenlan-amzn commented Feb 10, 2024 • edited Loading

peternied commented Feb 10, 2024

bowenlan-amzn commented Feb 12, 2024

bowenlan-amzn commented Feb 10, 2024 •

edited

Loading