Flaky-test: ExtensibleLoadManagerImplTest

[lhotari]

Search before asking

• I searched in the issues and found nothing similar.

Example failure

https://github.com/apache/pulsar/actions/runs/4604351215/jobs/8135398801?pr=20005#step:11:1081

Exception stacktrace

Error:  Tests run: 91, Failures: 2, Errors: 0, Skipped: 89, Time elapsed: 46.155 s <<< FAILURE! - in org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest
  Error:  stateCheck(org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest)  Time elapsed: 35.148 s  <<< FAILURE!
  java.lang.RuntimeException: org.apache.pulsar.metadata.api.MetadataStoreException$AlreadyClosedException: The metadata store is closed
  	at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.lambda$createPulsarTestContextBuilder$0(MockedPulsarServiceBaseTest.java:372)
  	at org.apache.pulsar.broker.testcontext.PulsarTestContext$AbstractCustomBuilder.build(PulsarTestContext.java:526)
  	at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.createMainPulsarTestContext(MockedPulsarServiceBaseTest.java:341)
  	at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.startBroker(MockedPulsarServiceBaseTest.java:293)
  	at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.init(MockedPulsarServiceBaseTest.java:213)
  	at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.internalSetup(MockedPulsarServiceBaseTest.java:133)
  	at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.internalSetup(MockedPulsarServiceBaseTest.java:150)
  	at org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.setup(ExtensibleLoadManagerImplTest.java:142)
  	at org.apache.pulsar.tests.TestRetrySupport.stateCheck(TestRetrySupport.java:60)
  	at jdk.internal.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
  	at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
  	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:590)
  	at org.testng.internal.invokers.TestInvoker.retryFailed(TestInvoker.java:263)
  	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:62)
  	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
  	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
  	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
  	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
  	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
  	at org.testng.TestRunner.privateRun(TestRunner.java:829)
  	at org.testng.TestRunner.run(TestRunner.java:602)
  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
  	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
  	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
  	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
  	at org.testng.TestNG.runSuites(TestNG.java:[1099](https://github.com/apache/pulsar/actions/runs/4604351215/jobs/8135398801?pr=20005#step:11:1100))
  	at org.testng.TestNG.run(TestNG.java:1067)
  	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:123)
  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:90)
  	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
  	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
  	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
  	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
  	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
  Caused by: org.apache.pulsar.metadata.api.MetadataStoreException$AlreadyClosedException: The metadata store is closed
  	at org.apache.pulsar.metadata.impl.AbstractMetadataStore.exists(AbstractMetadataStore.java:289)
  	at org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl.exists(MetadataCacheImpl.java:253)
  	at org.apache.pulsar.broker.resources.BaseResources.exists(BaseResources.java:184)
  	at org.apache.pulsar.broker.resources.TenantResources.tenantExists(TenantResources.java:58)
  	at org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.createNamespaceIfNotExists(ExtensibleLoadManagerImplTest.java:183)
  	at org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.beforePulsarStart(ExtensibleLoadManagerImplTest.java:173)
  	at org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest.lambda$createPulsarTestContextBuilder$0(MockedPulsarServiceBaseTest.java:370)
  	... 45 more
  
  Error:  testAssign(org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest)  Time elapsed: 0.014 s  <<< FAILURE!
  java.lang.NullPointerException: Cannot invoke "org.apache.pulsar.broker.PulsarService.getNamespaceService()" because "pulsar" is null
  	at org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.getBundleAsync(ExtensibleLoadManagerImplTest.java:852)
  	at org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.testAssign(ExtensibleLoadManagerImplTest.java:234)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
  	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:[1136](https://github.com/apache/pulsar/actions/runs/4604351215/jobs/8135398801?pr=20005#step:11:1137))
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  	at java.base/java.lang.Thread.run(Thread.java:833)

Are you willing to submit a PR?

• I'm willing to submit a PR!

[heesung-sn]

Expecting this change will fix this flaky test.

f129646

[Demogorgon314]

Fixed by #19945

[lhotari]

It would be useful to quarantine the flaky ExtensibleLoadManagerImplTest in the flaky or quarantine group since it's really a waste of time when a lot of builds fail. Once it's stable it could be moved to the tests that block merging PRs. @heesung-sn @Demogorgon314 would that make sense?

[heesung-sn]

The fix pr #19945 has been merged. I dont expect these tests to be flaky, moving forward. As we are very active working on this component, we will fix any future issues quickly.

[heesung-sn]

For sure, if this test continues to be flaky, I am okay with quarantining it.

[lhotari]

For sure, if this test continues to be flaky, I am okay with quarantining it.

@heesung-sn Great! Thanks for addressing the immediate issue.