[BUG] org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT intermittent failure #2064

dblock · 2022-02-07T21:22:44Z

Describe the bug


> Task :plugins:repository-s3:yamlRestTest

REPRODUCE WITH: ./gradlew ':plugins:repository-s3:yamlRestTest' --tests "org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT.test {yaml=repository_s3/20_repository_permanent_credentials/Register a repository with a non existing client}" -Dtests.seed=25A4942AE204EF9D -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-BH -Dtests.timezone=America/La_Paz -Druntime.java=17 -Dtests.rest.blacklist=repository_s3/50_repository_ecs_credentials/*

org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT > test {yaml=repository_s3/20_repository_permanent_credentials/Register a repository with a non existing client} FAILED
    java.lang.RuntimeException: Failure at [repository_s3/20_repository_permanent_credentials:7]: 60,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
        at __randomizedtesting.SeedInfo.seed([25A4942AE204EF9D:ADF0ABF04CF88265]:0)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:444)
        at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.test(OpenSearchClientYamlSuiteTestCase.java:406)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:833)

        Caused by:
        java.net.SocketTimeoutException: 60,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
            at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:905)
            at org.opensearch.client.RestClient.performRequest(RestClient.java:307)
            at org.opensearch.client.RestClient.performRequest(RestClient.java:310)
            at org.opensearch.client.RestClient.performRequest(RestClient.java:295)
            at org.opensearch.test.rest.yaml.ClientYamlTestClient.callApi(ClientYamlTestClient.java:212)
            at org.opensearch.test.rest.yaml.ClientYamlTestExecutionContext.callApiInternal(ClientYamlTestExecutionContext.java:263)
            at org.opensearch.test.rest.yaml.ClientYamlTestExecutionContext.callApi(ClientYamlTestExecutionContext.java:130)
            at org.opensearch.test.rest.yaml.section.DoSection.execute(DoSection.java:306)
            at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:431)
            ... 40 more

            Caused by:
            java.net.SocketTimeoutException: 60,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
                at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
                at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
                at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
                at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
                at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
                at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
                at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
                at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
                at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
                at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
                ... 1 more

Coming from #1917 (comment)

The text was updated successfully, but these errors were encountered:

dblock · 2022-05-10T16:54:17Z

#3258 (review)

andrross · 2022-07-26T15:40:27Z

Another failure here: https://build.ci.opensearch.org/job/gradle-check/977/

"root_cause" : [
          {
            "type" : "snapshot_exception",
            "reason" : "[repository_permanent:snapshot-one/B67YkZ99R9uEZ5BpNEeGJA] failed to update snapshot in repository",
            "stack_trace" : "SnapshotException[[repository_permanent:snapshot-one/B67YkZ99R9uEZ5BpNEeGJA] failed to update snapshot in repository]; nested: IllegalStateException[Duplicate key docs (attempted merging values [docs/wOU9hO-vSwOVcRDbp9c4DA] and [docs/Anw_fnI_R56Tjjx1Lb9fOQ])];
	at org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$37(BlobStoreRepository.java:1332)
	at org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:88)
	at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:82)
	at org.opensearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:81)
	at org.opensearch.action.ActionRunnable$1.doRun(ActionRunnable.java:61)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:798)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.IllegalStateException: Duplicate key docs (attempted merging values [docs/wOU9hO-vSwOVcRDbp9c4DA] and [docs/Anw_fnI_R56Tjjx1Lb9fOQ])
	at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135)
	at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:182)
	at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
	at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1707)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at org.opensearch.repositories.RepositoryData.<init>(RepositoryData.java:144)
	at org.opensearch.repositories.RepositoryData.addSnapshot(RepositoryData.java:371)
	at org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$39(BlobStoreRepository.java:1354)
	at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80)
	... 7 more

tlfeng · 2022-08-03T20:39:23Z

Other failure with different error message.
1
#4115 (comment)
https://build.ci.opensearch.org/job/gradle-check/1383/

> Task :plugins:repository-s3:yamlRestTest

REPRODUCE WITH: ./gradlew ':plugins:repository-s3:yamlRestTest' --tests "org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT.test {yaml=repository_s3/20_repository_permanent_credentials/Delete a non existing snapshot}" -Dtests.seed=6AEAC5574EE779EE -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=pt-BR -Dtests.timezone=Africa/Nairobi -Druntime.java=17 -Dtests.rest.denylist=repository_s3/30_repository_temporary_credentials/*,repository_s3/40_repository_ec2_credentials/*,repository_s3/50_repository_ecs_credentials/*,repository_s3/60_repository_eks_credentials/*

org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT > test {yaml=repository_s3/20_repository_permanent_credentials/Delete a non existing snapshot} FAILED
    java.lang.AssertionError: Failure at [repository_s3/20_repository_permanent_credentials:22]: expected [2xx] status code but api [snapshot.delete] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"repository_exception","reason":"[repository_permanent] concurrent modification of the index-N file, expected current generation [11716] but it was not found in the repository","stack_trace":"RepositoryException[[repository_permanent] concurrent modification of the index-N file, expected current generation [11716] but it was not found in the repository]
org.opensearch.repositories.blobstore.BlobStoreRepository.ensureSafeGenerationExists(BlobStoreRepository.java:2199)
org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$57(BlobStoreRepository.java:2080)
...

2
#4122 (comment)
https://build.ci.opensearch.org/job/gradle-check/1416/

> Task :plugins:repository-s3:yamlRestTest

REPRODUCE WITH: ./gradlew ':plugins:repository-s3:yamlRestTest' --tests "org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT.test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials}" -Dtests.seed=D2A4C945B309FD46 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-LB -Dtests.timezone=Africa/Dakar -Druntime.java=17 -Dtests.rest.denylist=repository_s3/30_repository_temporary_credentials/*,repository_s3/40_repository_ec2_credentials/*,repository_s3/50_repository_ecs_credentials/*,repository_s3/60_repository_eks_credentials/*

org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT > test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials} FAILED
    java.lang.AssertionError: Failure at [repository_s3/20_repository_permanent_credentials:244]: expected [2xx] status code but api [snapshot.restore] returned [404 Not Found] [{"error":{"root_cause":[{"type":"snapshot_missing_exception","reason":"[repository_permanent:snapshot-one/8mXh1LBqTjSq-uZOhcgV4Q] is missing","stack_trace":"SnapshotMissingException[[repository_permanent:snapshot-one/8mXh1LBqTjSq-uZOhcgV4Q] is missing]; nested: NoSuchFileException[Blob object [****_integration_tests/snap-8mXh1LBqTjSq-uZOhcgV4Q.dat] not found: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: HAZNQYZ4400KNKKC; S3 Extended Request ID: Lghc26ALGZhmkwujL6PLA/xP9N9eoizpyqOOJodxe/xpnruMtTJomktTx+QxrjmbX7nF/iNghUM=; Proxy: null)];
org.opensearch.repositories.blobstore.BlobStoreRepository.getSnapshotInfo(BlobStoreRepository.java:1505)\n\tat org.opensearch.snapshots.RestoreService.lambda$restoreSnapshot$2(RestoreService.java:231)\n\tat 
...

3
#4123 (comment)
https://build.ci.opensearch.org/job/gradle-check/1417/

> Task :plugins:repository-s3:yamlRestTest

REPRODUCE WITH: ./gradlew ':plugins:repository-s3:yamlRestTest' --tests "org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT.test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials}" -Dtests.seed=A25F370B6CEC23C4 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-MX -Dtests.timezone=America/Iqaluit -Druntime.java=17 -Dtests.rest.denylist=repository_s3/30_repository_temporary_credentials/*,repository_s3/40_repository_ec2_credentials/*,repository_s3/50_repository_ecs_credentials/*,repository_s3/60_repository_eks_credentials/*

org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT > test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials} FAILED
    java.lang.AssertionError: Failure at [repository_s3/20_repository_permanent_credentials:152]: expected [2xx] status code but api [snapshot.create] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"invalid_snapshot_name_exception","reason":"[repository_permanent:snapshot-one] Invalid snapshot name [snapshot-one], snapshot with the same name already exists","stack_trace":"InvalidSnapshotNameException[[repository_permanent:snapshot-one] Invalid snapshot name [snapshot-one], snapshot with the same name already exists]
org.opensearch.snapshots.SnapshotsService.ensureSnapshotNameAvailableInRepo(SnapshotsService.java:689)org.opensearch.snapshots.SnapshotsService$2.execute(SnapshotsService.java:428)
...

mariavalavan · 2022-09-13T19:32:06Z

Looks like I am facing similar issue when using open-search-high-level-client java client from android.
Lot of retries happens, works some times. From Another standalone java seems working fine.
Is there any workaround for this one.

2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.684 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.684 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.684 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.684 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.684 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.684 9068-9334/io.onenav.ceshostapp I/System.out: 2022-09-13 12:00:06.684 9068-9334/io.onenav.ceshostapp I/System.out: 12:00:06.637 [IntentService[ServiceIntent]] ERROR - openearch read: java.io.IOException: Unable to decrypt incoming data due to unexpected end of stream
at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:912)
at org.opensearch.client.RestClient.performRequest(RestClient.java:301)
at org.opensearch.client.RestClient.performRequest(RestClient.java:289)
at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1762)
at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1728)
at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696)
at org.opensearch.client.RestHighLevelClient.searchTemplate(RestHighLevelClient.java:1306)
at ai.onenav.opensearch.OpenSearchAccess.searchTemplateWithRetry(OpenSearchAccess.java:404)
at ai.onenav.opensearch.OpenSearchAccess.getLatestMessageByTimeStampAndEpoch(OpenSearchAccess.java:598)
at ai.onenav.opensearch.AtomicData.getSearchHits(AtomicData.java:117)
at ai.onenav.opensearch.AtomicData.queryForData(AtomicData.java:156)
at ai.onenav.opensearch.BdsAtomicData.getSateliteData(BdsAtomicData.java:61)
at ai.onenav.rtcm.RtcmMsgSet.loadRtcmMsgCacheFromDb(RtcmMsgSet.java:753)
at ai.onenav.am.AMManager.loadRtcmMsgCacheFromDb(AMManager.java:441)
at ai.onenav.pm.WeightedLeastSquares.updateAssistReplayMode(WeightedLeastSquares.java:3844)
at ai.onenav.pm.WeightedLeastSquares.calculateUserPositionVelocityLeastSquare(WeightedLeastSquares.java:283)
at ai.onenav.pm.PMController.computePositionVelocitySolutionsFromRawMeas(PMController.java:644)
at ai.onenav.pm.PE_Fix.handleStrobeToFix(PE_Fix.java:1696)
at io.onenav.ceshostapp.Rinse.processMeasForNewFixAndPrePos(Rinse.java:5253)
at io.onenav.ceshostapp.MainLoop.mainLoop(MainLoop.java:713)
at io.onenav.ceshostapp.ServiceIntent.onHandleIntent(ServiceIntent.java:345)
at android.app.IntentService$ServiceHandler.handleMessage(IntentService.java:78)
at android.os.Handler.dispatchMessage(Handler.java:107)
at android.os.Looper.loop(Looper.java:214)
at android.os.HandlerThread.run(HandlerThread.java:67)
Caused by: javax.net.ssl.SSLException: Unable to decrypt incoming data due to unexpected end of stream
at org.apache.http.nio.reactor.ssl.SSLIOSession.decryptData(SSLIOSession.java:497)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:526)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.lang.Thread.run(Thread.java:919)

peternied · 2022-10-20T22:25:40Z

Another failure, https://build.ci.opensearch.org/job/gradle-check/5123/ impacting #4843

REPRODUCE WITH: ./gradlew ':plugins:repository-s3:yamlRestTest' --tests "org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT.test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials}" -Dtests.seed=5A6D28597C359FE7 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-DO -Dtests.timezone=Atlantic/Canary -Druntime.java=17 -Dtests.rest.denylist=repository_s3/30_repository_temporary_credentials/*,repository_s3/40_repository_ec2_credentials/*,repository_s3/50_repository_ecs_credentials/*,repository_s3/60_repository_eks_credentials/*

dblock · 2022-11-08T15:32:03Z

#5143

dblock · 2022-11-11T21:22:14Z

I am going to close this issue and open individual ones for separate RepositoryS3ClientYamlTestSuiteIT failures.

#5219

dblock added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run labels Feb 7, 2022

dblock mentioned this issue Feb 7, 2022

Add regexp interval source #1917

Merged

5 tasks

anasalkouz removed the untriaged label Feb 9, 2022

kartg mentioned this issue May 10, 2022

[Backport 2.x] [Javadocs] add to o.o.dfs,fetch,internal,lookup,profile, and query packages #3262

Merged

This was referenced May 10, 2022

[Backport 2.x] Bump reactor-netty from 1.0.17 to 1.0.18 in /plugins/repository-azure #3258

Merged

Used set to make shell scripts more strict #3278

Merged

mch2 mentioned this issue Aug 24, 2022

Segment Replication - Implement segment replication event cancellation. #4225

Merged

5 tasks

dblock mentioned this issue Oct 24, 2022

Add a link to the flakey test dev guide. #4872

Merged

dblock closed this as completed Nov 11, 2022

Poojita-Raj mentioned this issue Nov 15, 2022

[Meta] Fix random test failures #1715

Closed

37 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT intermittent failure #2064

[BUG] org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT intermittent failure #2064

dblock commented Feb 7, 2022

dblock commented May 10, 2022

andrross commented Jul 26, 2022

tlfeng commented Aug 3, 2022 •

edited

Loading

mariavalavan commented Sep 13, 2022

peternied commented Oct 20, 2022 •

edited

Loading

dblock commented Nov 8, 2022

dblock commented Nov 11, 2022

[BUG] org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT intermittent failure #2064

[BUG] org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT intermittent failure #2064

Comments

dblock commented Feb 7, 2022

dblock commented May 10, 2022

andrross commented Jul 26, 2022

tlfeng commented Aug 3, 2022 • edited Loading

mariavalavan commented Sep 13, 2022

peternied commented Oct 20, 2022 • edited Loading

dblock commented Nov 8, 2022

dblock commented Nov 11, 2022

tlfeng commented Aug 3, 2022 •

edited

Loading

peternied commented Oct 20, 2022 •

edited

Loading