CredentialStoreClient$loadCredentials$2$1.invoke java.lang.IllegalStateException - Already resumed

[androidcodehunter]

Before opening, please confirm:

• I have searched for duplicate or closed issues and discussions.

Language and Async Model

Kotlin

Amplify Categories

Authentication

Gradle script dependencies

com.amplifyframework:core-kotlin:2.4.0
com.amplifyframework:aws-auth-cognito:2.4.0

Environment information

Welcome to Gradle 7.5!

Here are the highlights of this release:
 - Support for Java 18
 - Support for building with Groovy 4
 - Much more responsive continuous builds
 - Improved diagnostics for dependency resolution

For more details see https://docs.gradle.org/7.5/release-notes.html


------------------------------------------------------------
Gradle 7.5
------------------------------------------------------------

Build time:   2022-07-14 12:48:15 UTC
Revision:     c7db7b958189ad2b0c1472b6fe663e6d654a5103

Kotlin:       1.6.21
Groovy:       3.0.10
Ant:          Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM:          17 (Oracle Corporation 17+35-2724)
OS:           Mac OS X 13.2.1 x86_64

Please include any relevant guides or documentation you're referencing

No response

Describe the bug

I am using Amplify Authentication library. I am trying to call this function multiple times in the background when it return false,

private suspend fun getAuthSessionAndSetAccessTokenWhenRefreshed(): AWSCognitoAuthSession? {
        return try {
            val authSession = Amplify.Auth.fetchAuthSession() as AWSCognitoAuthSession
            val idResult = authSession.identityIdResult
            if (idResult.type == AuthSessionResult.Type.SUCCESS) {
                authSession
            } else {
                if (idResult.error is SessionExpiredException) {
                    if (idResult.error?.cause is NotAuthorizedException) {
                        logout()
                        authSession
                    } else {
                      AuthFetchSessionOptions.builder().forceRefresh(true).build().let { option ->
                            (Amplify.Auth.fetchAuthSession(option) as AWSCognitoAuthSession).also { session ->
                                val newIdentityIdResult = session.identityIdResult
                                if (newIdentityIdResult.type == AuthSessionResult.Type.FAILURE && newIdentityIdResult.error?.cause is NotAuthorizedException) {
                                       logout()
                                } else {
                                    val tokenResult = session.userPoolTokensResult
                                    if (tokenResult.type == AuthSessionResult.Type.SUCCESS) {
                                        setAccessToken(tokenResult.value?.accessToken)
                                    }
                                }
                            }
                        }
                    }
                } else {
                    authSession
                }
            }
        } catch (exception: AuthException) {
            null
        }
    }

But sometimes I found there are some crash happening in the Amplify library. Here is the crash output,

Fatal Exception: java.lang.IllegalStateException: Already resumed
       at kotlin.coroutines.SafeContinuation.resumeWith(SafeContinuationJvm.kt:44)
       at com.amplifyframework.auth.cognito.CredentialStoreClient$loadCredentials$2$1.invoke(CredentialStoreClient.kt:97)
       at com.amplifyframework.auth.cognito.CredentialStoreClient$loadCredentials$2$1.invoke(CredentialStoreClient.kt:95)
       at com.amplifyframework.auth.cognito.CredentialStoreClient$listenForResult$1.invoke(CredentialStoreClient.kt:77)
       at com.amplifyframework.auth.cognito.CredentialStoreClient$listenForResult$1.invoke(CredentialStoreClient.kt:60)
       at com.amplifyframework.statemachine.StateMachine.notifySubscribers(StateMachine.kt:176)
       at com.amplifyframework.statemachine.StateMachine.process(StateMachine.kt:191)
       at com.amplifyframework.statemachine.StateMachine.access$process(StateMachine.kt:49)
       at com.amplifyframework.statemachine.StateMachine$send$1.invokeSuspend(StateMachine.kt:160)
       at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
       at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:463)
       at java.util.concurrent.FutureTask.run(FutureTask.java:264)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:307)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:637)
       at java.lang.Thread.run(Thread.java:1012)

I went to check the library code to see where this crash happening. I found it is happening in below code where continuation gets called multiple time,

    override suspend fun loadCredentials(credentialType: CredentialType): AmplifyCredential {
        return suspendCoroutine { continuation ->
            listenForResult(
                CredentialStoreEvent(CredentialStoreEvent.EventType.LoadCredentialStore(credentialType)),
                { continuation.resumeWith(it) },
                { continuation.resumeWithException(it) }
            )
        }
    }

Could you tell me what to do to prevent this error happening as it crashing my app.

Reproduction steps (if applicable)

No response

Code Snippet

// Put your code below this line.

Log output

// Put your logs below this line

amplifyconfiguration.json

No response

GraphQL Schema

// Put your schema below this line

Additional information and screenshots

No response

[androidcodehunter]

@eeatonaws any update as I am getting lots of this error for my production customer. I cannot reproduce it locally but my production user getting this error which I am getting from crashlytics.

[gpanshu]

Hi @androidcodehunter thank you for reporting this. I am actively looking into this and will respond with an answer/fix once I have had the chance to dig deep into why this is happening.

[gpanshu]

@androidcodehunter can you enable logging in the application class by adding the following line BEFORE your auth plugin addition and please paste your logs when the exception occurs. This will help us understand the flow from where the problem is potentially occurring from and fix it since we are unable to reproduce it locally.

Amplify.addPlugin(AndroidLoggingPlugin(LogLevel.VERBOSE))

[androidcodehunter]

@gpanshu The issue is I am not able to reproduce it locally. Only this is happening for production user. Could you suggest to add any specific log in my crash analytics library before the described code execution.

[minaairsupport]

I have the same issue and this happens in the production environment
in my case, I have an app that runs in the background collecting sleep information and for sure the app will not be connected to the internet as the customer is sleeping he turns the connection when he gets up.

in my case that's happen after 5-6 days the users start using the app

Non-fatal Exception: java.lang.IllegalStateException: Already resumed at kotlin.coroutines.SafeContinuation.resumeWith(SafeContinuationJvm.kt:44) at com.amplifyframework.auth.cognito.CredentialStoreClient$loadCredentials$2$1.invoke(CredentialStoreClient.kt:97) at com.amplifyframework.auth.cognito.CredentialStoreClient$loadCredentials$2$1.invoke(CredentialStoreClient.kt:95) at com.amplifyframework.auth.cognito.CredentialStoreClient$listenForResult$1.invoke(CredentialStoreClient.kt:77) at com.amplifyframework.auth.cognito.CredentialStoreClient$listenForResult$1.invoke(CredentialStoreClient.kt:60) at com.amplifyframework.statemachine.StateMachine$addSubscription$1.invokeSuspend(StateMachine.kt:141) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)

   please help that's important 

[androidcodehunter]

@gpanshu I am also collecting data from background worker. Very few user getting this issue. Could you help me what logs I could collect before this crash happening. I might add some breadcumb or logs.

[tylerjroach]

@androidcodehunter @minaairsupport
This is a crash that we have been unable to produce and attempted to add mitigations to prevent.
While we further investigate, can you please confirm the version of Amplify these crashes are happening on?

I see version 2.4.0 mentioned, but is it possible these crashes are being observed on devices running older versions of the application (possibly running older versions of Amplify?)

[androidcodehunter]

@tylerjroach Thank you for quick update. I am using amplify version 2.4.0 and here is the impacted devices and os informations,

Operating systems
70% Android 13
18% Android 11
12% Android 10

Device states
88% background

Devices
70% Samsung
18% Gotron - FIH Foxconn
12% BLU - Unisoc

Looking at the stats, it's looks like this issue mostly happening when user in background state.

[tylerjroach]

@androidcodehunter Had you previously deployed your app running Amplify v2.0.0? I understand that the current version of your app is running Amplify v2.4.0, but curious if that is also the version you are seeing in your crash reports, or if these reports could be coming from customers that may not have updated to newer versions of your application.

[androidcodehunter]

@tylerjroach yes I did deployed old amplify version as well but I think both versions have this same issue. This crash is happening to v2.4.0 as well. I updated the Amplify hoping for it would be fixed but I think the latest version have the same issue.

[androidcodehunter]

@tylerjroach @gpanshu Looking at the Amplify source code, couldn't we use suspendCancellableCoroutine instead of suspendCoroutine to prevent this crash,

    override suspend fun loadCredentials(credentialType: CredentialType): AmplifyCredential {
        return suspendCancellableCoroutine { continuation ->
            listenForResult(
                CredentialStoreEvent(CredentialStoreEvent.EventType.LoadCredentialStore(credentialType)),
                { if (continuation.isActive) continuation.resumeWith(it) },
                { continuation.resumeWithException(it) }
            )
        }
    }

I don't have much idea about the internal amplify code but with suspendCancellableCoroutine we could check if the continuation is active or not. Because with frequent callback, this function could call again until it finishes. As a result it would give crash.

[tylerjroach]

listenForResult should only ever call it's onSuccess or onError 1 single time. As a result, the continuation should only be called 1 time.

While we can further mitigate the continuation being fired twice, it would only be masking the actual problem. We're still looking at all paths to determine what might be causing the continuation to fire a second time.

[reisfernanda]

Hello,
This crash also happened with one single user of our app, running Android 10 on Redmi Note 8.
We're using version 2.7.0 of the SDK.
Crashlytics says the app was in the background and this crash event happened in the first 2 seconds of the user's session.

[tylerjroach]

@reisfernanda Thank you for your report, especially around the details of the crash happening within the first 2 seconds of the users session.

With the other reports happening while the user is in the background, I'm wondering if its possible these crashes are happening with the application is being recreated, running through the app's onCreate, and calling Amplify.configure. @minaairsupport I know you have mentioned the crash is being seen 5-6 days after launch, but is it possible the app is being re-launched in the background, rather than a single continuous session?

There is a but that I worked on in Amplify v2.8.x that seems unrelated at first glance, but may resolve this issue. #2402

Here, It is possible that fetchAuthSession could be called before the sdk has fully configured (part of the configuration happens asynchronously). This PR ensures that customer calls are delayed until Amplify Auth has fully configured.

There's a migration process to migrate credentials from v1 -> v2 of Amplify that happens upon each Amplify.configure call. If fetchAuthSession (or a similar call is made while configuration is in process), it may be possible that in rare scenarios, both processes hit our CredentialStore state machine at the same time, causing multiple fires into the continuation.

While we will continue investigating and searching for a root cause, it may be beneficial to upgrade to the latest version of Amplify to see if the crash was resolved with the linked bug fix.

[androidcodehunter]

@tylerjroach thank you for the update. When the app is in background, android OS could kill app and restart if there are any memory constraints. We don't know what actually happening in the user end but kill app could happen anytime.

I am going to update the library and will let you know the results.

[tylerjroach]

@androidcodehunter Thank you, will be keeping an eye on the thread for updates.

[minaairsupport]

@tylerjroach thanks for the reply I am using 2.5.0

I have Amplify.DataStore.save called in the background I wonder how it deals when the token expired is it refresh it or I should call fetchAuthSession to refresh the token if it's expired?

I have a part of my code
https://gist.github.com/minaairsupport/d2b0fab5cef37bd9b775572ad5c4df06

also will try upgrading the lib to the latest and see
Thanks again

[reisfernanda]

Hello,
I updated the library to version 2.8.4 and the crash is still happening.

[tylerjroach]

@reisfernanda Thank you for letting us know. We will continue looking to find the root cause.

[androidcodehunter]

@tylerjroach I have updated the library and rolled out the release. Looks like the issue still persists.

[jamesclement]

Recently updated to (an admittedly far from latest) v2.2.2 from 1.35.1, and have now also started seeing this issue. Similarly haven't managed to reproduce it locally yet.

[Gesraha101]

Hi,
We are also having the exact same issue. I released an update for the app a week ago with the updated amplify SDK (upgraded from v1.33.2 to v2.8.7).

Crashlytics didn't report that error till after the update. We have around 5k daily users and in the last 7 days, 84 users had this crash 95 times. Crashlytics also reported that 55% of them were in the background and 80% of those crashes happened in the first 3 seconds after app launch. Couldn't reproduce it locally as everyone unfortunately but just thought the extra input might be helpful.

[tylerjroach]

@Gesraha101 Thank you for the additional info. Any info helps here. If 80% are in the first 3 seconds after app launch, and 55% are in the background, I'm assuming you have some background process that are waking up the application?

[Gesraha101]

@tylerjroach you might be on to something. To answer your question, the app should not be doing anything in the background.

But you know, your comment just reminded me of a strange behavior that I ignored a while ago. I noticed that when I kill the app, it launches again in the background in the next 10 seconds or so and keeps running for about 5 mins. At first, I thought it might be something done in the background by amplify and that it is normal. However, seeing your observation made me take a second look by copying Amplify's module to a new project and checking the same behavior and surely enough, it didn't happen in the new project so that might explain why 55% of them crash in the background. Will probably need to look for any memory leaks or other libraries that cause that behavior.

That's one mystery down for me :)

[reisfernanda]

Hello @tylerjroach, do you have any update on this issue? Thank you!

[tylerjroach]

I've taken some time today to diagnose and should have a PR up today, and once merged, a fix should go out in the next release.

[reisfernanda]

Nice, that would help us a lot. Thank you.

[tylerjroach]

This bug should be fixed in the latest release (2.11.2). Thanks everyone for your reports!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.