DataStore sync issue from offline saved models (custom type fields) #2710

999Rub · 2023-02-28T16:10:04Z

Description

This bug report is really close to aws-amplify/amplify-android#1199 from the amplify-android.

I've got a GraphQL model (wich is provided in schema section) that I can save offline or online. When I use the app online, absolutely everything works fine.

When I launch the app online, then turn off the internet connectivity, create and save a new model, it is locally saved properly. Then I end up the program (still offline), turn back the internet and run the program again (online).
The sync is processing, I can retrieve the data from my cloud, but the model that was saved in offline mode is not synced with the cloud.

It actually gives me a mix of DataStore and Amplify exception :

Regarding the Android issue mentioned above, I think my issue is really close to the same fix but I can't how to overpass it.

The thing that is really curious is, when I run the program, then turn off internet, create and save a model, and turn on internet (without closing the program this time), the model is correctly synced with my cloud.

I use the Auto Conflict resolver.

My Flutter doctor is fine.

If you need more details tell my and I will give you as much as I can.

Steps to Reproduce

Use a template wich implement Datastore and AppSync with GraphQL API
Create a model that has custom type fields (my custom type has only int and string fields)
Run the program online
Turn off internet
Create and save an object with the model you previously created
Close the program (offline)
Turn the internet back
Run the program
Here it is

Screenshots

No response

Platforms

Flutter Version

3.7.3

Amplify Flutter Version

10.7.3

Deployment Method

Amplify CLI

Schema

# Surgery model is the one where the exception comes from (every custom fields are mentioned in the exception, I've tried to delete them one by one and this is not one field especially).

type Surgery @model @auth(rules: [{allow: owner}]) {
  id: ID!
  localAnesthesia: [LocalAnesthesia]!
  procedure: String
  physicalExamination: PhysicalExamination!
  checkList: CheckList!
  charts: Chart!
}

# The Chart model is an exemple of my custom type field (they all use basics types like int or string)

type Chart {
  timer: Int!
  lowChart: String!
  highChart: String!
}

HuiSF · 2023-02-28T23:39:09Z

Hi @999Rub did the console log the stack trace when this error happens? Could you share more logs? I'd like to identity where the exception originated in the code and further to know where to look.

999Rub · 2023-03-01T08:18:22Z

Here's all I've got:

W/amplify:aws-datastore(17941): DataStoreException{message=Error encountered while creating model schema, cause=DataStoreException{message=Failed to get fields for model., cause=AmplifyException{message=An invalid CustomType field was provided. charts must be an instance of SerializedCustomType or a List of instances of SerializedCustomType, cause=null, recoverySuggestion=Check if this model schema is a correct representation of the fields in the provided Object}, recoverySuggestion=Validate your model file.}, recoverySuggestion=See attached exception for more details}
W/amplify:aws-datastore(17941): 	at com.amplifyframework.datastore.appsync.AppSyncClient.create(AppSyncClient.java:147)
W/amplify:aws-datastore(17941): 	at com.amplifyframework.datastore.syncengine.MutationProcessor.lambda$create$13$com-amplifyframework-datastore-syncengine-MutationProcessor(MutationProcessor.java:291)
W/amplify:aws-datastore(17941): 	at com.amplifyframework.datastore.syncengine.MutationProcessor$$ExternalSyntheticLambda3.publish(Unknown Source:4)
W/amplify:aws-datastore(17941): 	at com.amplifyframework.datastore.syncengine.MutationProcessor.lambda$publishWithStrategy$17(MutationProcessor.java:322)
W/amplify:aws-datastore(17941): 	at com.amplifyframework.datastore.syncengine.MutationProcessor$$ExternalSyntheticLambda13.subscribe(Unknown Source:4)
W/amplify:aws-datastore(17941): 	at io.reactivex.rxjava3.internal.operators.single.SingleCreate.subscribeActual(SingleCreate.java:40)
W/amplify:aws-datastore(17941): 	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
W/amplify:aws-datastore(17941): 	at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap.subscribeActual(SingleFlatMap.java:37)
W/amplify:aws-datastore(17941): 	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
W/amplify:aws-datastore(17941): 	at io.reactivex.rxjava3.internal.operators.single.SingleDelayWithObservable$OtherSubscriber.onComplete(SingleDelayWithObservable.java:87)
W/amplify:aws-datastore(17941): 	at io.reactivex.rxjava3.internal.operators.single.SingleDelayWithObservable$OtherSubscriber.onNext(SingleDelayWithObservable.java:68)
W/amplify:aws-datastore(17941): 	at io.reactivex.rxjava3.internal.operators.observable.ObservableTimer$TimerObserver.run(ObservableTimer.java:67)
W/amplify:aws-datastore(17941): 	at io.reactivex.rxjava3.internal.schedulers.ScheduledDirectTask.call(ScheduledDirectTask.java:41)
W/amplify:aws-datastore(17941): 	at io.reactivex.rxjava3.internal.schedulers.ScheduledDirectTask.call(ScheduledDirectTask.java:28)
W/amplify:aws-datastore(17941): 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
W/amplify:aws-datastore(17941): 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
W/amplify:aws-datastore(17941): 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W/amplify:aws-datastore(17941): 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
W/amplify:aws-datastore(17941): 	at java.lang.Thread.run(Thread.java:923)
W/amplify:aws-datastore(17941): Caused by: DataStoreException{message=Failed to get fields for model., cause=AmplifyException{message=An invalid CustomType field was provided. charts must be an instance of SerializedCustomType or a List of instances of SerializedCustomType, cause=null, recoverySuggestion=Check if this model schema is a correct representation of the fields in the provided Object}, recoverySuggestion=Validate your model file.}
W/amplify:aws-datastore(17941): 	at com.amplifyframework.datastore.appsync.AppSyncRequestFactory.buildCreationRequest(AppSyncRequestFactory.java:222)
W/amplify:aws-datastore(17941): 	at com.amplifyframework.datastore.appsync.AppSyncClient.create(AppSyncClient.java:144)
W/amplify:aws-datastore(17941): 	... 18 more
W/amplify:aws-datastore(17941): Caused by: AmplifyException{message=An invalid CustomType field was provided. charts must be an instance of SerializedCustomType or a List of instances of SerializedCustomType, cause=null, recoverySuggestion=Check if this model schema is a correct representation of the fields in the provided Object}
W/amplify:aws-datastore(17941): 	at com.amplifyframework.datastore.appsync.AppSyncRequestFactory.extractCustomTypeFieldValue(AppSyncRequestFactory.java:547)
W/amplify:aws-datastore(17941): 	at com.amplifyframework.datastore.appsync.AppSyncRequestFactory.extractFieldValue(AppSyncRequestFactory.java:500)
W/amplify:aws-datastore(17941): 	at com.amplifyframework.datastore.appsync.AppSyncRequestFactory.extractFieldLevelData(AppSyncRequestFactory.java:427)
W/amplify:aws-datastore(17941): 	at com.amplifyframework.datastore.appsync.AppSyncRequestFactory.getMapOfFieldNameAndValues(AppSyncRequestFactory.java:392)
W/amplify:aws-datastore(17941): 	at com.amplifyframework.datastore.appsync.AppSyncRequestFactory.buildCreationRequest(AppSyncRequestFactory.java:219)
W/amplify:aws-datastore(17941): 	... 19 more

W/amplify:aws-datastore(17941): Error ended observation of mutation outbox:
W/amplify:aws-datastore(17941): DataStoreException{message=Failed to process java.lang.RuntimeException: DataStoreException{message=Error encountered while creating model schema, cause=DataStoreException{message=Failed to get fields for model., cause=AmplifyException{message=An invalid CustomType field was provided. charts must be an instance of SerializedCustomType or a List of instances of SerializedCustomType, cause=null, recoverySuggestion=Check if this model schema is a correct representation of the fields in the provided Object}, recoverySuggestion=Validate your model file.}, recoverySuggestion=See attached exception for more details}, cause=null, recoverySuggestion=Check your internet connection.}
W/amplify:aws-datastore(17941): 	at com.amplifyframework.datastore.syncengine.MutationProcessor.drainMutationOutbox(MutationProcessor.java:122)
W/amplify:aws-datastore(17941): 	at com.amplifyframework.datastore.syncengine.MutationProcessor.lambda$startDrainingMutationOutbox$1$com-amplifyframework-datastore-syncengine-MutationProcessor(MutationProcessor.java:103)
W/amplify:aws-datastore(17941): 	at com.amplifyframework.datastore.syncengine.MutationProcessor$$ExternalSyntheticLambda17.apply(Unknown Source:4)
W/amplify:aws-datastore(17941): 	at io.reactivex.rxjava3.internal.operators.observable.ObservableFlatMapCompletableCompletable$FlatMapCompletableMainObserver.onNext(ObservableFlatMapCompletableCompletable.java:97)
W/amplify:aws-datastore(17941): 	at io.reactivex.rxjava3.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.drainNormal(ObservableObserveOn.java:201)
W/amplify:aws-datastore(17941): 	at io.reactivex.rxjava3.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.run(ObservableObserveOn.java:255)
W/amplify:aws-datastore(17941): 	at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65)
W/amplify:aws-datastore(17941): 	at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56)
W/amplify:aws-datastore(17941): 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
W/amplify:aws-datastore(17941): 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
W/amplify:aws-datastore(17941): 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W/amplify:aws-datastore(17941): 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
W/amplify:aws-datastore(17941): 	at java.lang.Thread.run(Thread.java:923)

As I can see in the AmplifyDatastore.db that my model is correct (and in fact it is correct because its working with full online).

999Rub · 2023-03-01T08:32:55Z

The problem seems comes from the serialization of custom type in this Model, but I've tried to delete the 'charts' field as it's mentioned in the exception and the error still occurs on every custom types of this Model.

The curious thing is, when I try to create the same issue with another Model (with the exact custom types too), this is working without any exception.

When one of its model is saved offline and try to be synced at the restart of the program online and failed, every next future instances of this model will fail to sync (even if they're saved online this time).

I can't get why it is working on a model and not on the other one, while they have both custom types with basics field types (string, int, double, lists..).

I do an update of my sync expressions at runtime (sync expressions doesn't have any effect regarding the behavior when I comment it or not).
However, when I do not use Amplify.DataStore.stop() and then Amplify.DataStore.start(), I don't receive any HubEvent if I previously saved a model offline (even those working), which I listen to for knowing when DataStore sent me ready and then switch from loading view to home page view.

When I only retrieve my data from the cloud (without offline saved before), and comment both Amplify.DataStore.stop() and Amplify.DataStore.start(), I've got the correct hubEvent in my listening method and everything works fine.

Maybe should I do not use Custom Types but general Models instead and just workaround with relation between each models ?

HuiSF · 2023-03-02T01:28:20Z

Thanks @999Rub the exception seems happening when clearing pending mutation box when your device is back online and deserializing stored data from the local DB (PendingMutationBox table rather than the actual Surgery table) I need to look into how the custom type data gets serialized and store in the PendingMutationBox table.

HuiSF · 2023-03-02T01:37:25Z

Quick follow up @999Rub which amplify-flutter version you are testing with?

999Rub · 2023-03-02T08:04:21Z

@HuiSF The amplify-flutter version is mentioned above but I use the 10.7.3 version.
We are not in the same time zone but if you need more details I'm available as much as I can. If you can, give me a set of details you'll probably need when you can answer me and I'll give a set of details back (If you need too), it will avoid waiting for each other to wake up ^^.
Will try to take a look about #2710 (comment) but I don't know whats happening under the hood in the PendingMutationBox so unfortunately I won't be the more efficient here...

The most mysterious thing to me (I don't know if you know why?), but why this issue occurs only on this model and not on the other ones (wich have custom types too) ?

HuiSF · 2023-03-02T16:56:46Z

Hi @999Rub sorry for the confusion, I believe 10.7.3 is the Amplify CLI version? Which you got from amplify --version?
You should be able to find the amplify-flutter packages version in your pubspec.yaml.

999Rub · 2023-03-03T08:47:39Z

@HuiSF You right this is the Amplify CLI version, sorry for the misunderstanding, my actual version is amplify_flutter: ^0.6.8

999Rub · 2023-03-07T08:10:20Z

Hi @HuiSF , is there anything new from your side ?

HuiSF · 2023-03-09T16:48:29Z

Hi @999Rub sorry for the delay, I will dig into this issue today.

999Rub · 2023-03-10T09:41:21Z

Hi @999Rub sorry for the delay, I will dig into this issue today.

Don't worry, I have at least 2 or 3 weeks left to solve this issue.
If there is no solution I will swap my custom types for classical models (I hope I don't have to do it).
Just keep me up if you find something.

kalismeras61 · 2023-03-14T09:32:14Z

@HuiSF i have also same issue, i am using latest one

amplify_auth_cognito: 1.0.0-next.6
amplify_api: 1.0.0-next.6
amplify_flutter: ^1.0.0-next.6
amplify_core: 1.0.0-next.6
amplify_storage_s3: 1.0.0-next.6
amplify_datastore: ^1.0.0-next.6

vgribok · 2023-03-16T18:03:46Z

Yep, same here with the 1.0.0-next.6. It's an Android only problem. For me it happens when the phone is online. save() succeeds but up-link sync fails.

999Rub · 2023-04-18T08:02:29Z

Hi @HuiSF , I saw your PR and how the issue's fix evolve last month. What are the news/state about it ?
Have you an estimated date of release for the fix ?
This only for me, to know if I need to plan some changes in my app to avoid this problem before it goes for bêta or if a fix will be available at time.

Thanks for you work and time.

HuiSF · 2023-04-18T17:02:42Z

Hi @999Rub we are working with amplify-android maintainers to test and merge the PRs, will update the progress, thank you for your patience.

999Rub · 2023-05-04T15:01:10Z

Hi @HuiSF , I've seen that the open issue has been merged into main branch, thanks for you work and Amplify team's work.
I think we can close this issue now.

Do you have any idea about when the fix will be available on the pub.dev ? Or maybe I don't have to wait that's available on pub.dev and simply pull the fix ?

Regards.

HuiSF · 2023-05-04T18:31:31Z

Hi @999Rub thanks for watching the progress! We are currently waiting for amplify-android releases, then we can integrate the fix into amplify-flutter. Amplify Android release should happen this week. We'll follow up ASAP once they release the new versions.

- Fixes #2710 - Improves #3017

999Rub · 2023-07-18T07:19:15Z

Hi @HuiSF , I've seen a new update on pub dev for the v0 of amplify_flutter wich include your fix for this issue (from https://github.com/aws-amplify/amplify-flutter/releases/tag/v0.6.14 ), thanks for the work.
Does the fix is also available on the latest v1 version (1.2.1) ? Or I need to wait until the v0 maintenance is done (19th July) to pull the fix ?

Regards.

HuiSF · 2023-07-19T17:38:50Z

Thank you for the feedback @999Rub !
And yes the fix is available in the latest v1 release v1.2.1.

Closing this issue 👍

dnys1 added datastore Issues related to the DataStore Category pending-triage This issue is in the backlog of issues to triage labels Feb 28, 2023

HuiSF self-assigned this Mar 2, 2023

This was referenced Mar 20, 2023

fix(datastore): SerializedCustomType de/serialization aws-amplify/amplify-android#2350

Merged

fix(datastore): SerializedCustomType de/serialization aws-amplify/amplify-android#2363

Merged

HuiSF added a commit that referenced this issue May 15, 2023

chore: bump amplify-android to 1.38.5

0910978

- Fixes #2710 - Improves #3017

HuiSF mentioned this issue May 15, 2023

chore: bump amplify-android to 1.38.6 #3050

Merged

dnys1 pushed a commit that referenced this issue May 22, 2023

chore: bump amplify-android to 1.38.5

64b6644

- Fixes #2710 - Improves #3017

HuiSF added a commit that referenced this issue Jun 16, 2023

chore: bump amplify-android to 1.38.6

90ad4b9

- Fixes #2710 - Improves #3017

dnys1 pushed a commit that referenced this issue Jun 19, 2023

chore: bump amplify-android to 1.38.6 (#3050)

9e3cd19

- Fixes #2710 - Improves #3017

HuiSF closed this as completed Jul 19, 2023

999Rub mentioned this issue Jul 26, 2023

DataStore cloud sync JsonSyntaxException #3472

Closed

14 tasks

999Rub mentioned this issue Aug 17, 2023

DataStore JSON exception on nullable arrays #3575

Closed

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DataStore sync issue from offline saved models (custom type fields) #2710

DataStore sync issue from offline saved models (custom type fields) #2710

999Rub commented Feb 28, 2023

HuiSF commented Feb 28, 2023

999Rub commented Mar 1, 2023

999Rub commented Mar 1, 2023 •

edited

Loading

HuiSF commented Mar 2, 2023

HuiSF commented Mar 2, 2023

999Rub commented Mar 2, 2023 •

edited

Loading

HuiSF commented Mar 2, 2023

999Rub commented Mar 3, 2023

999Rub commented Mar 7, 2023 •

edited

Loading

HuiSF commented Mar 9, 2023

999Rub commented Mar 10, 2023 •

edited

Loading

kalismeras61 commented Mar 14, 2023

vgribok commented Mar 16, 2023

999Rub commented Apr 18, 2023

HuiSF commented Apr 18, 2023

999Rub commented May 4, 2023

HuiSF commented May 4, 2023 •

edited

Loading

999Rub commented Jul 18, 2023

HuiSF commented Jul 19, 2023

DataStore sync issue from offline saved models (custom type fields) #2710

DataStore sync issue from offline saved models (custom type fields) #2710

Comments

999Rub commented Feb 28, 2023

Description

Categories

Steps to Reproduce

Screenshots

Platforms

Flutter Version

Amplify Flutter Version

Deployment Method

Schema

HuiSF commented Feb 28, 2023

999Rub commented Mar 1, 2023

999Rub commented Mar 1, 2023 • edited Loading

HuiSF commented Mar 2, 2023

HuiSF commented Mar 2, 2023

999Rub commented Mar 2, 2023 • edited Loading

HuiSF commented Mar 2, 2023

999Rub commented Mar 3, 2023

999Rub commented Mar 7, 2023 • edited Loading

HuiSF commented Mar 9, 2023

999Rub commented Mar 10, 2023 • edited Loading

kalismeras61 commented Mar 14, 2023

vgribok commented Mar 16, 2023

999Rub commented Apr 18, 2023

HuiSF commented Apr 18, 2023

999Rub commented May 4, 2023

HuiSF commented May 4, 2023 • edited Loading

999Rub commented Jul 18, 2023

HuiSF commented Jul 19, 2023

999Rub commented Mar 1, 2023 •

edited

Loading

999Rub commented Mar 2, 2023 •

edited

Loading

999Rub commented Mar 7, 2023 •

edited

Loading

999Rub commented Mar 10, 2023 •

edited

Loading

HuiSF commented May 4, 2023 •

edited

Loading