Couple of Bugs that are preventing AppSync from Synchronizing Local Datastore Changes Globally/Online

[b-cancel]

In Summary*
There are a couple of issues that I've run into using the (Amplify Datastore and API Combination)
and I've been reading many of the bug reports, and many seem to be related,
so I thought I'd post a compilation here since I assume that posting each individually would be annoying

Hopefully, I can also point many folks here and show them the workarounds I used,
so they can at least get over these hurdles for now until the issues are repaired

The Net Effect
What led me to discover all these problems it that I was able to use datastore locally,
but these changes would not be visible from the AWS Amplify Admin console

Bug 0: the null safety flag on amplify/cli.json unsets itself automatically
nothing much else to say here
before I pull every change from the database I need to create the flag again
**Workaround: make sure to set the flag every time before pulling

Bug 1: if the schema online and locally are different, synchronization will fail without proper explanation

1. I change the schema from AWS Amplify Admin Console
2. I set the null safety flag in amplify/cli.json
3. I pull using the command the Admin Console shows
4. Get an error code like this
   [VERBOSE-2:ui_dart_state.cc(199)] Unhandled Exception: DataStoreException(message: The operation couldn’t be completed. (SQLite.Result error 0.), recoverySuggestion: The operation couldn’t be completed. (SQLite.Result error 0.), underlyingException: The operation couldn’t be completed. (SQLite.Result error 0.))
   **Workaround: manually clear the datastore so that things are pulled cleanly from the global/online version

try {
     *first data store query I make in the app
} catch (e) {
     await Amplify.DataStore.clear();
     *reload the page that makes the query (in my case home)
}

Bug 2: If you just change the type of a field the sync will fail silently
**Workaround... same as for Bug 1

Bug 3: creating an object that CAN but not SHOULD have a reference through another object through an ID will work locally but not globally

1. I have a car model, I have an insurance model
2. each insurance can have many cars under it
3. I create the relationship "carsInsured" related to "Car" cardinality 1:n
4. the car model now gains an insuranceID
5. If I create a car with no insuranceID the change will save locally but not globally
   **Workaround... create an insurance that is empty... and check if the car has an insurance by simply checking a "hasInsurance" flag in the insurance object... then the car that I am saving will indeed have an insurance ID and the new car will be accepted globally

Bug 4: some types don't serialize properly

1. create any model with "AWSPhone" type (EX: user)
2. create a user with or without a phone
3. it will save locally but not globally
4. remove AWSPhone field (don't just change the type because then you will encounter Bug 2)
5. deploy, pull, adjust code accordingly
6. create another field that holds the same data but with a "trusted type" (for this example I used String)
   "Trusted Types" are the ones I've confirmed to work so far

*for anything that is a String in the dart models that are generated,
be careful when passing an empty string "" instead of null
(when I tried to create an object with a field of type AWSURL set to "" instead of null,
the data wouldn't push globally, the data would only stay on the local datastore)

--"Trusted Types"--
BASICS: [boolean, string, int]
AWS: [AWSEmail, AWSURL, AWSTimestamp]
CUSTOM: [any user creatable enum]

--I know there are issues with---
AWSPhone (So I use String)

--I've read that there have been issues with--
[float, AWSDate, AWSTime, AWSDateTime] (although I'm unsure if they are resolved)

--I haven't tested--
[AWSJSON, AWSIPAddress]

7. notice that it now creates locally and syncs globally

Platform
Amplify Flutter currently supports iOS and Android. This issue is reproducible in (check all that apply):
[X] Android
[X] iOS

[HuiSF]

Hi @b-cancel thanks for the details, much appreciated!

For bug 0
I encountered this issue myself as well. cli.json should be push when amplify status shows there is an update. Here's what I did:

1. amplify pull -> cli.json doesn't have the feature flag set
2. enable the feature flag, and make a non-breaking schema change (adding a unused model)
3. amplify push -> cli.json stored in s3 amplify bucket will update with the feature flag
4. remove the unused model and amplify push

For bug 1
According to the API doc, local DataStore should clear itself when ModelProvider.version changes, all local DB table will be recreated following new schema and cloud data will be synced back. When we expect schema changes, will need to amplify pull to generate the models and ModelProvider with the latest schema version, and build the into the App. Then the auto local DB clearing and recreation should be triggered.

Note: There is an issue #415 that may stop amplify-flutter doing in iOS platform. A fix PR #439 has been merged and pending release at this moment.

For bug 2
After type change please ensure ModelProvider is correctly generated and deploy latest schema changes into cloud. The mechanism described in above section should kick in automatically.

For bug 3
Could you elaborate what does "saved globally" mean? And could you please provide schema samples to help me understand the issue? Thanks in advance!

For bug 4
Is AWSPhone not @model annotated in your schema? Could you provide an example?
(Currently amplify-flutter doesn't support custom type (Map like type without @model annotation) we are working on bring this feature in near future.)

[b-cancel]

Thanks for the quick reply @HuiSF

below I refer to Amazon's Servers as whatever technology stores the data I can see from the AWS Admin Console

for bug 0
Thanks! this worked for me
but for those that don't know where or how to make the change because they've been generating their shcemas by just using the AWS Admin Console

what worked for me was adding (assuming you don't have a model named Repairer)

type Repairer @model @auth(rules: [{allow: public}]) {
  id: ID!
}

on the top of the 'amplify/backend/{api_name}/schema.graphql' file

for bug 1
"Then the auto local DB clearing and recreation should be triggered."
good to know that is the expected behavior! Looking forward to a fix soon
at least for now the workaround does the trick,
and hopefully after all my mentions whoever needs it will find it

for bug 2
the changes were made on AWS Amplify Admin Console, and I pulled the changes,
everything in code works properly, but If I don't clear the local database on the device... the database online has a slightly different schema as the local one... so they won't synchronize

for bug 3*
"saved globally" as in saved online in Amazon's servers

model examples (no fields are required except IDs of that object ofcourse)

Insurance
-id!
-name
-"carsInsured" related to "Car" cardinality 1:n

Car
-id!
-name
-insuranceID (added automatically to support the relationship created above)

//the code below will save locally, but that new car won't show up on the AWS Admin Console
Amplify.DataStore.save(Car(name: "cybertruck"))
//neither will
Amplify.DataStore.save(Car(name: "cybertruck", insuranceID: null))
Amplify.DataStore.save(Car(name: "cybertruck", insuranceID: ""))
//on the other hand this will work (ofcourse this might break down the line since I'm not storing the insurance object, but it proves my point)
Amplify.DataStore.save(Car(name: "cybertruck", insuranceID: Insurance().id))

for bug 4
Everything was created by pulling from the AWS Admin Console,
the AWS Admin Console allows me to select the "AWSPhone" type,
but it isn't serializing in a way that Amazon's servers would accept it,
or at least that is what seems to be happening.

I came up with this theory by...

1. create User model (id, name[String], phone[AWSPhone])
2. on AWS Admin Console create [a] user without phone [b] user with a phone
3. both are readable from a query from a flutter app, and I can edit both with a query and those changes will be push globally
4. but If from the flutter app I try to create [a] user without phone(ensuring only whole numbers than can start with 0) [b] user with phone... both users will save locally... but that data won't ever show up on the AWS Admin Console

[b-cancel]

Update on Bug 1
the try-catch worked because the mismatch was between my first query (against the User model)
but schema mismatches can happen anywhere
so to properly fix the issue you would have to wrap all of your queries in try to catches
and then handle clearly the database and then moving home appropriately in all cases

Needless to say this could be a lot of work depending on how your system is setup
and hard to avoid if your schema is changing as you are building out an application or adding new features

thankfully, like @HuiSF mentioned, a fix is comming soon

since my app isn't in production I just create a hidden manual button to clear the DB and reload my home
this button can be triggered after simply continuing when the exception is thrown

[b-cancel]

Bug 4 Extension
AWSURL doesn't serialize properly if an empty string is passed ("" not null)
but I think anyone would also expect that creating an object and setting the field to "" instead of null should still be valid
but doing so doesn't push the object to the Amazon Servers
(now that I think of it I haven't tried to pass "" instead of null for type String or AWSEmail... so that might be something else to watch out for)

[HuiSF]

Glad some tips helped.

For bug 3

I believe the insuranceId is a required field in your Car model?

If the schema is defined like this, then all GraphQL types (e.g. mutation input type) will also require insuranceId field, if you were not able to provide a valid value, GraphQL will fail. (You should be able to see the underlying GraphQL error logged in the App console)

If you want to create Car model freely without buying insurance, you can set the insuranceId in Car model as optional, and assign one later by updating this car model.

For bug 4

From your updated comment above, it should be similar to what I talked about above, if the filed is marked as required, you will have to provide the data for that field, otherwise the underlying GraphQL request will fail. (saved to local but cannot sync to cloud)

[kjones]

I can confirm that AWSJSON works as expected. It can be used in place of nested types but requires all clients to be updated since AppSync is expecting a JSON string instead of a GraphQL sub-type. The underlying data storage is identical so you can switch back to a nested type once the Flutter SDK supports them.

[b-cancel]

Thanks for that confirmation @kjones

And thanks for working through this with me @HuiSF

Unfortunately, I mentioned both bugs 3 and 4, because all the mentioned fields were not required

I just double checked the graph sql schema

Another bit of evidence is that I was able to create what I wanted from the AWS Amplify Admin Console
I could also create things from the flutter app, But it would simply not sync up to Amazon's servers

For example for bug 3:
I had multiple cars created on the admin console without insuranceID filled
And I could retrieve that from the flutter app
I could also create cars without insuranceID (AKA insurance reference) from the flutter app,
but these changes would not sync up to Amazon's servers

And for bug 4:
Same thing
I could create users with number with the AWSPhone type from the admin console, and I could read them from the flutter app
And I could create them from the flutter app

But in both cases, creating them from the flutter app would not sync the changes to Amazon's servers

Even when other models would sync edits, creates, and deletes

Please let me know if I need to get more specific, and thanks again for your patience

[HuiSF]

Thanks for the follow up @b-cancel sorry I misunderstood and thought AWSPhone is a custom type created by yourself 😅

Would you mind to post the model schema (You can find them in schema.graphql) of these problematic models?

Could you also paste the logs when invoking Amplify.DataStore.save()?

[b-cancel]

All the type I'm referring to by name are the ones allowed by AWS Amplify's Admin Console

As for the schema.graphql and the logs
I'll send those over on Monday

[b-cancel]

@HuiSF Below I show three thing

1. the code I use to create the logs that inform what is hapening
2. things working using a String type instead of AWSPhone
3. things not working using a AWSPhone type

---------------------------------------------------------------------------

Amplify.DataStore.save() doesn't usually log anything,
but I've set up the code below in relevant locations

dataHubSub = Amplify.Hub.listen([HubChannel.DataStore], (event) {
    if (event.eventName == "ready") {
      log("Data Hub Ready");
      afterDataHubReady();
    } else {
      print("Data Hub Other: " + event.eventName.toString());
    }
    if (event.runtimeType == DataStoreHubEvent) {
      DataStoreHubEvent dataStoreHubEvent = event;
      print("Data Hub Payload: " + dataStoreHubEvent.payload.toString());
    }
});

userStream = Amplify.DataStore.observe(User.classType);
    userStream.listen((SubscriptionEvent<User> event) {
      print("USER STREAM: " +
          event.eventType.toString() +
          " & " +
          event.item.toString());

      //if a user was
      //1. added -> it could not have been us
      //2. removed -> this shouldn't happen
      //3. updated -> it matters if they have our matching email
      if (event.eventType == EventType.update) {
        User userThatChanged = event.item;
        if (userThatChanged.email == Home.authUser.value.username) {
          print("\nOUR USER CHANGED\n");
          Home.userData.value = userThatChanged;
        }
      }
    }).onError((error) => print("USER STREAM ERROR: " + error));

---------------------------------------------------------------------------

Below is everything I am using that saves properly

*In AWS Amplify Admin Console

*schema.graphql

type User @model @auth(rules: [{allow: public}]) {
  id: ID!
  name: String
  email: AWSEmail
  phone: String
  specialAdmin: Boolean
}

*no logs coming from Amplify.DataStore.save()
but this is what my own logs are printing

flutter: Data Hub Other: outboxStatus
flutter: Data Hub Payload: Instance of 'OutboxStatusEvent'
flutter: Data Hub Other: outboxMutationEnqueued
flutter: Data Hub Payload: Instance of 'OutboxMutationEvent'
flutter: USER STREAM: EventType.create & User {id=883af41b-ef64-4ef6-a182-8df201c15a3a, name=working user, [email protected], phone=1231231234, specialAdmin=null}
flutter: Data Hub Other: outboxMutationProcessed
flutter: Data Hub Payload: Instance of 'OutboxMutationEvent'
flutter: Data Hub Other: outboxStatus
flutter: Data Hub Payload: Instance of 'OutboxStatusEvent'

---------------------------------------------------------------------------

Below is everything I am using that does not save properly

*In AWS Amplify Admin Console

*schema.graphql

type User @model @auth(rules: [{allow: public}]) {
  id: ID!
  name: String
  email: AWSEmail
  specialAdmin: Boolean
  amazonPhoneType: AWSPhone
}

*no logs coming from Amplify.DataStore.save()
but this is what my own logs are printing
Notice how its missing "outboxMutationProcessed"

flutter: Data Hub Other: outboxStatus
flutter: Data Hub Payload: Instance of 'OutboxStatusEvent'
flutter: Data Hub Other: outboxMutationEnqueued
flutter: Data Hub Payload: Instance of 'OutboxMutationEvent'
flutter: USER STREAM: EventType.create & User {id=fe03a72a-fbb4-4ec0-821f-320f4260394b, name=yes num, [email protected], specialAdmin=null, amazonPhoneType=1110004444}
flutter: USER STREAM: EventType.create & User {id=fe03a72a-fbb4-4ec0-821f-320f4260394b, name=yes num, [email protected], specialAdmin=null, amazonPhoneType=1110004444}
flutter: Data Hub Other: outboxStatus
flutter: Data Hub Payload: Instance of 'OutboxStatusEvent'

and yet, here is evidence that I can read information without the number (in other words, that field isn't required)

[b-cancel]

Alright @HuiSF I detailed everything above
I'll leave the rest in your hands...

Additionally, to deal with bugs 1 & 2...

Before making edits to a model

1. (Admin Console) delete everything in that model from amplifying admin
2. let the flutter app synchronize that
3. Amplify.Datastore.clear()
4. Amplify.Auth.signOut();
5. stop the app from running
6. (Admin Console) make the edits to the model
7. pull the new model to the flutter app
8. start the app running
9. (Admin Console) *create data if the app needs it to function & then reload the page (EX: at least 1 user is required)

Tedious yes... but at least it works for now...

[HuiSF]

Thanks @b-cancel I'll look into the issue around using AWSPhone.

In addition regarding this issue in bug 3

I could also create cars without insuranceID (AKA insurance reference) from the flutter app, but these changes would not sync up to Amazon's servers

The insuranceId a partition key of an index of the Car table, it might be related to this issue.

It has a workaround you can try, by defining the Car model as below:

type Car @model @key(name: "byInsuranceId", field: ["insuranceId"]) {
  id: ID!
  ...
  insuranceId: ID
  insurance: Insurance @connection(fields: ["insuranceId"])
}

This added insurance field should be able help platform libraries to generate can be processed GraphQL document when not assigning a non-null value to insuranceId field.

[HuiSF]

@b-cancel

For bug 4, when you are using type AWSPhone DynamoDB will verify if the value is a valid phone number. The example you are using is not a valid phone number, therefore DynamoDB rejected writing the record.

GraphQL request

{
	"query": "mutation CreatePerson($input: CreatePersonInput!) {\n  createPerson(input: $input) {\n    _deleted\n    _lastChangedAt\n    _version\n    department {\n      id\n    }\n    id\n    name\n    phone\n    posts {\n      items {\n        id\n      }\n      nextToken\n      startedAt\n    }\n  }\n}\n",
	"variables": {
		"input": {
			"name": "Test with Phone",
			"id": "1ea87481-dbc6-41a3-a6db-e1061e3f6d5a",
			"phone": "1110004444"
		}
	}
}

GraphQL response

{
	"data": null,
	"errors": [{
		"path": null,
		"locations": [{
			"line": 1,
			"column": 23,
			"sourceName": null
		}],
		"message": "Variable 'phone' has an invalid value. Unable to parse `1110004444` as a valid phone number."
	}]
}

This is behavior of DynamoDB is expected... however, underlying APIs didn't throw this DynamoDB error, and the underlying APIs should remove local record for this type of failure.

I will create issues for amplify-ios and amplify-android to track the issue.

Currently, the workaround is to ensure the phone number to be synced is valid number. (DataStore plugins doesn't have client data validation at this moment)

[b-cancel]

@HuiSF nice to know there are workarounds
I'll try them and get back to you if they don't work

As for the AWSPhone issue... how is 1110004444 not a valid phone number?
3 for area code, 7 for number
is it expecting the number in a particular format?
or does it actually check if that number is being used by somebody? (this seems unlikely)

I tried using my own phone number "9567772692" or 956 777 2692
and I was able to set that as the value from the AWS Amplify Admin Console
but again I was not able to create a user with that value from the flutter app

So I suspect this merits further investigation

[HuiSF]

According to AppSync documentation about the scalar type AWSPhone:

AWSPhone
A phone number. This value is stored as a string. Phone numbers can contain either spaces or hyphens to separate digit groups. Phone numbers without a country code are assumed to be US/North American numbers adhering to the North American Numbering Plan (NANP).

Other scalar types have certain validation rules as well.

[HuiSF]

amplify-flutter DataStore currently lacks support of configuring custom error handler like other platform libraries. Which could be challenging for developers to catch unexpected errors happened during DataStore operations. This is something that probably need to be prioritized.

[Jordan-Nelson]

FYI - Marking this as pending-triage because it needs to be split into separate issues so that they can be properly tracked.

[HuiSF]

Regarding scalar type related issue discussed above, certain level of client validation is required. As each client may have different requirement and design, a generic client validation cannot be implemented within Amplify layer.

Among the linked issues -

• Sync up issue for iOS with nullable field and enabled secondary index #306 depends on a fix will be implemented in Amplify CLI
• We will follow up DataStore Error having two relationships with the same model #494 separately

Therefore, I'm going to close this issue. Thanks again for the exemplary issue report, it helped amplify-flutter getting better!

[999Rub]

Thanks mate for this issues report, that saved me.
I can add for Bug 4:

If you use AWSJSON type , in Flutter it will locally works with simple String type (not casted as JSON object).
But it will failed when syncing to cloud.
I switch my AWSJSON type for a string type (with a JSON architecture).
My advice is to create a toJSON() and fromJSON() methods in your custom objects, but simply saved it as a String to the cloud, and keep you casting logic in the client side.