Datastore - Sync from DynamoDB to Client - Issue

[nithin-seenivasan]

Describe the bug

DynamoDB to Client sync is not happening unless local storage is cleared first.

Usage scenario:

1. Client uploads data into DynamoDB from Client manually:

• Data is written to local storage by passing an instance of the model to Amplify.DataStore.save() (WORKS)
• Data is immediately synced by the datastore sync engine to the DynamoDB table (WORKS)

2. Client uploads data into DynamoDB indirectly:

• The client can initiate a Lambda function, which processes some external data and writes it onto the DynamoDB table, while following the schema data types and formats (essentially 'mimicking' the Client writing locally to the DynamoDB table). The datastore Sync Engine on the client fails to download this data to the client's local storage.
• This was also tested by duplicating values in DynamoDB table generated by the client itself (with new uids). These duplicated values were not synced to the local storage of the client

The ONLY way I can force these externally written values to DynamoDB to download into Local Storage is if I clear the local storage using Amplify.DataStore.clear() and then do a read query Amplify.DataStore.query(Post.classType);.

To Reproduce
Steps to reproduce the behavior:

1. Followed the exact same steps as mentioned here - https://docs.amplify.aws/lib/datastore/getting-started/q/platform/flutter
2. Created a basic app using the Query, Create and Delete code from https://docs.amplify.aws/lib/datastore/data-access/q/platform/flutter#delete
3. Forced the app to resync with the DynamoDB table by using the Clear local storage code from https://docs.amplify.aws/lib/datastore/sync/q/platform/flutter#existing-backend

Expected behavior
I expected that external changes are made to the DynamoDB table will get synced to the local storage. This could either be -

1. The sync happens at a configurable time interval
2. The sync can be triggered by the client
3. The sync can be automatic, listening to any changes to the DynamoDB table

Platform
Amplify Flutter current supports iOS and Android. This issue is reproducible in (check all that apply):
[X] Android
[] iOS

Smartphone (please complete the following information):

• Device: [ZTE Axon 7]
• OS: [Android 8.0.0]

Additonal Data

From Android Studio's Console, showing the sequence from Clearing Local Storage to running the Read Query.

I/amplify:aws-datastore( 8329): Orchestrator lock released.
I/amplify:aws-datastore( 8329): Orchestrator lock acquired.
I/amplify:aws-datastore( 8329): Orchestrator transitioning from SYNC_VIA_API to STOPPED
I/amplify:aws-datastore( 8329): Setting currentState to LOCAL_ONLY
I/amplify:aws-datastore( 8329): Stopping subscription processor.
I/amplify:aws-api( 8329): No more active subscriptions. Closing web socket.
I/amplify:aws-datastore( 8329): Stopped subscription processor.
I/amplify:aws-datastore( 8329): Stopping observation of local storage changes.
I/amplify:aws-datastore( 8329): Setting currentState to STOPPED
I/amplify:aws-datastore( 8329): Orchestrator lock released.
I/amplify:aws-datastore( 8329): Creating table: PersistentModelVersion
I/amplify:aws-datastore( 8329): Creating table: Post
I/amplify:aws-datastore( 8329): Creating table: LastSyncMetadata
I/amplify:aws-datastore( 8329): Creating table: ModelMetadata
I/amplify:aws-datastore( 8329): Creating table: PersistentRecord
I/amplify:aws-datastore( 8329): Creating index for table: PersistentRecord
I/amplify:flutter:datastore( 8329): Successfully cleared the store
I/amplify:aws-datastore( 8329): Orchestrator lock acquired.
I/amplify:aws-datastore( 8329): Orchestrator transitioning from STOPPED to SYNC_VIA_API
I/amplify:aws-datastore( 8329): Starting to observe local storage changes.
I/amplify:aws-datastore( 8329): Now observing local storage. Local changes will be enqueued to mutation outbox.
I/amplify:aws-datastore( 8329): Setting currentState to LOCAL_ONLY
I/amplify:aws-datastore( 8329): Setting currentState to SYNC_VIA_API
I/amplify:aws-datastore( 8329): Orchestrator lock released.
I/amplify:aws-datastore( 8329): Starting API synchronization mode.
I/amplify:aws-datastore( 8329): Starting processing subscription events.
I/amplify:aws-datastore( 8329): Started subscription processor for models: [Post] of types [ON_CREATE, ON_UPDATE, ON_DELETE].
I/amplify:aws-datastore( 8329): Successfully sync'd down model state from cloud.
I/amplify:aws-datastore( 8329): Starting processing subscription data buffer.
I/amplify:aws-datastore( 8329): Started processing the mutation outbox. Pending mutations will be published to the cloud.
I/amplify:aws-datastore( 8329): Started the orchestrator in API sync mode.
I/amplify:aws-datastore( 8329): Orchestrator lock acquired.
I/amplify:aws-datastore( 8329): Orchestrator lock released.

Posted as an issue, discussed with @Amplifiyer on #5

[kjones]

Not an AWS dev but thought I would comment... DataStore client sync is built on top of AppSync subscription. Writing directly to a DynamoDB table bypasses AppSync. You should be able to make updates using AppSync GraphQL API requests and the clients will sync the new data. https://docs.amplify.aws/guides/functions/graphql-from-lambda/q/platform/js

[Amplifiyer]

Hi @nithin-seenivasan, @kjones is right, DataStore works with graphql and writing directly to the dynamoDB table bypasses it. It's not a recommended flow as well since client side DataStore manages the version numbers for a variety of use cases and writing directly to dynamodb will mess with the versioning and corrupt the data.

[nithin-seenivasan]

I suspected as much, thank you @kjones and @Amplifiyer for that. So can we conclude that the only way (currently) to trigger a sync on the Client side is by clearing local storage?

[Amplifiyer]

We don't recommend updating DDB directly and using the same tables with DataStore. As you pointed out, calling clear() will clear the local datastore and start a full sync with the DDB table fetching all the records in it.

[nithin-seenivasan]

@kjones Thanks for the heads up about GraphQL. I got my app working directly with Appsync, and everything works perfectly.

[kjones]

@nithin-seenivasan Good to hear. I'm certainly geeking out on this workflow at the moment where the backend can make changes that propagate automatically to the client. I'm using it in a couple of different ways:

1. Client DataStore->AppSync->DDB->DDB Stream Lambda->SQS->Data Analysis Lambda writes results to AppSync->Client DataStore
2. External events from 3rd party->API Gateway->Lambda writes updates to AppSync->Client DataStore

[nithin-seenivasan]

@kjones My workflow ended up very similar to what you wrote! I'm currently doing -

Client -> S3 -> Lambda Trigger -> Lambda -> Appsync -> DDB and Client Datastore

In your 2nd scenario, do you send out the data from 3rd party to ALL your clients via Lambda? I had the issue to only send the data that a client uploads to S3 back to the client, so I couldn't have all the users subscribe to the mutation. I ended up using a GraphQL schema with the CognitoUserPool authorized user having exclusive Read, Update and Delete permissions, while public API-key authenticated system (Lambda in this case) can Create. So Lambda, using an API-key writes a mutation to Appsync, mimicking the owner (whose ID i get from my uploaded file's filename in S3 bucket), and the corresponding user, who is subscribed to all mutations bearing their user-ID, receives it down to the client device. It works, but it's a bit roundabout.

[kjones]

The data coming from 3rd parties is from services the user has authorized via OAuth to send us data on their behalf. So that the data is only visible to a single user I set the models to require owner authorization. For Lambda function access I use IAM.

@auth(rules: [{ allow: owner }, { allow: private, provider: iam }])

When saving to AppSync from Lambda, the owner field needs to be set and the HTTP requests need to be AWS V4 signed.

[jukakoski]

I struggled two days with sync problems and tried all the tricks from these issue threads. Sync was working as expected when using Datastore API with clients, but external mutation using Appsync console didn't trigg the update on client side.
I had also the _version field in my mutation query as said in documentation, but it didn't help. https://docs.amplify.aws/lib/datastore/how-it-works/q/platform/js#conflict-resolution

Finally I monitored in Chrome developer tools how the Datastore client is making it's mutations and I realized the the client is passing all the fields of the model in mutation. So, after adding all fields into Appsync mutation query it started to work. In documentation it is said that you should include all the required fields, but I don't know how to define if the field is required or not. I didn't have exclamation mark after my fields so I though the fields were optional in mutation query. Apparently they were not.
I hope this helps someone..

[b-cancel]

This is a bug report I just filed that includes a couple of bugs I found and the workarounds to each
after reading your bug report I think it could help
#822