[WIP] SyncEngine for specific protocols + delegate sync. #836

LiranCohen · 2024-08-13T15:02:15Z

Resolves #826

This PR adds the ability to Sync a subset of data by providing specific protocols a user would like to sync. Additionally sync now supports using a grant and delegateDid to perform sync.

Introduced a CachedPermissions class to help cache the lookup of permission grants records from the DWN. This is used both in DwnApi and SyncEngineLevel.

changeset-bot · 2024-08-13T15:02:19Z

🦋 Changeset detected

Latest commit: 96944b7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages

Name	Type
@web5/agent	Minor
@web5/identity-agent	Minor
@web5/proxy-agent	Minor
@web5/user-agent	Minor
@web5/api	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

codecov · 2024-08-19T15:42:48Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.35%. Comparing base (34590b2) to head (96944b7).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #836      +/-   ##
==========================================
+ Coverage   93.27%   93.35%   +0.07%     
==========================================
  Files         115      116       +1     
  Lines       32447    32691     +244     
  Branches     2547     2598      +51     
==========================================
+ Hits        30266    30518     +252     
+ Misses       2143     2134       -9     
- Partials       38       39       +1

Components	Coverage Δ
agent	`87.20% <100.00%> (+0.31%)`	⬆️
api	`99.53% <100.00%> (+<0.01%)`	⬆️
common	`98.68% <ø> (ø)`
credentials	`94.95% <ø> (ø)`
crypto	`93.79% <ø> (ø)`
dids	`97.77% <ø> (ø)`
identity-agent	`96.42% <ø> (ø)`
crypto-aws-kms	`100.00% <ø> (ø)`
proxy-agent	`96.42% <ø> (ø)`
user-agent	`96.42% <ø> (ø)`

github-actions · 2024-08-19T17:21:39Z

TBDocs Report

🛑 Errors: 0
⚠️ Warnings: 5

@web5/api

Project entry file: packages/api/src/index.ts

@web5/crypto

Project entry file: packages/crypto/src/index.ts

📄 File: ./packages/crypto/src/utils.ts
⚠️ `extractor:typedoc:missing-docs`: CryptoUtils (Variable) does not have any documentation.
⚠️ `extractor:typedoc:missing-docs`: CryptoUtils.__type.randomPin (Property) does not have any documentation.
⚠️ `extractor:typedoc:missing-docs`: CryptoUtils.__type.randomUuid (Property) does not have any documentation.
⚠️ `extractor:typedoc:missing-docs`: CryptoUtils.__type.randomBytes (Property) does not have any documentation.
⚠️ `extractor:typedoc:missing-docs`: CryptoUtils.__type.getJoseSignatureAlgorithmFromPublicKey (Property) does not have any documentation.

@web5/crypto-aws-kms

Project entry file: packages/crypto-aws-kms/src/index.ts

@web5/dids

Project entry file: packages/dids/src/index.ts

@web5/credentials

Project entry file: packages/credentials/src/index.ts

TBDocs Report Updated at 2024-08-23T18:02:49Z 96944b7

This refactors a lot of what's in #824 with regards to creating/fetching grants. Satisfies: #827 Introduces a `PermissionsApi` interface and an `AgentPermissionsApi` concrete implementation. The interface implements the following methods `fetchGrants`, `fetchRequests`, `isGrantRevoked`, `createGrant`, `createRequest`, `createRevocation` as convenience methods for dealing with the built-in permission protocol records. The `AgentPermissionsApi` implements an additional static method `matchGrantFromArray` which was moved from a `PermissionsUtil` class, which is used to find the appropriate grant to use when authoring a message. A Private API usedin a connected state to find and cache the correct grants to use for the request. A Permissions API which implements `request`, `grant`, `queryRequests`, and `queryGrants` that a user can utilize The `Web5` permissions api introduces 3 helper classes to represent permissions: Class to represent a permission request record. It implements convenience methods similar to the `Record` class where you can `store()`, `import()` or `send()` the underlying request record. Additionally a `grant()` method will create a `PermissionGrant` object. Class to represent a grant record. It implements convenience methods similar to the `Record` class where you can `store()`, `import()` or `send()` the underlying grant record. Additionally a `revoke()` method will create a `GrantRevocation` object, and `isRevoked()` will check if the underlying grant has been revoked. Class to represent a permission grant revocation record. It implements convenience methods similar to the `Record` class where you can `store()` or `send()` the underlying revocation record.

…getting a permission

…the grants from wallet connect

…ethod

shamilovtim · 2024-08-23T19:17:05Z

packages/agent/src/sync-engine-level.ts

@@ -74,17 +93,19 @@ export class SyncEngineLevel implements SyncEngine {

  set agent(agent: Web5PlatformAgent) {
    this._agent = agent;
+    this._cachedPermissionsApi = new CachedPermissions({ agent: agent as Web5Agent, cachedDefault: true });


if agent is set to Web5PlatformAgent in CachedPermissions would it fix needing to cast this?

shamilovtim · 2024-08-23T19:21:47Z

packages/agent/src/sync-engine-level.ts

        }
      });

      let reply: MessagesReadReply;

      try {
        reply = await this.agent.rpc.sendDwnRequest({
-          dwnUrl,
-          targetDid : did,
+          dwnUrl,          targetDid : did,


formatting mistake

shamilovtim · 2024-08-23T19:27:29Z

packages/api/src/dwn-api.ts


  constructor(options: { agent: Web5Agent, connectedDid: string, delegateDid?: string }) {
    this.agent = options.agent;
    this.connectedDid = options.connectedDid;
    this.delegateDid = options.delegateDid;
    this.permissionsApi = new AgentPermissionsApi({ agent: this.agent });
+    this.cachedPermissionsApi = new CachedPermissions({ agent: this.agent, cachedDefault: true });


should it be named AgentCachedPermissionsApi for consistency? it seems like all the rest of the Agent...Api its a class that is coupled to agents and requires passing the Agent singleton into it

shamilovtim · 2024-08-23T19:29:07Z

packages/agent/src/sync-engine-level.ts

+  /**
+   * An instance of the `AgentPermissionsApi` that is used to interact with permissions grants used during sync
+   */
+  private _cachedPermissionsApi: CachedPermissions;


per the description this is kind of confusing. should there be some sort of inheritance?

shamilovtim · 2024-08-23T19:31:52Z

packages/api/src/web5.ts

+   *
+   * This will store the grants as the DWN owner to be used later when impersonating the connected DID.
+   */
+  static async processConnectedGrants({ grants, agent, delegateDid }: {


this doesn't seem like the right place for this method? it doesn't rely on any internal state from the Web5 class (from what I can tell) and seems to be pretty highly related to the connect work in Agent

shamilovtim

rubber stamp

LiranCohen force-pushed the lirancohen/grants-sync branch from 540a4ca to f7471b2 Compare August 13, 2024 15:39

LiranCohen force-pushed the lirancohen/grants-api branch 2 times, most recently from 090dd73 to 2b0eda3 Compare August 14, 2024 02:46

Base automatically changed from lirancohen/grants-api to lirancohen/grants-web5 August 14, 2024 03:03

LiranCohen force-pushed the lirancohen/grants-web5 branch from 2b9391c to 22d86bd Compare August 14, 2024 13:27

LiranCohen force-pushed the lirancohen/grants-sync branch 2 times, most recently from ed8a9e4 to 85ec6a5 Compare August 14, 2024 14:50

LiranCohen force-pushed the lirancohen/grants-web5 branch from 2f086dc to 9ab0463 Compare August 14, 2024 22:43

LiranCohen force-pushed the lirancohen/grants-sync branch from 3f632b8 to f264ddb Compare August 14, 2024 22:57

Base automatically changed from lirancohen/grants-web5 to main August 19, 2024 16:51

LiranCohen force-pushed the lirancohen/grants-sync branch from b0562bb to 903fdc2 Compare August 19, 2024 17:18

LiranCohen force-pushed the lirancohen/grants-sync branch 3 times, most recently from 5c9bfb1 to 01b5606 Compare August 22, 2024 00:27

LiranCohen marked this pull request as ready for review August 22, 2024 01:03

LiranCohen requested review from csuwildcat, shamilovtim, nitro-neal and thehenrytsai as code owners August 22, 2024 01:03

LiranCohen added 8 commits August 23, 2024 13:59

first pass at connect flow and grants api

e182aaa

first pass at connect flow and grants api

9e0c041

Sync by specific protocols and by delegate did

03fecb6

update dwn-sdk and account for 204 in sync

f44081b

sync enhancements

10699af

sync protocol test, used cached permissions in API package

3734cf9

fix test with cache clearing

a0ecae1

LiranCohen added 9 commits August 23, 2024 13:59

increase test coverage for cached permissions class

7a10732

fix after rebase

b2734c3

getPermissions does not return unefined, throws if there is an error …

b3e74c6

…getting a permission

skip when grants are not present in sync

ed24f2d

empty event log on error

f856d7f

assign specific protocols to sync based on the results of processing …

58cc290

…the grants from wallet connect

add response detail to error in DWN store

34ad0f7

increase test coverage, remove duplicate tests and clean up connect m…

1adf942

…ethod

add changesets

96944b7

LiranCohen force-pushed the lirancohen/grants-sync branch from 01b5606 to 96944b7 Compare August 23, 2024 18:00

shamilovtim reviewed Aug 23, 2024

View reviewed changes

shamilovtim approved these changes Aug 23, 2024

View reviewed changes

LiranCohen merged commit 3d1f825 into main Aug 23, 2024
35 checks passed

LiranCohen deleted the lirancohen/grants-sync branch August 23, 2024 20:53

github-actions bot mentioned this pull request Aug 23, 2024

Version Packages #848

Merged

This was referenced Sep 11, 2024

Version Packages #900

Closed

Version Packages #902

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] SyncEngine for specific protocols + delegate sync. #836

[WIP] SyncEngine for specific protocols + delegate sync. #836

LiranCohen commented Aug 13, 2024 •

edited

Loading

changeset-bot bot commented Aug 13, 2024 •

edited

Loading

codecov bot commented Aug 19, 2024 •

edited

Loading

github-actions bot commented Aug 19, 2024 •

edited

Loading

shamilovtim Aug 23, 2024

shamilovtim Aug 23, 2024

shamilovtim Aug 23, 2024

shamilovtim Aug 23, 2024

shamilovtim Aug 23, 2024

shamilovtim left a comment

[WIP] SyncEngine for specific protocols + delegate sync. #836

[WIP] SyncEngine for specific protocols + delegate sync. #836

Conversation

LiranCohen commented Aug 13, 2024 • edited Loading

changeset-bot bot commented Aug 13, 2024 • edited Loading

🦋 Changeset detected

codecov bot commented Aug 19, 2024 • edited Loading

Codecov Report

github-actions bot commented Aug 19, 2024 • edited Loading

@web5/api

@web5/crypto

@web5/crypto-aws-kms

@web5/dids

@web5/credentials

shamilovtim Aug 23, 2024

Choose a reason for hiding this comment

shamilovtim Aug 23, 2024

Choose a reason for hiding this comment

shamilovtim Aug 23, 2024

Choose a reason for hiding this comment

shamilovtim Aug 23, 2024

Choose a reason for hiding this comment

shamilovtim Aug 23, 2024

Choose a reason for hiding this comment

shamilovtim left a comment

Choose a reason for hiding this comment

LiranCohen commented Aug 13, 2024 •

edited

Loading

changeset-bot bot commented Aug 13, 2024 •

edited

Loading

codecov bot commented Aug 19, 2024 •

edited

Loading

github-actions bot commented Aug 19, 2024 •

edited

Loading