Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add merkleization of DID documents #492

Merged
merged 23 commits into from
Apr 6, 2023
Merged

feat: add merkleization of DID documents #492

merged 23 commits into from
Apr 6, 2023

Conversation

ntn-x2
Copy link
Member

@ntn-x2 ntn-x2 commented Mar 27, 2023

Fixes https://github.com/KILTprotocol/ticket/issues/2557 and fixes https://github.com/KILTprotocol/ticket/issues/2556.

This PR builds on top of the shell PR, and adds support for Merkle proof for DID documents.

Merkle proof structure

A DID merkle proof is, at the core, an order set of (key, value) pairs, on which proof-of-inclusion and proof-of-non-inclusion can be performed. This PR generates and validates merkle proofs where leaves are of two types:

  • a DID key reference leave, whose key is the tuple (key ID, key relationship) and the value is an empty tuple
  • a DID key details leave, whose key is the key ID and the value is the key details

For each key reference leaf with a given key ID, the proof also has to contain a key details leaf whose key is the key ID. Multiple reference leaves can reference the same key details leaf, optimising the storage size.

New runtime APIs

There is a new runtime API which the DIP sender would expose, and that allows anyone to generate a merkle proof for a given DID identifier and set of key IDs. The result contains the merkle root (which must match what other chains have stored in their pallet-dip-receiver map), and a merkle proof, which includes blinded values and a set of key reference and key details leaves for the keys identified by the provided key IDs.

How to test

The setup flow is similar to that of #489.
Specifically:

  • Set up the local Rococo network onboarding the sender and receiver chains with para IDs 2_000 and 2_001 respectively
  • Open an HRMP channel from sender 2_000 to receiver 2_001
  • Create a DID on the sender chain, e.g., using the kilt-did-utilities tool
    Screenshot 2023-03-27 at 10 00 48
  • Push the identity of the DID to the receiver chain via the pallet-dip-sender extrinsic
    Screenshot 2023-03-27 at 10 01 13
  • Call the runtime API to generate a proof for the created DID with some keys revealed
    Screenshot 2023-03-27 at 10 01 40
  • Use the generated proof to dispatch an extrinsic on the receiving chain

How to use the runtime API with polkadot apps

There is currently no support for the new runtime API in the public polkadot apps instance. To use the runtime APIs from UI, please use our fork from the aa/dip-sender-template branch, by running yarn && yarn build && yarn start, then connecting to the sender node WS socket.
For runtime augmentation within a Node script, please use our SDK repo from the aa/dip-merkle-proof branch.

@ntn-x2 ntn-x2 self-assigned this Mar 27, 2023
Copy link
Contributor

@weichweich weichweich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we build a custom merkle tree which contains only the keys of a DID?
Would it make sense to already include the whole DID document in here? If there are future use cases we would always need to come up with a custom merkle tree. I think I have a generalized DID Merkle Tree in mind where you could prove all info from the KILT chain.

@@ -45,13 +45,27 @@ impl<Identifier, Proof, Details> From<v1::IdentityProofAction<Identifier, Proof,
}

#[derive(Encode, Decode, RuntimeDebug, Clone, Eq, PartialEq, TypeInfo)]
pub enum VersionedIdentityProof<LeafKey, LeafValue> {
pub enum VersionedIdentityProof<BlindedValue, Leaf> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this should be marked as #[non_exhaustive]?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea is that consumers would try to convert their own version into this for message passing, and receivers would try to convert it back into a specific version. I took inspiration for how VersionedXcm works. Why do you think we need non_exhaustive?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking how a new version would be introduced. What happens if a VersionedIdentityProof::v2 get's send to someone that only supports v1? at the moment that would result in a decoding error. Adding non_exhaustive wouldn't change anything here.

But if you use the dip crate as a dependency and use the VersionedIdentityProof having non_exhaustive makes a difference. If VersionedIdentityProof is not marked as non_exhaustive, adding a new version is a breaking change (since match VersionedIdentityProof {} doesn't cover this new case). Having non_exhaustive makes adding new versions a non-breaking change since everyone is forced to already have code in place that covers new versions (and showing an error that this version is not supported).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. But isnt'a a compilation error what we actually want here? If you risk getting a versioned identity proof you don't support, you should probably return an error. Code won't need to be updated if you are using the old version, hence anything failing to decode will result in an error. Once you update the dependency, you should be at least aware that a new version could be around, and explicitly generate an error if it's not what you expect. Or am I still missing your point?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The non_exhaustive is only for compile time. If we introduce a new version this would be a breaking change. Since it fails to compile.

At runtime this doesn't change anything since if we talk to a newer version that we don't know, it fails to decode and won't reach any match statement where we check versions.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

343faf9. Let's see how this plays out 😄

runtimes/common/src/dip.rs Outdated Show resolved Hide resolved
runtimes/common/src/dip.rs Outdated Show resolved Hide resolved
Base automatically changed from aa/dip-take-2 to aa/dip April 4, 2023 14:48
@ntn-x2 ntn-x2 force-pushed the aa/did-proofs branch 2 times, most recently from 9e24bbf to fc58382 Compare April 5, 2023 07:32
@ntn-x2 ntn-x2 merged commit b79d697 into aa/dip Apr 6, 2023
@ntn-x2 ntn-x2 deleted the aa/did-proofs branch April 6, 2023 13:01
ntn-x2 added a commit that referenced this pull request Apr 17, 2023
Fixes KILTprotocol/ticket#2557 and fixes
KILTprotocol/ticket#2556.

This PR builds on top of the shell PR, and adds support for Merkle proof
for DID documents.

## Merkle proof structure

A DID merkle proof is, at the core, an order set of (key, value) pairs,
on which proof-of-inclusion and proof-of-non-inclusion can be performed.
This PR generates and validates merkle proofs where leaves are of two
types:

- a DID key reference leave, whose key is the tuple (key ID, key
relationship) and the value is an empty tuple
- a DID key details leave, whose key is the key ID and the value is the
key details

For each key reference leaf with a given key ID, the proof also has to
contain a key details leaf whose key is the key ID. Multiple reference
leaves can reference the same key details leaf, optimising the storage
size.

## New runtime APIs

There is a new runtime API which the DIP sender would expose, and that
allows anyone to generate a merkle proof for a given DID identifier and
set of key IDs. The result contains the merkle root (which must match
what other chains have stored in their `pallet-dip-receiver` map), and a
merkle proof, which includes blinded values and a set of key reference
and key details leaves for the keys identified by the provided key IDs.

## How to test

The setup flow is similar to that of
#489.
Specifically:

- Set up the local Rococo network onboarding the sender and receiver
chains with para IDs 2_000 and 2_001 respectively
- Open an HRMP channel from sender 2_000 to receiver 2_001
- Create a DID on the sender chain, e.g., using the
[kilt-did-utilities](https://github.com/KILTprotocol/kilt-did-utilities)
tool
![Screenshot 2023-03-27 at 10 00
48](https://user-images.githubusercontent.com/6704504/227900994-41f0f355-84bd-4b8a-a2a8-3a9c74447e59.png)
- Push the identity of the DID to the receiver chain via the
`pallet-dip-sender` extrinsic
![Screenshot 2023-03-27 at 10 01
13](https://user-images.githubusercontent.com/6704504/227901150-7e8c9c9d-8aac-4739-8ad3-fad4ba6ff5f8.png)
- Call the runtime API to generate a proof for the created DID with some
keys revealed
![Screenshot 2023-03-27 at 10 01
40](https://user-images.githubusercontent.com/6704504/227901309-94b4dbc9-ca83-4541-820d-d1bec6adc6f0.png)
- Use the generated proof to dispatch an extrinsic on the receiving
chain

### How to use the runtime API with polkadot apps

There is currently no support for the new runtime API in the public
polkadot apps instance. To use the runtime APIs from UI, please use [our
fork from the aa/dip-sender-template
branch](https://github.com/KILTprotocol/polkadot-apps/tree/aa/dip-sender-template),
by running `yarn && yarn build && yarn start`, then connecting to the
sender node WS socket.
For runtime augmentation within a Node script, please use our [SDK repo
from the aa/dip-merkle-proof
branch](https://github.com/KILTprotocol/sdk-js/tree/aa/dip-merkle-proof).
ntn-x2 added a commit that referenced this pull request Apr 19, 2023
Fixes KILTprotocol/ticket#2557 and fixes
KILTprotocol/ticket#2556.

This PR builds on top of the shell PR, and adds support for Merkle proof
for DID documents.

## Merkle proof structure

A DID merkle proof is, at the core, an order set of (key, value) pairs,
on which proof-of-inclusion and proof-of-non-inclusion can be performed.
This PR generates and validates merkle proofs where leaves are of two
types:

- a DID key reference leave, whose key is the tuple (key ID, key
relationship) and the value is an empty tuple
- a DID key details leave, whose key is the key ID and the value is the
key details

For each key reference leaf with a given key ID, the proof also has to
contain a key details leaf whose key is the key ID. Multiple reference
leaves can reference the same key details leaf, optimising the storage
size.

## New runtime APIs

There is a new runtime API which the DIP sender would expose, and that
allows anyone to generate a merkle proof for a given DID identifier and
set of key IDs. The result contains the merkle root (which must match
what other chains have stored in their `pallet-dip-receiver` map), and a
merkle proof, which includes blinded values and a set of key reference
and key details leaves for the keys identified by the provided key IDs.

## How to test

The setup flow is similar to that of
#489.
Specifically:

- Set up the local Rococo network onboarding the sender and receiver
chains with para IDs 2_000 and 2_001 respectively
- Open an HRMP channel from sender 2_000 to receiver 2_001
- Create a DID on the sender chain, e.g., using the
[kilt-did-utilities](https://github.com/KILTprotocol/kilt-did-utilities)
tool
![Screenshot 2023-03-27 at 10 00
48](https://user-images.githubusercontent.com/6704504/227900994-41f0f355-84bd-4b8a-a2a8-3a9c74447e59.png)
- Push the identity of the DID to the receiver chain via the
`pallet-dip-sender` extrinsic
![Screenshot 2023-03-27 at 10 01
13](https://user-images.githubusercontent.com/6704504/227901150-7e8c9c9d-8aac-4739-8ad3-fad4ba6ff5f8.png)
- Call the runtime API to generate a proof for the created DID with some
keys revealed
![Screenshot 2023-03-27 at 10 01
40](https://user-images.githubusercontent.com/6704504/227901309-94b4dbc9-ca83-4541-820d-d1bec6adc6f0.png)
- Use the generated proof to dispatch an extrinsic on the receiving
chain

### How to use the runtime API with polkadot apps

There is currently no support for the new runtime API in the public
polkadot apps instance. To use the runtime APIs from UI, please use [our
fork from the aa/dip-sender-template
branch](https://github.com/KILTprotocol/polkadot-apps/tree/aa/dip-sender-template),
by running `yarn && yarn build && yarn start`, then connecting to the
sender node WS socket.
For runtime augmentation within a Node script, please use our [SDK repo
from the aa/dip-merkle-proof
branch](https://github.com/KILTprotocol/sdk-js/tree/aa/dip-merkle-proof).
@ntn-x2 ntn-x2 mentioned this pull request Apr 20, 2023
30 tasks
ntn-x2 added a commit that referenced this pull request May 22, 2023
Fixes KILTprotocol/ticket#2557 and fixes
KILTprotocol/ticket#2556.

This PR builds on top of the shell PR, and adds support for Merkle proof
for DID documents.

## Merkle proof structure

A DID merkle proof is, at the core, an order set of (key, value) pairs,
on which proof-of-inclusion and proof-of-non-inclusion can be performed.
This PR generates and validates merkle proofs where leaves are of two
types:

- a DID key reference leave, whose key is the tuple (key ID, key
relationship) and the value is an empty tuple
- a DID key details leave, whose key is the key ID and the value is the
key details

For each key reference leaf with a given key ID, the proof also has to
contain a key details leaf whose key is the key ID. Multiple reference
leaves can reference the same key details leaf, optimising the storage
size.

## New runtime APIs

There is a new runtime API which the DIP sender would expose, and that
allows anyone to generate a merkle proof for a given DID identifier and
set of key IDs. The result contains the merkle root (which must match
what other chains have stored in their `pallet-dip-receiver` map), and a
merkle proof, which includes blinded values and a set of key reference
and key details leaves for the keys identified by the provided key IDs.

## How to test

The setup flow is similar to that of
#489.
Specifically:

- Set up the local Rococo network onboarding the sender and receiver
chains with para IDs 2_000 and 2_001 respectively
- Open an HRMP channel from sender 2_000 to receiver 2_001
- Create a DID on the sender chain, e.g., using the
[kilt-did-utilities](https://github.com/KILTprotocol/kilt-did-utilities)
tool
![Screenshot 2023-03-27 at 10 00
48](https://user-images.githubusercontent.com/6704504/227900994-41f0f355-84bd-4b8a-a2a8-3a9c74447e59.png)
- Push the identity of the DID to the receiver chain via the
`pallet-dip-sender` extrinsic
![Screenshot 2023-03-27 at 10 01
13](https://user-images.githubusercontent.com/6704504/227901150-7e8c9c9d-8aac-4739-8ad3-fad4ba6ff5f8.png)
- Call the runtime API to generate a proof for the created DID with some
keys revealed
![Screenshot 2023-03-27 at 10 01
40](https://user-images.githubusercontent.com/6704504/227901309-94b4dbc9-ca83-4541-820d-d1bec6adc6f0.png)
- Use the generated proof to dispatch an extrinsic on the receiving
chain

### How to use the runtime API with polkadot apps

There is currently no support for the new runtime API in the public
polkadot apps instance. To use the runtime APIs from UI, please use [our
fork from the aa/dip-sender-template
branch](https://github.com/KILTprotocol/polkadot-apps/tree/aa/dip-sender-template),
by running `yarn && yarn build && yarn start`, then connecting to the
sender node WS socket.
For runtime augmentation within a Node script, please use our [SDK repo
from the aa/dip-merkle-proof
branch](https://github.com/KILTprotocol/sdk-js/tree/aa/dip-merkle-proof).
ntn-x2 added a commit that referenced this pull request Dec 14, 2023
Feature branch for everything DIP. It will collect other PRs until we
are happy with the features, and will add the DIP to some of our
runtimes and merge this into `develop`.

## WIP Checklist for the open tasks for v1

- [x] Basic structure ->
#489
- [x] Merkleization of DID Documents ->
#492
- [x] `RuntimeCall` verification logic ->
#502
- [x] DID signature verification ->
#516
- [x] Add support for linked accounts and web3name ->
#525
- [x] Configurable origin for `commit_identity` ->
#526
- [x] Proper fee management ->
#528
- [x] Update to Polkadot 0.9.43 ->
c18a6ce
- [x] Replace XCM with state proofs ->
#543
- [x] Add support for relaychain consumer ->
#553 (part of
#543)
- [x] Proper error handling ->
#572
- [x] Add support for versioning ->
#573
- [x] Take deposits for identity commitments ->
#574
- [x] Expose common definitions usable by consumers ->
#577
- [x] Change ensure_signed! to configurable origin also for the
`dispatch_as` function ->
#577
- [x] Proper benchmarking and weights ->
#585
- [x] Comments and docs ->
#584
- [x] Revert Dockerfile changes in
#587
- [x] [OPTIONAL] Add support for Zombienet ->
#587
- [x] [OPTIONAL] Add chain spec loading from file for template runtimes
-> #587
- [x] Big, final review ->
#494 (review)
- [x] Improvements n.1 PR ->
#591
- [x] Improvements n.2 PR ->
#592
- [x] Add to Peregrine runtime ->
#594
- [ ] Deploy on Peregrine
- [ ] Unit tests
- [ ] Add to Spiritnet runtime
- [ ] Deploy on Spiritnet
- [ ] [OPTIONAL] Move DIP-related stuff into its own repo

---------

Co-authored-by: Adel Golghalyani <[email protected]>
Co-authored-by: Chris Chinchilla <[email protected]>
Co-authored-by: Albrecht <[email protected]>
webguru9178 pushed a commit to webguru9178/kilt-node that referenced this pull request Jan 8, 2024
Feature branch for everything DIP. It will collect other PRs until we
are happy with the features, and will add the DIP to some of our
runtimes and merge this into `develop`.

## WIP Checklist for the open tasks for v1

- [x] Basic structure ->
KILTprotocol/kilt-node#489
- [x] Merkleization of DID Documents ->
KILTprotocol/kilt-node#492
- [x] `RuntimeCall` verification logic ->
KILTprotocol/kilt-node#502
- [x] DID signature verification ->
KILTprotocol/kilt-node#516
- [x] Add support for linked accounts and web3name ->
KILTprotocol/kilt-node#525
- [x] Configurable origin for `commit_identity` ->
KILTprotocol/kilt-node#526
- [x] Proper fee management ->
KILTprotocol/kilt-node#528
- [x] Update to Polkadot 0.9.43 ->
KILTprotocol/kilt-node@c18a6ce
- [x] Replace XCM with state proofs ->
KILTprotocol/kilt-node#543
- [x] Add support for relaychain consumer ->
KILTprotocol/kilt-node#553 (part of
KILTprotocol/kilt-node#543)
- [x] Proper error handling ->
KILTprotocol/kilt-node#572
- [x] Add support for versioning ->
KILTprotocol/kilt-node#573
- [x] Take deposits for identity commitments ->
KILTprotocol/kilt-node#574
- [x] Expose common definitions usable by consumers ->
KILTprotocol/kilt-node#577
- [x] Change ensure_signed! to configurable origin also for the
`dispatch_as` function ->
KILTprotocol/kilt-node#577
- [x] Proper benchmarking and weights ->
KILTprotocol/kilt-node#585
- [x] Comments and docs ->
KILTprotocol/kilt-node#584
- [x] Revert Dockerfile changes in
KILTprotocol/kilt-node#587
- [x] [OPTIONAL] Add support for Zombienet ->
KILTprotocol/kilt-node#587
- [x] [OPTIONAL] Add chain spec loading from file for template runtimes
-> KILTprotocol/kilt-node#587
- [x] Big, final review ->
KILTprotocol/kilt-node#494 (review)
- [x] Improvements n.1 PR ->
KILTprotocol/kilt-node#591
- [x] Improvements n.2 PR ->
KILTprotocol/kilt-node#592
- [x] Add to Peregrine runtime ->
KILTprotocol/kilt-node#594
- [ ] Deploy on Peregrine
- [ ] Unit tests
- [ ] Add to Spiritnet runtime
- [ ] Deploy on Spiritnet
- [ ] [OPTIONAL] Move DIP-related stuff into its own repo

---------

Co-authored-by: Adel Golghalyani <[email protected]>
Co-authored-by: Chris Chinchilla <[email protected]>
Co-authored-by: Albrecht <[email protected]>
Ad96el added a commit that referenced this pull request Feb 7, 2024
Feature branch for everything DIP. It will collect other PRs until we
are happy with the features, and will add the DIP to some of our
runtimes and merge this into `develop`.

## WIP Checklist for the open tasks for v1

- [x] Basic structure ->
#489
- [x] Merkleization of DID Documents ->
#492
- [x] `RuntimeCall` verification logic ->
#502
- [x] DID signature verification ->
#516
- [x] Add support for linked accounts and web3name ->
#525
- [x] Configurable origin for `commit_identity` ->
#526
- [x] Proper fee management ->
#528
- [x] Update to Polkadot 0.9.43 ->
c18a6ce
- [x] Replace XCM with state proofs ->
#543
- [x] Add support for relaychain consumer ->
#553 (part of
#543)
- [x] Proper error handling ->
#572
- [x] Add support for versioning ->
#573
- [x] Take deposits for identity commitments ->
#574
- [x] Expose common definitions usable by consumers ->
#577
- [x] Change ensure_signed! to configurable origin also for the
`dispatch_as` function ->
#577
- [x] Proper benchmarking and weights ->
#585
- [x] Comments and docs ->
#584
- [x] Revert Dockerfile changes in
#587
- [x] [OPTIONAL] Add support for Zombienet ->
#587
- [x] [OPTIONAL] Add chain spec loading from file for template runtimes
-> #587
- [x] Big, final review ->
#494 (review)
- [x] Improvements n.1 PR ->
#591
- [x] Improvements n.2 PR ->
#592
- [x] Add to Peregrine runtime ->
#594
- [ ] Deploy on Peregrine
- [ ] Unit tests
- [ ] Add to Spiritnet runtime
- [ ] Deploy on Spiritnet
- [ ] [OPTIONAL] Move DIP-related stuff into its own repo

---------

Co-authored-by: Adel Golghalyani <[email protected]>
Co-authored-by: Chris Chinchilla <[email protected]>
Co-authored-by: Albrecht <[email protected]>
Ad96el added a commit that referenced this pull request Apr 2, 2024
Feature branch for everything DIP. It will collect other PRs until we
are happy with the features, and will add the DIP to some of our
runtimes and merge this into `develop`.

## WIP Checklist for the open tasks for v1

- [x] Basic structure ->
#489
- [x] Merkleization of DID Documents ->
#492
- [x] `RuntimeCall` verification logic ->
#502
- [x] DID signature verification ->
#516
- [x] Add support for linked accounts and web3name ->
#525
- [x] Configurable origin for `commit_identity` ->
#526
- [x] Proper fee management ->
#528
- [x] Update to Polkadot 0.9.43 ->
c18a6ce
- [x] Replace XCM with state proofs ->
#543
- [x] Add support for relaychain consumer ->
#553 (part of
#543)
- [x] Proper error handling ->
#572
- [x] Add support for versioning ->
#573
- [x] Take deposits for identity commitments ->
#574
- [x] Expose common definitions usable by consumers ->
#577
- [x] Change ensure_signed! to configurable origin also for the
`dispatch_as` function ->
#577
- [x] Proper benchmarking and weights ->
#585
- [x] Comments and docs ->
#584
- [x] Revert Dockerfile changes in
#587
- [x] [OPTIONAL] Add support for Zombienet ->
#587
- [x] [OPTIONAL] Add chain spec loading from file for template runtimes
-> #587
- [x] Big, final review ->
#494 (review)
- [x] Improvements n.1 PR ->
#591
- [x] Improvements n.2 PR ->
#592
- [x] Add to Peregrine runtime ->
#594
- [ ] Deploy on Peregrine
- [ ] Unit tests
- [ ] Add to Spiritnet runtime
- [ ] Deploy on Spiritnet
- [ ] [OPTIONAL] Move DIP-related stuff into its own repo

---------

Co-authored-by: Adel Golghalyani <[email protected]>
Co-authored-by: Chris Chinchilla <[email protected]>
Co-authored-by: Albrecht <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants