feat: basic DIP structure #489

ntn-x2 · 2023-03-15T16:28:26Z

Fixes https://github.com/KILTprotocol/ticket/issues/2550.

This PR will merge the changes into the aa/dip branch, which will be the base branch for all the future DIP improvements until we reach a version that we can merge into develop and deploy on some testnets in the Peregrine runtime.

Base structure for DIP

This PR contains the following components:

A pallet to send identity information to one or more destinations (DIP sender)
A pallet to receive such identity information from a source (DIP receiver)
A support crate that contains common (versioned) types, in future also traits and auxiliary functions, if needed
A template project consisting of 1. a template runtime for the dip-sender, with its node counterpart, 2. a template runtime for the dip-receiver, with its node counterpart, 3. an example XCM integration test using XCM emulator (not simulator) which connects the two template runtimes with a local Rococo relay chain.

Design

The current version assumes that anyone on the sender chain can dispatch identity information on some target chain (with which there exists an open HRMP channel). Ultimately, all the fees for dispatching the message and process it on the receiver side will be paid by the extrinsic submitter on the target chain. Right now, the submitter only pays for the extrinsic dispatch on the source chain, and the chain itself uses its own balance (liked to its sovereign account) on the target chain to pay for the Transact execution. The Transact operation that interacts with the dip-receiver pallet on the target chain has an OriginKind::Native, so that there is no way for any user on the source chain to dispatch such messages.

Right now, the basic structure and the templates use defaults for everything, from proof generation to proof verification. One step at the time, all these defaults will be replaced with the actual features. Nevertheless, for two chains that have the same concept of Identifier and AccountId, it is already possible to demo the complete e2e flow. For the sake of our demonstration, the flow would go as follow, assuming that both the source and target chains are registered as parachain on the same relaychain:

The source chain requests to open an HRMP channel with the target chain
The target chain accepts the request and the channel is open
The target chain issues some funds (enough to cover the demo use case) to the sender chain's sovereign account
A user on the source chain creates a DID with a single authentication key
The same user (but could be anyone else), calls the extrinsic on the dip-sender pallet to send the merkle root of the DID to the target chain, paying for the dispatch fee
The source chain sends the information to the target chain
The user calls the dispatch_as extrinsic of the dip-receiver pallet which is deployed on the target chain, providing a merkle proof of their DID document + details on the source chain. For now this check always returns true.
Inside, the dispatch_as has a nested call which is dispatched with the new DipOrigin, which pallets can now use if they expect extrinsics to be DID-authorized.

What's coming

I created tickets for each of the next steps, under the DIP milestone. The tickets will take care of:

Adding benchmarks and weights
Introducing proper error handling
Introducing proper DID signature verificaiton on the target template chain (right now it's just a default behaviour)
Introducing proper merkle proof generation (I tested that it works, but removed those pieces from this PR for the ease of understanding)
Introducing proper merkle proof verification (same as above)
Adding an emergency brake in the sender pallet to stop dispatching messages to a specific destination, e.g., if the sender sovereign account on the destination chain does not have any more or if the Transact encoding would fail
Adding support for metadata beyond the simple DID details, e.g., proof expiration time, etc.
Replacing user-provided weights and assets with per-chain configurations stored in the sender pallet
More comments (I can work on this first if the code as it is it's too hard to follow)
Adding support for version negotiation. There is a basic concept of versioning right now, but it can and should be expanded and improved.
More unit and integration tests.

Probably more tickets will come out of this work, and will be tracked separately in the same or in future milestones.

How to test

A basic integration test of the two template runtimes is present in the dip-templates-xcm-tests crate, inside the dip-template/runtimes/xcm-tests. It can be run as usual with cargo test -p dip-templates-xcm-tests.

wilwade · 2023-03-23T15:21:34Z

pallets/pallet-dip-receiver/src/lib.rs

+			let _ = T::ProofVerifier::verify_proof_against_digest(proof, proof_digest)
+				.map_err(|_| Error::<T>::InvalidProof)?;


I don't see the call as part of the proof verification. Shouldn't the proof or something check to make sure that the call is not someone front running and actually requested by the DID owner?

Hey, yes, that's in the making! This is the skeleton PR. There's no logic yet in here, but there will be a trait implemented for the RuntimeCall type which dictates which calls require the Dip origin, and if so, which key relationship must be used in the signature.
We already use it in our blockchain to define which calls require a DID origin and which don't ->

kilt-node/runtimes/peregrine/src/lib.rs

Line 996 in e862694

impl did::DeriveDidCallAuthorizationVerificationKeyRelationship for RuntimeCall {

.

Also DID signature verification is not yet in there, but it would work similarly to what we have on our chain in the submit_did_call extrinsic:

kilt-node/pallets/did/src/lib.rs

Line 1037 in e862694

pub fn submit_did_call(

.

So the receiver chain would keep track of the last used nonce, so that it's not possible to replay actions nor front-run anything if one action is dependent on another (i.e., the first nonce has to be 1 less than the next one).

I see. Cool! Thanks!

weichweich

Nice work.

My thoughts on this:

there are a few unrelated changes, would be great to move them to a separate PR
I don't understand the ReceiverParachainDipReceiverCalls and ReceiverParachainCalls
I feel like Receiver and Sender are not the best wordings here?
- Isn't Provider and relying-party what openID uses? I feel like this would describe it better? In the end sending and receiving is only done to read the storage. If we use another strategy to access the parachains storage, this wouldn't be sending/receiving anymore
Is this future proof for new technology?
- The consensus in the community seems to be, that storage proofs should be used to read another chains storage. Here we use XCM. We should at least make sure we can switch later to the other mechanism?

dip-template/runtimes/dip-sender/src/dip.rs

nodes/parachain/Cargo.toml

pallets/pallet-dip-receiver/src/lib.rs

pallets/pallet-dip-receiver/src/origin.rs

weichweich · 2023-03-27T08:29:41Z

pallets/pallet-dip-receiver/src/traits.rs

+}
+
+// Always returns success.
+pub struct SuccessfulProofVerifier<ProofDigest, LeafKey, LeafValue>(PhantomData<(ProofDigest, LeafKey, LeafValue)>);


this should probably only be used in tests and benchmarks?

This is used in the shell template runtime, inside this PR. It could also be used in some mock environment. How would you suggest to change or feature-gate this?

in some pallets with have a mock feature for that

It is used in the actual runtime template, so we can't use any features. It will be deprecated with the new merkle proof PR, but here it's used here: https://github.com/KILTprotocol/kilt-node/pull/489/files#diff-6c40e72378ea467c86621567df431cc2dabb74d5b76ef2a25fa43a16cf568e00R32

ntn-x2 · 2023-03-27T13:53:59Z

Isn't Provider and relying-party what openID uses? I feel like this would describe it better? In the end sending and receiving is only done to read the storage. If we use another strategy to access the parachains storage, this wouldn't be sending/receiving anymore

Gotcha. I will rename them everywhere. In order to avoid merge hell, I made a ticket and will tackle this in the future. But your is a valid point, as other people have had the same confusion.

Is this future proof for new technology?

As mentioned elsewhere, versioning is not completely baked in here, and will be part of a different ticket, most likely after the other big feature milestone have been reached, e.g., call-based verification and actual DID signature verification. But we already have a ticket for versioning, where we can discuss how future-proof the implementation will be, at that point.

…ake-2

Fixes KILTprotocol/ticket#2550. **This PR will merge the changes into the `aa/dip` branch, which will be the base branch for all the future DIP improvements until we reach a version that we can merge into `develop` and deploy on some testnets in the Peregrine runtime.** ## Base structure for DIP This PR contains the following components: * A pallet to send identity information to one or more destinations (DIP sender) * A pallet to receive such identity information from a source (DIP receiver) * A support crate that contains common (versioned) types, in future also traits and auxiliary functions, if needed * A template project consisting of 1. a template runtime for the dip-sender, with its node counterpart, 2. a template runtime for the dip-receiver, with its node counterpart, 3. an example XCM integration test using [XCM emulator](https://github.com/shaunxw/xcm-simulator) (not simulator) which connects the two template runtimes with a local Rococo relay chain. ### Design The current version assumes that anyone on the sender chain can dispatch identity information on some target chain (with which there exists an open HRMP channel). Ultimately, all the fees for dispatching the message and process it on the receiver side will be paid by the extrinsic submitter on the target chain. Right now, the submitter only pays for the extrinsic dispatch on the source chain, and the chain itself uses its own balance (liked to its sovereign account) on the target chain to pay for the `Transact` execution. The `Transact` operation that interacts with the `dip-receiver` pallet on the target chain has an `OriginKind::Native`, so that there is no way for any user on the source chain to dispatch such messages. Right now, the basic structure and the templates use defaults for everything, from proof generation to proof verification. One step at the time, all these defaults will be replaced with the actual features. Nevertheless, for two chains that have the same concept of `Identifier` and `AccountId`, it is already possible to demo the complete e2e flow. For the sake of our demonstration, the flow would go as follow, **assuming that both the source and target chains are registered as parachain on the same relaychain**: 1. The source chain requests to open an HRMP channel with the target chain 2. The target chain accepts the request and the channel is open 3. The target chain issues some funds (enough to cover the demo use case) to the sender chain's sovereign account 4. A user on the source chain creates a DID with a single authentication key 5. The same user (but could be anyone else), calls the extrinsic on the `dip-sender` pallet to send the merkle root of the DID to the target chain, paying for the dispatch fee 6. The source chain sends the information to the target chain 7. The user calls the `dispatch_as` extrinsic of the `dip-receiver` pallet which is deployed on the target chain, providing a merkle proof of their DID document + details on the source chain. For now this check always returns true. 8. Inside, the `dispatch_as` has a nested call which is dispatched with the new `DipOrigin`, which pallets can now use if they expect extrinsics to be DID-authorized. ## What's coming I created tickets for each of the next steps, under the DIP milestone. The tickets will take care of: * Adding benchmarks and weights * Introducing proper error handling * Introducing proper DID signature verificaiton on the target template chain (right now it's just a default behaviour) * Introducing proper merkle proof generation (I tested that it works, but removed those pieces from this PR for the ease of understanding) * Introducing proper merkle proof verification (same as above) * Adding an emergency brake in the sender pallet to stop dispatching messages to a specific destination, e.g., if the sender sovereign account on the destination chain does not have any more or if the `Transact` encoding would fail * Adding support for metadata beyond the simple DID details, e.g., proof expiration time, etc. * Replacing user-provided weights and assets with per-chain configurations stored in the sender pallet * More comments (I can work on this first if the code as it is it's too hard to follow) * Adding support for version negotiation. There is a basic concept of versioning right now, but it can and should be expanded and improved. * More unit and integration tests. Probably more tickets will come out of this work, and will be tracked separately in the same or in future milestones. ## How to test A basic integration test of the two template runtimes is present in the `dip-templates-xcm-tests` crate, inside the `dip-template/runtimes/xcm-tests`. It can be run as usual with `cargo test -p dip-templates-xcm-tests`.

Fixes KILTprotocol/ticket#2557 and fixes KILTprotocol/ticket#2556. This PR builds on top of the shell PR, and adds support for Merkle proof for DID documents. ## Merkle proof structure A DID merkle proof is, at the core, an order set of (key, value) pairs, on which proof-of-inclusion and proof-of-non-inclusion can be performed. This PR generates and validates merkle proofs where leaves are of two types: - a DID key reference leave, whose key is the tuple (key ID, key relationship) and the value is an empty tuple - a DID key details leave, whose key is the key ID and the value is the key details For each key reference leaf with a given key ID, the proof also has to contain a key details leaf whose key is the key ID. Multiple reference leaves can reference the same key details leaf, optimising the storage size. ## New runtime APIs There is a new runtime API which the DIP sender would expose, and that allows anyone to generate a merkle proof for a given DID identifier and set of key IDs. The result contains the merkle root (which must match what other chains have stored in their `pallet-dip-receiver` map), and a merkle proof, which includes blinded values and a set of key reference and key details leaves for the keys identified by the provided key IDs. ## How to test The setup flow is similar to that of #489. Specifically: - Set up the local Rococo network onboarding the sender and receiver chains with para IDs 2_000 and 2_001 respectively - Open an HRMP channel from sender 2_000 to receiver 2_001 - Create a DID on the sender chain, e.g., using the [kilt-did-utilities](https://github.com/KILTprotocol/kilt-did-utilities) tool ![Screenshot 2023-03-27 at 10 00 48](https://user-images.githubusercontent.com/6704504/227900994-41f0f355-84bd-4b8a-a2a8-3a9c74447e59.png) - Push the identity of the DID to the receiver chain via the `pallet-dip-sender` extrinsic ![Screenshot 2023-03-27 at 10 01 13](https://user-images.githubusercontent.com/6704504/227901150-7e8c9c9d-8aac-4739-8ad3-fad4ba6ff5f8.png) - Call the runtime API to generate a proof for the created DID with some keys revealed ![Screenshot 2023-03-27 at 10 01 40](https://user-images.githubusercontent.com/6704504/227901309-94b4dbc9-ca83-4541-820d-d1bec6adc6f0.png) - Use the generated proof to dispatch an extrinsic on the receiving chain ### How to use the runtime API with polkadot apps There is currently no support for the new runtime API in the public polkadot apps instance. To use the runtime APIs from UI, please use [our fork from the aa/dip-sender-template branch](https://github.com/KILTprotocol/polkadot-apps/tree/aa/dip-sender-template), by running `yarn && yarn build && yarn start`, then connecting to the sender node WS socket. For runtime augmentation within a Node script, please use our [SDK repo from the aa/dip-merkle-proof branch](https://github.com/KILTprotocol/sdk-js/tree/aa/dip-merkle-proof).

Fixes KILTprotocol/ticket#2550. **This PR will merge the changes into the `aa/dip` branch, which will be the base branch for all the future DIP improvements until we reach a version that we can merge into `develop` and deploy on some testnets in the Peregrine runtime.** ## Base structure for DIP This PR contains the following components: * A pallet to send identity information to one or more destinations (DIP sender) * A pallet to receive such identity information from a source (DIP receiver) * A support crate that contains common (versioned) types, in future also traits and auxiliary functions, if needed * A template project consisting of 1. a template runtime for the dip-sender, with its node counterpart, 2. a template runtime for the dip-receiver, with its node counterpart, 3. an example XCM integration test using [XCM emulator](https://github.com/shaunxw/xcm-simulator) (not simulator) which connects the two template runtimes with a local Rococo relay chain. ### Design The current version assumes that anyone on the sender chain can dispatch identity information on some target chain (with which there exists an open HRMP channel). Ultimately, all the fees for dispatching the message and process it on the receiver side will be paid by the extrinsic submitter on the target chain. Right now, the submitter only pays for the extrinsic dispatch on the source chain, and the chain itself uses its own balance (liked to its sovereign account) on the target chain to pay for the `Transact` execution. The `Transact` operation that interacts with the `dip-receiver` pallet on the target chain has an `OriginKind::Native`, so that there is no way for any user on the source chain to dispatch such messages. Right now, the basic structure and the templates use defaults for everything, from proof generation to proof verification. One step at the time, all these defaults will be replaced with the actual features. Nevertheless, for two chains that have the same concept of `Identifier` and `AccountId`, it is already possible to demo the complete e2e flow. For the sake of our demonstration, the flow would go as follow, **assuming that both the source and target chains are registered as parachain on the same relaychain**: 1. The source chain requests to open an HRMP channel with the target chain 2. The target chain accepts the request and the channel is open 3. The target chain issues some funds (enough to cover the demo use case) to the sender chain's sovereign account 4. A user on the source chain creates a DID with a single authentication key 5. The same user (but could be anyone else), calls the extrinsic on the `dip-sender` pallet to send the merkle root of the DID to the target chain, paying for the dispatch fee 6. The source chain sends the information to the target chain 7. The user calls the `dispatch_as` extrinsic of the `dip-receiver` pallet which is deployed on the target chain, providing a merkle proof of their DID document + details on the source chain. For now this check always returns true. 8. Inside, the `dispatch_as` has a nested call which is dispatched with the new `DipOrigin`, which pallets can now use if they expect extrinsics to be DID-authorized. ## What's coming I created tickets for each of the next steps, under the DIP milestone. The tickets will take care of: * Adding benchmarks and weights * Introducing proper error handling * Introducing proper DID signature verificaiton on the target template chain (right now it's just a default behaviour) * Introducing proper merkle proof generation (I tested that it works, but removed those pieces from this PR for the ease of understanding) * Introducing proper merkle proof verification (same as above) * Adding an emergency brake in the sender pallet to stop dispatching messages to a specific destination, e.g., if the sender sovereign account on the destination chain does not have any more or if the `Transact` encoding would fail * Adding support for metadata beyond the simple DID details, e.g., proof expiration time, etc. * Replacing user-provided weights and assets with per-chain configurations stored in the sender pallet * More comments (I can work on this first if the code as it is it's too hard to follow) * Adding support for version negotiation. There is a basic concept of versioning right now, but it can and should be expanded and improved. * More unit and integration tests. Probably more tickets will come out of this work, and will be tracked separately in the same or in future milestones. ## How to test A basic integration test of the two template runtimes is present in the `dip-templates-xcm-tests` crate, inside the `dip-template/runtimes/xcm-tests`. It can be run as usual with `cargo test -p dip-templates-xcm-tests`.

Fixes KILTprotocol/ticket#2557 and fixes KILTprotocol/ticket#2556. This PR builds on top of the shell PR, and adds support for Merkle proof for DID documents. ## Merkle proof structure A DID merkle proof is, at the core, an order set of (key, value) pairs, on which proof-of-inclusion and proof-of-non-inclusion can be performed. This PR generates and validates merkle proofs where leaves are of two types: - a DID key reference leave, whose key is the tuple (key ID, key relationship) and the value is an empty tuple - a DID key details leave, whose key is the key ID and the value is the key details For each key reference leaf with a given key ID, the proof also has to contain a key details leaf whose key is the key ID. Multiple reference leaves can reference the same key details leaf, optimising the storage size. ## New runtime APIs There is a new runtime API which the DIP sender would expose, and that allows anyone to generate a merkle proof for a given DID identifier and set of key IDs. The result contains the merkle root (which must match what other chains have stored in their `pallet-dip-receiver` map), and a merkle proof, which includes blinded values and a set of key reference and key details leaves for the keys identified by the provided key IDs. ## How to test The setup flow is similar to that of #489. Specifically: - Set up the local Rococo network onboarding the sender and receiver chains with para IDs 2_000 and 2_001 respectively - Open an HRMP channel from sender 2_000 to receiver 2_001 - Create a DID on the sender chain, e.g., using the [kilt-did-utilities](https://github.com/KILTprotocol/kilt-did-utilities) tool ![Screenshot 2023-03-27 at 10 00 48](https://user-images.githubusercontent.com/6704504/227900994-41f0f355-84bd-4b8a-a2a8-3a9c74447e59.png) - Push the identity of the DID to the receiver chain via the `pallet-dip-sender` extrinsic ![Screenshot 2023-03-27 at 10 01 13](https://user-images.githubusercontent.com/6704504/227901150-7e8c9c9d-8aac-4739-8ad3-fad4ba6ff5f8.png) - Call the runtime API to generate a proof for the created DID with some keys revealed ![Screenshot 2023-03-27 at 10 01 40](https://user-images.githubusercontent.com/6704504/227901309-94b4dbc9-ca83-4541-820d-d1bec6adc6f0.png) - Use the generated proof to dispatch an extrinsic on the receiving chain ### How to use the runtime API with polkadot apps There is currently no support for the new runtime API in the public polkadot apps instance. To use the runtime APIs from UI, please use [our fork from the aa/dip-sender-template branch](https://github.com/KILTprotocol/polkadot-apps/tree/aa/dip-sender-template), by running `yarn && yarn build && yarn start`, then connecting to the sender node WS socket. For runtime augmentation within a Node script, please use our [SDK repo from the aa/dip-merkle-proof branch](https://github.com/KILTprotocol/sdk-js/tree/aa/dip-merkle-proof).

Fixes KILTprotocol/ticket#2550. **This PR will merge the changes into the `aa/dip` branch, which will be the base branch for all the future DIP improvements until we reach a version that we can merge into `develop` and deploy on some testnets in the Peregrine runtime.** ## Base structure for DIP This PR contains the following components: * A pallet to send identity information to one or more destinations (DIP sender) * A pallet to receive such identity information from a source (DIP receiver) * A support crate that contains common (versioned) types, in future also traits and auxiliary functions, if needed * A template project consisting of 1. a template runtime for the dip-sender, with its node counterpart, 2. a template runtime for the dip-receiver, with its node counterpart, 3. an example XCM integration test using [XCM emulator](https://github.com/shaunxw/xcm-simulator) (not simulator) which connects the two template runtimes with a local Rococo relay chain. ### Design The current version assumes that anyone on the sender chain can dispatch identity information on some target chain (with which there exists an open HRMP channel). Ultimately, all the fees for dispatching the message and process it on the receiver side will be paid by the extrinsic submitter on the target chain. Right now, the submitter only pays for the extrinsic dispatch on the source chain, and the chain itself uses its own balance (liked to its sovereign account) on the target chain to pay for the `Transact` execution. The `Transact` operation that interacts with the `dip-receiver` pallet on the target chain has an `OriginKind::Native`, so that there is no way for any user on the source chain to dispatch such messages. Right now, the basic structure and the templates use defaults for everything, from proof generation to proof verification. One step at the time, all these defaults will be replaced with the actual features. Nevertheless, for two chains that have the same concept of `Identifier` and `AccountId`, it is already possible to demo the complete e2e flow. For the sake of our demonstration, the flow would go as follow, **assuming that both the source and target chains are registered as parachain on the same relaychain**: 1. The source chain requests to open an HRMP channel with the target chain 2. The target chain accepts the request and the channel is open 3. The target chain issues some funds (enough to cover the demo use case) to the sender chain's sovereign account 4. A user on the source chain creates a DID with a single authentication key 5. The same user (but could be anyone else), calls the extrinsic on the `dip-sender` pallet to send the merkle root of the DID to the target chain, paying for the dispatch fee 6. The source chain sends the information to the target chain 7. The user calls the `dispatch_as` extrinsic of the `dip-receiver` pallet which is deployed on the target chain, providing a merkle proof of their DID document + details on the source chain. For now this check always returns true. 8. Inside, the `dispatch_as` has a nested call which is dispatched with the new `DipOrigin`, which pallets can now use if they expect extrinsics to be DID-authorized. ## What's coming I created tickets for each of the next steps, under the DIP milestone. The tickets will take care of: * Adding benchmarks and weights * Introducing proper error handling * Introducing proper DID signature verificaiton on the target template chain (right now it's just a default behaviour) * Introducing proper merkle proof generation (I tested that it works, but removed those pieces from this PR for the ease of understanding) * Introducing proper merkle proof verification (same as above) * Adding an emergency brake in the sender pallet to stop dispatching messages to a specific destination, e.g., if the sender sovereign account on the destination chain does not have any more or if the `Transact` encoding would fail * Adding support for metadata beyond the simple DID details, e.g., proof expiration time, etc. * Replacing user-provided weights and assets with per-chain configurations stored in the sender pallet * More comments (I can work on this first if the code as it is it's too hard to follow) * Adding support for version negotiation. There is a basic concept of versioning right now, but it can and should be expanded and improved. * More unit and integration tests. Probably more tickets will come out of this work, and will be tracked separately in the same or in future milestones. ## How to test A basic integration test of the two template runtimes is present in the `dip-templates-xcm-tests` crate, inside the `dip-template/runtimes/xcm-tests`. It can be run as usual with `cargo test -p dip-templates-xcm-tests`.

Fixes KILTprotocol/ticket#2557 and fixes KILTprotocol/ticket#2556. This PR builds on top of the shell PR, and adds support for Merkle proof for DID documents. ## Merkle proof structure A DID merkle proof is, at the core, an order set of (key, value) pairs, on which proof-of-inclusion and proof-of-non-inclusion can be performed. This PR generates and validates merkle proofs where leaves are of two types: - a DID key reference leave, whose key is the tuple (key ID, key relationship) and the value is an empty tuple - a DID key details leave, whose key is the key ID and the value is the key details For each key reference leaf with a given key ID, the proof also has to contain a key details leaf whose key is the key ID. Multiple reference leaves can reference the same key details leaf, optimising the storage size. ## New runtime APIs There is a new runtime API which the DIP sender would expose, and that allows anyone to generate a merkle proof for a given DID identifier and set of key IDs. The result contains the merkle root (which must match what other chains have stored in their `pallet-dip-receiver` map), and a merkle proof, which includes blinded values and a set of key reference and key details leaves for the keys identified by the provided key IDs. ## How to test The setup flow is similar to that of #489. Specifically: - Set up the local Rococo network onboarding the sender and receiver chains with para IDs 2_000 and 2_001 respectively - Open an HRMP channel from sender 2_000 to receiver 2_001 - Create a DID on the sender chain, e.g., using the [kilt-did-utilities](https://github.com/KILTprotocol/kilt-did-utilities) tool ![Screenshot 2023-03-27 at 10 00 48](https://user-images.githubusercontent.com/6704504/227900994-41f0f355-84bd-4b8a-a2a8-3a9c74447e59.png) - Push the identity of the DID to the receiver chain via the `pallet-dip-sender` extrinsic ![Screenshot 2023-03-27 at 10 01 13](https://user-images.githubusercontent.com/6704504/227901150-7e8c9c9d-8aac-4739-8ad3-fad4ba6ff5f8.png) - Call the runtime API to generate a proof for the created DID with some keys revealed ![Screenshot 2023-03-27 at 10 01 40](https://user-images.githubusercontent.com/6704504/227901309-94b4dbc9-ca83-4541-820d-d1bec6adc6f0.png) - Use the generated proof to dispatch an extrinsic on the receiving chain ### How to use the runtime API with polkadot apps There is currently no support for the new runtime API in the public polkadot apps instance. To use the runtime APIs from UI, please use [our fork from the aa/dip-sender-template branch](https://github.com/KILTprotocol/polkadot-apps/tree/aa/dip-sender-template), by running `yarn && yarn build && yarn start`, then connecting to the sender node WS socket. For runtime augmentation within a Node script, please use our [SDK repo from the aa/dip-merkle-proof branch](https://github.com/KILTprotocol/sdk-js/tree/aa/dip-merkle-proof).

Fixes KILTprotocol/ticket#2550. **This PR will merge the changes into the `aa/dip` branch, which will be the base branch for all the future DIP improvements until we reach a version that we can merge into `develop` and deploy on some testnets in the Peregrine runtime.** ## Base structure for DIP This PR contains the following components: * A pallet to send identity information to one or more destinations (DIP sender) * A pallet to receive such identity information from a source (DIP receiver) * A support crate that contains common (versioned) types, in future also traits and auxiliary functions, if needed * A template project consisting of 1. a template runtime for the dip-sender, with its node counterpart, 2. a template runtime for the dip-receiver, with its node counterpart, 3. an example XCM integration test using [XCM emulator](https://github.com/shaunxw/xcm-simulator) (not simulator) which connects the two template runtimes with a local Rococo relay chain. ### Design The current version assumes that anyone on the sender chain can dispatch identity information on some target chain (with which there exists an open HRMP channel). Ultimately, all the fees for dispatching the message and process it on the receiver side will be paid by the extrinsic submitter on the target chain. Right now, the submitter only pays for the extrinsic dispatch on the source chain, and the chain itself uses its own balance (liked to its sovereign account) on the target chain to pay for the `Transact` execution. The `Transact` operation that interacts with the `dip-receiver` pallet on the target chain has an `OriginKind::Native`, so that there is no way for any user on the source chain to dispatch such messages. Right now, the basic structure and the templates use defaults for everything, from proof generation to proof verification. One step at the time, all these defaults will be replaced with the actual features. Nevertheless, for two chains that have the same concept of `Identifier` and `AccountId`, it is already possible to demo the complete e2e flow. For the sake of our demonstration, the flow would go as follow, **assuming that both the source and target chains are registered as parachain on the same relaychain**: 1. The source chain requests to open an HRMP channel with the target chain 2. The target chain accepts the request and the channel is open 3. The target chain issues some funds (enough to cover the demo use case) to the sender chain's sovereign account 4. A user on the source chain creates a DID with a single authentication key 5. The same user (but could be anyone else), calls the extrinsic on the `dip-sender` pallet to send the merkle root of the DID to the target chain, paying for the dispatch fee 6. The source chain sends the information to the target chain 7. The user calls the `dispatch_as` extrinsic of the `dip-receiver` pallet which is deployed on the target chain, providing a merkle proof of their DID document + details on the source chain. For now this check always returns true. 8. Inside, the `dispatch_as` has a nested call which is dispatched with the new `DipOrigin`, which pallets can now use if they expect extrinsics to be DID-authorized. ## What's coming I created tickets for each of the next steps, under the DIP milestone. The tickets will take care of: * Adding benchmarks and weights * Introducing proper error handling * Introducing proper DID signature verificaiton on the target template chain (right now it's just a default behaviour) * Introducing proper merkle proof generation (I tested that it works, but removed those pieces from this PR for the ease of understanding) * Introducing proper merkle proof verification (same as above) * Adding an emergency brake in the sender pallet to stop dispatching messages to a specific destination, e.g., if the sender sovereign account on the destination chain does not have any more or if the `Transact` encoding would fail * Adding support for metadata beyond the simple DID details, e.g., proof expiration time, etc. * Replacing user-provided weights and assets with per-chain configurations stored in the sender pallet * More comments (I can work on this first if the code as it is it's too hard to follow) * Adding support for version negotiation. There is a basic concept of versioning right now, but it can and should be expanded and improved. * More unit and integration tests. Probably more tickets will come out of this work, and will be tracked separately in the same or in future milestones. ## How to test A basic integration test of the two template runtimes is present in the `dip-templates-xcm-tests` crate, inside the `dip-template/runtimes/xcm-tests`. It can be run as usual with `cargo test -p dip-templates-xcm-tests`.

Fixes KILTprotocol/ticket#2557 and fixes KILTprotocol/ticket#2556. This PR builds on top of the shell PR, and adds support for Merkle proof for DID documents. ## Merkle proof structure A DID merkle proof is, at the core, an order set of (key, value) pairs, on which proof-of-inclusion and proof-of-non-inclusion can be performed. This PR generates and validates merkle proofs where leaves are of two types: - a DID key reference leave, whose key is the tuple (key ID, key relationship) and the value is an empty tuple - a DID key details leave, whose key is the key ID and the value is the key details For each key reference leaf with a given key ID, the proof also has to contain a key details leaf whose key is the key ID. Multiple reference leaves can reference the same key details leaf, optimising the storage size. ## New runtime APIs There is a new runtime API which the DIP sender would expose, and that allows anyone to generate a merkle proof for a given DID identifier and set of key IDs. The result contains the merkle root (which must match what other chains have stored in their `pallet-dip-receiver` map), and a merkle proof, which includes blinded values and a set of key reference and key details leaves for the keys identified by the provided key IDs. ## How to test The setup flow is similar to that of #489. Specifically: - Set up the local Rococo network onboarding the sender and receiver chains with para IDs 2_000 and 2_001 respectively - Open an HRMP channel from sender 2_000 to receiver 2_001 - Create a DID on the sender chain, e.g., using the [kilt-did-utilities](https://github.com/KILTprotocol/kilt-did-utilities) tool ![Screenshot 2023-03-27 at 10 00 48](https://user-images.githubusercontent.com/6704504/227900994-41f0f355-84bd-4b8a-a2a8-3a9c74447e59.png) - Push the identity of the DID to the receiver chain via the `pallet-dip-sender` extrinsic ![Screenshot 2023-03-27 at 10 01 13](https://user-images.githubusercontent.com/6704504/227901150-7e8c9c9d-8aac-4739-8ad3-fad4ba6ff5f8.png) - Call the runtime API to generate a proof for the created DID with some keys revealed ![Screenshot 2023-03-27 at 10 01 40](https://user-images.githubusercontent.com/6704504/227901309-94b4dbc9-ca83-4541-820d-d1bec6adc6f0.png) - Use the generated proof to dispatch an extrinsic on the receiving chain ### How to use the runtime API with polkadot apps There is currently no support for the new runtime API in the public polkadot apps instance. To use the runtime APIs from UI, please use [our fork from the aa/dip-sender-template branch](https://github.com/KILTprotocol/polkadot-apps/tree/aa/dip-sender-template), by running `yarn && yarn build && yarn start`, then connecting to the sender node WS socket. For runtime augmentation within a Node script, please use our [SDK repo from the aa/dip-merkle-proof branch](https://github.com/KILTprotocol/sdk-js/tree/aa/dip-merkle-proof).

Feature branch for everything DIP. It will collect other PRs until we are happy with the features, and will add the DIP to some of our runtimes and merge this into `develop`. ## WIP Checklist for the open tasks for v1 - [x] Basic structure -> #489 - [x] Merkleization of DID Documents -> #492 - [x] `RuntimeCall` verification logic -> #502 - [x] DID signature verification -> #516 - [x] Add support for linked accounts and web3name -> #525 - [x] Configurable origin for `commit_identity` -> #526 - [x] Proper fee management -> #528 - [x] Update to Polkadot 0.9.43 -> c18a6ce - [x] Replace XCM with state proofs -> #543 - [x] Add support for relaychain consumer -> #553 (part of #543) - [x] Proper error handling -> #572 - [x] Add support for versioning -> #573 - [x] Take deposits for identity commitments -> #574 - [x] Expose common definitions usable by consumers -> #577 - [x] Change ensure_signed! to configurable origin also for the `dispatch_as` function -> #577 - [x] Proper benchmarking and weights -> #585 - [x] Comments and docs -> #584 - [x] Revert Dockerfile changes in #587 - [x] [OPTIONAL] Add support for Zombienet -> #587 - [x] [OPTIONAL] Add chain spec loading from file for template runtimes -> #587 - [x] Big, final review -> #494 (review) - [x] Improvements n.1 PR -> #591 - [x] Improvements n.2 PR -> #592 - [x] Add to Peregrine runtime -> #594 - [ ] Deploy on Peregrine - [ ] Unit tests - [ ] Add to Spiritnet runtime - [ ] Deploy on Spiritnet - [ ] [OPTIONAL] Move DIP-related stuff into its own repo --------- Co-authored-by: Adel Golghalyani <[email protected]> Co-authored-by: Chris Chinchilla <[email protected]> Co-authored-by: Albrecht <[email protected]>

Feature branch for everything DIP. It will collect other PRs until we are happy with the features, and will add the DIP to some of our runtimes and merge this into `develop`. ## WIP Checklist for the open tasks for v1 - [x] Basic structure -> KILTprotocol/kilt-node#489 - [x] Merkleization of DID Documents -> KILTprotocol/kilt-node#492 - [x] `RuntimeCall` verification logic -> KILTprotocol/kilt-node#502 - [x] DID signature verification -> KILTprotocol/kilt-node#516 - [x] Add support for linked accounts and web3name -> KILTprotocol/kilt-node#525 - [x] Configurable origin for `commit_identity` -> KILTprotocol/kilt-node#526 - [x] Proper fee management -> KILTprotocol/kilt-node#528 - [x] Update to Polkadot 0.9.43 -> KILTprotocol/kilt-node@c18a6ce - [x] Replace XCM with state proofs -> KILTprotocol/kilt-node#543 - [x] Add support for relaychain consumer -> KILTprotocol/kilt-node#553 (part of KILTprotocol/kilt-node#543) - [x] Proper error handling -> KILTprotocol/kilt-node#572 - [x] Add support for versioning -> KILTprotocol/kilt-node#573 - [x] Take deposits for identity commitments -> KILTprotocol/kilt-node#574 - [x] Expose common definitions usable by consumers -> KILTprotocol/kilt-node#577 - [x] Change ensure_signed! to configurable origin also for the `dispatch_as` function -> KILTprotocol/kilt-node#577 - [x] Proper benchmarking and weights -> KILTprotocol/kilt-node#585 - [x] Comments and docs -> KILTprotocol/kilt-node#584 - [x] Revert Dockerfile changes in KILTprotocol/kilt-node#587 - [x] [OPTIONAL] Add support for Zombienet -> KILTprotocol/kilt-node#587 - [x] [OPTIONAL] Add chain spec loading from file for template runtimes -> KILTprotocol/kilt-node#587 - [x] Big, final review -> KILTprotocol/kilt-node#494 (review) - [x] Improvements n.1 PR -> KILTprotocol/kilt-node#591 - [x] Improvements n.2 PR -> KILTprotocol/kilt-node#592 - [x] Add to Peregrine runtime -> KILTprotocol/kilt-node#594 - [ ] Deploy on Peregrine - [ ] Unit tests - [ ] Add to Spiritnet runtime - [ ] Deploy on Spiritnet - [ ] [OPTIONAL] Move DIP-related stuff into its own repo --------- Co-authored-by: Adel Golghalyani <[email protected]> Co-authored-by: Chris Chinchilla <[email protected]> Co-authored-by: Albrecht <[email protected]>

Feature branch for everything DIP. It will collect other PRs until we are happy with the features, and will add the DIP to some of our runtimes and merge this into `develop`. ## WIP Checklist for the open tasks for v1 - [x] Basic structure -> #489 - [x] Merkleization of DID Documents -> #492 - [x] `RuntimeCall` verification logic -> #502 - [x] DID signature verification -> #516 - [x] Add support for linked accounts and web3name -> #525 - [x] Configurable origin for `commit_identity` -> #526 - [x] Proper fee management -> #528 - [x] Update to Polkadot 0.9.43 -> c18a6ce - [x] Replace XCM with state proofs -> #543 - [x] Add support for relaychain consumer -> #553 (part of #543) - [x] Proper error handling -> #572 - [x] Add support for versioning -> #573 - [x] Take deposits for identity commitments -> #574 - [x] Expose common definitions usable by consumers -> #577 - [x] Change ensure_signed! to configurable origin also for the `dispatch_as` function -> #577 - [x] Proper benchmarking and weights -> #585 - [x] Comments and docs -> #584 - [x] Revert Dockerfile changes in #587 - [x] [OPTIONAL] Add support for Zombienet -> #587 - [x] [OPTIONAL] Add chain spec loading from file for template runtimes -> #587 - [x] Big, final review -> #494 (review) - [x] Improvements n.1 PR -> #591 - [x] Improvements n.2 PR -> #592 - [x] Add to Peregrine runtime -> #594 - [ ] Deploy on Peregrine - [ ] Unit tests - [ ] Add to Spiritnet runtime - [ ] Deploy on Spiritnet - [ ] [OPTIONAL] Move DIP-related stuff into its own repo --------- Co-authored-by: Adel Golghalyani <[email protected]> Co-authored-by: Chris Chinchilla <[email protected]> Co-authored-by: Albrecht <[email protected]>

ntn-x2 added 30 commits February 24, 2023 10:50

Update dependencies

afe92fd

Update node side

2538942

Peregrine runtime compiling

c585f30

Spiritnet runtime compiling

bc1da7a

Everything compiling

b427a6d

All fixes

d865e0f

Compiles

0cd10f7

Last fixes

8a43ba1

More

23ac524

fmt

71732cc

Add pallet-xcm to runtime bench script

bb58447

Clippy

f5bc271

New pallet skeleton

a266edc

First iteration

12b4fc9

Test setup

39f5696

Before switching to the Parity example

fb69836

Integrate Parity example

6e6c59a

Parity example refactoring in progress

11d6fcf

Test setup refactoring complete

a3b9236

Test environment compiling successfully

ff76493

Messages going from parachain A to parachain B

7fc85c6

Transact works! Must add WithdrawAsset instruction first

873c2d9

E2e flow works

708f249

More progress on the receiver side

e0fe62a

Compiling

7760d02

IT WORKS

53c04b7

Working again

10faad2

Add support for additional details

6c0518b

Add error handling

8646bbc

Change Transact origin to be the parachain itself

b2b0d72

Add XCM pallet to template runtimes

2097d27

ntn-x2 force-pushed the aa/dip-take-2 branch from 5bc6b39 to 2097d27 Compare March 16, 2023 13:43

ntn-x2 added ✨ new feature feature: new feature ⛓ KILT node labels Mar 16, 2023

wilwade reviewed Mar 23, 2023

View reviewed changes

weichweich reviewed Mar 27, 2023

View reviewed changes

ntn-x2 mentioned this pull request Mar 27, 2023

feat: add merkleization of DID documents #492

Merged

ntn-x2 added 2 commits March 28, 2023 10:19

Merge commit '7234aee203bc488b7e4bef7cc9cf69bfb1ba3ea4' into aa/dip-t…

22ffcc9

…ake-2

Merge commit 'df9cef8e473a0512c17954baa6e2363ce8c65231' into aa/dip-t…

41d5538

…ake-2

ntn-x2 removed ✨ new feature feature: new feature ⛓ KILT node labels Mar 28, 2023

Remove unnecessary code duplication

4d8f14d

weichweich approved these changes Apr 4, 2023

View reviewed changes

ntn-x2 merged commit 98de546 into aa/dip Apr 4, 2023

ntn-x2 deleted the aa/dip-take-2 branch April 4, 2023 14:48

ntn-x2 added the 🗺️ DIP label Apr 19, 2023

ntn-x2 mentioned this pull request Apr 20, 2023

feat: DIP v1 features #494

Merged

30 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: basic DIP structure #489

feat: basic DIP structure #489

ntn-x2 commented Mar 15, 2023

wilwade Mar 23, 2023

ntn-x2 Mar 23, 2023

ntn-x2 Mar 23, 2023

wilwade Mar 23, 2023

weichweich left a comment

weichweich Mar 27, 2023

ntn-x2 Mar 27, 2023

weichweich Mar 29, 2023

ntn-x2 Mar 30, 2023

ntn-x2 commented Mar 27, 2023

		let _ = T::ProofVerifier::verify_proof_against_digest(proof, proof_digest)
		.map_err(\|_\| Error::<T>::InvalidProof)?;

feat: basic DIP structure #489

feat: basic DIP structure #489

Conversation

ntn-x2 commented Mar 15, 2023

Base structure for DIP

Design

What's coming

How to test

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weichweich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ntn-x2 commented Mar 27, 2023