XCM v5 #4826

franciscoaguirre · 2024-06-19T04:13:24Z

Context

This PR aims to introduce XCMv5, for now it's in progress and will be updated over time.
This branch will serve as a milestone branch for merging in all features we want to add to XCM, roughly outlined here.
More features could be added.

TODO

Fixes #3434
Fixes #4190
Fixes #5209
Fixes #5241
Fixes #4284

paritytech-cicd-pr · 2024-09-04T15:21:07Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable 3/3
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/7256155

Implements [RFC#105](polkadot-fellows/RFCs#105) which, at the time of this PR, has not been approved yet. Some aspects might be changed as a result of discussion. ## TODO - [x] Add new instruction and use it in conversion functions - [x] Implement in xcm-executor - [x] Setup for xcm-executor unit tests - [x] Actual xcm-executor unit tests - [x] Happy path - [x] Unhappy path - [x] Emulated tests - [x] Asset hub westend - [x] Asset hub rococo - [x] Benchmarks - [x] Dummy values - [x] Actual benchmarks - [x] PRDoc --------- Co-authored-by: command-bot <> Co-authored-by: Adrian Catangiu <[email protected]>

Added SetAssetClaimer instruction --------- Co-authored-by: Francisco Aguirre <[email protected]> Co-authored-by: command-bot <> Co-authored-by: Adrian Catangiu <[email protected]>

A new instruction `InitiateTransfer` is introduced that initiates an assets transfer from the chain it is executed on, to another chain. The executed transfer is point-to-point (chain-to-chain) with all of the transfer properties specified in the instruction parameters. The instruction also allows specifying another XCM program to be executed on the remote chain. If a transfer requires going through multiple hops, an XCM program can compose this instruction to be used at every chain along the path, on each hop describing that specific leg of the transfer. **Note:** Transferring assets that require different paths (chains along the way) is _not supported within same XCM_ because of the async nature of cross chain messages. This new instruction, however, enables initiating transfers for multiple assets that take the same path even if they require different transfer types along that path. The usage and composition model of `InitiateTransfer` is the same as with existing `DepositReserveAsset`, `InitiateReserveWithdraw` and `InitiateTeleport` instructions. The main difference comes from the ability to handle assets that have different point-to-point transfer type between A and B. The other benefit is that it also allows specifying remote fee payment and transparently appends the required remote fees logic to the remote XCM. We can specify the desired transfer type for some asset(s) using: ```rust /// Specify which type of asset transfer is required for a particular `(asset, dest)` combination. pub enum AssetTransferFilter { /// teleport assets matching `AssetFilter` to `dest` Teleport(AssetFilter), /// reserve-transfer assets matching `AssetFilter` to `dest`, using the local chain as reserve ReserveDeposit(AssetFilter), /// reserve-transfer assets matching `AssetFilter` to `dest`, using `dest` as reserve ReserveWithdraw(AssetFilter), } ``` This PR adds 1 new XCM instruction: ```rust /// Cross-chain transfer matching `assets` in the holding register as follows: /// /// Assets in the holding register are matched using the given list of `AssetTransferFilter`s, /// they are then transferred based on their specified transfer type: /// /// - teleport: burn local assets and append a `ReceiveTeleportedAsset` XCM instruction to /// the XCM program to be sent onward to the `dest` location, /// /// - reserve deposit: place assets under the ownership of `dest` within this consensus system /// (i.e. its sovereign account), and append a `ReserveAssetDeposited` XCM instruction /// to the XCM program to be sent onward to the `dest` location, /// /// - reserve withdraw: burn local assets and append a `WithdrawAsset` XCM instruction /// to the XCM program to be sent onward to the `dest` location, /// /// The onward XCM is then appended a `ClearOrigin` to allow safe execution of any following /// custom XCM instructions provided in `remote_xcm`. /// /// The onward XCM also potentially contains a `BuyExecution` instruction based on the presence /// of the `remote_fees` parameter (see below). /// /// If a transfer requires going through multiple hops, an XCM program can compose this instruction /// to be used at every chain along the path, describing that specific leg of the transfer. /// /// Parameters: /// - `dest`: The location of the transfer next hop. /// - `remote_fees`: If set to `Some(asset_xfer_filter)`, the single asset matching /// `asset_xfer_filter` in the holding register will be transferred first in the remote XCM /// program, followed by a `BuyExecution(fee)`, then rest of transfers follow. /// This guarantees `remote_xcm` will successfully pass a `AllowTopLevelPaidExecutionFrom` barrier. /// - `remote_xcm`: Custom instructions that will be executed on the `dest` chain. Note that /// these instructions will be executed after a `ClearOrigin` so their origin will be `None`. /// /// Safety: No concerns. /// /// Kind: *Command*. /// InitiateTransfer { destination: Location, remote_fees: Option<AssetTransferFilter>, assets: Vec<AssetTransferFilter>, remote_xcm: Xcm<()>, } ``` An `InitiateTransfer { .. }` instruction shall transfer to `dest`, all assets in the `holding` register that match the provided `assets` and `remote_fees` filters. These filters identify the assets to be transferred as well as the transfer type to be used for transferring them. It shall handle the local side of the transfer, then forward an onward XCM to `dest` for handling the remote side of the transfer. It should do so using same mechanisms as existing `DepositReserveAsset`, `InitiateReserveWithdraw`, `InitiateTeleport` instructions but practically combining all required XCM instructions to be remotely executed into a _single_ remote XCM program to be sent over to `dest`. Furthermore, through `remote_fees: Option<AssetTransferFilter>`, it shall allow specifying a single asset to be used for fees on `dest` chain. This single asset shall be remotely handled/received by the **first instruction** in the onward XCM and shall be followed by a `BuyExecution` instruction using it. If `remote_fees` is set to `None`, the **first instruction** in the onward XCM shall be a `UnpaidExecution` instruction. The rest of the assets shall be handled by subsequent instructions, thus also finally allowing [single asset buy execution](#2423) barrier security recommendation. The `BuyExecution` appended to the onward XCM specifies `WeightLimit::Unlimited`, thus being limited only by the `remote_fees` asset "amount". This is a deliberate decision for enhancing UX - in practice, people/dApps care about limiting the amount of fee asset used and not the actually used weight. The onward XCM, following the assets transfers instructions, `ClearOrigin` or `DescendOrigin` instructions shall be appended to stop acting on behalf of the source chain, then the caller-provided `remote_xcm` shall also be appended, allowing the caller to control what to do with the transferred assets. Closes #5209 --------- Co-authored-by: Francisco Aguirre <[email protected]> Co-authored-by: command-bot <>

Relates to: #4826 Relates to: #3214 ## Description `pallet-xcm` stores some operational data that uses `Versioned*` XCM types. When we add a new XCM version (XV), we deprecate XV-2 and remove XV-3. Without proper migration, this can lead to issues with [undecodable storage](https://github.com/paritytech/polkadot-sdk/actions/runs/11381324568/job/31662577532?pr=6092), as was identified on the XCMv5 branch where XCMv2 was removed. This PR extends the existing `MigrateToLatestXcmVersion` to include migration for the `Queries`, `LockedFungibles`, and `RemoteLockedFungibles` storage types. Additionally, more checks were added to `try_state` for these types. ## TODO - [x] create tracking issue for `polkadot-fellows` polkadot-fellows/runtimes#492 - [x] Add missing `MigrateToLatestXcmVersion` for westend - [x] fix pallet-xcm `Queries` - fails for Westend https://github.com/paritytech/polkadot-sdk/actions/runs/11381324568/job/31662577532?pr=6092 - `V2` was removed from `Versioned*` stuff, but we have a live data with V2 e.g. Queries - e.g. Kusama or Polkadot relay chains ``` VersionNotifier: { origin: { V2: { parents: 0 interior: { X1: { Parachain: 2,124 } } } } isActive: true } ``` ![image](https://github.com/user-attachments/assets/f59f761b-46a7-4def-8aea-45c4e41c0a00) - [x] fix also for `RemoteLockedFungibles` - [x] fix also for `LockedFungibles` ## Follow-ups - [ ] deploy on Westend chains before XCMv5 - [ ] #6188 --------- Co-authored-by: command-bot <> Co-authored-by: GitHub Action <[email protected]> Co-authored-by: Francisco Aguirre <[email protected]>

Relates to: #4826 Relates to: #3214 `pallet-xcm` stores some operational data that uses `Versioned*` XCM types. When we add a new XCM version (XV), we deprecate XV-2 and remove XV-3. Without proper migration, this can lead to issues with [undecodable storage](https://github.com/paritytech/polkadot-sdk/actions/runs/11381324568/job/31662577532?pr=6092), as was identified on the XCMv5 branch where XCMv2 was removed. This PR extends the existing `MigrateToLatestXcmVersion` to include migration for the `Queries`, `LockedFungibles`, and `RemoteLockedFungibles` storage types. Additionally, more checks were added to `try_state` for these types. - [x] create tracking issue for `polkadot-fellows` polkadot-fellows/runtimes#492 - [x] Add missing `MigrateToLatestXcmVersion` for westend - [x] fix pallet-xcm `Queries` - fails for Westend https://github.com/paritytech/polkadot-sdk/actions/runs/11381324568/job/31662577532?pr=6092 - `V2` was removed from `Versioned*` stuff, but we have a live data with V2 e.g. Queries - e.g. Kusama or Polkadot relay chains ``` VersionNotifier: { origin: { V2: { parents: 0 interior: { X1: { Parachain: 2,124 } } } } isActive: true } ``` ![image](https://github.com/user-attachments/assets/f59f761b-46a7-4def-8aea-45c4e41c0a00) - [x] fix also for `RemoteLockedFungibles` - [x] fix also for `LockedFungibles` - [ ] deploy on Westend chains before XCMv5 - [ ] #6188 --------- Co-authored-by: command-bot <> Co-authored-by: GitHub Action <[email protected]> Co-authored-by: Francisco Aguirre <[email protected]> (cherry picked from commit efd6603)

yrong · 2024-10-29T01:22:14Z

cumulus/parachains/runtimes/bridge-hubs/common/src/message_queue.rs

@@ -23,7 +23,7 @@ use frame_support::{
 use pallet_message_queue::OnQueueChanged;
 use scale_info::TypeInfo;
 use snowbridge_core::ChannelId;
-use xcm::v4::{Junction, Location};
+use xcm::v5::{Junction, Location};


Would it be better to use prelude or latest here?

Btw: there is one minor fix in yrong@32ac7aa

@yrong yes, sure, please, you can open PR directly to the master with fixing message_queue.rs to use latest - it will help reducing XCMv5 PR size.

and to your minor fix, it is not that correct, because pools are using xcm::v4::Location, we need possibly migration for that, so using there xcm:v4 there is correct

Addresses #4284 For V5, removed `required_weight_at_most` from `Transact`. The weigher now has to decode the call inside of a transaction in order to know it's dispatch weight. It's harder to make mistakes now, since the user no longer specifies a weight value which might be wrong.

…rigin on destination (#5971) Built on top of #5876 # Description Currently, all XCM asset transfer instructions ultimately clear the origin in the remote XCM message by use of the `ClearOrigin` instruction. This is done for security considerations to ensure that subsequent (user-controlled) instructions cannot command the authority of the sending chain. The problem with this approach is that it limits what can be achieved on remote chains through XCM. Most XCM operations require having an origin, and following any asset transfer the origin is lost, meaning not much can be done other than depositing the transferred assets to some local account or transferring them onward to another chain. For example, we cannot transfer some funds for buying execution, then do a `Transact` (all in the same XCM message). In the case of XCM programs going from source-chain directly to dest-chain without an intermediary hop, we can enable scenarios such as above by using the AliasOrigin instruction instead of the ClearOrigin instruction. Instead of clearing the source-chain origin, the destination chain shall attempt to alias source-chain to "original origin" on the source chain. Most common such origin aliasing would be X1(Parachain(source-chain)) -> X2(Parachain(source-chain), AccountId32(origin-account)) for the case of a single hop transfer where the initiator is a (signed/pure/proxy) account origin-account on source-chain. This is equivalent to using the DescendOrigin instruction in this case, but also usable in the multi hop case. This allows an actor on chain A to Transact on chain B without having to prefund its SA account on chain B, instead they can simply transfer the required fees in the same XCM program as the Transact. As long as the asset transfer has the same XCM route/hops as the rest of the program, this pattern of usage can be composed across multiple hops, to ultimately Transact on the final hop using the original origin on the source chain, effectively abstracting away any intermediary hops. ### XCM `InitiateAssetsTransfer` instruction changes A new parameter `preserve_origin` to be added to the `InitiateAssetsTransfer` XCM instruction that specifies if the original origin should be preserved or cleared. ```diff InitiateAssetsTransfer { destination: Location, assets: Vec<AssetTransferFilter>, remote_fees: Option<AssetTransferFilter>, + preserve_origin: bool, remote_xcm: Xcm<()>, } ``` This parameter is explicitly necessary because the instruction should be usable between any two chains regardless of their origin-aliasing trust relationship. Preserving the origin requires some level of trust, while clearing it works regardless of that relationship. Specifying `preserve_origin: false` will always work regardless of the configured alias filters of the involved chains. # Testing - [x] e2e test: User on PenpalA registers foreign token (transacts) on PenpalB through XCM, while paying all fees using USDT (meaning XCM has to go through AssetHub) - AH carries over the original origin, effectively being a transparent proxy, - [x] e2e test: User/contract on Ethereum registers foreign token (transacts) on Polkadot-PenpalA through XCM (over bridge), while paying all fees using DOT (has to go through AssetHub) - AH carries over the original origin, effectively being a transparent proxy for Ethereum, --------- Signed-off-by: Adrian Catangiu <[email protected]> Co-authored-by: Francisco Aguirre <[email protected]> Co-authored-by: Branislav Kontur <[email protected]>

yrong · 2024-10-29T16:09:43Z

...parachains/integration-tests/emulated/tests/bridges/bridge-hub-westend/src/tests/transact.rs

+	let require_weight_at_most = Weight::from_parts(1000000000, 200000);
+	// xcm to be executed at dest
+	let xcm_on_dest = Xcm(vec![
+		Transact { origin_kind: OriginKind::Xcm, require_weight_at_most, call },


require_weight_at_most is deprecated.

Yeah, still haven't integrated that here it seems

Relates to: paritytech#4826 Relates to: paritytech#3214 ## Description `pallet-xcm` stores some operational data that uses `Versioned*` XCM types. When we add a new XCM version (XV), we deprecate XV-2 and remove XV-3. Without proper migration, this can lead to issues with [undecodable storage](https://github.com/paritytech/polkadot-sdk/actions/runs/11381324568/job/31662577532?pr=6092), as was identified on the XCMv5 branch where XCMv2 was removed. This PR extends the existing `MigrateToLatestXcmVersion` to include migration for the `Queries`, `LockedFungibles`, and `RemoteLockedFungibles` storage types. Additionally, more checks were added to `try_state` for these types. ## TODO - [x] create tracking issue for `polkadot-fellows` polkadot-fellows/runtimes#492 - [x] Add missing `MigrateToLatestXcmVersion` for westend - [x] fix pallet-xcm `Queries` - fails for Westend https://github.com/paritytech/polkadot-sdk/actions/runs/11381324568/job/31662577532?pr=6092 - `V2` was removed from `Versioned*` stuff, but we have a live data with V2 e.g. Queries - e.g. Kusama or Polkadot relay chains ``` VersionNotifier: { origin: { V2: { parents: 0 interior: { X1: { Parachain: 2,124 } } } } isActive: true } ``` ![image](https://github.com/user-attachments/assets/f59f761b-46a7-4def-8aea-45c4e41c0a00) - [x] fix also for `RemoteLockedFungibles` - [x] fix also for `LockedFungibles` ## Follow-ups - [ ] deploy on Westend chains before XCMv5 - [ ] paritytech#6188 --------- Co-authored-by: command-bot <> Co-authored-by: GitHub Action <[email protected]> Co-authored-by: Francisco Aguirre <[email protected]>

yrong · 2024-10-31T07:48:09Z

...ains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/hybrid_transfers.rs

+			PayFees { asset: fees.clone() },
+			InitiateTransfer {
+				destination: t.args.dest,
+				remote_fees: Some(AssetTransferFilter::ReserveWithdraw(fees.into())),


Furthermore, through remote_fees: Option, it shall allow specifying a single asset to be used for fees on dest chain.

Can remote_fees contains multiple assets and also to allow PayFees with multiple assets?

No, this was discussed in the ‘PayFees’ RFC and decided to keep it simple with single asset. There aren’t real examples of paying multiple fees in different assets on a single chain.

There aren’t real examples of paying multiple fees in different assets on a single chain.

Fee 1 in DOT dry run here for the execution cost(i.e. merkle the message tree) on BH.

Fee 2 in WETH to dry run submit on Ethereum.

We need these two kinds of fees as user input for building the transfer xcm on AH.

But the XCM "executed" on Ethereum and the one executed on BH are different and can have different fee assets:

Let's assume a simple WETH transfer from AH to eth (the XCM intended for Ethereum is actually irrelevant):

XcmAtEthereum = Xcm::new(vec![ WithdrawAsset(WETH, 10), PayFees(WETH, 0.01), DepositAsset(Wild(All), beneficiary), ]);

The XCM executing on BH is actually (assuming current SovereignPaidExporter):

XcmAtBridgeHub = Xcm::new(vec![ WithdrawAsset(DOT, 0.01), PayFees(DOT, 0.01), ExportMessage(XcmAtEthereum), RefundSurplus, DepositAsset(Wild(All), origin), ]);

So you can see that BH is using PayFees(DOT) and Ethereum is using PayFees(WETH).

AH executor is not aware of BH, so AH "sends directly" to Ethereum.

On AH, the instructions to be executed would have to be:

XcmAtAssetHub = Xcm::new(vec![ WhateverInstructionsBefore, ..., InitiateTransfer( destination: BridgeHub, assets: vec![AssetTransferFilter::DestinationReserve(WETH, 9.99)], remote_fees: Some(AssetTransferFilter::DestinationReserve(WETH, 0.01)), preserve_origin: false, remote_xcm: Xcm::new(vec![DepositAsset(Wild(All), beneficiary)]), ), ]);

Then the AH exporter knows it needs to send to BH, builds XcmAtBridgeHub from above, charges 0.01 DOT transport fee and sends to BH.

But it's never two assets for fees in one chain, it's one asset in two different chains

charges 0.01 DOT transport fee and sends to BH with SovereignPaidExporter

The 0.01 DOT is only to cover the execution of xcm on BH until delivered to message queue here

After that the message is async dispatched on idle and committed to header. This should be the main cost of the execution on BH and it's what the dry run here estimated for.

Say it's 0.1 DOT(10x more), so the intention here is not to mix the two portions of fees together(i.e. 0.1+0.01=0.11 DOT), we minimize the static config of delivery fee here as 0.01 DOT and make the 0.1 DOT a user input.

Anyway, we can workaround it with the single fee asset. UI just convert the 0.1 DOT to WETH and added that to the remote fee portion when building xcm on AH.

franciscoaguirre added 5 commits June 5, 2024 15:07

feat(xcm-v5): skeleton

b01161f

feat(xcm-v3): no longer use v2 types

0b83e68

feat: remove XCMv2

e917d67

Merge branch 'master' into xcm-v5

9005e04

fix(xcm-v5): fix most compilation errors

5fc4a32

franciscoaguirre added the T6-XCM This PR/Issue is related to XCM. label Jun 26, 2024

franciscoaguirre added 12 commits June 26, 2024 17:39

Merge branch 'master' into xcm-v5

337e26e

fix(pallet-xcm): change V4 to V5

19cc1da

Merge branch 'master' into xcm-v5

5cfd108

Merge branch 'master' into xcm-v5

96b5b4b

Merge branch 'master' into xcm-v5

6de7a42

fix(xcm-v5): change v4 to v5

126ef2f

fix: more v4 to v5 conversions

a16e19f

Merge branch 'master' into xcm-v5

a3a09af

fix: more v4 to v5 conversions

bbba323

fix: more v4 to v5 conversions

98c3906

fix(xcm-builder): use previously unused function

08a82d1

fix(xcm-executor): change v4 to v5

1f031ec

acatangiu mentioned this pull request Aug 19, 2024

[XCMv5]Remove XCM testnet NetworkIds #5390

Open

franciscoaguirre force-pushed the xcm-v5 branch from 9bec263 to 1f031ec Compare August 20, 2024 13:07

franciscoaguirre added 5 commits August 21, 2024 18:08

Merge branch 'master' into xcm-v5

5b423ad

Merge branch 'master' into xcm-v5

1116cbb

Merge branch 'master' into xcm-v5

cfa2ef4

Merge branch 'master' into xcm-v5

241e992

Merge branch 'master' into xcm-v5

d421d3f

franciscoaguirre added 2 commits October 1, 2024 08:04

Merge branch 'master' into xcm-v5

3e85791

fix: v4 -> v5

f80c76a

bkontur mentioned this pull request Oct 8, 2024

Enhance interoperability for pallet-assets #5969

Open

franciscoaguirre self-assigned this Oct 8, 2024

bkontur mentioned this pull request Oct 17, 2024

Fix migrations for pallet-xcm #6099

Closed

5 tasks

franciscoaguirre and others added 5 commits October 17, 2024 15:35

Merge branch 'master' of github.com:paritytech/polkadot-sdk into xcm-v5

f481f8e

Merge branch 'master' into xcm-v5

678a379

Merge branch 'master' of github.com:paritytech/polkadot-sdk into xcm-v5

4d9cc77

fix merge dmg

cdfecb7

bkontur mentioned this pull request Oct 21, 2024

Fix migrations for pallet-xcm #6148

Merged

7 tasks

x3c41a and others added 4 commits October 21, 2024 16:16

Added SetAssetClaimer Instruction to XCM v5 (#5585)

446465b

Added SetAssetClaimer instruction --------- Co-authored-by: Francisco Aguirre <[email protected]> Co-authored-by: command-bot <> Co-authored-by: Adrian Catangiu <[email protected]>

Merge branch 'master' into xcm-v5

4a6e85c

Merge branch 'master' into xcm-v5

10d6795

yrong reviewed Oct 29, 2024

View reviewed changes

franciscoaguirre and others added 3 commits October 29, 2024 11:43

Merge branch 'master' into xcm-v5

e8807a3

yrong reviewed Oct 29, 2024

View reviewed changes

Merge branch 'master' into xcm-v5

4c29910

yrong reviewed Oct 31, 2024

View reviewed changes

Merge branch 'master' into xcm-v5

d2a98f4

franciscoaguirre marked this pull request as ready for review October 31, 2024 13:20

franciscoaguirre requested review from a team as code owners October 31, 2024 13:20

paritytech-review-bot bot requested a review from a team October 31, 2024 13:21

Merge branch 'master' into xcm-v5

a9762d6

yrong mentioned this pull request Nov 1, 2024

Outbound queue v2 yrong/polkadot-sdk#4

Open

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

XCM v5 #4826

XCM v5 #4826

franciscoaguirre commented Jun 19, 2024 •

edited

Loading

paritytech-cicd-pr commented Sep 4, 2024

yrong Oct 29, 2024 •

edited

Loading

bkontur Oct 29, 2024

yrong Oct 29, 2024

franciscoaguirre Oct 31, 2024

yrong Oct 31, 2024

acatangiu Oct 31, 2024

yrong Oct 31, 2024

acatangiu Oct 31, 2024

franciscoaguirre Oct 31, 2024

yrong Oct 31, 2024 •

edited

Loading

XCM v5 #4826

Are you sure you want to change the base?

XCM v5 #4826

Conversation

franciscoaguirre commented Jun 19, 2024 • edited Loading

Context

TODO

paritytech-cicd-pr commented Sep 4, 2024

yrong Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yrong Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

franciscoaguirre commented Jun 19, 2024 •

edited

Loading

yrong Oct 29, 2024 •

edited

Loading

yrong Oct 31, 2024 •

edited

Loading