Dynamic fees for bridges-v1

[svyatonik]

https://gist.github.com/svyatonik/6d4d9f58b800ac951dcac6c1aa827c67#file-bridgequeues-png

This PR introduces all logic and components required for dynamic fees in v1, that we may build in this repo:

• message delivery transaction will be rejected if message dispatcher is "inactive". Message dispatcher must become "inactive" when the XCM channel (XCMP) between target bridge hub and target asset hub is suspended. This suspension occurs e.g. when the target asset hub is unable to process inbound XCM messages in a timely fashion. Review hints: XcmBlobMessageDispatch and messages pallet;
• added artificial limit to the bridge outbound queue. When it is full, we stop accepting new bridge messages. This can occur e.g. if relayers are not running or if dispatcher at the target bridge hub is "inactive" and relayers can't deliver any messages. Review hints: XcmBlobHaulerAdapter, LocalXcmQueueManager, LocalXcmQueueMessageProcessor and LocalXcmQueueSuspender;
• a new pallet that must be deployed at asset hubs (at sending chains) - pallet-xcm-bridge-hub-router. Now it exists only to support dynamic fees. In the future, we'll (probably) add a support for dynamic bridges to it. Dynamic fee is based on XCM: Properly set the pricing for the DMP router polkadot#6843. We start increasing fee factor when the channel between sending asset hub and its sibling bridge hub is suspended and/or has many queued messages.

TODOs in this PR:

• tests;
• review remaining TODOs in the code;
• weights for the new pallet;
• add support of delivery transaction logic changes to our signed extensions;
• add support of delivery transaction logic changes to our relay I'd like to do it in a separate PR. It isn't actually necessary, because signed extensions will do the job, but desirable to make relayer "polite".

But before investing a time in that ^^^, I'd like to check if all my assumptions are correct and test it using local chains. For that, we need:

• add new branch to Cumulus, with this code and Use Message Queue as DMP and XCMP dispatch queue cumulus#2157 (started work on that: https://github.com/paritytech/cumulus/compare/sv-bridges-and-new-message-queue: this branch contains updated bridges subtree (this PR) + oty-message-queue + bko-transfer-asset-via-bridge-pallet-xcm + bridge-hub-kusama-polkadot);
• implement IsChannelActive for outbound part of XCM queue between target BH and target AH;
• implement LocalXcmQueue for outbound part of XCM queue between source AH and source BH;
• add some scripts to send many messages and check how the whole scheme reacts to that. E.g.: ensure that every queue has a backpressure mechanism and it works as expected, ensure that all bridge queues works as a whole (they're able to trigger backpressure mechanism at the first queue eventually). See Dynamic fees for bridges-v1 #2294 (comment)

[svyatonik]

Another dependency we'll need for that is paritytech/cumulus#2342. So idea is:

1. bridge backpressure mechanism involves many chains, so it isn't instant - it may take some to suspend the queue between source AH and source BH. And we can't allow senders to send messages over the bridge for a constant cost while this mechanism activates;
2. so we need to charge dynamic fees for sending messages from source AH and source BH even when we don't have any info about other bridge queues. That's where the mentioned PR should help us - the total transport cost that the sender pays is the BridgeFee + HrmpFee. So BridgeFee will grow when we have a report about overloaded bridge queues and HrmpFee shall be maintained independently

[svyatonik]

I've been able finally to test the whole bridge pipeline with backpressure mechanism enabled. Here's how it works (from the practice perspective):

1. inbound messages at target AH qre queued for later processing in the pallet_message_queue. When the number of queued messages reaches 200 (see cumulus_pallet_xcmp_queue::QueueConfigData::suspend_threshold), the inbound channel with the sibling BH is suspended;
2. when the target BH receives the "queue-suspended" signal from the sibling AH, it starts rejecting all further message delivery transactions. I've also added a (questionable) limit to the number of outbound "pages" (page = concatenated XCM messages, limited by size) in the target BH -> target AH queue, which will also trigger the reject of all further message delivery transacitons;
3. since relayer can't deliver messages, they're piling up at the outbound bridge queue of source BH (the same happens if target chains are OK, but there's no relayer running). When the number of queued messages reaches the MAX_ENQUEUED_MESSAGES_AT_OUTBOUND_LANE (currently set to 300, but probably needs to be increased to maximal number of messages in delivery tx, which is 4096 iirc), it stops processing new XCM messages from the sibling (source) AH;
4. since inbound messages are not processing, they keep piling up at the pallet_message_queue of the source BH. As in (1), when the number of unprocessed messages reaches 200, suspension signal is sent to the sibling AH;
5. when sibling AH receives this signal, it stops send messages to the source BH and they start piling up at the outbound XCM queue (cumulus_pallet_xcmp_queue). When next message is sent over the bridge, we will start increasing the fee.

Things to notice:

1. it may take some time to propagate the the-pipeline-is-overloaded fact back to the source AH (e.g. if the target BH -> target AH queue is overloaded). During that time, the messages can be sent over the bridge with the base fee. However, there's also (needs to be) Increase xcm fees when busy cumulus#2342, which will trigger exponential cost for sending messages from source AH to source BH. So the total fee will be BridgeFee + HrmpFee. They both have their own exponential factor, changing independently;
2. there are other (sometimes faster) ways to report bridge queues state to the source AH to increase the fee factor. But they are all non-instant (e.g. because they depend on finality proofs of multiple chains). So the backpressure seems the most natural mechanism here
3. in the end, the main "theoretical" question here is what we're going to achieve with dynamic fees. One of parameters here is limits of all involved queues. Do we think it is ok to fill up queues to their limits with the base fee? Or maybe we want to start increasing fees before reaching limits? If the latter, then maybe we need to apply some (much smaller comparing to overloaded) exponential factor for every sent message. And decrease the factor by "normal messages per block" from the on_initialize?

[svyatonik]

Marked onice because of dependencies

[alvicsam]

bot fmt

[alvicsam]

bot clean

[svyatonik]

This code is executed at the target bridge hub. I suggest to use the following code for the Channel implementation:

pub struct LocalXcmChannelWithSiblingAssetHub;

impl bp_xcm_bridge_hub_router::XcmChannelStatusProvider for LocalXcmChannelWithSiblingAssetHub {
	fn is_congested() -> bool {
		// let's find the channel with the sibling AH
		let sibling_asset_hub_id: cumulus_primitives_core::ParaId = xcm_config::SiblingAssetHubParId::get().into();
		let outbound_channels = cumulus_pallet_xcmp_queue::OutboundXcmpStatus::<Runtime>::get();
		let channel_with_sibling_asset_hub = outbound_channels.iter()
			.filter(|c| c.recipient() == sibling_asset_hub_id)
			.next();

		// no channel => it is empty, so not congested
		let channel_with_sibling_asset_hub = match channel_with_sibling_asset_hub {
			Some(channel_with_sibling_asset_hub) => channel_with_sibling_asset_hub,
			None => return false,
		};

		// suspended channel => it is congested
		if channel_with_sibling_asset_hub.is_suspended() {
			return true;
		}

		// TODO: the following restriction is arguable, we may live without that, assuming that there
		// can't be more than some `N` messages queued at the bridge queue (at the source BH) AND before
		// accepting next (or next-after-next) delivery transaction, we'll receive the suspension signal
		// from the target AH and stop accepting delivery transactions

		// it takes some time for target AH to suspend inbound channel with the target BH and during that
		// we will keep accepting new message delivery transactions. Let's also reject new deliveries if
		// there are too many "pages" (concatenated XCM messages) in the target BH -> target AH queue.
		const MAX_QUEUED_PAGES_BEFORE_DEACTIVATION: u16 = 4;
		if channel_with_sibling_asset_hub.queued_pages() > MAX_QUEUED_PAGES_BEFORE_DEACTIVATION {
			return true;
		}

		true
	}
}

[svyatonik]

This code is executed at the source (aka sending) chain. In our case it is the source asset hub. I suggest to use the following code as theWithBridgeHubChannel:

	pub struct LocalXcmpChannelAdapter;

	impl bp_xcm_bridge_hub_router::XcmChannelStatusProvider for LocalXcmpChannelAdapter {
		fn is_congested() -> bool {
			// if the outbound channel with recipient is suspended, it means that one of further
			// bridge queues (e.g. bridge queue between two bridge hubs) is overloaded, so we shall
			// take larger fee for our outbound messages
			let sibling_bridge_hub_id: cumulus_primitives_core::ParaId = BridgeHubKusamaParaId::get().into();
			let outbound_channels = cumulus_pallet_xcmp_queue::OutboundXcmpStatus::<Runtime>::get();
			let outbound_channel = outbound_channels.iter()
				.filter(|c| c.recipient() == sibling_bridge_hub_id)
				.next();
			let is_outbound_channel_suspended = outbound_channel.clone().map(|c| c.is_suspended()).unwrap_or(false);
			if is_outbound_channel_suspended {
				return true;
			}

			// if the inbound channel with recipient is suspended, it means that we are unable to receive
			// congestion reports from the bridge hub. So we assume the bridge pipeline is congested too
			let inbound_suspended = cumulus_pallet_xcmp_queue::InboundXcmpSuspended::<Runtime>::get();
			let is_inbound_channel_suspended = inbound_suspended.contains(&sibling_bridge_hub_id);
			if is_inbound_channel_suspended {
				return true;
			}

			// TODO: https://github.com/paritytech/cumulus/pull/2342 - once this PR is merged, we may
			// remove the following code
			//
			// if the outbound channel has at least `N` pages enqueued, let's assume it is congested.
			// Normally, the chain with a few opened HRMP channels, will "send" pages at every block.
			// Having `N` pages means that for last `N` blocks we either have not sent any messages,
			// or have sent signals.
			const MAX_OUTBOUND_PAGES_BEFORE_CONGESTION: u16 = 4;
			let is_outbound_channel_congested = outbound_channel.map(|c| c.queued_pages()).unwrap_or(0);
			is_outbound_channel_congested > MAX_OUTBOUND_PAGES_BEFORE_CONGESTION
		}
	}

[svyatonik]

The base fee then need to include both bridge fee and HRMP fee. And then instead of the BridgeFee * BridgeFeeFactor + HrmpFee * HrmpFeeFactor, we'll temporary have the (BridgeFee + HrmpFee) * BridgeFeeFactor until paritytech/cumulus#2342 is implemented. This is fine, because the exponential factor is present in the formula.

Also keep in mind that the actual HRMP fee is currently set to zero (type PriceForSiblingDelivery = ()), so this is even better for us.

[bkontur]

bot fmt

[command-bot]

@bkontur https://gitlab.parity.io/parity/mirrors/parity-bridges-common/-/jobs/3360002 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/fmt/fmt.sh". Check out https://gitlab.parity.io/parity/mirrors/parity-bridges-common/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 22-ad69847f-27aa-4c4d-b127-6977bd06c682 to cancel this command or bot cancel to cancel all commands in this pull request.

[command-bot]

@bkontur Command "$PIPELINE_SCRIPTS_DIR/commands/fmt/fmt.sh" has finished. Result: https://gitlab.parity.io/parity/mirrors/parity-bridges-common/-/jobs/3360002 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/parity-bridges-common/-/jobs/3360002/artifacts/download.

[bkontur]

bot rebase

[Polkadot-Forum]

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/polkadot-kusama-bridge/2971/5