Introduce Pallet paged-list

[ggwpez]

Abstract

Introduces a pallet to provide a linked list data structure: pallet-paged-list. The elements are aggregated into pages to keep a low PoV footprint on reading. The main advantage is a smaller memory footprint within the runtime when a transaction errors and causes a storage rollback.

(Detailed implementation documentation is in the Rust Code)

API

The external API is defined by trait StoragePagedList:

pub trait StoragePagedList<V: FullCodec> {
	/// Iterator type to loop over all elements in FIFO order.
	type Iterator: Iterator<Item = V>;
	/// Fast append iterator to append multiple elements.
	type Appender: StorageAppender<V>;

	/// List all elements in append (FIFO) order.
	fn iter() -> Self::Iterator;

	/// Drain the elements.
	fn drain() -> Self::Iterator;

	/// A fast append iterator.
	fn appender() -> Self::Appender;

	/// Append a single element.
	///
	/// Should not be called repeatedly; use `append_many` instead.
	fn append_one<EncodeLikeValue>(item: EncodeLikeValue)
	where
		EncodeLikeValue: EncodeLike<V>,
	{
		Self::append_many(core::iter::once(item));
	}

	/// Append many elements.
	///
	/// Should not be called repeatedly; use `appender` instead.
	fn append_many<EncodeLikeValue, I>(items: I)
	where
		EncodeLikeValue: EncodeLike<V>,
		I: IntoIterator<Item = EncodeLikeValue>,
	{
		let mut ap = Self::appender();
		ap.append_many(items);
	}
}

And specifically trait StorageAppender to conveniently append values:

pub trait StorageAppender<V: FullCodec> {
	fn append<EncodeLikeValue>(&mut self, item: EncodeLikeValue)
	where
		EncodeLikeValue: EncodeLike<V>;

	// Note: this has a default impl, as an `Appender` is already assumed to be fast for appending.
	fn append_many<EncodeLikeValue, I>(&mut self, items: I)
	where
		EncodeLikeValue: EncodeLike<V>,
		I: IntoIterator<Item = EncodeLikeValue>,
	{
		for item in items.into_iter() {
			self.append(item);
		}
	}
}

Testing

Fuzzer tests are in place to test random sequences of appending and removal of elements. The fuzzer tests are currently not running in the CI.

[kianenigma]

In terms of organization, this is very similar to what pallet-bags-list is providing.

Perhaps we can put their associated traits in frame-support::collectio ns or something similar, with a more holistic documentation, explaining what each does and when they should each be used. If not done here, would be great if you make an issue about it and assign me.

I recall the queue in message-queue was also something that I imagined could be extracted as one of these "data structure as a pallet" thingies.

[ggwpez]

bot rebase

[paritytech-processbot]

Rebased

[ggwpez]

bot merge

[paritytech-processbot]

Waiting for commit status.

[paritytech-processbot]

Merge cancelled due to error. Error: Statuses failed for dbceffa

[ggwpez]

bot merge

[paritytech-processbot]

Waiting for commit status.

[paritytech-processbot]

Merge cancelled due to error. Error: Statuses failed for 551ad22

[ggwpez]

bot merge

[xlc]

This pallet only holds a single storage. It should be possible to make the PagedList struct to be generic to take storage as generic parameter. So that the storage can be defined in other pallets and have no need to have an actual paged list pallet here. This can help reduce the complexity (all related storages are defined in a single pallet) and reduce numbers of pallets, which we have a hard cap of 255.

This is an example https://github.com/open-web3-stack/open-runtime-module-library/blob/58146e4a227c4cc4c49669a749dc04ee54d8f009/utilities/src/linked_item.rs

[bkchr]

Yes, why isn't this just another storage primitive?

[ggwpez]

Yes the pallet could provide multiple lists per instance. We can still add that. Currently it was designed in the simplest way possible to allow for a fix to be deployed in a timely manner.

It could also have been done as FRAME primitive, but then it needs support in the metadata.

[bkchr]

But now it is also not supported in metadata :P