[AIP-47][Discussion] Aggregators V2

[igor-aptos]

AIP Discussion

Abstraction for enabling efficient concurrent modifications of a counter, that has an optional limit defined, and be able to efficiently extract it's value and store it in another resource.
It revamps and expands on current notion of Aggregators, and adds new notion of AggregatorSnapshots.

Specifically, allow aggregators to be used efficiently for control flow based on whether numeric operations would be executed or overflow/underflow, as well as allowing values from aggregators to be stored elsewhere, without incurring performance impact.

Read more about it here: https://github.com/aptos-foundation/AIPs/blob/main/aips/aip-47.md

[alnoki]

@igor-aptos could we use this to parallelize serial ID assignment?

E.g. every order submitted to an order book gets a unique order ID, and the txns can parallelize?

[vusirikala]

@alnoki Aggregators can be used when you are performing add/sub operations to a resource frequently. For example, for each NFT mint and burn operation, we need increment or decrement the token_supply variable. If we do this naively, the code looks like

token_supply = token_supply + 1;

Here, the transaction is reading token_supply and then writing the updated value to the token_supply. Due to this operation, the NFT mint/burn transactions become inherently sequential. Aggregators are a way to circumvent this and parallelize these types of transactions. Essentially, the idea is to let token_supply be a Aggregator<u64> instead of u64 type. The above increment operation will by replaced by aggregator_v2::add(token_supply, 1). Internally, when the virtual machine encounters such an add/sub operation on aggregator_v2, it will remember that the token_supply has to be incremented but doesn't increment immediately. Later, when the transaction is to be committed, all these aggregator operations are done.

If you want to generate universally unique 256 bit integers in a parallelizable manner, then you can use AUIDs which were introduced in AIP-36.

[alnoki]

@alnoki Aggregators can be used when you are performing add/sub operations to a resource frequently. For example, for each NFT mint and burn operation, we need increment or decrement the token_supply variable. If we do this naively, the code looks like

token_supply = token_supply + 1;

Here, the transaction is reading token_supply and then writing the updated value to the token_supply. Due to this operation, the NFT mint/burn transactions become inherently sequential. Aggregators are a way to circumvent this and parallelize these types of transactions. Essentially, the idea is to let token_supply be a Aggregator<u64> instead of u64 type. The above increment operation will by replaced by aggregator_v2::add(token_supply, 1). Internally, when the virtual machine encounters such an add/sub operation on aggregator_v2, it will remember that the token_supply has to be incremented but doesn't increment immediately. Later, when the transaction is to be committed, all these aggregator operations are done.

If you want to generate universally unique 256 bit integers in a parallelizable manner, then you can use AUIDs which were introduced in AIP-36.

@vusirikala thanks for the tip. I asked a related question in #154 (comment)

Namely I was hoping to get sequential parallelizable IDs

[igor-aptos]

Yes, you can use this (aggregators v2) to get sequential parallelizable IDs with this! that is exactly what token minting will be doing for populating index field, you can check more about that in AIP-43

[vusirikala]

@alnoki Great question. That's actually the reason we introduced the notion of AggregatorSnapshot. In the above example, the virtual machine postpones the aggregator_v2::add(token_supply, 1) operation to the very end. But what if the user wants to read token supply during the transaction? In aggregators v1, when the aggregator is read during the transaction, we will first perform all the postponed add/sub operations on the aggregator and then read the latest value of the aggregator. So, we lose parallelization benefits of the aggregator whenever the aggregator is read. In order to circumvent this, we introduced snapshot mechanism in aggregators v2. Whenever you want to read the value of the aggregator, we create a snapshot of the aggregator's state. This is done by let token_supply_snpashot = aggregators_v2::snapshot(token_supply). The token_supply_snapshot is of type AggregatorSnapshot<u64>. Here the token_supply_snapshot is only a placeholder what the aggregator's value would be when the snapshot is created. At the very end, when the transaction is committed, we materialize the snapshot's value, i.e., we perform all the outstanding operations to be done on the aggregator and compute the actual value of the aggregator.

Does that make sense? The conclusion is you could potentially use AggregatorSnapshot feature to generate sequential IDs in a parallelizable way. But how much you can parallelize the serial ID generation depends on whether you are reading or performing any operations on the serial ID after it is generated.

[alnoki]

@igor-aptos @vusirikala thank you for the explanations above

But how much you can parallelize the serial ID generation depends on whether you are reading or performing any operations on the serial ID after it is generated

In my particular case, I'd be storing the assigned serial ID in a struct field, then inserting the struct to a custom data structure. How would this affect parallelism?

[vusirikala]

@alnoki That's perfect! Aggregators are designed to parallelize these types of cases. You can instantiate your Serial ID to be an Aggregator and it should be parallel. If you are facing any issues with aggregators or if you don't feel you are achieving parallelizability for some reason, please msg me your code and I can take a look.

[alnoki]

@alnoki That's perfect! Aggregators are designed to parallelize these types of cases. You can instantiate your Serial ID to be an Aggregator and it should be parallel. If you are facing any issues with aggregators or if you don't feel you are achieving parallelizability for some reason, please msg me your code and I can take a look.

Got it, thanks!

This is moreso for a future design, but I've got this discussion bookmarked on our org repo for future consideration

[alinush]

A few concerns:

1. It's not clear to me why you need both an Aggregator and an AggregatorSnapshot. I wonder if this will confuse developers.

2. I think the problem for me is I don't really understand, from this AIP, what aggregators are for anyway. Are they just an atomic counter?

3. Is this an AIP for (1) what the API of aggregators should like, (2) the implementation of such an API, or (3) both?

4. The AIP lacks meaningful examples. I wonder how many people in our community will understand this.

[davidiw]

What elements are supported and what is the outcome of using an unsupported element?

[igor-aptos]

@davidiw - IntElement is of u64 or u128 type. calling any method with unsupported type will return EUNSUPPORTED_AGGREGATOR_TYPE / EUNSUPPORTED_AGGREGATOR_SNAPSHOT_TYPE error. Note also about a third struct that was recently added - DerivedStringSnapshot, with a padding field that makes sure it has constant serialized size. I've updated the AIP text with more details, and code references.

@alinush :

1. Aggregator is a mutable counter, that can be concurrently modified. AggregatorSnapshot is immutable value, that refers to a snapshot of an aggregator value at a given point in time. For example, in AIP-43, inside TokenV2, supply counter in the collection is of Aggregator type, while index inside of an individual token is of AggregatorSnapshot type. AggregatorSnapshot is there to replace primitive type, because if you take u64 out of an Aggregator, you loose any concurrency advantages.
2. check out AIP-43, hopefully that shows actual usage. But yeah it is a counter that you can modify without creating read-write conflicts, and that you can read in only restricted set of ways (to keep not having read-write conflicts)
3. both
4. We have a series of blog posts comming out. first one is out on medium