Abnormal gas usage of ProveCommitAggregate compared with ProveCommitSector

[zhiqiangxu]

We found that the average cost of ProveCommitAggregate is higher than ProveCommitSector, mainly because the gas usage of ProveCommitAggregate is higher than N times the gas usage of ProveCommitSector, here're two typical transactions:

ProveCommitAggregate(which aggregates 16 sectors):
https://filfox.info/en/message/bafy2bzacebejx2n2jjzfn3d5xtwe54qy6j35txjiofdgcsuwt54cmh2uox6du

ProveCommitSector:
https://filfox.info/en/message/bafy2bzacedf6z7ll6t5wgy34tltebjk7ucsm5fahqquy32vpnwwgw2gz3spzy

The gas usage of ProveCommitAggregate is 2,778,005,962, of ProveCommitSector is 63,750,572, the average gas cost(ignoring aggregate_fee for now) is 2,778,005,962/(63,750,572*16)=2.723510820028407 times that of ProveCommitSector.

A possible reason is that for ProveCommitSector, the gas cost is mostly decided by this empirical constant, and activation is delayed to cron for free; But for ProveCommitAggregate, even though there's also a empirical constant, the activation gas is also paid by the miner directly.

So we guess that current activation gas is much higher than anticipated earlier, and we've stopped using ProveCommitAggregate for DC sectors(CC sectors don't have deals to activate).

[jennijuju]

Your observation is correct and we do have a regression on this feature atm - have you seen filecoin-project/builtin-actors#1276, we have some context there?

[zhiqiangxu]

Thanks for sharing the link, good to know this has been confirmed.

[anorth]

While optimisations to sector activation probably won't need a FIP, they won't resolve the imbalance between the methods. That will require a FIP so this is the appropriate discussion venue.

[steven004]

This looks a high priority issue. The base-fee will get higher if storage providers stop prove-aggregation, which may make the chain quality worse.

[anorth]

This note expands an item from Cron safety - survey of the problem and solutions. It describes why and how to to remove from cron the expensive sector and deal activation work associated with onboarding via the single-sector ProveCommit method. This:

• resolves an imbalance in the sector activation gas subsidy available to different onboarding methods; and
• is a pre-requisite for any kind of push-style events associated with sector activation for user-programmed actors/contracts;
• removes some unnecessary work from cron, as one step towards bounding its cost.

Developed in collaboration with @ZenGround0.

Background

Single PoRep proofs are validated asynchronously via a cron call to the power actor. The asynchrony is to enable the use of multiple CPU cores by verifying multiple proofs submitted that epoch in parallel. Blockchain execution is usually single-threaded, so multiple processor cores are seen as an otherwise wasted resource. After a proof is verified the sector is activated back with the miner actor, all in the power actor’s cron handler.

By contrast, when submitting an aggregated PoRep, all of the proof validation and sector activation is executed synchronously and billed to the caller.

Problems

We explicitly charge gas (discounted) for the single proof validation when it is submitted, so it’s both bounded and paid for. However, the state updates associated with sector activation are not metered, because they execute in cron. The execution cost of this activation varies significantly with the deal content of the sector, but the storage provider doesn’t pay gas for this. On the other hand, aggregated proof validation also activates the sector synchronously, for which the SP pays the gas cost.

Thus single-sector onboarding is subsidised relative to aggregation, but aggregation would otherwise efficiently support much higher onboarding rates.

Deal activation is cron only (kind-of) ok at the moment because the built-in market is the only market, and it offers no facility to synchronously invoke (notify) user actors/contracts during deal activation. To better support deal-related user actors we would like support activation notifications to untrusted code, which we cannot do from cron.

Proposal

Ultimately the SP must pay the gas costs associated with sector activation, including deal activation. The present cron subsidy must be removed.

Synchronous sector activation

Add a new ProveCommitAndActivate method which validates the PoRep and synchronously activates a single sector. This method will not enjoy the verification gas discount for asynchronous validation, but is essentially the same workflow as today. (Assumption: this is cheaper than using ProveCommitAggregate with a single proof).

The existing ProveCommitSector method must be deprecated in a subsequent upgrade.

This will have the effect of removing the subsidy to single ProveCommitSector calls relative to aggregate proofs. The result is increased cost of the single-verification method. This will resolve the imbalance and make ProveCommitAggregate again preferred when onboarding multiple sectors. However, it won’t reduce any SP costs.

Optimise deal activation costs

Resolving the imbalance results in an aggregate increase in onboarding gas costs, as the SP must pay for the deal activation work currently hidden in cron.

We should also optimise the on-chain work that’s now uniformly paid for by the SP. Details of are deferred to filecoin-project/builtin-actors#1276 (we need to profile and understand the costs first).

Asynchronous sector activation (optional)

Add a new ProveCommitAsync method which schedules a proof for asynchronous validation via the power actor cron, but does not activate the sector. This method can offer proof verification at a discount, accounting for the parallel verification potential.

Internally, instead of calling the current ConfirmSectorProofsValid method (which activates the sector), the power actor calls a new miner actor method which stores the validated sector number in miner state but does not activate the sector. This introduces a new state in sector onboarding: validated but not yet active. Such a sector has no power and is not scheduled for WindowPoSt.

Add a new ActivateSectors method which activates sectors that have validated proofs. This includes activating the deals which, today, are referenced in the sectors’ on-chain info. The SP calls this and pays the gas cost.

This will restore the global efficiency gain of asynchronous, batched proof validation, but requires an SP to submit a total of three messages to complete sector onboarding (or four, with deals).

This would introduce protocol and operational complexity which may not be worth the gain. It’s also a stepping stone to more functionality.

Sealing as a service support (optional)

Add ProveCommitAggregateOnly method which verifies an aggregated PoRep but does not activate the sectors. The sectors may be subsequently activated via ActivateSectors. The existing ProveCommitAggregate method continues to do both at once.

This method efficiently supports sealing as a service. The sealer may submit proof verification on chain, then take their time to transport the physical sector to the miner, who then activates it when they are ready to submit regular WindowPost.

The motivation for this is from discussion #570.

Deadline assignment support (optional)

Add parameters to ActivateSectors to support the SP nominating a WindowPoSt deadline to which the sectors are to be assigned. SPs have requested this, and it may significantly reduce off-chain staffing costs associated with production monitoring and failure resolution. See #505.

TODO: determine if SP control over WindowPoSt scheduling is safe; initial design assumed not.

Future-proofing

Future changes for improved storage programmability (sector events/notifications) will need additional parameters to sector activation methods (ActivateSector, ProveCommitSync, and the existing methods) to include a list of PieceCIDs and associated actor addresses and piece identifiers to be notified about piece inclusion in an activated sector. We could probably add these to the parameter object for this method now, but require them to be empty.

[anorth]

An even simpler variant is to always use ProveCommitAggregate, even for single sectors. I assume that this is notably more expensive than verifying a proof directly, but if that assumption is wrong then this could work well and further simplify things.

[f8-ptrk]

we need to take care that all that works for the replica update route of doing deals too

[anorth]

Replica updates already pay activation gas, so there's no "problem" there.

Is separating proof from activation relevant for replica updates? This seems less likely a candidate for SAAS.

[anorth]

Note that asynchronous sector activation (single or batch) will require some security analysis. The ability to defer Window PoSt arbitrarily, and to activate a very large amount of storage at once without the throughput limitations of sealing may affect security properties. cc @nicola @irenegia @lucaniz

[jennijuju]

Quick question, while the optimization part of filecoin-project/builtin-actors#1276 is being worked on. Our team is currently not prioritizing filecoin-project/builtin-actors#1277:

• The original issue is a protocol bug described in the issue and should eventually be fixed.
• However, one of the top complaints we hear from the SP community is that the onboarding cost is high - there are multiple efforts ongoing to see how we can reduce it from the protocol level.
• Now, while Remove deal/sector activation from cron builtin-actors#1277 is the right thing to do, given currently ProveCommit sector is activating deals for free, and it wont be possible to make deal activarion ProveCommitAggregation to be cheaper than free.
• In other words, today SP can choose to use ProveCommit over ProveCommitAggregation even when the base fee reaches the batch balancer, so to leverage the FREE activation for a much cheaper gas cost, fixing #1277 or Abnormal gas usage of `ProveCommitAggregate` compared with `ProveCommitSector` #689 today will remove that advantage.

With that being said, I am curious if the community prefers correctness soon on this so that you can leverage Aggregation!

[steven004]

From my personal opinion, I would think this is the right direction, to make it more reasonable.

[jennijuju]

opened #820 for mitigating this undesired behaviour

[jennijuju]

@anorth - have a q when writing #820.

In the past, we have always been keeping an outdated method for an upgrade or two after the introduction of the new method before dropping it. However, sometimes we forgot about the deprecation which leads unnecessary protocol methods and code in actors. Thus im wondering, can we introduce the new method and drop the old method in the same upgrade. The old method deprecation will be activated 1-2 days after the upgrade so to avoid messages that are inflight around upgrade to fail & give a bit more time for clients to adopt if they haven't already?

[anorth]

It's not about in-flight messages. It's more about complex SP operations which may take time to migrate. SP's can't really commit to migrating to a new method until after it's live on mainnet. We can't know how complex some migration might be for bespoke operations. This is not a trivial like-for-like method swap. I think we're right to eventually force migration, but a short timeline isn't reasonable.

We could hardcode the deprecation in a single upgrade, but this limits options. I think we would have to set something like 60-90 days upgrade period for SPs. If there's a bug in the new thing, or in clients interacting with the new thing, now we need an emergency upgrade to avoid the programmatic deprecation of the old way that still works. I don't see any real benefit to a short timeline or a pre-determined one.

[jennijuju]

fair points!