Implement sector->deal mapping in built-in market actor

[anorth]

Adds provider_sectors to market state, a HAMT[Address]HAMT[SectorNumber]SectorDeals. The value is a sorted vec of deal IDs and the associated sector's expiration epoch. Also adds sector number to deal state, to permit finding and removing that mapping when a deal expires.

Prototype for part of direct data onboarding.

[codecov-commenter]

Codecov Report

Merging #1347 (611c18e) into integration/direct-onboarding (ae4fa9a) will increase coverage by 1.59%.
The diff coverage is 73.02%.

Additional details and impacted files

@@                        Coverage Diff                        @@
##           integration/direct-onboarding    #1347      +/-   ##
=================================================================
+ Coverage                          84.27%   85.86%   +1.59%     
=================================================================
  Files                                143      144       +1     
  Lines                              29205    28443     -762     
=================================================================
- Hits                               24613    24424     -189     
+ Misses                              4592     4019     -573     

Files Changed	Coverage Δ
actors/market/src/deal.rs	78.33% <ø> (-10.35%)	⬇️
actors/miner/src/ext.rs	40.00% <ø> (-18.34%)	⬇️
runtime/src/runtime/policy.rs	98.37% <ø> (ø)
test_vm/src/util/workflows.rs	93.79% <50.00%> (-5.53%)	⬇️
runtime/src/test_utils.rs	80.17% <54.54%> (-2.65%)	⬇️
actors/miner/src/lib.rs	66.98% <62.50%> (-15.50%)	⬇️
actors/market/src/state.rs	69.89% <70.58%> (-24.09%)	⬇️
actors/market/src/lib.rs	75.58% <90.90%> (-16.91%)	⬇️
actors/market/src/types.rs	57.40% <100.00%> (-4.60%)	⬇️
test_vm/src/expects.rs	95.50% <100.00%> (-0.24%)	⬇️

... and 7 files with indirect coverage changes

[ZenGround0]

Haven't reviewed tests yet. Leaving approved to unblock you in case you address all review before I finish reviewing tests.

[ZenGround0]

Are we planning to send an expiration extension event from the miner actor on every extension? If not we should consider dropping this field or renaming it to "sector_expiration_hint" or something.

[anorth]

Not every extension event, only when the SP wants to notify this market of it. The right interpretation of this is "the last expiration epoch for this sector that I heard about" or "the minimum committed duration".

However, it's not strictly necessary for direct data onboarding. I will remove it, but leave the SectorDealIDs struct wrapping the deal IDs so there's a place to insert it later if desired.

[ZenGround0]

Do you have a good sense of the worst case of bitfield size vs a vector of integers? My bitfield knowledge is lacking but I'm thinking that terminations are more likely than other operations to be random access which could limit bitfield compression benefits.

[anorth]

When there are many terminations together (e.g. partition faulting out) I think they're actually reasonably likely to be dense. Sectors in the same partition are likely activated around the same time, and there are small gas savings to the SP for dense bitfields.

RLE+ uses varint to encode ranges, so I believe in most cases is more concise than CBOR which uses a power-of-two number of bytes (e.g. 8 bytes for anything > 2^32). @Kubuxu do you have any more insight into cases where a bitfield may perform worse than a vector?

[ZenGround0]

It is worth figuring out if we can hit internal parameter size limits and making that case impossible. If it's already impossible I don't see a reason to chunk. If we are worried about market call costs I think the right mechanism is to reduce the per-epoch termination processing limit.

[anorth]

Looks like it's 1MiB from https://github.com/filecoin-project/ref-fvm/blob/ac2b9b499b9d6e7de4068bbbe8f26545557edb60/fvm/src/machine/mod.rs#L158

If the sector numbers were represented as a vec of 8-byte uints, this gives room for 130k of them (55 full partitions). Both cron and manual processing of early termination are currently limited by MAX_ADDRESSED_SECTORS = 25k, so I think we're good. I will add a note to that parameter though.

[ZenGround0]

@Kubuxu do we have a guarantee that in the worst case bitfield will degrade to <= 8 bytes per int? Are there any edge cases where it can perform worse we should worry about?

[ZenGround0]

These three tests cases still seem relevant and I think should still have tests after this change.

[anorth]

fail_when_caller_is_not_the_provider_of_the_deal: this is impossible now since the caller does not specify the deals, they are looked up. Only inconsistent internal state from activation could cause this. Specifying a non-existent sector is already tested.

fail_when_deal_has_been_published_but_not_activated: impossible since deal IDs are not specified by the caller. Only activated deals exist in the provider_sectors map. Specifying a sector with an already-expired deal is tested.

termination_of_all_deals_should_fail_when_one_deal_fails: impossible to specify a deal that can fail to be terminated, e.g. because not activated.

These removed tests reflect fundamental simplifications we gained from decoupling.

[ZenGround0]

Nice, thanks for explaining