New API for built-in actors accessible to user-programmed actors

[anorth]

This is a proposal for the built-in actors to present a thoughtful API to user actors on the FVM, while retaining abstraction and flexibility for future protocol upgrades.

Motivation

User-programmed actors on the FVM will want to interact with the built-in actors, especially those related to storage. The built-in actors were not designed with calls from user actors in mind, so do not present a good interface to them.

In the past, we have been able to alter the APIs presented by the built-in actors because callers were either other built in actors (which we could atomically upgrade) or off-chain parties (who can change their code). User-programmed actors will be a new class of caller which will generally be unable or unwilling to upgrade to new APIs. This means that if we ever change a built-in actor method that a user actor depends on, we would break that actor. Network stakeholders will likely be extremely reluctant to break existing actors, with the implication that built-in actor methods reachable by user-programmed actors are effectively permanent.

Indefinite compatibility requirements on a legacy or poorly-conceived interface could present a very significant restriction on future improvements to Filecoin’s protocol and capabilities.

Goals

• Built-in actors present a well-design API to user-programmed actors
• Built-in actor APIs are restricted to methods we feel confident about supporting indefinitely
• Maximise flexibility for built-in actors to change behaviour and internal representations in the future to support core protocol enhancements, without breaking user actors.

Proposal

New public API

I propose that we thoughtfully design a new API for built-in actors to expose to user actors, independent of the existing API that is used for internal messaging. This API would conform to a recommended calling convention (discussion in #382) to present the same calling pattern as user-programmed actors are expected to expose and use between themselves.

I would expect this new API to present much more of the built-in actors internal state than is currently accessible. The current built-in API is limited only to methods actually necessary for use by other built-in actors. Off-chain tools use direct state inspection. A new API should have relatively wide coverage over the data that another actor might reasonably need access to, and potentially replace much state inspection with invocation of getter-type methods.

At the same time, this new API should abstract over the concrete representation of internal state as much as possible. There are opportunities to improve the internal state representation for many built-in actors. Hiding the concrete details while exposing useful information can bring us utility and flexibility.

Restricted internal API

I propose we restrict the currently implemented API of all built-in actors by rejecting calls from any caller which is not a built-in actor, except for a selection of those methods intended to be invoked by external entities.

This API is primarily designed for messaging between mutually-trusting built-in actors. It may expose information or capabilities that are not intended for general consumption. If we allow user-actors to call these methods, we will be locked into their signatures and behaviour indefinitely. This lock-in would severely restrict our ability to improve Filecoin’s core protocol capabilities in the future.

It’s possible that a subset of existing API methods are suitable for export. In this case, I think it is better to consciously re-export those methods under a more flexible calling convention than to support direct calls to them. This opt-in approach to exporting will also reduce the burden of analysis of which methods are both safe and supportable indefinitely.

Limits on public API scope

It is likely that the first version of a public API for built-in actors will not expose everything that a user programmer may want. This is because some core functionalities are targets for immediate change (e.g. #313, #298). We should scope initial API access to those data and capabilities of the built-in actors that we think are most stable and least likely to interact with future protocol improvements. These will be tough decisions in many instances. Future protocol upgrade can add new API functionality (but probably never remove or significantly alter existing methods).

Discussion

Calling convention

As discussed in #382, the existing sequential-method-number dispatch mechanism used by built-in actors is inappropriate for the development of APIs and standards for interoperability. User-programmed actors are expected to adopt some alternative and more flexible convention.

But the built-in actors must support some calling style for invocations from others. So we have to pick one, and composition of multiple actors will be most straightforward if they all use the same convention. This will also permit the built-in actors to implement standard APIs intended to be widely implemented, such as token APIs (e.g. storage deals as standardised NFTs).

While the FVM team are keen that the VM itself shouldn’t preclude the development of alternative conventions, this need not apply to the built-in actors. It’s acknowledged that support from the built-in actors would represent a significant endorsement of some convention, and promote its adoption more broadly. But, again, they have to support something. So it may as well be consciously designed, flexible, and suitable for wide use.

API scope

I don’t have a concrete API scope to propose yet, but we should probably develop an initial one for the FIP describing this proposal. Even if small, we can always extend it later.

[Tom-OriginStorage]

I strongly agree that the API for built-in actors should follow whichever calling convention we eventually decide for user programmed actors.

Nonetheless, I do not advocate for an EVM based calling convention. FVM is WASM-based virtual machine, it simply makes more sense to adopt other WASM-based virtual machine calling conventions such as Ink from Polkadot, Cosmwasm from Cosmos, etc. And if we do choose these WASM-based conventions, we will also realized the overhead costs in terms of development and maintenance can be significantly lowered compared to what we have right now.

Let me use Cosmwasm internal internal message dispatch (yes, two internals as we are doing an internal dispatch within an internal dispatch) to illustrate this:

#[no_mangle]
pub fn invoke(params: u32) -> u32 {
    // Conduct method dispatch. Handle input parameters and return data.
    let ret: Option<RawBytes> = match sdk::message::method_number() {
        1 => constructor(),
        2 => execute(params),
        3 => query(params),
        _ => abort!(USR_UNHANDLED_MESSAGE, "unrecognized method"),
    };

    // Insert the return data block if necessary, and return the correct
    // block ID.
    match ret {
        None => sdk::message::NO_DATA_BLOCK_ID,
        Some(v) => match sdk::ipld::put_block(DAG_CBOR, v.bytes()) {
            Ok(id) => id,
            Err(err) => abort!(USR_SERIALIZATION, "failed to store return value: {}", err),
        },
    }
}

pub fn query(params: u32) -> Option<RawBytes> {
    let params = sdk::message::params_raw(params).unwrap().1;
    let params = RawBytes::new(params);
    match params.deserialize().unwrap() {
        QueryMsg::BalanceOf { owner } => balance_of(owner),
        QueryMsg::OwnerOf { token_id } => owner_of(token_id),
        QueryMsg::GetApproved { token_id } => get_approved(token_id),
        QueryMsg::IsApprovalForAll { owner, operator } => is_approved_for_all(owner, operator),
    }
}

...

First of all, this is a snippet of the Cosmwasm calling convention that has been implemented on FVM. The ugly method numbers will only be used to define on what kind of methods we are calling (1 for constructor methods, 2 for state modifying methods, 3 for state querying methods, etc). This means that it will be pre-defined since the start, and developers writing actors should never need or be required to modify it. We can also replace this numbers with readable macros.

And to resolve the method internally, here's how:
lotus chain invoke t01001 2 oWpiYWxhbmNlX29moWVvd25lcmVhbGljZQ==

whereby oWpiYWxhbmNlX29moWVvd25lcmVhbGljZQ== is the cbor-base64 encoding of {"balance_of": {"owner": "alice"}}. If lotus automatically does cbor-base64 encoding for us in the future, we can directly pass the json.

We then used the Rust's Enum Matching pattern to perform the resolution of the json object to whatever method it was suppose to call, and it is a widely used pattern in Rust to make code readable and beginner friendly.

And the advantage of this is pretty obvious, its super easy to extend. If we have a new API, we just need to declare an additional enum with the expected input params. If we need to rename or change existing parameters of an existing API, we just modify the corresponding enum. And for user created actors that depend on them, the json used to interact with the current actor just needs to be upgraded.

And guess what, we can also generate an ABI-equivalent template by piggybacking cosmwasm tools from the enums. And that effect spillovers to other tools including frontend wallet SDKs and more.

Therefore I believe by adopting a widely used WASM-based calling convention from existing WASM-based virtual machines, it will help the Filecoin ecosystem to save a lot of unnecessary work. We should work smart, not work hard.

[anorth]

Thanks, but let's keep the calling convention discussion in #382 (I will respond). I think the main point here is that these methods should follow a convention like that, and we're aligned here.

[arajasek]

First impressions:

@anorth Thanks a lot for getting this started! I think the need for this is well-motivated.

I propose we restrict the currently implemented API of all built-in actors by rejecting calls from any caller which is not a built-in actor, except for a selection of those methods intended to be invoked by external entities.

Strong support for this idea.

[raulk]

Thanks for getting this started, @anorth.

Feedback

I concur with the suggested approach of restricting currently exposed APIs so they can only be consumed by other built-in actors (we might need to add a syscall to efficiently evaluate this condition, as it'll gate every call; likely something we can evaluate in the invoke entrypoint). As you say, these APIs were not implemented with the view that they'll be etched in stone forever (as a consequence of user-deployed actors relying on them). We also will not violate the basic blockchain guarantee of not breaking already-deployed actors with a future protocol upgrade that may change these APIs.

I'm also onboard with deliberately re-exporting functionality under a new future-proof API. One technicality that I want to point out to confirm that our thinking is aligned:

In this case, I think it is better to consciously re-export those methods under a more flexible calling convention than to support direct calls to them.

I would rephrase this as: "I think it's better to conciously re-export the functionality under a new set of methods that use a more flexible and future-proof schema". [I don't expect the call convention to change besides what will come out #382 and #409 re: dispatch; that is, builtin actors will use CBOR for params and return values, errors are communicated through aborts, etc.]

Action

I'm keen to get started with this initiative ASAP -- I think FEVM cannot ship without a public builtin-actors API that enables devs to build the use cases they are intending to. Since we expect to open up the API progressively over time, we should ensure that the first iteration follows Pareto (let's expose the 20% that will enable 80% of use cases) to avoid user disappointment. Aadi (Product Manager from our team) is conducting research to catalogue the use cases being pursued by the community. This research should heavily influence the decisions of which APIs we open up, with understanding of the consequences.

However, here comes the bad news: the FVM team is not equipped to tackle this for nv18. We are not planning to make functional changes to built-in actors, this would significantly expand our scope and dillute our focus -- which is on the runtime itself. Would Protocol Dev be keen to drive this for nv18?

[anorth]

In this case, I think it is better to consciously re-export those methods under a more flexible calling convention than to support direct calls to them.

I would rephrase this as: "I think it's better to conciously re-export the functionality under a new set of methods that use a more flexible and future-proof schema".

Sure. I was addressing a potential set where the existing schema is already fine, and the only thing to change is the method number / dispatch. E.g payment channel or multisig probably have some of these.

I think nv18 is a good target for this, but let's not shard that discussion out from the governance forums.

[anorth]

Some work on this has begun

• a PR demonstrating a mechanism: Proof of concept exported API for Account actor builtin-actors#797
• an audit of current methods to export, and notes about new ones to add short term

@arajasek and I will roll through this list to produce some APIs that can be exercised in upcoming dev/testnets, and then write up a quick FIP as it settles.

[FroghubMan]

Is it possible to define non-stable APIs and warn users of restricted usage rules, and when APIs will be changed, users upgrade actors to achieve new compatibility?

[Tom-OriginStorage]

Any particular non-stable APIs u hav in mind?

[FroghubMan]

A well-developed api iterative upgrade strategy is more valuable than defining some seemingly thoughtful apis.
I don't think any API is thoughtful!

[FroghubMan]

If you only provide a thoughtful API for user-defined actors, you may now have a richness of functions.

[anorth]

This is an interesting idea, thanks. There may be some APIs for example that are mainly intended to be invoked by external parties (accounts) but that we would want to export so that smart-contract wallets and other proxies for external parties can still use them. But we'd reserve the ability to change those APIs (because external code that calls them can be updated). An example might be ReportConsensusFault, and maybe even most of the miner actor's sector onboarding/proving methods.

[Destore2023]

This is an interesting idea, thanks. There may be some APIs for example that are mainly intended to be invoked by external parties (accounts) but that we would want to export so that smart-contract wallets and other proxies for external parties can still use them. But we'd reserve the ability to change those APIs (because external code that calls them can be updated). An example might be ReportConsensusFault, and maybe even most of the miner actor's sector onboarding/proving methods.

How about a fork (AB testing) to see what can @FroghubMan do in the new sector ?

[anorth]

One open question is whether to export the methods on the current payment channel actor. The big picture line is that, once user-programmed native actors are possible, we expect people to write new and better payment channel actors, and there's no reason for the built-in one to have any special status. So by refusing to export its API, we encourage that development.

However, given that the actor does exist and work, a reasonable argument could be made that it's more helpful just to open it up. We don't expect to do any further development on it, so API lock-in is a low-level concern. FYI @arajasek @ZenGround0

Note that we have exported methods on the multisig actor, for which similar arguments could be made both ways.

[anorth]

My initial pass did not export any methods from the power or reward actor. However, I can see it would be useful for the sector initial pledge calculation to be possible in user-programmed actors. To enable this, we'd need to export either:

• the network total power, circulating supply, block reward values, or
• a new method that specifically does the pledge calculation, but abstracts how it's performed

[anorth]

I'd like to see a design for a lending market that definitely needs this function, to motivate committing to it indefinitely.

[anorth]

Progress update

@arajasek and I have done a first quick pass at restricting the current internal API, exporting a set of existing methods, and adding some basic new methods to the market and miner actors. The main goal in this initial pass was to establish a pattern and get a conservative safe set of methods out quickly for testing and downstream SDK development.

We don't think this is the complete API for FVM 2.1 milestone, and we'll work on that next. One particular focus will be miner actor orchestration methods, so that another actor could meaningfully function as a miner owner or controller. Please let us know other major needs either here or in the #fil-actors slack channel.

This API is being developed on the integration/builtin-api branch of the built-in actors repo. We'll draft a FIP for the API as it gets closer to finished.

Account	
AuthenticateMessage	#797
UniversalReceiverHook	#797
	
Datacap Token	
Name	#807
Symbol	#807
TotalSupply	#807
BalanceOf	#807
Transfer	#807
TransferFrom	#807
IncreaseAllowance	#807
DecreaseAllowance	#807
RevokeAllowance	#807
Burn	#807
BurnFrom	#807
Allowance	#807
	
Init	
Exec	#807
	
Market	
AddBalance	#807
WithdrawBalance	#807
GetBalance	#812
GetDealDataCommitment	#818
GetDealClient	#818
GetDealProvider	#818
GetDealLabel	#818
GetDealTerm	#818
GetDealEpochPrice	#818
GetDealClientCollateral	#818
GetDealProviderCollateral	#818
GetDealVerified	#818
GetDealActivation	#819
	
Miner	
ChangeBeneficiary	#807
GetBeneficiary	#807
GetOwner	#811
IsControllingAddress	#811
GetSectorSize	#811
GetVestingFunds	#811
GetAvailableBalance	#811
	
Multisig	
Propose	#807
Approve	#807
Cancel	#807
AddSigner	#807
RemoveSigner	#807
SwapSigner	#807
ChangeNumApprovalsThreshold	#807
LockBalance	#807
UniversalReceiverHook	#807
	
Power	
NetworkRawPower	#810
MinerRawPower	#810
	
RemoveExpiredAllocations	#807
ExtendClaimTerms	#807
RemoveExpiredClaims	#807
UniversalReceiverHook	#807

[anorth]

I've written up a longer motivation for why to avoid exposing many parts of miner and sector state: #539

[anorth]

What about exposing QA power (e.g this request)?

The reasons to expose it are that it is (today) a core feature of the network and a miner, and corresponds to consensus power, share of reward, and (more roughly) stake. I think it's clear there is at least some utility, though raw byte power would suffice for many other things.

The reason not to expose it is that it would lock in the concept and approximate mechanism of QA power (callers would not expect the meaning to change a lot over time). While I don't see QA power changing in the short term, there have been multiple proposals for various ways of decoupling stake and reward distribution from raw storage commitments. These proposals would generally do away with QA power as it's the coupling mechanism.

• FIP-0033 Explicit FIL+ Subsidy
• Decoupling Storage and Deal with independent POW and POS mechanisms Decoupling Storage and Deal with independent POW and POS mechanisms #442

There have also been multiple changes to how QA power is calculated, rather than how it is used. It's unclear whether change like this might be impeded by wide adoption of QA power for other uses by actors.

• FIP-0045
• FIP-0036 and FIP-0054 (forthcoming)

cc @arajasek @kikakkz @ZenGround0 @robertl0705

[anorth]

I am weakly opposed to exposing it in the API now. I do not think the supposed benefits are sufficiently clear or high impact, given the uncertainty around the lock-in and constraints that would be created.

I am only weakly opposed because it seems feasible that large changes that impacted or removed QA power could (and maybe would anyway) be implemented in a forward-only manner, affecting new SPs and sectors rather than changing the rules for all existing sectors at one point in time. This would preserve the meaning of QA power, but slowly phase out its relevance over a period of years, and this might be something that we're sufficiently confident in doing. A blocker to doing that would be some highly significant actor or application with code that can't be upgraded, and also usage that can't be migrated to a new version of the application. I'm not aware of much precedence like that in other ecosystems (gas tokens in Ethereum, that depend on gas refunds from SELFDESTRUCT, are one).

Thus I expect that more evidence for the utility of exposing QA power could tip me over to thinking the tradeoff worthwhile. But I haven't see a clear demonstration of a high level behaviour that requires it yet.

[kaitlin-beegle]

Hey @anorth! Catching up after the holidays and want to see if this FIP has successfully completed Last Call - can you let me know if you were able to follow up on this request offline, or if it's worth holding the FIP up for? We have wiggle room if you feel it's important to get clarity here before deeming the FIP accepted.

I suspect that this may be a future issue, but wanted to double check.

[anorth]

I would like to see any counter that @arajasek may wish to offer, but otherwise no I don't think we need to hold up the FIP for this.

[arajasek]

I think we leave this as-is for now. We will definitely be adding more APIs in future network upgrades, and I suspect we'll get much more feedback about what's missing after the upcoming network upgrade (when people actually start deploying contracts on mainnet).

Until then, I think the only change I would consider is if someone raised a concern with the security or long-livedness of any existing exported methods.

[wenxinnnnn]

User-defined actors read SP's initial pledge total, which I think needs to be opened early.
I think my Dapps need it.

[anorth]

Could you please expand on what they need it for? (Feel free to engage me in Slack if this needs more back-and-forth).

[kaitlin-beegle]

Hey @wenxinnnnn! This FIP was in Last Call, and I want to be sure your issue/concern has been addressed. It's been a few weeks since (to my knowledge) anyone has been able to get in touch with you via Slack or Github.

Can you let me know ASAP if you have any other details or concerns related to this flag? Thanks!

[anorth]

One reason this might be useful is to calculate what a miner's balance would be if (all) sectors were terminated and paid maximum penalties, with remaining pledge released.

• I think you can get a pretty good approximation of this using total balance, available, and vesting total, noting that penalties are paid first from the free and vesting totals.
• But in any case it won't be possible to get an accurate on-chain estimate of termination fees, which vary per sector. We're almost certainly not going to expose the per-sector fee information, since that sector metadata is an area of active change.

[anorth]

In summary, this is still worth considering in the future, but not for this initial API FIP.

[wenxinnnnn]

My purpose is not to calculate the cost of terminating a sector. Instead, it is used to calculate the tokens provided by users in exchange for the share of sp rights.
"Read initial pledge total" is the basic concept of filecoin sp. I don't think it is an unstable API. I think its development workload is not large at all, and it will not affect the progress of the current version.
In addition,regarding the cost of terminating a sector, I agree with you that it doesn't need an exact value. As long as there is a certain range of penalties, contract participants are prevented from choosing lose-lose.

[wenxinnnnn]

Why not provide a simpler GetLockedFunds?
Is it inappropriate for GetLockedFunds and GetVestingFunds to coexist?

[arajasek]

@wenxinnnnn That's a fair question. The reason is that we're trying to have exported methods correspond to simple concepts, instead of coupling them into larger types that might be more prone to change). So we've exported a method to get a miner's available balance, and a list of any vesting funds that will be unlocked.

Note that we have not exported a GetLockedFunds method.

[anorth]

You can (or will be able to) check the actual token balance, so you can also calculate balance - available = locked

[wenxinnnnn]

balance - available = locked

I think it's a trick. In some future FIP, if the lock-up rules are redistributed, it will be a disaster for dapps that use this method to calculate the lock-up amount.
Either sacrifice FIP, or sacrifice them.
Obviously an accurate concept, an accurate API should be provided for ease of use.

[ZenGround0]

I am recommending removal of PublishStorageDeals API export. IIRC I mistakenly suggested including this on a preliminary spreadsheet planning this document. The reason to not include is that the confusion cost is high and the programmability benefit is low.

Since FIP 0050 was merged to next brach of builtin actors late last week it has generated confusion among the internal FVM team. I've also recently received messages from community members trying to use it incorrectly. The main problem is that PSD as it exists today is an infrastructure method restricted to be called by a control address of an SP actor that acts as provider of the deal. The reason for this is that publish storage deals kicks off a process in the system that requires a storage miner to follow through and actually store the data, so there is a good reason for the system to force that this is called by the storage provider marked in the deal. However the name is very inviting to developers that want to create deals in the network with smart contracts and it is the natural first place to start playing around.

There is definitely room for innovation on top of PSD from the storage provider pool side but I don't think its worth the cost of constantly fighting the misconception that a developer can use PSD to make a storage deal for the next few months.

If we don't remove this method we should at least rename it to something that indicates it is a storage provider infrastructure method.

[raulk]

However this is straight up wrong:

and would cripple the system so that a third party is always needed (e.g. a bounty chaser)

as the client contract model (as opposed to deal bounty model) has precisely the same SP-drives-storage-through-PSD property you're describing without a third party.

Right, that was too bold a statement. I don't understand exactly what you meant there, but keeping PSD enables deal formalization/acceptance flows that live entirely on chain so that a third party is not needed to facilitate.

[aashidham]

I would also vote to retain.

Could we simply rename the function? Confusion comes from the name. For a layman builder the method simply does not do what it says on the tin.

What if we call it something like MinerSubmitDealProposal or MinerDealSubmit to make it clear this is something for SP actors (or a solidity actor that is authorized to act on behalf of an SP)?

[arajasek]

@aashidham I like the proposal to rename the method to clarify its behaviour. I think if we do wind up retaining, we should strongly consider that.

Can you expand on your vote to retain? I'd like to know more about your motivation.

[aashidham]

Dropping PSD here might reduce developer confusion in the short term, but in the long term can constrain what gets built and how these building patterns can calcify over time. For example, it never allows a flow where a solidity actor acting on behalf of an SP can call PSD directly.

Deal bounties are kind of the FVM version of the model-view-controller paradigm — useful but can be limiting frameworks if that’s all you are familiar with.

[raulk]

@ZenGround0 unless you want to pursue this further, I think the emerging consensus here is to retain the method to (a) avoid ossification of patterns, (b) prevent innovation blockage where devs actually want to use this correctly in the next months and they just can't.

An interesting suggestion was made to possibly rename this method to make it less of a "false friend". But I don't think that's the right heuristic to drive protocol-level decisions. I believe this method is correctly named today: it allows a Miner to Publish Storage Deals it has committed to undertake. IMO what happened here is that devs applied inductive reasoning to arrive to a wrong conclusion, because:

1. Good deal-making docs don't exist.
2. They are not referenced from the Solidityi API docs.
3. This method might not be correctly explained in Solidity API docs.
4. High-level APIs are not available in the Solidity library. The library is explosing the low-level protocol APIs as-is, i.e. it is exposing the plumbing.

I think addressing the points above would correct this situation.

[arajasek]

I'm having concerns about the utility of exporting the Init actor's Exec method. While this does have some use-cases today (eg. having a contract create a multisig actor), it's likely not very useful in its current form. The fact that it takes a code_cid as an input means it's gonna be pretty hard to ever write an unchanging contract that uses this method reliably.

Further, this method receives a serialization of the relevant constructor's params as input. Exporting this method risks locking the current shape of the constructor params for our built-in actors.

Lastly, the only actors that can actually be created by contracts using this method are multisigs and payment channels, both of which we're interested in dropping long-term support for.

@Stebalien also has concerns about potential unforeseen consequences of exporting this method, and would rather delay its exporting until a pressing use-case is identified.

[arajasek]

UPDATE: This method is being dropped from the FIP-0050 exports: #616

[samuerio]

Since we support fevm actor to call native actor with opcode CALL,Then we can implement a native actor that supports both the FRC46 standard API and the ERC20 standard API.(eg. This is an example of native actor implements the erc20-transfer method.)

From a longer-term perspective, Is it possible extend built-in actors to provide New APIs that receives and returns data in ABI-encoded format? In this way, fevm actors can directly call built-in actors conveniently.

[anorth]

Yes I think it will be entirely possible and encouraged to implement both FVM native and EVM ABI interfaces within a single native actor.

Maybe we'll consider it a good idea to do so in the built-in actors, but this case is not clear. A library could do it from the EVM side. We generally want to keep the built-in actors as small scope as possible, and presenting multiple APIs would conflict with that. What about future guest VMs?