Execute CrossMessages

[aakoshh]

Implement the mechanism for executing cross messages in Fendermint, both top-down and bottom-up.

Background

Lotus

When IPC is running in Lotus, where it's just a smart contract, we have no choice but to send the bottom-up checkpoint as a single "full fat" message to the Gateway actor, which will do its damnedest to execute all messages, and it's all paid for by the submitter. The contract, by Solidity internal forwarding semantics, makes sure it does not fail the batch if one of the messages in the batch fail.

There is also a question of gas accounting: how do we make the sender of the message pay for it? The FVM SDK has a method for returning the gas used in the transaction, but we can't access it from Solidity. However, there seems to be a gasleft function built into Solidity itself.

One thing we don't have to deal with on Lotus is paying for the execution of a fund message, because it's the rootnet, which doesn't need to be funded.

Fendermint

When IPC is running in Fendermint, we have more control over how we execute cross messages, ie. how we call the FVM itself, and we can even manipulate the state tree. The more we do, however, the more non-standard this whole thing is. We can execute messages one by one, or in a batch, by driving the process from the host, rather than delegating everything to the Gateway.

FVM

Here are the execution modes in FVM:

• explicit: The sender has to be a regular Account, its nonce is checked, it balance is checked to see that it covers the gas limit, it's charged, it gets a refund in the end. The nonce is increased, even if the message fails due to out-of-gas error.
• implicit: The sender is checked to exist, but its nonce and balance are ignored. Gas limit is respected, but it doesnt' cost anything. The SYSTEM account can only use this one. It's not possible to invoke this for normal user transactions.

Desired semantics

The things we want from the execution are:

• The sender of the cross message (not the relayer of the checkpoint) pays for the execution on the subnet where the execution takes place. We don't want the relayer or an unlucky validator to have to pay for the whole thing.
• fund messages must be exempt from this, because otherwise how do you pay for your first fund message that comes from the parent to establish your account
• Regardless of the outcome of a cross message execution, the nonce in the Gateway must be incremented, so that it accepts the next incoming message
• The cross message must be executed regardless of the current nonce of the sender account, and it must leave it untouched, so that the sender operating directly on the subnet does not get disrupted

Possible mechanisms

Option 1: Arm twisting the nonce

1. We execute messages one by one
2. We parse the message from the Solidity type and check if it's a fund message; if it is, execute it implicitly by the SYSTEM account to mint some funds and credit it to the user; also adjust the circ_supply which lives in the host.
3. If it's not a fund message, we change the nonce of the sender to match what's in the message, then use explicit execution where gas is charged (but the nonce won't cause a problem), finally we change the nonce back to what it was in the state tree.
4. Regardless of the outcome, we use an implicit message to bump the nonce of the Gateway account.
5. GOTO 1

Option 2: Arm twisting the gas/balance checks and refunds

1. We execute messages one by one
2. We parse the message from the Solidity type and check if it's a fund message; if it is, execute it implicitly by the SYSTEM account to mint some funds and credit it to the user; also adjust the circ_supply which lives in the host.
3. If it's not a fund message, we check the and charge the gas limit to the sender account in the host, then user implicit execution so nonce and gas is ignored (but gas limit is respected), then we refund unused gas. So we lift some of the things the FVM does into the host.
4. Regardless of the outcome, we use an implicit message to bump the nonce of the Gateway account.
5. GOTO 1

Option 3: Pass the whole batch and execute in Solidity

If the gasleft method works in Solidity, we can use almost the same mechanism as on Lotus to let the Gateway execute all messages and charge the cost to the sender internally based on delta gas.

1. We execute all messages at one.
2. We use implicit execution by the SYSTEM actor so the overall execution doesn't cost any gas to anyone, it's all internal. (Note that this is different from Lotus where the relayer has to pay, and expects full rewards).
3. We adjust the circ_supply based on what the Gateway returns after processing all the fund. fund thus does not cost gas, but if we just pass everything to the Gateway, it must return the total.

[aakoshh]

#202

consider for staggered execution

[aakoshh]

Option 3 is the currently agreed best approach, if it works.

[aakoshh]

I am not well versed with Solidity, so I hope I am wrong but I am worried about a trivial vector of attack with Option 3 above.

I imagine that cross messages have some kind of gas limit field, and the gas limit of the entire batch is the sum total of limits of the messages in it. The problem is, if we pass all messages as a batch, then only the batch-level limit can and will be enforced by host system. Within Solidity, I don't know about any means of enforcing that a particular message in the batch should not be able to consume more than the gas limit it specified.

Therefore a malicious actor can send a cross message with a low gas limit, then simply start an infinite loop until all the remaining gas for the batch is consumed, preventing any other message from running, and ultimately causing the entire batch to be reverted with an out-of-gas error.

On Lotus, the relayer will have to pay for this batch, yet receive no reward in return. The attacker doesn't have to pay anything as their message was just an internal construct of the gateway.

What's the story here? Is there any way for the gateway to protect itself from gas overruns on cross messages?

[aakoshh]

Here are the conclusions of our little powwow.

This is indeed a problem with no known solutions in Lotus. The consequences are severe: if we cannot control the gas of the messages in the batch and they cause the entire relayed checkpoint to fail with out of gas error, that means that checkpoint becomes impossible to relay, and the subnet can no longer make progress in terms of checkpoints.

The only way to avoid the problem is to restrict cross messages to fund and release and nothing that would trigger execution on the Lotus rootnet.

On Fendermint we have to consider Option 1 and Option 2 instead, so that we can execute messages one by one and let the FVM enforce their individual gas limit.

[aakoshh]

It turns out there is a way in Solidity to limit the gas of delegate calls: https://forum.openzeppelin.com/t/a-brief-analysis-of-the-new-try-catch-functionality-in-solidity-0-6/2564

See DELEGATECALL at https://www.evm.codes/#f4?fork=shanghai

From the Tangerine Whistle fork, gas is capped at all but one 64th (remaining_gas / 64) of the remaining gas of the current context. If a call tries to send more, the gas is changed to match the maximum allowed.
Stack input
gas: amount of gas to send to the sub context to execute. The gas that is not used by the sub context is returned to this one.