[EPIC] Local Mpool Management

[dineshshenoy]

Edited by @jennijuju

Introduction

For miners, messages often get stuck in the mempool because the budget for sending messages is too low when the base fee spikes.
This can cause a backup of other messages and requires manual intervention and results in a weak user experience or node unavailability for a period of time. Many node operators manually increase the feeCap in order to send the message through or attempt creative mechanisms to cancel the message.

Product & Design Requirements

Make sending out a message and have it landed on chain less problematic for all node operators. Proposed implementations should answer the following:

• What's the best practices on how to react?
• What is a smarter logic within Lotus to manage the fee threshold?
• Possible to add daily budget that can be used for individual messages throughout the day?

Related issues

• Mpool monitoring tool for checking message selection
• Log message execution error
• Set messages' fees interactively

[f8-ptrk]

Best practices on how to react?

yes, spend as much time as you guys/girls can find on lowering the baseFee.

do not mix node operators and miners here, i think the problems these two groups of users have a very different

miners have 3 control addresses:

• post
• precommit
• commit

and having messages stuck in the *commit control addresses is business as usual if you do not want to pay the high fees. miners can handle it, miners know what they are doing. the solution for them is a lower baseFee -> fewer messages, messages with lower gas consumption. In addition miners can control the prices for PublishDeals and AddBalance messages via config file.

having new miners run into this problem with the first message they send (CreateMiner) is a good exercise in what is coming to haunt them in the next weeks/month...

short: miners are fine, basically.

if you implement something that adds "budgets", "smartness", ... please take care that it can be turned off via config flags - most of miners have sophisticated methods to optimize fee costs - you won't be able to implement that into lotus to everyone's advantage (lets start with calculating the probability a certain level of baseFees occurs within a given time frame...).

node operators on the other hand want messages gone. transactions, deals, addbalance...

first: merge this filecoin-project/filecoin-docs#623 then the config param that controls the fees now is at least documented. then it will be good to have a "send-at-all-costs" flag so messages aren't getting stuck at all. a flag param that just applies an acceptable baseFee to the messages, regardless of the estimated gas limit, would be good too (pay up to 4nFIl baseFee, no matter the gas limit - this might be a nice gimmick for miners too...).

[edit] budgets will perform very badly with miners because messages are created at the end of (or during) the sealing process. i'd rather have a message stuck after i sealed a sector for 8-12h then having it rejected by a budget that looked very different 8-12h ago.

[rjan90]

One idea is to have a better UX for miners trying to replace the fee's on a message. I think that would at least help people that are just initializing their miners.

So right now the --auto function does not work very well, if this got updated to work again, this command could have some kind of feedback-message that the user had to confirm before the message where replace:

miner@miner:~$ lotus mpool replace --auto <from> <nonce>
Replacing fees on message <nonce>, estimated cost/burn-fee ~0.2FIL
Do you want to continue? [Y/n] Y
Message replaced

[f8-ptrk]

@rjan90 right now it isn't even clear what exactly --auto is doing. i assume it stops raising the gas-feecap at the, undocumented, setting in the daemons config.toml. but i might be wrong about that.

if we can turn that question off with a --non-interactive or a --assume-yes (-y) more clarification in the process of replacing can never be wrong

Replacing fees on message , estimated cost/burn-fee ~0.2FIL

we actually can calculate the max. costs in advance from the setting in the message

[jennijuju]

Few things to look into:

• More user interactive cmds for pushing messages, i.e: asking user for acceptable max fee before pushing messages; or giving user suggested gas values and let user decide whether they accept the fees or not.
• If the message has been pushed to the network mpool and fees are too low for being executed, inform users in the log.
• More user controllable configuration with suggested values.
• ETH Gas Station-like tool that proposes feecap/premium values for different estimated execution speeds, broken down per type of message.
• Mpool monitoring tool for checking message selection

[f8-ptrk]

ETH Gas Station-like tool that proposes feecap/premium values for different estimated execution speeds, broken down per type of message.

that is a hell of a task. we tried looking at a week of baseFee values while monitoring the current overall mpool trying to predict when a certain baseFee level occurs and what premium will be needed to get the message on chain when that level arrives. (look at messages/tipset values too if considering this)

[raulk]

To expand on the interactivity. Lotus could offer configurable "fee tiers", with potentially 3 tiers by default: e.g. aggressive, standard, frugal. Users could create custom tier policies in their config.toml. When submitting a transaction, a user could specify the name of the tier they want to use. The client would then calculate the fee cap and premium accordingly using the base fee of the last tipset (and potentially of the previous N tipsets to calculate a projection of where the basefee could go), and the current state of the mpool.

[raulk]

Mpool monitoring tool for checking message selection

To expand on this. A tool like this would be useful to understand if message selection in reality matches the expectation: it could help identify if there are bugs lurking that may skew message selection, and/or if miners have modified the message selection logic in any way.

Mechanically, this tool would comprise two components:

1. an append-only ledger of including: mpool operations (using MpoolSuscribe), message inclusions in mined blocks (watching the chain).
2. a Jupyter / ObservableHQ notebook that would enable us to query this append-only ledger and ask questions like "given the state of the mpool at this time, what messages would Lotus select", or "given the state of the mpool at this time, did the messages selected in the next tipset matched what was expected".

[f8-ptrk]

@raulk the normal transaction has a --gas-feecap setting of around 11 nFIl with the default DefaultMaxFee. that's not a problem at all.

from ideling/reading/answering the slack - the problem, for miners, the daemon has is obvious: the CreateMiner messages get always stuck with the default DefaultMaxFee settings.

the CreateMiner message is, besides withdrawing funds, the only message a miner ever needs to send using the daemons fee configuration.

there are several issues open regarding adding a --max-fee, or a --gas-feecap option to the lotus-miner init command. one hardly see other "stuck messages" problems on stack anymore and if - they are related to the clear command.

About the tiers: if they do not guarantee message sending - they just delay the problem, at possibly high computational costs trying to project where a basefee might be

To expand on this. A tool like this would be useful to understand if message selection in reality matches the expectation: it could help identify if there are bugs lurking that may skew message selection, and/or if miners have modified the message selection logic in any way.

they do. field observations: big miner with the capability to run their own code tend to include their own messages first, sometimes just no messages at all. "normal" miners mostly choose by how much FIL is in it for them (--> higher gas-premium increases the chance to get on chain fast)

maybe talk to juan @ zondax - they had an mpool watch tool in the pipeline, i didn't follow up in recent weeks so status and capabilities aren't clear to me right now.