Investigate the effects of decreasing function call gas/compute cost

[aborg-dev]

This issue tracks the implications of decreasing the action_function_call costs from 4.6TGas (send + execution) to 700GGas. This work is a part of near/near-one-project-tracking#59.

UPD: To avoid undercharging, the plan is also to increase wasm_contract_loading_bytes from 216KGas to the current estimator estimation of 13.5MGas.
This will affect all calculations below and also can break some of the smart contracts (as this is a price increase).

The main questions are:

• How will the throughput of the network (specifically Shart 2) change?
• How will the chunk production time change?
• How many smart contracts will break due to a gas price increase?

Child tracking issues:

• Document setup to test protocol changes on mainnet traffic #10849

The results of the investigation so far:

• We would need to modify the refund estimation logic that relies on function call base cost.
  One solution would be to set minimum_new_receipt_gas to 108_059_500_000 + 2_319_861_500_000 + 2_319_861_500_000 = 4.7TGas which would limit the overcharging to 6x which is the current level.

For the approach relying on compute costs we would need to implement #8806.

Related issues:

• True cost of function calls #7227
• Base gas costs for function calls are too high #7741

[aborg-dev]

The first step to answer these questions will be to understand how much gas in the chunk is spent on action_function_call. We had a similar question in the past #8258 and I'll do the investigation for this specific action.

[aborg-dev]

It looks like this data is currently not collected by Prometheus, so I'll look into using the NEAR Indexer database for the estimate. Unfortunately, Indexer DB has been deprecated and https://github.com/PagodaPlatform/congestion-analysis does not work anymore. There is an alternative BigQuery-based database https://docs.near.org/bos/queryapi/big-query, I'll see if it contains the necessary data.

[aborg-dev]

We mined the necessary data from DataBricks databases together @khorolets (we first did it on DataBricks and then I repeated the same queries on BigQuery). Here is the bottom line for shard 2 only:

• On 18th of March, shard 2 produced 65k blocks with total capacity of 65000PGas and utilized capacity of 26000PGas
• There have been 3.2M function call actions, which only in base costs yield 15000PGas

Based on that, decreasing the function call base cost to 700GGas would result in 12800PGas reduction in usage from 26000PGas to 13200PGas, roughly halving the total gas usage in the shard.

It's tricky to extrapolate from these numbers to the effects on the congestion, but a naive estimate would be that this would double the throughput and cut in half the queuing time at the peak load (assuming the usage pattern remains the same).

---

Query:

SELECT count(*) as count, sum(gas_limit) as gas_limit, sum(gas_used) as gas_used
FROM `bigquery-public-data.crypto_near_mainnet_us.chunks`
WHERE block_date = "2024-03-18" AND shard_id=2

Result:

[{
  "count": "65149",
  "gas_limit": "6.5149e+19",
  "gas_used": "2.6066742283255353e+19"
}]

Query:

SELECT action_kind, count(*) as count
FROM `bigquery-public-data.crypto_near_mainnet_us.receipt_actions`
WHERE block_date = "2024-03-18"
AND shard_id=2
group by action_kind
LIMIT 10

Result:

[{
  "action_kind": "ADD_KEY",
  "count": "6614"
}, {
  "action_kind": "DELETE_ACCOUNT",
  "count": "12"
}, {
  "action_kind": "TRANSFER",
  "count": "4415908"
}, {
  "action_kind": "DELETE_KEY",
  "count": "2199"
}, {
  "action_kind": "CREATE_ACCOUNT",
  "count": "109"
}, {
  "action_kind": "DEPLOY_CONTRACT",
  "count": "22"
}, {
  "action_kind": "STAKE",
  "count": "756"
}, {
  "action_kind": "FUNCTION_CALL",
  "count": "3281705"
}]

[aborg-dev]

Answering the question about chunk production time is much trickier as it heavily depends on the validator hardware.
We collect chunk apply time from some validators, for example here is the P99 graph of chunk apply time for shard 2 for a set of 6 validators (that are also chunk producers):

If we assume that chunk throughput increased by 2x and latency of chunk apply to increase by 2x, we would expect the P99 to reach:

• 2s for 2 validators
• 1s for 4 validators

This will lead to 2 of the validators skipping chunks for 10 minutes every hour. To avoid this, we would need to offset the gas cost change with a 2x performance improvement during function call execution.

[aborg-dev]

One more idea from @tayfunelmas - we can mirror the traffic from the mainnet to check the effects of changing the gas price. I'll work on this next.

Idea from @bowenwang1996 - study the spikes in P99 metric for validators to see if there are systematic issues and try to replay and profile these blocks.

[aborg-dev]

Following up on our discussion with @akhi3030, the mainnet validator metrics that we should be looking at to understand how much the performance improvements help and how much we can increase the throughput in congested shards:

Validator chunk processing latency

We have a dashboard showing these latencies across a few validators: https://nearinc.grafana.net/goto/RT5ocLJIR?orgId=1

Specifically for this investigation, we are interested in Shard 2:

• This shard experiences regular congestion with a large number of delayed receipts: https://nearinc.grafana.net/goto/dlRfcL1SR?orgId=1
• The latency of chunk processing across validators is spiking in unison with congestion: https://nearinc.grafana.net/goto/krMs5Y1Sg?orgId=1

After the performance improvements are deployed to these validators, we expect lower P50 and P99 peaks in chunk processing latency. These peaks give us an upper bound of how much we can expect to increase the network's throughput during the congested periods before the chunk processing on these validators starts taking >1s and they start lagging behind the chain head.

The new shard 5 is experiencing similar spikes in latency https://nearinc.grafana.net/goto/3lZ3cY1IR?orgId=1, but no consistent congestion yet https://nearinc.grafana.net/goto/Z2fe5LJIR?orgId=1.

We might also be interested in looking at these metrics on one of our RPC nodes: https://nearinc.grafana.net/goto/27QWtLJSg?orgId=1 as we have more control over the nodes. From my observations, these metrics largely agree with the trends on the validators.

At the moment, P99 latency regularly spikes to 1 second, and P50 latency spikes to 400ms - this signals that any increase in throughput during congested times will bring P99 over 1 second.

[bowenwang1996]

I think we need to fix #10907 before reducing function call base cost

[aborg-dev]

To echo what I've posted on Zulip:
TL;DR: I looked through data in Grafana and switch to 1.38.1 does not yield any visible improvements in chunk apply latency on Shard 2, but switch to 1.38.2 significantly improves P90 and P99 latency. See full analysis below.

This likely means that either:

• Our hypothesis 1 about the distribution of receipts during congestion is wrong
• Or contract cache does not significantly improve the performance of validators

We need to look deeper to distinguish these two.

---

Validator mainnet-legends-1 going through 1.38.0 -> 1.38.1 -> 1.38.2: https://nearinc.grafana.net/goto/O18B6JbSg?orgId=1

• No clear changes in latencies on Shard 2 after switch to 1.38.1
• Visible decrease of P90 and P99 latency on Shard 2 after switch to 1.38.2. Hard to tell how much
  yet, need to wait for more data

Two validators mainnet-cryptium and mainnet-chorusone going through 1.38.0 -> 1.38.1: https://nearinc.grafana.net/goto/c8BUeJxSR?orgId=1

• No clear changes in latencies on Shard 2 after switch to 1.38.1

Two RPC nodes sre-mainnet-rpc-01-asia-east1-a-85d71fe2 at 1.38.0 and one38-1-mainnet-rpc-01-asia-east1-a-7a542519 at 1.38.1: https://nearinc.grafana.net/goto/hMt2gbbSR?orgId=1

• No clear changes in latencies on Shard 2 after switch to 1.38.1
• I'm not sure these nodes are comparable, but I haven't found any better way to track how a
  particular RPC node performed before and after switch (maybe @Razvan Barbascu (vanbarbascu) has any suggestions how to
  find that comparison?)