Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Measure biggest contributors to state witness size #11537

Closed
Longarithm opened this issue Jun 10, 2024 · 0 comments · Fixed by #11567
Closed

Measure biggest contributors to state witness size #11537

Longarithm opened this issue Jun 10, 2024 · 0 comments · Fixed by #11567
Assignees
Labels
A-stateless-validation Area: stateless validation

Comments

@Longarithm
Copy link
Member

Longarithm commented Jun 10, 2024

  1. Measure which trie keys are the biggest contributors to the state witness size.

Basically we need metric (trie key first byte, shard id) -> u64. The easy way to measure it is to add (2 * key.len() + value.len()) at each Trie::get (2 because of nibbles). Alternatively we could add how much bytes each new key adds to the current state witness size, but the first way already looks like a good approximation. This can help to see why we often hit state witness size limit: https://nearone.grafana.net/goto/FaAFtMUIg?orgId=1

In the good case, if this is because of contract codes, we should do #11099 after that. Otherwise we should look for other ways to improve throughput.

  1. Add metric measuring how many bytes is contributed by state items (2 * key.len() + value.len()) vs. inner trie items (everything else - basically, sizes of all CryptoHashes). This gives the merkle trie overhead.

Zulip

@Longarithm Longarithm added the A-stateless-validation Area: stateless validation label Jun 10, 2024
@Longarithm Longarithm changed the title Measure trie key types contribution to state witness size Measure biggest contributors to state witness size Jun 10, 2024
@jancionear jancionear self-assigned this Jun 11, 2024
github-merge-queue bot pushed a commit that referenced this issue Jul 9, 2024
…d storage proof (#11567)

Fixes [#11537](#11537)

The new metric allows to see how much of the recorded storage proof
belongs to each trie column.

Here's an example visualisation on a localnet node with ft.py workload
at ~500 TPS:


![image](https://github.com/near/nearcore/assets/149345204/115ebe03-f839-4c2b-b290-7afba5bbf849)

Refs [zulip
discussion](https://near.zulipchat.com/#narrow/stream/308695-nearone.2Fprivate/topic/Large.20number.20of.20delayed.20receipts.20in.20statelessnet/near/443831650)
jancionear added a commit to jancionear/nearcore that referenced this issue Jul 10, 2024
…d storage proof (near#11567)

Fixes [near#11537](near#11537)

The new metric allows to see how much of the recorded storage proof
belongs to each trie column.

Here's an example visualisation on a localnet node with ft.py workload
at ~500 TPS:


![image](https://github.com/near/nearcore/assets/149345204/115ebe03-f839-4c2b-b290-7afba5bbf849)

Refs [zulip
discussion](https://near.zulipchat.com/#narrow/stream/308695-nearone.2Fprivate/topic/Large.20number.20of.20delayed.20receipts.20in.20statelessnet/near/443831650)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-stateless-validation Area: stateless validation
Projects
None yet
2 participants