refactor: Combine contract accesses and deployments into ContractUpdates #12326

tayfunelmas · 2024-10-28T13:36:31Z

The propagation of contract accesses (calls) and deployments are done together. Thus, introducing a wrapper called ContractUpdates that contains the accesses and deployments and this is passed through the call stack from the chunk application up to the point we send the messages.

Also address a TODO to change BTreeSet to HashSet to contain the code hashes.

We do NOT change the representation of code hashes in database and network messages, but only the internal data structures.

This reverts commit a4e861a.

codecov · 2024-10-28T13:59:36Z

Codecov Report

Attention: Patch coverage is 58.92857% with 23 lines in your changes missing coverage. Please review.

Project coverage is 71.26%. Comparing base (daa67a3) to head (f97d0e0).
Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
...src/stateless_validation/state_witness_producer.rs	67.85%	4 Missing and 5 partials ⚠️
.../src/stateless_validation/contract_distribution.rs	0.00%	9 Missing ⚠️
...idation/partial_witness/partial_witness_tracker.rs	0.00%	3 Missing ⚠️
...alidation/partial_witness/partial_witness_actor.rs	0.00%	1 Missing ⚠️
runtime/runtime/src/lib.rs	80.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #12326       +/-   ##
===========================================
+ Coverage   38.11%   71.26%   +33.15%     
===========================================
  Files         837      838        +1     
  Lines      168711   169346      +635     
  Branches   168711   169346      +635     
===========================================
+ Hits        64298   120686    +56388     
+ Misses     100525    43414    -57111     
- Partials     3888     5246     +1358

Flag	Coverage Δ
backward-compatibility	`0.16% <0.00%> (?)`
db-migration	`0.16% <0.00%> (?)`
genesis-check	`1.23% <0.00%> (?)`
integration-tests	`39.04% <58.92%> (+0.93%)`	⬆️
linux	`70.65% <55.35%> (+32.54%)`	⬆️
linux-nightly	`70.84% <58.92%> (?)`
macos	`50.39% <55.35%> (?)`
pytests	`1.54% <0.00%> (?)`
sanity-checks	`1.35% <0.00%> (?)`
unittests	`64.20% <55.35%> (?)`
upgradability	`0.21% <0.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

shreyan-gupta

Looks good!

shreyan-gupta · 2024-10-28T14:16:28Z

chain/chain/src/store/mod.rs

    ) {
        if let Some(partial_storage) = partial_storage {
+            let ContractUpdates { contract_accesses, contract_deploys } = contract_updates;


This is the interaction layer with store, verifying whether it's okay to not have contract_accesses and contract_deploys sorted?

Just checked, I think we are storing vec here, so should be fine

We do not rely on sorted-ness of the hashes (eg. not keeping a hash of the list etc.) so using HashSet should be fine. We store the hashes in the DB as Vector but the storage is temporary and again does not depend on the sortedness.

By the way, do you know the reason to store hashes as vectors? Is it to save space?

yeah two considerations, first I started using vector all over the place in the first PR and did not want to change the protocol and DB structures. Second, I thought vec would store them more compactly and we turn them into hashset when processing anyways, thus did not want to deal with a migration.

shreyan-gupta · 2024-10-28T14:19:39Z

chain/client/src/stateless_validation/state_witness_producer.rs

-    /// Code-hashes of contracts deployed while applying the previous chunk.
-    pub(crate) contract_deploys: BTreeSet<CodeHash>,
+    /// Contracts accessed and deployed while applying the chunk.
+    pub contract_updates: ContractUpdates,


pub(crate)?

done (will send a commit)

shreyan-gupta · 2024-10-28T14:21:27Z

chain/client/src/stateless_validation/state_witness_producer.rs

@@ -422,13 +415,15 @@ impl Client {
    /// Sends the contract accesses to the same chunk validators
    /// (except for the chunk producers that track the same shard),
    /// which will receive the state witness for the new chunk.  
-    fn send_contract_accesses_to_chunk_validators(
+    fn send_contract_updates_to_validators(


Seems like we are still only sending contract_accesses and not contract_updates. Do we want to rename this function?

I am planning to also send the deployments, but yeah I can rename in the later PRs. Reverted the naming change here.

…as/nearcore into contract-dist-refactor

Trisfald

🚀

Trisfald · 2024-10-28T15:12:31Z

chain/chain/src/store/mod.rs

    ) {
        if let Some(partial_storage) = partial_storage {
+            let ContractUpdates { contract_accesses, contract_deploys } = contract_updates;


By the way, do you know the reason to store hashes as vectors? Is it to save space?

robin-near · 2024-10-28T17:35:10Z

chain/client/src/stateless_validation/partial_witness/partial_witness_tracker.rs

@@ -43,7 +43,7 @@ enum AccessedContractsState {
    Unknown,
    /// Received `ChunkContractAccesses` and sent `ContractCodeRequest`,
    /// waiting for response from the chunk producer.
-    Requested { contract_hashes: BTreeSet<CodeHash>, requested_at: Instant },
+    Requested { contract_hashes: HashSet<CodeHash>, requested_at: Instant },


What're the pros and cons of BTreeSet vs HashSet here? Why are we using one vs the other? Would Vec also work?

The reason for set vs vector is to indicate using the type that the items are unique.

HashSet is known to be faster than BTreeSet and also since we do not need order/sortedness, thus I wanted to replace it with HashSet.

tayfunelmas added 20 commits October 14, 2024 04:48

Add nayduck test for kicking out validators

a4e861a

Revert "Add nayduck test for kicking out validators"

9c023d4

This reverts commit a4e861a.

Merge branch 'near:master' into master

f6dcdb8

Merge branch 'near:master' into master

2b710b9

Merge branch 'near:master' into master

8288d28

Merge branch 'near:master' into master

83d09b7

Merge branch 'near:master' into master

0bb5f0c

Merge branch 'near:master' into master

47f348b

Merge branch 'near:master' into master

6569508

Merge branch 'near:master' into master

5f50a51

Merge branch 'near:master' into master

d3514ef

Merge branch 'near:master' into master

e3e685a

Merge branch 'near:master' into master

e1b9c99

Merge branch 'near:master' into master

9c609ae

Merge branch 'near:master' into master

45a89d1

Merge branch 'near:master' into master

a8591dc

Merge branch 'near:master' into master

1943fa7

Merge branch 'near:master' into master

94092d6

Merge branch 'near:master' into master

2c43a6e

Introduce ContractUpdates

aaff662

tayfunelmas requested review from pugachAG, shreyan-gupta, robin-near and Trisfald October 28, 2024 13:36

tayfunelmas marked this pull request as ready for review October 28, 2024 13:38

tayfunelmas requested a review from a team as a code owner October 28, 2024 13:38

Merge branch 'master' into contract-dist-refactor

9903349

shreyan-gupta approved these changes Oct 28, 2024

View reviewed changes

Revert naming change for function

7c5bd82

Merge branch 'contract-dist-refactor' of https://github.com/tayfunelm…

edd733d

…as/nearcore into contract-dist-refactor

Trisfald approved these changes Oct 28, 2024

View reviewed changes

robin-near reviewed Oct 28, 2024

View reviewed changes

robin-near approved these changes Oct 28, 2024

View reviewed changes

tayfunelmas added 3 commits October 30, 2024 00:03

Change visibility of ContractUpdates in one place.

2ffa191

Remove unnecessary HashSet::from_iter

d0dbe45

Merge branch 'master' into contract-dist-refactor

f97d0e0

tayfunelmas enabled auto-merge October 30, 2024 08:14

tayfunelmas disabled auto-merge October 30, 2024 08:44

tayfunelmas added this pull request to the merge queue Oct 30, 2024

tayfunelmas removed this pull request from the merge queue due to a manual request Oct 30, 2024

tayfunelmas added this pull request to the merge queue Oct 30, 2024

Merged via the queue into near:master with commit 1e41b11 Oct 30, 2024
29 checks passed

tayfunelmas deleted the contract-dist-refactor branch October 30, 2024 09:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: Combine contract accesses and deployments into ContractUpdates #12326

refactor: Combine contract accesses and deployments into ContractUpdates #12326

tayfunelmas commented Oct 28, 2024 •

edited

Loading

codecov bot commented Oct 28, 2024 •

edited

Loading

shreyan-gupta left a comment

shreyan-gupta Oct 28, 2024

shreyan-gupta Oct 28, 2024

tayfunelmas Oct 28, 2024

Trisfald Oct 28, 2024

tayfunelmas Oct 30, 2024

shreyan-gupta Oct 28, 2024

tayfunelmas Oct 30, 2024

shreyan-gupta Oct 28, 2024

tayfunelmas Oct 28, 2024

Trisfald left a comment

Trisfald Oct 28, 2024

robin-near Oct 28, 2024

tayfunelmas Oct 30, 2024

refactor: Combine contract accesses and deployments into ContractUpdates #12326

refactor: Combine contract accesses and deployments into ContractUpdates #12326

Conversation

tayfunelmas commented Oct 28, 2024 • edited Loading

codecov bot commented Oct 28, 2024 • edited Loading

Codecov Report

shreyan-gupta left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Trisfald left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tayfunelmas commented Oct 28, 2024 •

edited

Loading

codecov bot commented Oct 28, 2024 •

edited

Loading