Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(storage): save enum indices in RocksDB #162

Conversation

perekopskiy
Copy link
Contributor

@perekopskiy perekopskiy commented Oct 5, 2023

What ❔

Enumeration indices now are saved along with values in the same column family. Indices are added gradually for old DB entries. The number of keys processed each L1 batch is configurable.

Why ❔

Enumeration indices in storage are necessary for boojum upgrade.

Checklist

  • PR title corresponds to the body of PR (we generate changelog entries from PRs).
  • Tests for the changes have been added / updated.
  • Documentation comments have been added / updated.
  • Code has been formatted via zk fmt and zk lint.

@perekopskiy perekopskiy requested a review from a team as a code owner October 5, 2023 11:06
@codecov
Copy link

codecov bot commented Oct 5, 2023

Codecov Report

Attention: 17 lines in your changes are missing coverage. Please review.

Comparison is base (fe2d6ad) 31.77% compared to head (f40f5fa) 31.87%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #162      +/-   ##
==========================================
+ Coverage   31.77%   31.87%   +0.10%     
==========================================
  Files         481      481              
  Lines       25593    25688      +95     
==========================================
+ Hits         8132     8189      +57     
- Misses      17461    17499      +38     
Files Coverage Δ
core/bin/external_node/src/config/mod.rs 18.18% <ø> (ø)
core/bin/external_node/src/main.rs 0.00% <ø> (ø)
...lib/zksync_core/src/metadata_calculator/helpers.rs 86.66% <100.00%> (ø)
core/lib/config/src/configs/chain.rs 37.93% <0.00%> (-2.81%) ⬇️
core/lib/state/src/in_memory.rs 93.75% <85.71%> (-2.55%) ⬇️
...zksync_core/src/state_keeper/batch_executor/mod.rs 69.23% <0.00%> (-0.42%) ⬇️
core/lib/zksync_core/src/state_keeper/mod.rs 0.00% <0.00%> (ø)
core/lib/dal/src/storage_logs_dal.rs 79.01% <83.33%> (+1.16%) ⬆️
core/lib/storage/src/db.rs 52.41% <50.00%> (-0.11%) ⬇️
core/lib/state/src/rocksdb/mod.rs 91.07% <90.24%> (-2.87%) ⬇️

... and 21 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

core/lib/dal/src/storage_logs_dedup_dal.rs Outdated Show resolved Hide resolved
core/lib/state/src/in_memory.rs Outdated Show resolved Hide resolved
core/lib/state/src/in_memory.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/dal/src/storage_logs_dedup_dal.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@slowli slowli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally, this functionality should be tested. One approach that comes to mind is syncing RocksdbStorage from Postgres (you can test enum indices at this point) and then manually removing enum indices for all keys in the state CF (by directly using RocksDB operations), so that they can be synced again.

core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/storage/src/db.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/dal/src/storage_logs_dedup_dal.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
@perekopskiy perekopskiy requested a review from slowli October 6, 2023 12:31
Copy link
Contributor

@slowli slowli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice test! 👍 Other than some nits, looks good.

core/lib/dal/src/storage_logs_dal.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Outdated Show resolved Hide resolved
@perekopskiy perekopskiy requested a review from slowli October 9, 2023 08:21
slowli
slowli previously approved these changes Oct 9, 2023
Copy link
Contributor

@slowli slowli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than minor nits, looks good. Please update the PR description; it describes an outdated implementation.

core/bin/external_node/src/config/mod.rs Outdated Show resolved Hide resolved
core/lib/dal/src/storage_logs_dal.rs Show resolved Hide resolved
core/lib/dal/src/storage_logs_dedup_dal.rs Outdated Show resolved Hide resolved
core/lib/state/src/rocksdb/mod.rs Show resolved Hide resolved
@perekopskiy perekopskiy requested a review from slowli October 9, 2023 11:46
@perekopskiy perekopskiy added this pull request to the merge queue Oct 10, 2023
Merged via the queue into main with commit bab099d Oct 10, 2023
29 checks passed
@perekopskiy perekopskiy deleted the perekopskiy-pla-563-implement-state-keeper-rocksdb-migration-for-enum-indices branch October 10, 2023 11:35
github-merge-queue bot pushed a commit that referenced this pull request Oct 11, 2023
🤖 I have created a release *beep* *boop*
---


##
[16.0.0](core-v15.1.1...core-v16.0.0)
(2023-10-11)


### ⚠ BREAKING CHANGES

* **vm:** Update Refund model
([#181](#181))

### Features

* change chainId to u64
([#167](#167))
([f14bf68](f14bf68))
* **merkle tree:** Provide Merkle proofs for tree entries and entry
ranges ([#119](#119))
([1e30d0b](1e30d0b))
* **storage:** save enum indices in RocksDB
([#162](#162))
([bab099d](bab099d))
* **vm:** Update Refund model
([#181](#181))
([92b6f59](92b6f59))


### Bug Fixes

* **db:** drop constraint prover_jobs_fri_l1_batch_number_fkey
([#173](#173))
([fa71650](fa71650))
* **vm:** Make execution status and stop reason public
([#169](#169))
([f98c4fa](f98c4fa))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants