feat(prover): Refactor WitnessGenerator #2845
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Artemka374
changed the title
EmilLuta
approved these changes
Sep 12, 2024
There was a problem hiding this comment.
I've gone through all of it. AFAICT, these are just structural changes. I'm not sure ArtifactsManager is the right abstraction, but definitely helps, compared to what we had before. Unclear why we need update_database in the ArtifactsManager interface.
All in all, this codebase is much better than what we had before, but we may need to revisit it in the near future.
|
Yes, the changes are structural for now, no logic was touched. |
itegulov
pushed a commit
that referenced
this pull request
Sep 13, 2024
## What ❔ Introduce new structure for witness generators. Introduce `ArtifactsManager` trait responsible for operations with object store and artifacts. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`.
gianbelinche
added a commit
to lambdaclass/zksync-era
that referenced
this pull request
Sep 13, 2024
* fix(tee-prover): passthrough env vars to the SGX enclave (matter-labs#2824) ## What ❔ Passthrough env vars to the SGX enclave. Relevant logs showcasing the issue: https://grafana.matterlabs.dev/goto/1iFHMIeIg?orgId=1 ## Why ❔ To fix the bug. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): Add setup keys step to prover init (matter-labs#2811) ## What ❔ Add setup keys step to prover init * feat(zk_toolbox): Allow running docker images for provers (matter-labs#2800) ## What ❔ Add `--docker` flag for `zk_inception prover run` which will allow prover components to run from docker images. ## Why ❔ To decrease setup time/improve UX ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore: Add README for verified sources fetcher (matter-labs#2829) ## What ❔ Adds README for verified sources fetcher ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore(ci): Limit tokio/rayon pools for zk_toolbox CI (matter-labs#2828) ## What ❔ Limits rayon threadpool size to 2 and tokio threadpool size to 4 in zk_toolbox CI. I have checked locally, and with this configuration time to run integration tests is pretty close to the default configuration. ## Why ❔ By default, both tokio and rayon will try to use all the CPUs. When we run multiple Rust binaries at the same time (3 servers and 3 ENs in our case), it causes a lot of conflict for resources, regardless of the number of CPUs. * feat: attester committees data extractor (BFT-434) (matter-labs#2684) Extraction of the attester committee from consensus registry state. If consensus registry address is not specified, we fall back to attester committee from genesis. This pr does NOT enable the dynamic attestation, as the registry address needs to be added to the main node config first. --------- Co-authored-by: Moshe Shababo <[email protected]> Co-authored-by: Igor Aleksanov <[email protected]> * test: Improve revert integration test (matter-labs#2822) ## What ❔ - Removes a data race from the revert integration test (a single node one); a batch gets executed after it is verified that not all batches are executed. - Removes copy-paste between single-node and EN tests. - Structures revert tests into smaller steps. ## Why ❔ Makes the test less flaky and easier to maintain. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * test: Fix "missing revert data" error; fix / debug integration tests (matter-labs#2804) ## What ❔ - Fixes the "missing revert data" error by updating the used reth Docker image. The error is probably caused by [this issue](paradigmxyz/reth#7381) fixed in the new reth versions. - Removes "web3 API compatibility tests › Should check API returns error when there are too many logs in eth_getLogs" test as fundamentally flaky and able to poison other tests. - Adds logging for upgrade test to investigate L1 "nonce too low" errors. ## Why ❔ Flaky CI bad. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * refactor(api): Extract oneshot VM executor to executor crate (matter-labs#2806) ## What ❔ Extracts oneshot VM executor to the executor crate. ## Why ❔ To make executor logic more reusable and maintainable. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix(en): Fix connection starvation during snapshot recovery (matter-labs#2836) ## What ❔ - Fixes DB connection starvation during snapshot recovery. Caused by the insufficient number of connections in the DB pool provided to recovery logic. - Additionally, fixes max concurrency of recovery not being set. ## Why ❔ Connection starvation errors degrade UX. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix: handling of HTTP 403 thrown by proxyd (matter-labs#2835) When a method is missing: proxyd returns HTTP 403: methodnotfound while api server returns HTTP 200: methodnotfound we need to handle both. * chore: log the whole error message when the task fails (matter-labs#2834) "{err}" is not useful, because it truncates the stack trace of anyhow errors. * feat: allow specifying what tests to run with zks (matter-labs#2841) `zks t i` now accepts a `-t` flag that can can be used to specify a pattern. Only matching tests are run. * chore: clean up dependencies (matter-labs#2839) ## What ❔ Resolves matter-labs#2783 ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(utils): Rework locate_workspace, introduce Workspace type (matter-labs#2830) ## What - Removes `locate_workspace` and `workspace_dir_or_current_dir` methods. - Instead, introduces `Workspace` type that is aware of different Cargo workspaces in the codebase. ## Why The approach with a single `locate_workspace` doesn't work well for our codebase, since we have multiple workspaces. It resulted in some very implicit and convoluted code (see the removed `get_base_dir` in prover workspace). New approach handles all 3 workspaces _plus_ the lack of a workspace. * feat(zk_toolbox): `zk_supervisor prover` subcommand (matter-labs#2820) ## What ❔ Add prover subcommand for `zk_supervisor`. Add the following subcommand: * `zk_supervisor prover info` - Prints information about current prover setup. * `zk_supervisor prover insert-version` - Insert new protocol version in prover database(integration with `prover_cli`). * `zk_supervisor prover insert-batch` - Insert new batch in prover database(integration with `prover_cli`). Add automatic creation of `prover/artifacts/witness_inputs` dirs if the storage is file backed on init. ## Why ❔ To improve UX of working with provers. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk-toolbox): Make token multiplier optional (matter-labs#2843) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Danil <[email protected]> * feat: Smaller zk_environment image (matter-labs#1920) ## What ❔ * Reduced uncompressed size from 7.5GB to 4.81 GB. * Changes to make zk_environment image smaller * compile solc with release mode * remove unnecessary components from google cli * build things first, and then only copy the artifacts. ## Why ❔ * This image is used on ALL of our CI builds - so reducing its size will speed up the CIs. * fix(zk_toolbox): secrets path, artifacts path (matter-labs#2850) ## What ❔ Fix getting chain config in zk_supervisor prover Fix artifacts path when initializing. Setup data path for provers ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * ci: fix using cargo nextest (matter-labs#2855) ## What ❔ `cargo install --list` doesn't work properly in CI because we copy only binaries to zk-environment also increase reth block-time and fixes upgrade-test which improves tests stability ## Why ❔ we rely on cargo-nextest in tests ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat: add da clients (matter-labs#2743) ## What ❔ This PR changes the approach to managing 3rd party DA clients. It was assumed before that they will be stored in a separate repository (hyperchain-da), but to simplify the processes and solve a recursive dependency problem, we decided to manage those within `zksync-era`. The config now defines which DA client will be used, for proto-based configuration it requires adding these lines to general.yaml: ``` da_client: avail: api_node_url: wss://turing-rpc.avail.so/ws bridge_api_url: undefined seed: SEED_PHRASE app_id: 82 timeout: 3 max_retries: 5 ``` for env-based: ``` DA_CLIENT="Avail" DA_API_NODE_URL="localhost:12345" DA_BRIDGE_API_URL="localhost:54321" DA_SEED="SEED_PHRASE" DA_APP_ID=1 DA_TIMEOUT=2 DA_MAX_RETRIES=3 ``` If no config is provided - the default behavior is to use NoDA client (same as now). The `da_client` config might be merged with `da_dispatcher` at some point as the second depends on the first one, so their separation does not make much sense (apart from simplification of the configs). But I'd prefer to do it as a separate PR in case we decide to merge them. The client was reimplemented using only lightweight libraries from crates.io, so it doesn't have any visible impact on build time. ## Why ❔ To enable seamless integration with 3rd party DA clients in `zksync-era`. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): deploy legacy bridge (matter-labs#2837) ## What ❔ Support legacy bridges in zk toolbox ## Why ❔ We still have some legacy bridges in production, so we have to have an opportunity to test it ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> * feat(zk_toolbox): Add external_node consensus support (matter-labs#2821) ## What ❔ Add external_node consensus support --------- Signed-off-by: Danil <[email protected]> Co-authored-by: Danil <[email protected]> * feat: Selector generator tool (matter-labs#2844) ## What ❔ * A small tool to generate the selector hashes based on the ABI from json files ## Why ❔ * The output json can be useful for humans to better understand some of the errors (and calldata) * It can also be read by our tools, to make the debugging easier. In the future, we could call this tool regularly on each contracts version change, but for now it can stay as manual. * feat(prover): Optimize setup keys loading (matter-labs#2847) ## What ❔ - Loads setup keys to memory in parallel (for GPU prover only). - Refactors a bunch of related code for simplicity. ## Why ❔ - Locally I've observed load time going from ~30s to ~12s, so ~60% improvement for prover start time. - Readability & maintainability. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat: Bump crypto and protocol deps (matter-labs#2825) ## What ❔ - Use latest versions of crypto, protocol, gpu, and consensus crates. - Remove solved cargo deny advisories from the allowlist. ## Why ❔ - A bunch of fixes/improvements were done. - Optimization of dependency graph. - Solving cargo deny advisories. * feat(prover): Refactor WitnessGenerator (matter-labs#2845) ## What ❔ Introduce new structure for witness generators. Introduce `ArtifactsManager` trait responsible for operations with object store and artifacts. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * Move hyperchain-da to zksync-era * Uncomment validium config checks --------- Signed-off-by: Danil <[email protected]> Co-authored-by: Patrick <[email protected]> Co-authored-by: Matías Ignacio González <[email protected]> Co-authored-by: Artem Fomiuk <[email protected]> Co-authored-by: perekopskiy <[email protected]> Co-authored-by: Igor Aleksanov <[email protected]> Co-authored-by: Grzegorz Prusak <[email protected]> Co-authored-by: Moshe Shababo <[email protected]> Co-authored-by: Alex Ostrovski <[email protected]> Co-authored-by: Joonatan Saarhelo <[email protected]> Co-authored-by: Danil <[email protected]> Co-authored-by: Marcin M <[email protected]> Co-authored-by: Dima Zhornyk <[email protected]>
juan518munoz
pushed a commit
to lambdaclass/zksync-era
that referenced
this pull request
Sep 17, 2024
* fix(tee-prover): passthrough env vars to the SGX enclave (matter-labs#2824) ## What ❔ Passthrough env vars to the SGX enclave. Relevant logs showcasing the issue: https://grafana.matterlabs.dev/goto/1iFHMIeIg?orgId=1 ## Why ❔ To fix the bug. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): Add setup keys step to prover init (matter-labs#2811) ## What ❔ Add setup keys step to prover init * feat(zk_toolbox): Allow running docker images for provers (matter-labs#2800) ## What ❔ Add `--docker` flag for `zk_inception prover run` which will allow prover components to run from docker images. ## Why ❔ To decrease setup time/improve UX ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore: Add README for verified sources fetcher (matter-labs#2829) ## What ❔ Adds README for verified sources fetcher ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore(ci): Limit tokio/rayon pools for zk_toolbox CI (matter-labs#2828) ## What ❔ Limits rayon threadpool size to 2 and tokio threadpool size to 4 in zk_toolbox CI. I have checked locally, and with this configuration time to run integration tests is pretty close to the default configuration. ## Why ❔ By default, both tokio and rayon will try to use all the CPUs. When we run multiple Rust binaries at the same time (3 servers and 3 ENs in our case), it causes a lot of conflict for resources, regardless of the number of CPUs. * feat: attester committees data extractor (BFT-434) (matter-labs#2684) Extraction of the attester committee from consensus registry state. If consensus registry address is not specified, we fall back to attester committee from genesis. This pr does NOT enable the dynamic attestation, as the registry address needs to be added to the main node config first. --------- Co-authored-by: Moshe Shababo <[email protected]> Co-authored-by: Igor Aleksanov <[email protected]> * test: Improve revert integration test (matter-labs#2822) ## What ❔ - Removes a data race from the revert integration test (a single node one); a batch gets executed after it is verified that not all batches are executed. - Removes copy-paste between single-node and EN tests. - Structures revert tests into smaller steps. ## Why ❔ Makes the test less flaky and easier to maintain. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * test: Fix "missing revert data" error; fix / debug integration tests (matter-labs#2804) ## What ❔ - Fixes the "missing revert data" error by updating the used reth Docker image. The error is probably caused by [this issue](paradigmxyz/reth#7381) fixed in the new reth versions. - Removes "web3 API compatibility tests › Should check API returns error when there are too many logs in eth_getLogs" test as fundamentally flaky and able to poison other tests. - Adds logging for upgrade test to investigate L1 "nonce too low" errors. ## Why ❔ Flaky CI bad. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * refactor(api): Extract oneshot VM executor to executor crate (matter-labs#2806) ## What ❔ Extracts oneshot VM executor to the executor crate. ## Why ❔ To make executor logic more reusable and maintainable. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix(en): Fix connection starvation during snapshot recovery (matter-labs#2836) ## What ❔ - Fixes DB connection starvation during snapshot recovery. Caused by the insufficient number of connections in the DB pool provided to recovery logic. - Additionally, fixes max concurrency of recovery not being set. ## Why ❔ Connection starvation errors degrade UX. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix: handling of HTTP 403 thrown by proxyd (matter-labs#2835) When a method is missing: proxyd returns HTTP 403: methodnotfound while api server returns HTTP 200: methodnotfound we need to handle both. * chore: log the whole error message when the task fails (matter-labs#2834) "{err}" is not useful, because it truncates the stack trace of anyhow errors. * feat: allow specifying what tests to run with zks (matter-labs#2841) `zks t i` now accepts a `-t` flag that can can be used to specify a pattern. Only matching tests are run. * chore: clean up dependencies (matter-labs#2839) ## What ❔ Resolves matter-labs#2783 ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(utils): Rework locate_workspace, introduce Workspace type (matter-labs#2830) ## What - Removes `locate_workspace` and `workspace_dir_or_current_dir` methods. - Instead, introduces `Workspace` type that is aware of different Cargo workspaces in the codebase. ## Why The approach with a single `locate_workspace` doesn't work well for our codebase, since we have multiple workspaces. It resulted in some very implicit and convoluted code (see the removed `get_base_dir` in prover workspace). New approach handles all 3 workspaces _plus_ the lack of a workspace. * feat(zk_toolbox): `zk_supervisor prover` subcommand (matter-labs#2820) ## What ❔ Add prover subcommand for `zk_supervisor`. Add the following subcommand: * `zk_supervisor prover info` - Prints information about current prover setup. * `zk_supervisor prover insert-version` - Insert new protocol version in prover database(integration with `prover_cli`). * `zk_supervisor prover insert-batch` - Insert new batch in prover database(integration with `prover_cli`). Add automatic creation of `prover/artifacts/witness_inputs` dirs if the storage is file backed on init. ## Why ❔ To improve UX of working with provers. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk-toolbox): Make token multiplier optional (matter-labs#2843) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Danil <[email protected]> * feat: Smaller zk_environment image (matter-labs#1920) ## What ❔ * Reduced uncompressed size from 7.5GB to 4.81 GB. * Changes to make zk_environment image smaller * compile solc with release mode * remove unnecessary components from google cli * build things first, and then only copy the artifacts. ## Why ❔ * This image is used on ALL of our CI builds - so reducing its size will speed up the CIs. * fix(zk_toolbox): secrets path, artifacts path (matter-labs#2850) ## What ❔ Fix getting chain config in zk_supervisor prover Fix artifacts path when initializing. Setup data path for provers ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * ci: fix using cargo nextest (matter-labs#2855) ## What ❔ `cargo install --list` doesn't work properly in CI because we copy only binaries to zk-environment also increase reth block-time and fixes upgrade-test which improves tests stability ## Why ❔ we rely on cargo-nextest in tests ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat: add da clients (matter-labs#2743) ## What ❔ This PR changes the approach to managing 3rd party DA clients. It was assumed before that they will be stored in a separate repository (hyperchain-da), but to simplify the processes and solve a recursive dependency problem, we decided to manage those within `zksync-era`. The config now defines which DA client will be used, for proto-based configuration it requires adding these lines to general.yaml: ``` da_client: avail: api_node_url: wss://turing-rpc.avail.so/ws bridge_api_url: undefined seed: SEED_PHRASE app_id: 82 timeout: 3 max_retries: 5 ``` for env-based: ``` DA_CLIENT="Avail" DA_API_NODE_URL="localhost:12345" DA_BRIDGE_API_URL="localhost:54321" DA_SEED="SEED_PHRASE" DA_APP_ID=1 DA_TIMEOUT=2 DA_MAX_RETRIES=3 ``` If no config is provided - the default behavior is to use NoDA client (same as now). The `da_client` config might be merged with `da_dispatcher` at some point as the second depends on the first one, so their separation does not make much sense (apart from simplification of the configs). But I'd prefer to do it as a separate PR in case we decide to merge them. The client was reimplemented using only lightweight libraries from crates.io, so it doesn't have any visible impact on build time. ## Why ❔ To enable seamless integration with 3rd party DA clients in `zksync-era`. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): deploy legacy bridge (matter-labs#2837) ## What ❔ Support legacy bridges in zk toolbox ## Why ❔ We still have some legacy bridges in production, so we have to have an opportunity to test it ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> * feat(zk_toolbox): Add external_node consensus support (matter-labs#2821) ## What ❔ Add external_node consensus support --------- Signed-off-by: Danil <[email protected]> Co-authored-by: Danil <[email protected]> * feat: Selector generator tool (matter-labs#2844) ## What ❔ * A small tool to generate the selector hashes based on the ABI from json files ## Why ❔ * The output json can be useful for humans to better understand some of the errors (and calldata) * It can also be read by our tools, to make the debugging easier. In the future, we could call this tool regularly on each contracts version change, but for now it can stay as manual. * feat(prover): Optimize setup keys loading (matter-labs#2847) ## What ❔ - Loads setup keys to memory in parallel (for GPU prover only). - Refactors a bunch of related code for simplicity. ## Why ❔ - Locally I've observed load time going from ~30s to ~12s, so ~60% improvement for prover start time. - Readability & maintainability. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat: Bump crypto and protocol deps (matter-labs#2825) ## What ❔ - Use latest versions of crypto, protocol, gpu, and consensus crates. - Remove solved cargo deny advisories from the allowlist. ## Why ❔ - A bunch of fixes/improvements were done. - Optimization of dependency graph. - Solving cargo deny advisories. * feat(prover): Refactor WitnessGenerator (matter-labs#2845) ## What ❔ Introduce new structure for witness generators. Introduce `ArtifactsManager` trait responsible for operations with object store and artifacts. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * Move hyperchain-da to zksync-era * Uncomment validium config checks --------- Signed-off-by: Danil <[email protected]> Co-authored-by: Patrick <[email protected]> Co-authored-by: Matías Ignacio González <[email protected]> Co-authored-by: Artem Fomiuk <[email protected]> Co-authored-by: perekopskiy <[email protected]> Co-authored-by: Igor Aleksanov <[email protected]> Co-authored-by: Grzegorz Prusak <[email protected]> Co-authored-by: Moshe Shababo <[email protected]> Co-authored-by: Alex Ostrovski <[email protected]> Co-authored-by: Joonatan Saarhelo <[email protected]> Co-authored-by: Danil <[email protected]> Co-authored-by: Marcin M <[email protected]> Co-authored-by: Dima Zhornyk <[email protected]>
juan518munoz
added a commit
to lambdaclass/zksync-era
that referenced
this pull request
Sep 17, 2024
* fix(tee-prover): passthrough env vars to the SGX enclave (matter-labs#2824) ## What ❔ Passthrough env vars to the SGX enclave. Relevant logs showcasing the issue: https://grafana.matterlabs.dev/goto/1iFHMIeIg?orgId=1 ## Why ❔ To fix the bug. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): Add setup keys step to prover init (matter-labs#2811) ## What ❔ Add setup keys step to prover init * feat(zk_toolbox): Allow running docker images for provers (matter-labs#2800) ## What ❔ Add `--docker` flag for `zk_inception prover run` which will allow prover components to run from docker images. ## Why ❔ To decrease setup time/improve UX ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore: Add README for verified sources fetcher (matter-labs#2829) ## What ❔ Adds README for verified sources fetcher ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore(ci): Limit tokio/rayon pools for zk_toolbox CI (matter-labs#2828) ## What ❔ Limits rayon threadpool size to 2 and tokio threadpool size to 4 in zk_toolbox CI. I have checked locally, and with this configuration time to run integration tests is pretty close to the default configuration. ## Why ❔ By default, both tokio and rayon will try to use all the CPUs. When we run multiple Rust binaries at the same time (3 servers and 3 ENs in our case), it causes a lot of conflict for resources, regardless of the number of CPUs. * feat: attester committees data extractor (BFT-434) (matter-labs#2684) Extraction of the attester committee from consensus registry state. If consensus registry address is not specified, we fall back to attester committee from genesis. This pr does NOT enable the dynamic attestation, as the registry address needs to be added to the main node config first. --------- Co-authored-by: Moshe Shababo <[email protected]> Co-authored-by: Igor Aleksanov <[email protected]> * test: Improve revert integration test (matter-labs#2822) ## What ❔ - Removes a data race from the revert integration test (a single node one); a batch gets executed after it is verified that not all batches are executed. - Removes copy-paste between single-node and EN tests. - Structures revert tests into smaller steps. ## Why ❔ Makes the test less flaky and easier to maintain. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * test: Fix "missing revert data" error; fix / debug integration tests (matter-labs#2804) ## What ❔ - Fixes the "missing revert data" error by updating the used reth Docker image. The error is probably caused by [this issue](paradigmxyz/reth#7381) fixed in the new reth versions. - Removes "web3 API compatibility tests › Should check API returns error when there are too many logs in eth_getLogs" test as fundamentally flaky and able to poison other tests. - Adds logging for upgrade test to investigate L1 "nonce too low" errors. ## Why ❔ Flaky CI bad. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * refactor(api): Extract oneshot VM executor to executor crate (matter-labs#2806) ## What ❔ Extracts oneshot VM executor to the executor crate. ## Why ❔ To make executor logic more reusable and maintainable. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix(en): Fix connection starvation during snapshot recovery (matter-labs#2836) ## What ❔ - Fixes DB connection starvation during snapshot recovery. Caused by the insufficient number of connections in the DB pool provided to recovery logic. - Additionally, fixes max concurrency of recovery not being set. ## Why ❔ Connection starvation errors degrade UX. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix: handling of HTTP 403 thrown by proxyd (matter-labs#2835) When a method is missing: proxyd returns HTTP 403: methodnotfound while api server returns HTTP 200: methodnotfound we need to handle both. * chore: log the whole error message when the task fails (matter-labs#2834) "{err}" is not useful, because it truncates the stack trace of anyhow errors. * feat: allow specifying what tests to run with zks (matter-labs#2841) `zks t i` now accepts a `-t` flag that can can be used to specify a pattern. Only matching tests are run. * chore: clean up dependencies (matter-labs#2839) ## What ❔ Resolves matter-labs#2783 ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(utils): Rework locate_workspace, introduce Workspace type (matter-labs#2830) ## What - Removes `locate_workspace` and `workspace_dir_or_current_dir` methods. - Instead, introduces `Workspace` type that is aware of different Cargo workspaces in the codebase. ## Why The approach with a single `locate_workspace` doesn't work well for our codebase, since we have multiple workspaces. It resulted in some very implicit and convoluted code (see the removed `get_base_dir` in prover workspace). New approach handles all 3 workspaces _plus_ the lack of a workspace. * feat(zk_toolbox): `zk_supervisor prover` subcommand (matter-labs#2820) ## What ❔ Add prover subcommand for `zk_supervisor`. Add the following subcommand: * `zk_supervisor prover info` - Prints information about current prover setup. * `zk_supervisor prover insert-version` - Insert new protocol version in prover database(integration with `prover_cli`). * `zk_supervisor prover insert-batch` - Insert new batch in prover database(integration with `prover_cli`). Add automatic creation of `prover/artifacts/witness_inputs` dirs if the storage is file backed on init. ## Why ❔ To improve UX of working with provers. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk-toolbox): Make token multiplier optional (matter-labs#2843) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Danil <[email protected]> * feat: Smaller zk_environment image (matter-labs#1920) ## What ❔ * Reduced uncompressed size from 7.5GB to 4.81 GB. * Changes to make zk_environment image smaller * compile solc with release mode * remove unnecessary components from google cli * build things first, and then only copy the artifacts. ## Why ❔ * This image is used on ALL of our CI builds - so reducing its size will speed up the CIs. * fix(zk_toolbox): secrets path, artifacts path (matter-labs#2850) ## What ❔ Fix getting chain config in zk_supervisor prover Fix artifacts path when initializing. Setup data path for provers ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * ci: fix using cargo nextest (matter-labs#2855) ## What ❔ `cargo install --list` doesn't work properly in CI because we copy only binaries to zk-environment also increase reth block-time and fixes upgrade-test which improves tests stability ## Why ❔ we rely on cargo-nextest in tests ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat: add da clients (matter-labs#2743) ## What ❔ This PR changes the approach to managing 3rd party DA clients. It was assumed before that they will be stored in a separate repository (hyperchain-da), but to simplify the processes and solve a recursive dependency problem, we decided to manage those within `zksync-era`. The config now defines which DA client will be used, for proto-based configuration it requires adding these lines to general.yaml: ``` da_client: avail: api_node_url: wss://turing-rpc.avail.so/ws bridge_api_url: undefined seed: SEED_PHRASE app_id: 82 timeout: 3 max_retries: 5 ``` for env-based: ``` DA_CLIENT="Avail" DA_API_NODE_URL="localhost:12345" DA_BRIDGE_API_URL="localhost:54321" DA_SEED="SEED_PHRASE" DA_APP_ID=1 DA_TIMEOUT=2 DA_MAX_RETRIES=3 ``` If no config is provided - the default behavior is to use NoDA client (same as now). The `da_client` config might be merged with `da_dispatcher` at some point as the second depends on the first one, so their separation does not make much sense (apart from simplification of the configs). But I'd prefer to do it as a separate PR in case we decide to merge them. The client was reimplemented using only lightweight libraries from crates.io, so it doesn't have any visible impact on build time. ## Why ❔ To enable seamless integration with 3rd party DA clients in `zksync-era`. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): deploy legacy bridge (matter-labs#2837) ## What ❔ Support legacy bridges in zk toolbox ## Why ❔ We still have some legacy bridges in production, so we have to have an opportunity to test it ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> * feat(zk_toolbox): Add external_node consensus support (matter-labs#2821) ## What ❔ Add external_node consensus support --------- Signed-off-by: Danil <[email protected]> Co-authored-by: Danil <[email protected]> * feat: Selector generator tool (matter-labs#2844) ## What ❔ * A small tool to generate the selector hashes based on the ABI from json files ## Why ❔ * The output json can be useful for humans to better understand some of the errors (and calldata) * It can also be read by our tools, to make the debugging easier. In the future, we could call this tool regularly on each contracts version change, but for now it can stay as manual. * feat(prover): Optimize setup keys loading (matter-labs#2847) ## What ❔ - Loads setup keys to memory in parallel (for GPU prover only). - Refactors a bunch of related code for simplicity. ## Why ❔ - Locally I've observed load time going from ~30s to ~12s, so ~60% improvement for prover start time. - Readability & maintainability. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat: Bump crypto and protocol deps (matter-labs#2825) ## What ❔ - Use latest versions of crypto, protocol, gpu, and consensus crates. - Remove solved cargo deny advisories from the allowlist. ## Why ❔ - A bunch of fixes/improvements were done. - Optimization of dependency graph. - Solving cargo deny advisories. * feat(prover): Refactor WitnessGenerator (matter-labs#2845) ## What ❔ Introduce new structure for witness generators. Introduce `ArtifactsManager` trait responsible for operations with object store and artifacts. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk-toolbox): use chain admin for bridgehub (matter-labs#2857) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> * fix: count SECP256 precompile to account validation gas limit as well (matter-labs#2859) Account validation counts all the precompile's extra gas costs. This PR adds a missing precompile. * chore(vm): Bump `zksync_vm2` revision (matter-labs#2838) ## What ❔ Updates the fast VM revision to incorporate latest changes. ## Why ❔ Mainly to check that these changes work. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat: Add 80 arch to cuda archs (matter-labs#2871) ## What ❔ Add CUDA ARCH 80 to list of CUDA archs. ## Why ❔ To be able to run provers on A100. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix: Docker build for compressor (matter-labs#2874) ## What ❔ Pass CUDA_ARCH to compressor image build. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(prover): fix setup_metadata_to_setup_data_key (matter-labs#2875) * fix(zksync_tee_prover): strip binary for nix package (matter-labs#2876) ## What ❔ Strip `zksync_tee_prover` binary for nix package. ## Why ❔ Somehow llvm decided to produce different symbol names each build ```diff - 12916: 000000000194d628 48 OBJECT LOCAL HIDDEN 23 anon.c880a573512c356e53163839ee7cd669.742.llvm.17979232152659092167 + 12916: 000000000194d628 48 OBJECT LOCAL HIDDEN 23 anon.c880a573512c356e53163839ee7cd669.742.llvm.14235143304249741099 [...] ``` Remove those to make the builds reproducible again. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Harald Hoyer <[email protected]> * fix(eth_watch): fix `get_events_inner` (matter-labs#2882) ## What ❔ - `get_events_inner` should recursively call itself - `get_events_inner` should allow passing `None` as topics and/or addresses ## Why ❔ bug fix ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * refactor(vm-runner): Improve VM runner / VM playground (matter-labs#2840) ## What ❔ Various minor improvements to VM runner / VM playground: - Get batch storage asynchronously, so that it works efficiently with snapshot storage. - Add metrics / logs to ensure that snapshot storage works as expected. ## Why ❔ Improves usability. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk_toolbox): increase confirmations in testing (matter-labs#2878) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> Increase confirmations in testing. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> Should make CI workflows less flaky. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): Verbose version message (matter-labs#2884) Modifies the version message to include stuff like git commit, branch, build timestamp, and submodules versions (non-recursive). Sample output: ``` $ zki --version zk_inception v0.1.0-c03229138 Branch: popzxc-version-with-git-commit Submodules: - contracts: 3a1b5d4b94ffb00f03d436a7db7e48589eb74d39 Build timestamp: 2024-09-16 12:07:09 ``` * feat(en): Resume incomplete snapshot in snapshot creator in more cases (matter-labs#2886) ## What ❔ Resumes an incomplete snapshot in the snapshot creator if the creator config doesn't specify an L1 batch. ## Why ❔ This effectively reverts the relevant changes from matter-labs#2256. It makes the snapshot creator resilient by default without additional setup, at the cost of parallel creator jobs working incorrectly (unless they all specify L1 batches). ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(build-base): Add sccache to build-base Dockerfile (matter-labs#2889) ## What ❔ Adding installation of sccache to build-base Dockerfile ## Why ❔ To speed-up docker builds ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * refactor(selector-generator): Polish the code (matter-labs#2888) ## What ❔ - Changes procedural approach to more idiomatic for rust (introduces `Selectors` + `App` structures). `Selectors` are not FS aware, `App` is. - Introduces more strong typing -- when both selectors and function names are `String`, it's easy to misuse them (I did several times, while working on this PR). - Removes manual selector calculation and uses `ethabi` for that. - Makes the code async. - Adds docs for CLI (e.g. to see with `--help` argument). - Adds a test for `Selectors`. - Reduces number of panics and uses `anyhow` to propagate errors. - Adds new selectors (I guess I have more contracts compiled locally). ## Why ❔ Maintainability and testability, mostly. With the refactoring, it should be easier to add new functionality and tests for it. Also, the runtime has improved. Before ``` $ time cargo run --bin selector_generator -- contracts etc/selector-generator-data/selectors.json Compiling selector_generator v0.1.0 (/home/popzxc/workspace/current/zksync-era/core/bin/selector_generator) Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.54s Running `target/debug/selector_generator contracts etc/selector-generator-data/selectors.json` Analyzed 401 files. Added 452 selectors (before: 516 after: 968) cargo run --bin selector_generator -- contracts 4.48s user 7.02s system 100% cpu 11.486 total ``` After ``` $ time cargo run --bin selector_generator -- contracts etc/selector-generator-data/selectors.json Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.20s Running `target/debug/selector_generator contracts etc/selector-generator-data/selectors.json` Analyzed 1087 files. Added 0 selectors (before: 1023 after: 1023) cargo run --bin selector_generator -- contracts 1.24s user 0.12s system 100% cpu 1.360 total ``` ## Next steps I would love to see `--check` mode introduced (which checks that 0 new entries were added) and this tool is added to CI, so that we continously run it and prevent from rotting. @mm-zk leaving it to you. * feat(zk_toolbox): add `zki ecosystem build` subcommand (matter-labs#2787) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> Add `zki ecosystem build` subcommand, which builds L1 transactions without signing and broadcasting them. This allows a security team to sign them using keys in cold storage and multisig. ```bash Create transactions to build ecosystem contracts Usage: zki ecosystem build-transactions [OPTIONS] Options: --sender <SENDER> Address of the transaction sender --l1-rpc-url <L1_RPC_URL> L1 RPC URL -o, --out <OUT> Output directory for the generated files -h, --help Print help (see a summary with '-h') ``` ```bash Create unsigned transactions for chain deployment Usage: zki chain build-transactions [OPTIONS] Options: -o, --out <OUT> Output directory for the generated files -h, --help Print help (see a summary with '-h') ``` ### Output ```json # /transactions/deploy.json { "transactions": [ { "hash": null, "transactionType": "CREATE", "contractName": null, "contractAddress": "0xddca24376aa96d8f56667d78306a3bbbdc65b1ff", "function": null, "arguments": null, "transaction": { "from": "0xaf9d732e8a5607caccb72df525851849c33edf9e", "gas": "0x15a02", "value": "0x0", "input": "0x604580600e600039806000f350fe7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe03601600081602082378035828234f58015156039578182fd5b8082525050506014600cf3", "nonce": "0x0", "chainId": "0x9" }, "additionalContracts": [], "isFixedGasLimit": false }, { "hash": null, "transactionType": "CALL", "contractName": null, "contractAddress": "0xddca24376aa96d8f56667d78306a3bbbdc65b1ff", "function": null, "arguments": null, "transaction": { "from": "0xaf9d732e8a5607caccb72df525851849c33edf9e", "to": "0xddca24376aa96d8f56667d78306a3bbbdc65b1ff", "gas": "0xfb603", "value": "0x0", "input": ... ``` ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat: deployment of multicall3 on L2 by zk_toolbox (matter-labs#2872) For complex atomic operations on consensus registry. Should be merged after matter-labs/era-contracts#805 Tested manually by running `cargo -p zk_inception -- e init --dev`, which populated the multicall3 address in contract.yaml to non-zero value. * fix(zk_toolbox): Clone era observability (matter-labs#2892) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Danil <[email protected]> * feat: updated the docker examples to generate consensus keys and use gossipnet (matter-labs#2476) Tested on a hetzner instance. --------- Co-authored-by: Bruno França <[email protected]> * fix: token multiplier setter is taken from chain config (matter-labs#2893) Fix `zki chain update-toen-multiplier-setter` to take the address from the chain config instead of the ecosystem. * feat: emit errors in prover API metrics (matter-labs#2890) ## What ❔ Report reasons of failed calls in prover API metrics. ## Why ❔ To have more observability ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * ci: Deprecate zk in ci tests (matter-labs#2860) ## What ❔ Do not use old zk tool inside our ci. Only zk docker is still in place will be replaced soon by DevOps team ## Why ❔ We are deprecating old zk tool and now it's not needed anymore ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> Co-authored-by: perekopskiy <[email protected]> * chore: increase max_nonce_ahead (matter-labs#2896) ## What ❔ increases max_nonce_ahead in default config ## Why ❔ fix flaky test ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): Add zk_supervisor config-writer command (matter-labs#2866) ## What ❔ Add zk_supervisor override configs command ## Why ❔ In some cases we want to override some default configs. Right now it is done manually and we need proper automation for regular use/ci. --------- Signed-off-by: Danil <[email protected]> Signed-off-by: Harald Hoyer <[email protected]> Co-authored-by: Patrick <[email protected]> Co-authored-by: Matías Ignacio González <[email protected]> Co-authored-by: Artem Fomiuk <[email protected]> Co-authored-by: perekopskiy <[email protected]> Co-authored-by: Igor Aleksanov <[email protected]> Co-authored-by: Grzegorz Prusak <[email protected]> Co-authored-by: Moshe Shababo <[email protected]> Co-authored-by: Alex Ostrovski <[email protected]> Co-authored-by: Joonatan Saarhelo <[email protected]> Co-authored-by: Danil <[email protected]> Co-authored-by: Marcin M <[email protected]> Co-authored-by: Dima Zhornyk <[email protected]> Co-authored-by: Harald Hoyer <[email protected]> Co-authored-by: Manuel Mauro <[email protected]> Co-authored-by: Artem Makhortov <[email protected]> Co-authored-by: Bruno França <[email protected]> Co-authored-by: Ivan Schasny <[email protected]> Co-authored-by: perekopskiy <[email protected]>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Oct 31, 2024
🤖 I have created a release *beep* *boop* --- ## [16.6.0](prover-v16.5.0...prover-v16.6.0) (2024-10-31) ### Features * (DB migration) Rename recursion_scheduler_level_vk_hash to snark_wrapper_vk_hash ([#2809](#2809)) ([64f9551](64f9551)) * Add initial version prover_autoscaler ([#2993](#2993)) ([ebf9604](ebf9604)) * added seed_peers to consensus global config ([#2920](#2920)) ([e9d1d90](e9d1d90)) * attester committees data extractor (BFT-434) ([#2684](#2684)) ([92dde03](92dde03)) * Bump crypto and protocol deps ([#2825](#2825)) ([a5ffaf1](a5ffaf1)) * **circuit_prover:** Add circuit prover ([#2908](#2908)) ([48317e6](48317e6)) * **consensus:** Support for syncing blocks before consensus genesis over p2p network ([#3040](#3040)) ([d3edc3d](d3edc3d)) * **da-clients:** add secrets ([#2954](#2954)) ([f4631e4](f4631e4)) * gateway preparation ([#3006](#3006)) ([16f2757](16f2757)) * Integrate tracers and implement circuits tracer in vm2 ([#2653](#2653)) ([87b02e3](87b02e3)) * Move prover data to /home/popzxc/workspace/current/zksync-era/prover/data ([#2778](#2778)) ([62e4d46](62e4d46)) * Prover e2e test ([#2975](#2975)) ([0edd796](0edd796)) * **prover:** add CLI option to run prover with max allocation ([#2794](#2794)) ([35e4cae](35e4cae)) * **prover:** Add endpoint to PJM to get queue reports ([#2918](#2918)) ([2cec83f](2cec83f)) * **prover:** Add error to panic message of prover ([#2807](#2807)) ([6e057eb](6e057eb)) * **prover:** Add min_provers and dry_run features. Improve metrics and test. ([#3129](#3129)) ([7c28964](7c28964)) * **prover:** Add scale failure events watching and pods eviction. ([#3175](#3175)) ([dd166f8](dd166f8)) * **prover:** Add sending scale requests for Scaler targets ([#3194](#3194)) ([767c5bc](767c5bc)) * **prover:** Add support for scaling WGs and compressor ([#3179](#3179)) ([c41db9e](c41db9e)) * **prover:** Autoscaler sends scale request to appropriate agents. ([#3150](#3150)) ([bfedac0](bfedac0)) * **prover:** Extract keystore into a separate crate ([#2797](#2797)) ([e239260](e239260)) * **prover:** Optimize setup keys loading ([#2847](#2847)) ([19887ef](19887ef)) * **prover:** Refactor WitnessGenerator ([#2845](#2845)) ([934634b](934634b)) * **prover:** Update witness generator to zkevm_test_harness 0.150.6 ([#3029](#3029)) ([2151c28](2151c28)) * **prover:** Use query macro instead string literals for queries ([#2930](#2930)) ([1cf959d](1cf959d)) * **prover:** WG refactoring [#3](#3) ([#2942](#2942)) ([df68762](df68762)) * **prover:** WitnessGenerator refactoring [#2](#2) ([#2899](#2899)) ([36e5340](36e5340)) * Refactor metrics/make API use binaries ([#2735](#2735)) ([8ed086a](8ed086a)) * Remove prover db from house keeper ([#2795](#2795)) ([85b7346](85b7346)) * **tee:** use hex serialization for RPC responses ([#2887](#2887)) ([abe0440](abe0440)) * **utils:** Rework locate_workspace, introduce Workspace type ([#2830](#2830)) ([d256092](d256092)) * vm2 tracers can access storage ([#3114](#3114)) ([e466b52](e466b52)) * **vm:** Do not panic on VM divergence ([#2705](#2705)) ([7aa5721](7aa5721)) * **vm:** EVM emulator support – base ([#2979](#2979)) ([deafa46](deafa46)) * **vm:** Extract batch executor to separate crate ([#2702](#2702)) ([b82dfa4](b82dfa4)) * **zk_toolbox:** `zk_supervisor prover` subcommand ([#2820](#2820)) ([3506731](3506731)) * **zk_toolbox:** Add external_node consensus support ([#2821](#2821)) ([4a10d7d](4a10d7d)) * **zk_toolbox:** Add SQL format for zk supervisor ([#2950](#2950)) ([540e5d7](540e5d7)) * **zk_toolbox:** deploy legacy bridge ([#2837](#2837)) ([93b4e08](93b4e08)) * **zk_toolbox:** Redesign zk_toolbox commands ([#3003](#3003)) ([114834f](114834f)) * **zkstack_cli:** Build dependencies at zkstack build time ([#3157](#3157)) ([724d9a9](724d9a9)) ### Bug Fixes * allow compilation under current toolchain ([#3176](#3176)) ([89eadd3](89eadd3)) * **api:** Return correct flat call tracer ([#2917](#2917)) ([218646a](218646a)) * count SECP256 precompile to account validation gas limit as well ([#2859](#2859)) ([fee0c2a](fee0c2a)) * Fix Doc lint. ([#3158](#3158)) ([c79949b](c79949b)) * ignore unknown fields in rpc json response ([#2962](#2962)) ([692ea73](692ea73)) * **prover:** Do not exit on missing watcher data. ([#3119](#3119)) ([76ed6d9](76ed6d9)) * **prover:** fix setup_metadata_to_setup_data_key ([#2875](#2875)) ([4ae5a93](4ae5a93)) * **prover:** Run for zero queue to allow scaling down to 0 ([#3115](#3115)) ([bbe1919](bbe1919)) * **tee_verifier:** correctly initialize storage for re-execution ([#3017](#3017)) ([9d88373](9d88373)) * **vm:** Prepare new VM for use in API server and fix divergences ([#2994](#2994)) ([741b77e](741b77e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What ❔
Introduce new structure for witness generators.
Introduce
ArtifactsManagertrait responsible for operations with object store and artifacts.Why ❔
Checklist
zk fmtandzk lint.