feat(prover): Optimize setup keys loading #2847
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
EmilLuta
reviewed
Sep 11, 2024
EmilLuta
reviewed
Sep 11, 2024
EmilLuta
previously approved these changes
Sep 12, 2024
EmilLuta
approved these changes
Sep 12, 2024
This was referenced Sep 12, 2024
itegulov
pushed a commit
that referenced
this pull request
Sep 13, 2024
## What ❔ - Loads setup keys to memory in parallel (for GPU prover only). - Refactors a bunch of related code for simplicity. ## Why ❔ - Locally I've observed load time going from ~30s to ~12s, so ~60% improvement for prover start time. - Readability & maintainability. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`.
gianbelinche
added a commit
to lambdaclass/zksync-era
that referenced
this pull request
Sep 13, 2024
* fix(tee-prover): passthrough env vars to the SGX enclave (matter-labs#2824) ## What ❔ Passthrough env vars to the SGX enclave. Relevant logs showcasing the issue: https://grafana.matterlabs.dev/goto/1iFHMIeIg?orgId=1 ## Why ❔ To fix the bug. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): Add setup keys step to prover init (matter-labs#2811) ## What ❔ Add setup keys step to prover init * feat(zk_toolbox): Allow running docker images for provers (matter-labs#2800) ## What ❔ Add `--docker` flag for `zk_inception prover run` which will allow prover components to run from docker images. ## Why ❔ To decrease setup time/improve UX ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore: Add README for verified sources fetcher (matter-labs#2829) ## What ❔ Adds README for verified sources fetcher ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore(ci): Limit tokio/rayon pools for zk_toolbox CI (matter-labs#2828) ## What ❔ Limits rayon threadpool size to 2 and tokio threadpool size to 4 in zk_toolbox CI. I have checked locally, and with this configuration time to run integration tests is pretty close to the default configuration. ## Why ❔ By default, both tokio and rayon will try to use all the CPUs. When we run multiple Rust binaries at the same time (3 servers and 3 ENs in our case), it causes a lot of conflict for resources, regardless of the number of CPUs. * feat: attester committees data extractor (BFT-434) (matter-labs#2684) Extraction of the attester committee from consensus registry state. If consensus registry address is not specified, we fall back to attester committee from genesis. This pr does NOT enable the dynamic attestation, as the registry address needs to be added to the main node config first. --------- Co-authored-by: Moshe Shababo <[email protected]> Co-authored-by: Igor Aleksanov <[email protected]> * test: Improve revert integration test (matter-labs#2822) ## What ❔ - Removes a data race from the revert integration test (a single node one); a batch gets executed after it is verified that not all batches are executed. - Removes copy-paste between single-node and EN tests. - Structures revert tests into smaller steps. ## Why ❔ Makes the test less flaky and easier to maintain. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * test: Fix "missing revert data" error; fix / debug integration tests (matter-labs#2804) ## What ❔ - Fixes the "missing revert data" error by updating the used reth Docker image. The error is probably caused by [this issue](paradigmxyz/reth#7381) fixed in the new reth versions. - Removes "web3 API compatibility tests › Should check API returns error when there are too many logs in eth_getLogs" test as fundamentally flaky and able to poison other tests. - Adds logging for upgrade test to investigate L1 "nonce too low" errors. ## Why ❔ Flaky CI bad. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * refactor(api): Extract oneshot VM executor to executor crate (matter-labs#2806) ## What ❔ Extracts oneshot VM executor to the executor crate. ## Why ❔ To make executor logic more reusable and maintainable. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix(en): Fix connection starvation during snapshot recovery (matter-labs#2836) ## What ❔ - Fixes DB connection starvation during snapshot recovery. Caused by the insufficient number of connections in the DB pool provided to recovery logic. - Additionally, fixes max concurrency of recovery not being set. ## Why ❔ Connection starvation errors degrade UX. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix: handling of HTTP 403 thrown by proxyd (matter-labs#2835) When a method is missing: proxyd returns HTTP 403: methodnotfound while api server returns HTTP 200: methodnotfound we need to handle both. * chore: log the whole error message when the task fails (matter-labs#2834) "{err}" is not useful, because it truncates the stack trace of anyhow errors. * feat: allow specifying what tests to run with zks (matter-labs#2841) `zks t i` now accepts a `-t` flag that can can be used to specify a pattern. Only matching tests are run. * chore: clean up dependencies (matter-labs#2839) ## What ❔ Resolves matter-labs#2783 ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(utils): Rework locate_workspace, introduce Workspace type (matter-labs#2830) ## What - Removes `locate_workspace` and `workspace_dir_or_current_dir` methods. - Instead, introduces `Workspace` type that is aware of different Cargo workspaces in the codebase. ## Why The approach with a single `locate_workspace` doesn't work well for our codebase, since we have multiple workspaces. It resulted in some very implicit and convoluted code (see the removed `get_base_dir` in prover workspace). New approach handles all 3 workspaces _plus_ the lack of a workspace. * feat(zk_toolbox): `zk_supervisor prover` subcommand (matter-labs#2820) ## What ❔ Add prover subcommand for `zk_supervisor`. Add the following subcommand: * `zk_supervisor prover info` - Prints information about current prover setup. * `zk_supervisor prover insert-version` - Insert new protocol version in prover database(integration with `prover_cli`). * `zk_supervisor prover insert-batch` - Insert new batch in prover database(integration with `prover_cli`). Add automatic creation of `prover/artifacts/witness_inputs` dirs if the storage is file backed on init. ## Why ❔ To improve UX of working with provers. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk-toolbox): Make token multiplier optional (matter-labs#2843) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Danil <[email protected]> * feat: Smaller zk_environment image (matter-labs#1920) ## What ❔ * Reduced uncompressed size from 7.5GB to 4.81 GB. * Changes to make zk_environment image smaller * compile solc with release mode * remove unnecessary components from google cli * build things first, and then only copy the artifacts. ## Why ❔ * This image is used on ALL of our CI builds - so reducing its size will speed up the CIs. * fix(zk_toolbox): secrets path, artifacts path (matter-labs#2850) ## What ❔ Fix getting chain config in zk_supervisor prover Fix artifacts path when initializing. Setup data path for provers ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * ci: fix using cargo nextest (matter-labs#2855) ## What ❔ `cargo install --list` doesn't work properly in CI because we copy only binaries to zk-environment also increase reth block-time and fixes upgrade-test which improves tests stability ## Why ❔ we rely on cargo-nextest in tests ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat: add da clients (matter-labs#2743) ## What ❔ This PR changes the approach to managing 3rd party DA clients. It was assumed before that they will be stored in a separate repository (hyperchain-da), but to simplify the processes and solve a recursive dependency problem, we decided to manage those within `zksync-era`. The config now defines which DA client will be used, for proto-based configuration it requires adding these lines to general.yaml: ``` da_client: avail: api_node_url: wss://turing-rpc.avail.so/ws bridge_api_url: undefined seed: SEED_PHRASE app_id: 82 timeout: 3 max_retries: 5 ``` for env-based: ``` DA_CLIENT="Avail" DA_API_NODE_URL="localhost:12345" DA_BRIDGE_API_URL="localhost:54321" DA_SEED="SEED_PHRASE" DA_APP_ID=1 DA_TIMEOUT=2 DA_MAX_RETRIES=3 ``` If no config is provided - the default behavior is to use NoDA client (same as now). The `da_client` config might be merged with `da_dispatcher` at some point as the second depends on the first one, so their separation does not make much sense (apart from simplification of the configs). But I'd prefer to do it as a separate PR in case we decide to merge them. The client was reimplemented using only lightweight libraries from crates.io, so it doesn't have any visible impact on build time. ## Why ❔ To enable seamless integration with 3rd party DA clients in `zksync-era`. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): deploy legacy bridge (matter-labs#2837) ## What ❔ Support legacy bridges in zk toolbox ## Why ❔ We still have some legacy bridges in production, so we have to have an opportunity to test it ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> * feat(zk_toolbox): Add external_node consensus support (matter-labs#2821) ## What ❔ Add external_node consensus support --------- Signed-off-by: Danil <[email protected]> Co-authored-by: Danil <[email protected]> * feat: Selector generator tool (matter-labs#2844) ## What ❔ * A small tool to generate the selector hashes based on the ABI from json files ## Why ❔ * The output json can be useful for humans to better understand some of the errors (and calldata) * It can also be read by our tools, to make the debugging easier. In the future, we could call this tool regularly on each contracts version change, but for now it can stay as manual. * feat(prover): Optimize setup keys loading (matter-labs#2847) ## What ❔ - Loads setup keys to memory in parallel (for GPU prover only). - Refactors a bunch of related code for simplicity. ## Why ❔ - Locally I've observed load time going from ~30s to ~12s, so ~60% improvement for prover start time. - Readability & maintainability. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat: Bump crypto and protocol deps (matter-labs#2825) ## What ❔ - Use latest versions of crypto, protocol, gpu, and consensus crates. - Remove solved cargo deny advisories from the allowlist. ## Why ❔ - A bunch of fixes/improvements were done. - Optimization of dependency graph. - Solving cargo deny advisories. * feat(prover): Refactor WitnessGenerator (matter-labs#2845) ## What ❔ Introduce new structure for witness generators. Introduce `ArtifactsManager` trait responsible for operations with object store and artifacts. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * Move hyperchain-da to zksync-era * Uncomment validium config checks --------- Signed-off-by: Danil <[email protected]> Co-authored-by: Patrick <[email protected]> Co-authored-by: Matías Ignacio González <[email protected]> Co-authored-by: Artem Fomiuk <[email protected]> Co-authored-by: perekopskiy <[email protected]> Co-authored-by: Igor Aleksanov <[email protected]> Co-authored-by: Grzegorz Prusak <[email protected]> Co-authored-by: Moshe Shababo <[email protected]> Co-authored-by: Alex Ostrovski <[email protected]> Co-authored-by: Joonatan Saarhelo <[email protected]> Co-authored-by: Danil <[email protected]> Co-authored-by: Marcin M <[email protected]> Co-authored-by: Dima Zhornyk <[email protected]>
juan518munoz
pushed a commit
to lambdaclass/zksync-era
that referenced
this pull request
Sep 17, 2024
* fix(tee-prover): passthrough env vars to the SGX enclave (matter-labs#2824) ## What ❔ Passthrough env vars to the SGX enclave. Relevant logs showcasing the issue: https://grafana.matterlabs.dev/goto/1iFHMIeIg?orgId=1 ## Why ❔ To fix the bug. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): Add setup keys step to prover init (matter-labs#2811) ## What ❔ Add setup keys step to prover init * feat(zk_toolbox): Allow running docker images for provers (matter-labs#2800) ## What ❔ Add `--docker` flag for `zk_inception prover run` which will allow prover components to run from docker images. ## Why ❔ To decrease setup time/improve UX ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore: Add README for verified sources fetcher (matter-labs#2829) ## What ❔ Adds README for verified sources fetcher ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore(ci): Limit tokio/rayon pools for zk_toolbox CI (matter-labs#2828) ## What ❔ Limits rayon threadpool size to 2 and tokio threadpool size to 4 in zk_toolbox CI. I have checked locally, and with this configuration time to run integration tests is pretty close to the default configuration. ## Why ❔ By default, both tokio and rayon will try to use all the CPUs. When we run multiple Rust binaries at the same time (3 servers and 3 ENs in our case), it causes a lot of conflict for resources, regardless of the number of CPUs. * feat: attester committees data extractor (BFT-434) (matter-labs#2684) Extraction of the attester committee from consensus registry state. If consensus registry address is not specified, we fall back to attester committee from genesis. This pr does NOT enable the dynamic attestation, as the registry address needs to be added to the main node config first. --------- Co-authored-by: Moshe Shababo <[email protected]> Co-authored-by: Igor Aleksanov <[email protected]> * test: Improve revert integration test (matter-labs#2822) ## What ❔ - Removes a data race from the revert integration test (a single node one); a batch gets executed after it is verified that not all batches are executed. - Removes copy-paste between single-node and EN tests. - Structures revert tests into smaller steps. ## Why ❔ Makes the test less flaky and easier to maintain. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * test: Fix "missing revert data" error; fix / debug integration tests (matter-labs#2804) ## What ❔ - Fixes the "missing revert data" error by updating the used reth Docker image. The error is probably caused by [this issue](paradigmxyz/reth#7381) fixed in the new reth versions. - Removes "web3 API compatibility tests › Should check API returns error when there are too many logs in eth_getLogs" test as fundamentally flaky and able to poison other tests. - Adds logging for upgrade test to investigate L1 "nonce too low" errors. ## Why ❔ Flaky CI bad. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * refactor(api): Extract oneshot VM executor to executor crate (matter-labs#2806) ## What ❔ Extracts oneshot VM executor to the executor crate. ## Why ❔ To make executor logic more reusable and maintainable. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix(en): Fix connection starvation during snapshot recovery (matter-labs#2836) ## What ❔ - Fixes DB connection starvation during snapshot recovery. Caused by the insufficient number of connections in the DB pool provided to recovery logic. - Additionally, fixes max concurrency of recovery not being set. ## Why ❔ Connection starvation errors degrade UX. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix: handling of HTTP 403 thrown by proxyd (matter-labs#2835) When a method is missing: proxyd returns HTTP 403: methodnotfound while api server returns HTTP 200: methodnotfound we need to handle both. * chore: log the whole error message when the task fails (matter-labs#2834) "{err}" is not useful, because it truncates the stack trace of anyhow errors. * feat: allow specifying what tests to run with zks (matter-labs#2841) `zks t i` now accepts a `-t` flag that can can be used to specify a pattern. Only matching tests are run. * chore: clean up dependencies (matter-labs#2839) ## What ❔ Resolves matter-labs#2783 ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(utils): Rework locate_workspace, introduce Workspace type (matter-labs#2830) ## What - Removes `locate_workspace` and `workspace_dir_or_current_dir` methods. - Instead, introduces `Workspace` type that is aware of different Cargo workspaces in the codebase. ## Why The approach with a single `locate_workspace` doesn't work well for our codebase, since we have multiple workspaces. It resulted in some very implicit and convoluted code (see the removed `get_base_dir` in prover workspace). New approach handles all 3 workspaces _plus_ the lack of a workspace. * feat(zk_toolbox): `zk_supervisor prover` subcommand (matter-labs#2820) ## What ❔ Add prover subcommand for `zk_supervisor`. Add the following subcommand: * `zk_supervisor prover info` - Prints information about current prover setup. * `zk_supervisor prover insert-version` - Insert new protocol version in prover database(integration with `prover_cli`). * `zk_supervisor prover insert-batch` - Insert new batch in prover database(integration with `prover_cli`). Add automatic creation of `prover/artifacts/witness_inputs` dirs if the storage is file backed on init. ## Why ❔ To improve UX of working with provers. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk-toolbox): Make token multiplier optional (matter-labs#2843) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Danil <[email protected]> * feat: Smaller zk_environment image (matter-labs#1920) ## What ❔ * Reduced uncompressed size from 7.5GB to 4.81 GB. * Changes to make zk_environment image smaller * compile solc with release mode * remove unnecessary components from google cli * build things first, and then only copy the artifacts. ## Why ❔ * This image is used on ALL of our CI builds - so reducing its size will speed up the CIs. * fix(zk_toolbox): secrets path, artifacts path (matter-labs#2850) ## What ❔ Fix getting chain config in zk_supervisor prover Fix artifacts path when initializing. Setup data path for provers ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * ci: fix using cargo nextest (matter-labs#2855) ## What ❔ `cargo install --list` doesn't work properly in CI because we copy only binaries to zk-environment also increase reth block-time and fixes upgrade-test which improves tests stability ## Why ❔ we rely on cargo-nextest in tests ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat: add da clients (matter-labs#2743) ## What ❔ This PR changes the approach to managing 3rd party DA clients. It was assumed before that they will be stored in a separate repository (hyperchain-da), but to simplify the processes and solve a recursive dependency problem, we decided to manage those within `zksync-era`. The config now defines which DA client will be used, for proto-based configuration it requires adding these lines to general.yaml: ``` da_client: avail: api_node_url: wss://turing-rpc.avail.so/ws bridge_api_url: undefined seed: SEED_PHRASE app_id: 82 timeout: 3 max_retries: 5 ``` for env-based: ``` DA_CLIENT="Avail" DA_API_NODE_URL="localhost:12345" DA_BRIDGE_API_URL="localhost:54321" DA_SEED="SEED_PHRASE" DA_APP_ID=1 DA_TIMEOUT=2 DA_MAX_RETRIES=3 ``` If no config is provided - the default behavior is to use NoDA client (same as now). The `da_client` config might be merged with `da_dispatcher` at some point as the second depends on the first one, so their separation does not make much sense (apart from simplification of the configs). But I'd prefer to do it as a separate PR in case we decide to merge them. The client was reimplemented using only lightweight libraries from crates.io, so it doesn't have any visible impact on build time. ## Why ❔ To enable seamless integration with 3rd party DA clients in `zksync-era`. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): deploy legacy bridge (matter-labs#2837) ## What ❔ Support legacy bridges in zk toolbox ## Why ❔ We still have some legacy bridges in production, so we have to have an opportunity to test it ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> * feat(zk_toolbox): Add external_node consensus support (matter-labs#2821) ## What ❔ Add external_node consensus support --------- Signed-off-by: Danil <[email protected]> Co-authored-by: Danil <[email protected]> * feat: Selector generator tool (matter-labs#2844) ## What ❔ * A small tool to generate the selector hashes based on the ABI from json files ## Why ❔ * The output json can be useful for humans to better understand some of the errors (and calldata) * It can also be read by our tools, to make the debugging easier. In the future, we could call this tool regularly on each contracts version change, but for now it can stay as manual. * feat(prover): Optimize setup keys loading (matter-labs#2847) ## What ❔ - Loads setup keys to memory in parallel (for GPU prover only). - Refactors a bunch of related code for simplicity. ## Why ❔ - Locally I've observed load time going from ~30s to ~12s, so ~60% improvement for prover start time. - Readability & maintainability. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat: Bump crypto and protocol deps (matter-labs#2825) ## What ❔ - Use latest versions of crypto, protocol, gpu, and consensus crates. - Remove solved cargo deny advisories from the allowlist. ## Why ❔ - A bunch of fixes/improvements were done. - Optimization of dependency graph. - Solving cargo deny advisories. * feat(prover): Refactor WitnessGenerator (matter-labs#2845) ## What ❔ Introduce new structure for witness generators. Introduce `ArtifactsManager` trait responsible for operations with object store and artifacts. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * Move hyperchain-da to zksync-era * Uncomment validium config checks --------- Signed-off-by: Danil <[email protected]> Co-authored-by: Patrick <[email protected]> Co-authored-by: Matías Ignacio González <[email protected]> Co-authored-by: Artem Fomiuk <[email protected]> Co-authored-by: perekopskiy <[email protected]> Co-authored-by: Igor Aleksanov <[email protected]> Co-authored-by: Grzegorz Prusak <[email protected]> Co-authored-by: Moshe Shababo <[email protected]> Co-authored-by: Alex Ostrovski <[email protected]> Co-authored-by: Joonatan Saarhelo <[email protected]> Co-authored-by: Danil <[email protected]> Co-authored-by: Marcin M <[email protected]> Co-authored-by: Dima Zhornyk <[email protected]>
juan518munoz
added a commit
to lambdaclass/zksync-era
that referenced
this pull request
Sep 17, 2024
* fix(tee-prover): passthrough env vars to the SGX enclave (matter-labs#2824) ## What ❔ Passthrough env vars to the SGX enclave. Relevant logs showcasing the issue: https://grafana.matterlabs.dev/goto/1iFHMIeIg?orgId=1 ## Why ❔ To fix the bug. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): Add setup keys step to prover init (matter-labs#2811) ## What ❔ Add setup keys step to prover init * feat(zk_toolbox): Allow running docker images for provers (matter-labs#2800) ## What ❔ Add `--docker` flag for `zk_inception prover run` which will allow prover components to run from docker images. ## Why ❔ To decrease setup time/improve UX ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore: Add README for verified sources fetcher (matter-labs#2829) ## What ❔ Adds README for verified sources fetcher ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore(ci): Limit tokio/rayon pools for zk_toolbox CI (matter-labs#2828) ## What ❔ Limits rayon threadpool size to 2 and tokio threadpool size to 4 in zk_toolbox CI. I have checked locally, and with this configuration time to run integration tests is pretty close to the default configuration. ## Why ❔ By default, both tokio and rayon will try to use all the CPUs. When we run multiple Rust binaries at the same time (3 servers and 3 ENs in our case), it causes a lot of conflict for resources, regardless of the number of CPUs. * feat: attester committees data extractor (BFT-434) (matter-labs#2684) Extraction of the attester committee from consensus registry state. If consensus registry address is not specified, we fall back to attester committee from genesis. This pr does NOT enable the dynamic attestation, as the registry address needs to be added to the main node config first. --------- Co-authored-by: Moshe Shababo <[email protected]> Co-authored-by: Igor Aleksanov <[email protected]> * test: Improve revert integration test (matter-labs#2822) ## What ❔ - Removes a data race from the revert integration test (a single node one); a batch gets executed after it is verified that not all batches are executed. - Removes copy-paste between single-node and EN tests. - Structures revert tests into smaller steps. ## Why ❔ Makes the test less flaky and easier to maintain. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * test: Fix "missing revert data" error; fix / debug integration tests (matter-labs#2804) ## What ❔ - Fixes the "missing revert data" error by updating the used reth Docker image. The error is probably caused by [this issue](paradigmxyz/reth#7381) fixed in the new reth versions. - Removes "web3 API compatibility tests › Should check API returns error when there are too many logs in eth_getLogs" test as fundamentally flaky and able to poison other tests. - Adds logging for upgrade test to investigate L1 "nonce too low" errors. ## Why ❔ Flaky CI bad. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * refactor(api): Extract oneshot VM executor to executor crate (matter-labs#2806) ## What ❔ Extracts oneshot VM executor to the executor crate. ## Why ❔ To make executor logic more reusable and maintainable. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix(en): Fix connection starvation during snapshot recovery (matter-labs#2836) ## What ❔ - Fixes DB connection starvation during snapshot recovery. Caused by the insufficient number of connections in the DB pool provided to recovery logic. - Additionally, fixes max concurrency of recovery not being set. ## Why ❔ Connection starvation errors degrade UX. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix: handling of HTTP 403 thrown by proxyd (matter-labs#2835) When a method is missing: proxyd returns HTTP 403: methodnotfound while api server returns HTTP 200: methodnotfound we need to handle both. * chore: log the whole error message when the task fails (matter-labs#2834) "{err}" is not useful, because it truncates the stack trace of anyhow errors. * feat: allow specifying what tests to run with zks (matter-labs#2841) `zks t i` now accepts a `-t` flag that can can be used to specify a pattern. Only matching tests are run. * chore: clean up dependencies (matter-labs#2839) ## What ❔ Resolves matter-labs#2783 ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(utils): Rework locate_workspace, introduce Workspace type (matter-labs#2830) ## What - Removes `locate_workspace` and `workspace_dir_or_current_dir` methods. - Instead, introduces `Workspace` type that is aware of different Cargo workspaces in the codebase. ## Why The approach with a single `locate_workspace` doesn't work well for our codebase, since we have multiple workspaces. It resulted in some very implicit and convoluted code (see the removed `get_base_dir` in prover workspace). New approach handles all 3 workspaces _plus_ the lack of a workspace. * feat(zk_toolbox): `zk_supervisor prover` subcommand (matter-labs#2820) ## What ❔ Add prover subcommand for `zk_supervisor`. Add the following subcommand: * `zk_supervisor prover info` - Prints information about current prover setup. * `zk_supervisor prover insert-version` - Insert new protocol version in prover database(integration with `prover_cli`). * `zk_supervisor prover insert-batch` - Insert new batch in prover database(integration with `prover_cli`). Add automatic creation of `prover/artifacts/witness_inputs` dirs if the storage is file backed on init. ## Why ❔ To improve UX of working with provers. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk-toolbox): Make token multiplier optional (matter-labs#2843) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Danil <[email protected]> * feat: Smaller zk_environment image (matter-labs#1920) ## What ❔ * Reduced uncompressed size from 7.5GB to 4.81 GB. * Changes to make zk_environment image smaller * compile solc with release mode * remove unnecessary components from google cli * build things first, and then only copy the artifacts. ## Why ❔ * This image is used on ALL of our CI builds - so reducing its size will speed up the CIs. * fix(zk_toolbox): secrets path, artifacts path (matter-labs#2850) ## What ❔ Fix getting chain config in zk_supervisor prover Fix artifacts path when initializing. Setup data path for provers ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * ci: fix using cargo nextest (matter-labs#2855) ## What ❔ `cargo install --list` doesn't work properly in CI because we copy only binaries to zk-environment also increase reth block-time and fixes upgrade-test which improves tests stability ## Why ❔ we rely on cargo-nextest in tests ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat: add da clients (matter-labs#2743) ## What ❔ This PR changes the approach to managing 3rd party DA clients. It was assumed before that they will be stored in a separate repository (hyperchain-da), but to simplify the processes and solve a recursive dependency problem, we decided to manage those within `zksync-era`. The config now defines which DA client will be used, for proto-based configuration it requires adding these lines to general.yaml: ``` da_client: avail: api_node_url: wss://turing-rpc.avail.so/ws bridge_api_url: undefined seed: SEED_PHRASE app_id: 82 timeout: 3 max_retries: 5 ``` for env-based: ``` DA_CLIENT="Avail" DA_API_NODE_URL="localhost:12345" DA_BRIDGE_API_URL="localhost:54321" DA_SEED="SEED_PHRASE" DA_APP_ID=1 DA_TIMEOUT=2 DA_MAX_RETRIES=3 ``` If no config is provided - the default behavior is to use NoDA client (same as now). The `da_client` config might be merged with `da_dispatcher` at some point as the second depends on the first one, so their separation does not make much sense (apart from simplification of the configs). But I'd prefer to do it as a separate PR in case we decide to merge them. The client was reimplemented using only lightweight libraries from crates.io, so it doesn't have any visible impact on build time. ## Why ❔ To enable seamless integration with 3rd party DA clients in `zksync-era`. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): deploy legacy bridge (matter-labs#2837) ## What ❔ Support legacy bridges in zk toolbox ## Why ❔ We still have some legacy bridges in production, so we have to have an opportunity to test it ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> * feat(zk_toolbox): Add external_node consensus support (matter-labs#2821) ## What ❔ Add external_node consensus support --------- Signed-off-by: Danil <[email protected]> Co-authored-by: Danil <[email protected]> * feat: Selector generator tool (matter-labs#2844) ## What ❔ * A small tool to generate the selector hashes based on the ABI from json files ## Why ❔ * The output json can be useful for humans to better understand some of the errors (and calldata) * It can also be read by our tools, to make the debugging easier. In the future, we could call this tool regularly on each contracts version change, but for now it can stay as manual. * feat(prover): Optimize setup keys loading (matter-labs#2847) ## What ❔ - Loads setup keys to memory in parallel (for GPU prover only). - Refactors a bunch of related code for simplicity. ## Why ❔ - Locally I've observed load time going from ~30s to ~12s, so ~60% improvement for prover start time. - Readability & maintainability. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat: Bump crypto and protocol deps (matter-labs#2825) ## What ❔ - Use latest versions of crypto, protocol, gpu, and consensus crates. - Remove solved cargo deny advisories from the allowlist. ## Why ❔ - A bunch of fixes/improvements were done. - Optimization of dependency graph. - Solving cargo deny advisories. * feat(prover): Refactor WitnessGenerator (matter-labs#2845) ## What ❔ Introduce new structure for witness generators. Introduce `ArtifactsManager` trait responsible for operations with object store and artifacts. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk-toolbox): use chain admin for bridgehub (matter-labs#2857) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> * fix: count SECP256 precompile to account validation gas limit as well (matter-labs#2859) Account validation counts all the precompile's extra gas costs. This PR adds a missing precompile. * chore(vm): Bump `zksync_vm2` revision (matter-labs#2838) ## What ❔ Updates the fast VM revision to incorporate latest changes. ## Why ❔ Mainly to check that these changes work. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat: Add 80 arch to cuda archs (matter-labs#2871) ## What ❔ Add CUDA ARCH 80 to list of CUDA archs. ## Why ❔ To be able to run provers on A100. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix: Docker build for compressor (matter-labs#2874) ## What ❔ Pass CUDA_ARCH to compressor image build. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(prover): fix setup_metadata_to_setup_data_key (matter-labs#2875) * fix(zksync_tee_prover): strip binary for nix package (matter-labs#2876) ## What ❔ Strip `zksync_tee_prover` binary for nix package. ## Why ❔ Somehow llvm decided to produce different symbol names each build ```diff - 12916: 000000000194d628 48 OBJECT LOCAL HIDDEN 23 anon.c880a573512c356e53163839ee7cd669.742.llvm.17979232152659092167 + 12916: 000000000194d628 48 OBJECT LOCAL HIDDEN 23 anon.c880a573512c356e53163839ee7cd669.742.llvm.14235143304249741099 [...] ``` Remove those to make the builds reproducible again. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Harald Hoyer <[email protected]> * fix(eth_watch): fix `get_events_inner` (matter-labs#2882) ## What ❔ - `get_events_inner` should recursively call itself - `get_events_inner` should allow passing `None` as topics and/or addresses ## Why ❔ bug fix ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * refactor(vm-runner): Improve VM runner / VM playground (matter-labs#2840) ## What ❔ Various minor improvements to VM runner / VM playground: - Get batch storage asynchronously, so that it works efficiently with snapshot storage. - Add metrics / logs to ensure that snapshot storage works as expected. ## Why ❔ Improves usability. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk_toolbox): increase confirmations in testing (matter-labs#2878) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> Increase confirmations in testing. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> Should make CI workflows less flaky. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): Verbose version message (matter-labs#2884) Modifies the version message to include stuff like git commit, branch, build timestamp, and submodules versions (non-recursive). Sample output: ``` $ zki --version zk_inception v0.1.0-c03229138 Branch: popzxc-version-with-git-commit Submodules: - contracts: 3a1b5d4b94ffb00f03d436a7db7e48589eb74d39 Build timestamp: 2024-09-16 12:07:09 ``` * feat(en): Resume incomplete snapshot in snapshot creator in more cases (matter-labs#2886) ## What ❔ Resumes an incomplete snapshot in the snapshot creator if the creator config doesn't specify an L1 batch. ## Why ❔ This effectively reverts the relevant changes from matter-labs#2256. It makes the snapshot creator resilient by default without additional setup, at the cost of parallel creator jobs working incorrectly (unless they all specify L1 batches). ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(build-base): Add sccache to build-base Dockerfile (matter-labs#2889) ## What ❔ Adding installation of sccache to build-base Dockerfile ## Why ❔ To speed-up docker builds ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * refactor(selector-generator): Polish the code (matter-labs#2888) ## What ❔ - Changes procedural approach to more idiomatic for rust (introduces `Selectors` + `App` structures). `Selectors` are not FS aware, `App` is. - Introduces more strong typing -- when both selectors and function names are `String`, it's easy to misuse them (I did several times, while working on this PR). - Removes manual selector calculation and uses `ethabi` for that. - Makes the code async. - Adds docs for CLI (e.g. to see with `--help` argument). - Adds a test for `Selectors`. - Reduces number of panics and uses `anyhow` to propagate errors. - Adds new selectors (I guess I have more contracts compiled locally). ## Why ❔ Maintainability and testability, mostly. With the refactoring, it should be easier to add new functionality and tests for it. Also, the runtime has improved. Before ``` $ time cargo run --bin selector_generator -- contracts etc/selector-generator-data/selectors.json Compiling selector_generator v0.1.0 (/home/popzxc/workspace/current/zksync-era/core/bin/selector_generator) Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.54s Running `target/debug/selector_generator contracts etc/selector-generator-data/selectors.json` Analyzed 401 files. Added 452 selectors (before: 516 after: 968) cargo run --bin selector_generator -- contracts 4.48s user 7.02s system 100% cpu 11.486 total ``` After ``` $ time cargo run --bin selector_generator -- contracts etc/selector-generator-data/selectors.json Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.20s Running `target/debug/selector_generator contracts etc/selector-generator-data/selectors.json` Analyzed 1087 files. Added 0 selectors (before: 1023 after: 1023) cargo run --bin selector_generator -- contracts 1.24s user 0.12s system 100% cpu 1.360 total ``` ## Next steps I would love to see `--check` mode introduced (which checks that 0 new entries were added) and this tool is added to CI, so that we continously run it and prevent from rotting. @mm-zk leaving it to you. * feat(zk_toolbox): add `zki ecosystem build` subcommand (matter-labs#2787) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> Add `zki ecosystem build` subcommand, which builds L1 transactions without signing and broadcasting them. This allows a security team to sign them using keys in cold storage and multisig. ```bash Create transactions to build ecosystem contracts Usage: zki ecosystem build-transactions [OPTIONS] Options: --sender <SENDER> Address of the transaction sender --l1-rpc-url <L1_RPC_URL> L1 RPC URL -o, --out <OUT> Output directory for the generated files -h, --help Print help (see a summary with '-h') ``` ```bash Create unsigned transactions for chain deployment Usage: zki chain build-transactions [OPTIONS] Options: -o, --out <OUT> Output directory for the generated files -h, --help Print help (see a summary with '-h') ``` ### Output ```json # /transactions/deploy.json { "transactions": [ { "hash": null, "transactionType": "CREATE", "contractName": null, "contractAddress": "0xddca24376aa96d8f56667d78306a3bbbdc65b1ff", "function": null, "arguments": null, "transaction": { "from": "0xaf9d732e8a5607caccb72df525851849c33edf9e", "gas": "0x15a02", "value": "0x0", "input": "0x604580600e600039806000f350fe7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe03601600081602082378035828234f58015156039578182fd5b8082525050506014600cf3", "nonce": "0x0", "chainId": "0x9" }, "additionalContracts": [], "isFixedGasLimit": false }, { "hash": null, "transactionType": "CALL", "contractName": null, "contractAddress": "0xddca24376aa96d8f56667d78306a3bbbdc65b1ff", "function": null, "arguments": null, "transaction": { "from": "0xaf9d732e8a5607caccb72df525851849c33edf9e", "to": "0xddca24376aa96d8f56667d78306a3bbbdc65b1ff", "gas": "0xfb603", "value": "0x0", "input": ... ``` ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat: deployment of multicall3 on L2 by zk_toolbox (matter-labs#2872) For complex atomic operations on consensus registry. Should be merged after matter-labs/era-contracts#805 Tested manually by running `cargo -p zk_inception -- e init --dev`, which populated the multicall3 address in contract.yaml to non-zero value. * fix(zk_toolbox): Clone era observability (matter-labs#2892) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Danil <[email protected]> * feat: updated the docker examples to generate consensus keys and use gossipnet (matter-labs#2476) Tested on a hetzner instance. --------- Co-authored-by: Bruno França <[email protected]> * fix: token multiplier setter is taken from chain config (matter-labs#2893) Fix `zki chain update-toen-multiplier-setter` to take the address from the chain config instead of the ecosystem. * feat: emit errors in prover API metrics (matter-labs#2890) ## What ❔ Report reasons of failed calls in prover API metrics. ## Why ❔ To have more observability ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * ci: Deprecate zk in ci tests (matter-labs#2860) ## What ❔ Do not use old zk tool inside our ci. Only zk docker is still in place will be replaced soon by DevOps team ## Why ❔ We are deprecating old zk tool and now it's not needed anymore ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> Co-authored-by: perekopskiy <[email protected]> * chore: increase max_nonce_ahead (matter-labs#2896) ## What ❔ increases max_nonce_ahead in default config ## Why ❔ fix flaky test ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): Add zk_supervisor config-writer command (matter-labs#2866) ## What ❔ Add zk_supervisor override configs command ## Why ❔ In some cases we want to override some default configs. Right now it is done manually and we need proper automation for regular use/ci. --------- Signed-off-by: Danil <[email protected]> Signed-off-by: Harald Hoyer <[email protected]> Co-authored-by: Patrick <[email protected]> Co-authored-by: Matías Ignacio González <[email protected]> Co-authored-by: Artem Fomiuk <[email protected]> Co-authored-by: perekopskiy <[email protected]> Co-authored-by: Igor Aleksanov <[email protected]> Co-authored-by: Grzegorz Prusak <[email protected]> Co-authored-by: Moshe Shababo <[email protected]> Co-authored-by: Alex Ostrovski <[email protected]> Co-authored-by: Joonatan Saarhelo <[email protected]> Co-authored-by: Danil <[email protected]> Co-authored-by: Marcin M <[email protected]> Co-authored-by: Dima Zhornyk <[email protected]> Co-authored-by: Harald Hoyer <[email protected]> Co-authored-by: Manuel Mauro <[email protected]> Co-authored-by: Artem Makhortov <[email protected]> Co-authored-by: Bruno França <[email protected]> Co-authored-by: Ivan Schasny <[email protected]> Co-authored-by: perekopskiy <[email protected]>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Sep 19, 2024
🤖 I have created a release *beep* *boop* --- ## [24.25.0](core-v24.24.0...core-v24.25.0) (2024-09-19) ### Features * (DB migration) Rename recursion_scheduler_level_vk_hash to snark_wrapper_vk_hash ([#2809](#2809)) ([64f9551](64f9551)) * add da clients ([#2743](#2743)) ([9218612](9218612)) * attester committees data extractor (BFT-434) ([#2684](#2684)) ([92dde03](92dde03)) * emit errors in prover API metrics ([#2890](#2890)) ([2ac7cc5](2ac7cc5)) * **en:** Resume incomplete snapshot in snapshot creator in more cases ([#2886](#2886)) ([f095b4a](f095b4a)) * make `to` address optional for transaction data ([#2852](#2852)) ([8363c1d](8363c1d)) * **prover:** Optimize setup keys loading ([#2847](#2847)) ([19887ef](19887ef)) * Selector generator tool ([#2844](#2844)) ([b359b08](b359b08)) * **tee:** use hex serialization for RPC responses ([#2887](#2887)) ([abe0440](abe0440)) * **utils:** Rework locate_workspace, introduce Workspace type ([#2830](#2830)) ([d256092](d256092)) * **zk_toolbox:** Add external_node consensus support ([#2821](#2821)) ([4a10d7d](4a10d7d)) ### Bug Fixes * count SECP256 precompile to account validation gas limit as well ([#2859](#2859)) ([fee0c2a](fee0c2a)) * **en:** Fix connection starvation during snapshot recovery ([#2836](#2836)) ([52f4f76](52f4f76)) * **eth_watch:** fix `get_events_inner` ([#2882](#2882)) ([c957dd8](c957dd8)) * handling of HTTP 403 thrown by proxyd ([#2835](#2835)) ([2d71c74](2d71c74)) * **state-keeper:** Restore processed tx metrics in state keeper ([#2815](#2815)) ([4d8862b](4d8862b)) * **tee-prover:** fix deserialization of `std::time::Duration` in `envy` config ([#2817](#2817)) ([df8641a](df8641a)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <[email protected]>
juan518munoz
added a commit
to lambdaclass/zksync-era
that referenced
this pull request
Oct 8, 2024
* feat(ci): Deprecate use of zk tool in docker-build process (#2901) ## What ❔ Remove use of ZK tool in docker-build process ## Why ❔ Improve readability of CI, speeds up building process ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * test: Investigate L1 "nonce too low" errors (#2848) ## What ❔ - Resolves "nonce too low" errors on L1 by retrying transaction requests. This is a temporary measure until a new version of `reth` is released. - Reduces spammy logs for integration tests. Enables verbose output for integration tests in CI. - Fixes some other issues with tests (e.g., a data race in the snapshot recovery test). ## Why ❔ Makes CI more stable. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix(ci): vm perf to prometheus (#2909) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Danil <[email protected]> * feat(ci): Add building base docker image (#2913) ## What ❔ Workflow to build base-image ## Why ❔ Added ability to push to ghcr.io ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(ci): Fix multiarch for build-base CI (#2914) ## What ❔ Fix multiarch command for build-base-image CI ## Why ❔ ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(ci): Switch build-base to ghcr.io (#2912) ## What ❔ Change build-base image location from dockerhub to ghcr.io ## Why ❔ Workaround for dockerhub rate-limiting ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat: make `to` address optional for transaction data (#2852) ## What ❔ This PR is the first part of [this one](https://github.com/matter-labs/zksync-era/pull/2631). We decided to separate it into smaller PRs to make it easier to read and review. This PR contains only the changes to the `TransactionData` structs for the different VM versions, modifying them for an optional field. The other parts are currently available in our repo fork, as they are based on this branch. The other PRs are: - [feat: evm simulator as base system contract](https://github.com/lambdaclass/zksync-era/pull/256): integrates the EVM interpreter contract as a base system contract and includes all necessary updates for zksync-era to run with it. - [feat: make evm simulator optional for the server](https://github.com/lambdaclass/zksync-era/pull/257): makes the evm simulator optional to run using a config file. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(prover): WitnessGenerator refactoring #2 (#2899) ## What ❔ Introduce WitnessGenerator trait Rename some methods ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(tee): use hex serialization for RPC responses (#2887) ## What ❔ Following Anton's suggestion, we have switched to hex serialization for API/RPC requests and responses. Previously, we used default JSON serialization for Vec<u8>, which resulted in a lengthy comma-separated list of integers. This change standardizes serialization, making it more efficient and reducing the size of the responses. The previous format, with a series of comma-separated integers for pubkey-like fields, looked odd. Then: ``` curl -X POST\ -H "Content-Type: application/json" \ --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [491882, "sgx"] }' \ https://mainnet.era.zksync.io {"jsonrpc":"2.0","result":[{"attestation":[3,0,2,0,0,0,0,0,10,<dozens of comma-separated integers here> ``` Now: ``` $ curl -X POST \ -H "Content-Type: application/json" \ --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [1, "sgx"] }' \ http://localhost:3050 {"jsonrpc":"2.0","result":[{"l1BatchNumber":1,"teeType":"sgx","pubkey":"0506070809","signature":"0001020304","proof":"0a0b0c0d0e","provedAt":"2024-09-16T11:53:38.253033Z","attestation":"0403020100"}],"id":1} ``` This change needs to be deployed in lockstep with: https://github.com/matter-labs/teepot/pull/209. ## Why ❔ To improve user experience (UX) and optimize the RPC response size. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. Co-authored-by: Artem Makhortov <[email protected]> * feat(ci): Add runtime-base image (#2919) ## What ❔ Adding image which will be used as runtime-base for services ## Why ❔ As a workaround for dockerhub ratelimits ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(ci): Rename workflows for runner and builder base (#2921) ## What ❔ Rename workflows for runner and builder base ## Why ❔ Easier to find =) ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(ci): zksync-runtime-base proper platforms names (#2923) ## What ❔ Proper platform names for docker build action ## Why ❔ Docker documentation (which is linked in docker-build-action docs) [points to non-working reference](https://docs.docker.com/reference/cli/docker/buildx/build/#platform) ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(ci): Proper type of var for platforms in docker build action (#2924) ## What ❔ Proper type of var for platforms in docker build action ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk_toolbox): show readable log (#2911) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> * feat(ci): Migrate to vendored image as a zksync-runtime-base (#2926) ## What ❔ Move building images to zksync-runtime-base ## Why ❔ Workaround for dockerhub rate limits ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore(main): release core 24.25.0 (#2816) :robot: I have created a release *beep* *boop* --- ## [24.25.0](https://github.com/matter-labs/zksync-era/compare/core-v24.24.0...core-v24.25.0) (2024-09-19) ### Features * (DB migration) Rename recursion_scheduler_level_vk_hash to snark_wrapper_vk_hash ([#2809](https://github.com/matter-labs/zksync-era/issues/2809)) ([64f9551](https://github.com/matter-labs/zksync-era/commit/64f95514c99f95da2a19a97ff064c29a97efc22f)) * add da clients ([#2743](https://github.com/matter-labs/zksync-era/issues/2743)) ([9218612](https://github.com/matter-labs/zksync-era/commit/9218612fdb2b63c20841e2e2e5a45bbd23c01fbc)) * attester committees data extractor (BFT-434) ([#2684](https://github.com/matter-labs/zksync-era/issues/2684)) ([92dde03](https://github.com/matter-labs/zksync-era/commit/92dde039ee8a0bc08e2019b7fa6f243a34d9816f)) * emit errors in prover API metrics ([#2890](https://github.com/matter-labs/zksync-era/issues/2890)) ([2ac7cc5](https://github.com/matter-labs/zksync-era/commit/2ac7cc5836e69fc82c98df2005fedee01c1084e1)) * **en:** Resume incomplete snapshot in snapshot creator in more cases ([#2886](https://github.com/matter-labs/zksync-era/issues/2886)) ([f095b4a](https://github.com/matter-labs/zksync-era/commit/f095b4a3223222ac712de53592fe1e68f766600f)) * make `to` address optional for transaction data ([#2852](https://github.com/matter-labs/zksync-era/issues/2852)) ([8363c1d](https://github.com/matter-labs/zksync-era/commit/8363c1d8697ad9bd2fe5d326218476bc3dad38af)) * **prover:** Optimize setup keys loading ([#2847](https://github.com/matter-labs/zksync-era/issues/2847)) ([19887ef](https://github.com/matter-labs/zksync-era/commit/19887ef21a8bbd26977353f8ee277b711850dfd2)) * Selector generator tool ([#2844](https://github.com/matter-labs/zksync-era/issues/2844)) ([b359b08](https://github.com/matter-labs/zksync-era/commit/b359b085895da6582f1d28722107bc5b25f1232c)) * **tee:** use hex serialization for RPC responses ([#2887](https://github.com/matter-labs/zksync-era/issues/2887)) ([abe0440](https://github.com/matter-labs/zksync-era/commit/abe0440811ae4daf4a0f307922a282e9664308e0)) * **utils:** Rework locate_workspace, introduce Workspace type ([#2830](https://github.com/matter-labs/zksync-era/issues/2830)) ([d256092](https://github.com/matter-labs/zksync-era/commit/d2560928cc67b40a97a5497ac8542915bf6f91a9)) * **zk_toolbox:** Add external_node consensus support ([#2821](https://github.com/matter-labs/zksync-era/issues/2821)) ([4a10d7d](https://github.com/matter-labs/zksync-era/commit/4a10d7d9554d6c1aa2f4fc46557d40baaad8ff2f)) ### Bug Fixes * count SECP256 precompile to account validation gas limit as well ([#2859](https://github.com/matter-labs/zksync-era/issues/2859)) ([fee0c2a](https://github.com/matter-labs/zksync-era/commit/fee0c2ad08a5ab4a04252765b367eb9fbb1f3db7)) * **en:** Fix connection starvation during snapshot recovery ([#2836](https://github.com/matter-labs/zksync-era/issues/2836)) ([52f4f76](https://github.com/matter-labs/zksync-era/commit/52f4f763674d25f8a5e7f3a111354a559f798d52)) * **eth_watch:** fix `get_events_inner` ([#2882](https://github.com/matter-labs/zksync-era/issues/2882)) ([c957dd8](https://github.com/matter-labs/zksync-era/commit/c957dd8011213e0e95fa5962e2310321b29a0d16)) * handling of HTTP 403 thrown by proxyd ([#2835](https://github.com/matter-labs/zksync-era/issues/2835)) ([2d71c74](https://github.com/matter-labs/zksync-era/commit/2d71c7408a0eed3662fc51f70fa9f525d66e4c6f)) * **state-keeper:** Restore processed tx metrics in state keeper ([#2815](https://github.com/matter-labs/zksync-era/issues/2815)) ([4d8862b](https://github.com/matter-labs/zksync-era/commit/4d8862b76a55ac78edd481694fefd2107736ffd9)) * **tee-prover:** fix deserialization of `std::time::Duration` in `envy` config ([#2817](https://github.com/matter-labs/zksync-era/issues/2817)) ([df8641a](https://github.com/matter-labs/zksync-era/commit/df8641a912a8d480ceecff58b0bfaef05e04f0c8)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <[email protected]> * docs: Update system requirements (#2894) ## What ❔ - 30 GB of free disk space are no longer sufficient to run a testnet EN. Upped the system requirements to 50 GB. - State growth is normally used to refer to the growth of the state tree, not of the historical state. What we have is historical state pruning, not state pruning. * chore(vm): Bump vm2 repo revision (#2883) ## What ❔ Bumps the `vm2` crate revision and updates new VM usage correspondingly. ## Why ❔ To keep the `vm2` dependency up to date and track possible regressions. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(nix): use precompiled `rocksdb` and `snappy` (#2925) ## What ❔ Use precompiled `rocksdb` and `snappy` in the nix build. ## Why ❔ Speeds up compilation and is usually better maintained. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Harald Hoyer <[email protected]> * feat(prover): Add endpoint to PJM to get queue reports (#2918) ## What ❔ Add `/queue_report` endpoint, which will get the data about queue and send it. ## Why ❔ To work with new autoscaler ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Co-authored-by: EmilLuta <[email protected]> * fix(eth-sender): print better error message in case of missing blob prices (#2927) Signed-off-by: tomg10 <[email protected]> Co-authored-by: perekopskiy <[email protected]> * feat: added seed_peers to consensus global config (#2920) It will allow us to announce the recommended default list of peers to all ENs without manual intervention. Fixes BFT-509. --------- Co-authored-by: Bruno França <[email protected]> * feat(vm): Do not panic on VM divergence (#2705) ## What ❔ - Allows to continue batch execution on divergence via new `ShadowLenient` VM mode. - Dumps VM state to logs and optionally a file on divergence. ## Why ❔ Allows to detect divergencies in multiple batches w/o blockers. The dumped VM state will hopefully allow investigating divergencies locally, although this logic isn't implemented yet. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(prover): Use query macro instead string literals for queries (#2930) ## What ❔ In some places our sqlx queries are using string literals instead of query macros. This PR changes this behaviour in places it is possible. ## Why ❔ To prevent possible SQL injections. It also will cache the queries, which should make them faster. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(circuit_prover): Add circuit prover (#2908) ### Summary This PR introduces a new component `circuit_prover`, which is multiple WVGs & GPU prover running together, groupless. The changes are non-destructive, old setup and new setup must work together in-tandem. ### What? Circuit prover is a component that runs X WVGs alongside a GPU prover. Making full use of CPUs on the GPU machine, WVGs as a component can be removed altogether. Prover groups are not needed anymore. Based on empiric testing we can (almost -- there will be follow-up fixes to make it efficient) fully run everything on a single machine. The current implementation can sunset the old setup. Current metrics show that circuit prover is > 60% efficient than old one (but quirks are needed for node proofs to unlock it -- will be treated as a follow-up). The purpose is to have the `circuit_prover` deprecate old `prover_fri` & `witness_vector_generator`. ### Why? The changes will allow us to reduce our infrastructure footprint by ~2x and fix plenty of issues we had in the past. Namely: - fully decoupled of GCP - better resource utilization & reduce costs - reduce overall infrastructure needs (which solves the GPU unavailability we've been facing) - reduce complexity & other inefficiencies (no more prover groups!) - and more ### Ask We want to unblock folks running on AWS. This PR is done as is to speed up release process on DevOps side, as it's the longest pole. NOTE: This is the first PR out of a longer set of PRs. Comments are more than welcome, but the following concerns will be addressed in follow-up PRs and are out of scope for this PR: - tracing implementation is subpar; in fact, I'm confident that most metrics could be done via traces - there's a lot of code duplication (both old/new prover, but runner interface between new WVG & CP) - tests - concern separation between job scheduling & job execution - job priority based on resource consumption - other nits (such as no README, constants being hard-coded instead of configuration, etc.) ### Reviewer suggestion This is basically a merge between `prover_fri`, `witness_vector_generation` and `JobProcessor`. Checking code alongside should give you a better view of what's going on. Sorry for making this hard. :/ * refactor(vm): Refactor L1 batch params provider (#2891) ## What ❔ Refactors `L1BatchParamsProvider`, in particular its construction. ## Why ❔ To have more intuitive DevEx. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(vm): Extract oneshot VM executor – environment types (#2885) ## What ❔ Continues oneshot VM executor extraction by moving generic environment types to the `zksync_vm_executor` crate. ## Why ❔ Makes it possible to use the VM executor outside the API server. E.g., it's now used in `zksync_node_consensus`. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix(api_server): fix blob_gas length (#2673) Discovered during testing Gateway integration (our own asserts from query.rs started being triggered by our eth api implementation :D ) --------- Co-authored-by: perekopskiy <[email protected]> * refactor: Improve EN Grafana dashboards (#2931) ## What ❔ Adding more information to the EN Grafana dashboards, which barely had any. ## Why ❔ Better observability. Fixes BFT-507 * chore(main): release core 24.26.0 (#2928) :robot: I have created a release *beep* *boop* --- ## [24.26.0](https://github.com/matter-labs/zksync-era/compare/core-v24.25.0...core-v24.26.0) (2024-09-23) ### Features * added seed_peers to consensus global config ([#2920](https://github.com/matter-labs/zksync-era/issues/2920)) ([e9d1d90](https://github.com/matter-labs/zksync-era/commit/e9d1d905f1ce86f9de2cf39d79be4b5aada4a81d)) * **circuit_prover:** Add circuit prover ([#2908](https://github.com/matter-labs/zksync-era/issues/2908)) ([48317e6](https://github.com/matter-labs/zksync-era/commit/48317e640a00b016bf7bf782cc94fccaf077ed6d)) * **prover:** Add endpoint to PJM to get queue reports ([#2918](https://github.com/matter-labs/zksync-era/issues/2918)) ([2cec83f](https://github.com/matter-labs/zksync-era/commit/2cec83f26e0b9309387135ca43718af4fcd6f6b1)) * **vm:** Do not panic on VM divergence ([#2705](https://github.com/matter-labs/zksync-era/issues/2705)) ([7aa5721](https://github.com/matter-labs/zksync-era/commit/7aa5721d22e253d05d369a60d5bcacbf52021c48)) * **vm:** Extract oneshot VM executor – environment types ([#2885](https://github.com/matter-labs/zksync-era/issues/2885)) ([a2d4126](https://github.com/matter-labs/zksync-era/commit/a2d4126f9e0c9dc46f49a861549d076fdbcf66d3)) ### Bug Fixes * **api_server:** fix blob_gas length ([#2673](https://github.com/matter-labs/zksync-era/issues/2673)) ([44a8f79](https://github.com/matter-labs/zksync-era/commit/44a8f79739704e1216a7b34a0580ad7ba1cce3bd)) * **eth-sender:** print better error message in case of missing blob prices ([#2927](https://github.com/matter-labs/zksync-era/issues/2927)) ([38fc824](https://github.com/matter-labs/zksync-era/commit/38fc824f75e8b0e84f10348d1502fc8a26d12015)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <[email protected]> * chore(ci): Fix CI building (#2936) After new prover was introduced, a new flag is needed for code that touches GPU related code that has no CUDA available. This PR adds the flag to CI pipelines that were missed. [Sample](https://github.com/matter-labs/zksync-era/actions/runs/10958933706/job/30430256489) broken CI. * feat(vm): Split old and new VM implementations (#2915) ## What ❔ Splits old and new VM implementations in the `multivm` crate: - Old VMs are encapsulated in the `LegacyVmInstance` enum, while new ones in the `FastVmInstance` enum (which includes plain and shadowed VM variants). - Fast VM and `FastVmInstance` now expose a tracer type. - Usage of the Fast VM in the batch executor are updated correspondingly. ## Why ❔ It seems infeasible to unify the tracer model for old and new VMs, so keeping them all in a single enum makes little sense. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): Add options for selective installation of zk_inception and zk_supervisor (#2934) ## What ❔ - Introduced `--inception` and `--supervisor` options to allow users to specify which binaries to install. - Removed the `ZKUP_SKIP_ZK_SUPERVISOR` option. ## Why ❔ Make it clear what to install, focusing on what to include rather what to exclude * feat(zk_toolbox): Add zk toolbox unit tests to zks test (#2935) ## What ❔ Add zk toolbox unit tests to zks test * feat(circuit-prover): Add circuit_prover Dockerfile and build rules (#2939) This will enable us to build images for circuit proving (new version). Not tested. --------- Co-authored-by: Yury Akudovich <[email protected]> * feat(contract-verifier): add new compilers (#2947) * ci: Copy setup_keys to docker/circuit-prover-gpu-gar (#2951) ## What ❔ Copy setup keys to be available for docker/circuit-prover-gpu-gar. <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ To build circuit-prover-gpu image with keys. <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * docs: fix invalid links to "namespaces/eth.rs" and "execution_sandbox.rs" (#2945) ## What ❔ This PR fixes 2 invalid links in the documents which was caused by refactor (refactor: Split the rest of the zksync_core (https://github.com/matter-labs/zksync-era/pull/1940)) ## Why ❔ Unclarity in documentation ## Checklist - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore(prover): Fixes broken non-GPU setup (#2953) Previously, this was done via a compile flag. This was deemed poor taste and we've moved a specific env-var - `ZKSYNC_USE_CUDA_STUBS`. This is wired deep within crypto dependencies. This PR updates the dependencies, adds documentation on how to work with the flag and updates all workflows (fixing the broken ones from RUSTFLAGS time). --------- Co-authored-by: Igor Aleksanov <[email protected]> * chore(prover-gateway): Speed up polling (#2957) 16 minutes is just ridiculous. This should've been done a long time ago, but has been avoided due to other priorities and always put on the back-burner/forgotten. Today we change it. * fix(api): Return correct flat call tracer (#2917) ## What ❔ It was an inconsistency between call tracer results ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> * chore(main): release core 24.27.0 (#2940) :robot: I have created a release *beep* *boop* --- ## [24.27.0](https://github.com/matter-labs/zksync-era/compare/core-v24.26.0...core-v24.27.0) (2024-09-25) ### Features * **vm:** Split old and new VM implementations ([#2915](https://github.com/matter-labs/zksync-era/issues/2915)) ([93bc66f](https://github.com/matter-labs/zksync-era/commit/93bc66f21f9f67a440f06f1c4402e0d687698741)) ### Bug Fixes * **api:** Return correct flat call tracer ([#2917](https://github.com/matter-labs/zksync-era/issues/2917)) ([218646a](https://github.com/matter-labs/zksync-era/commit/218646aa1c56200f4ffee99b7f83366e2689354f)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <[email protected]> * feat(eth-watch): redesign to support multiple chains (#2867) This PR contains code from sync-layer-stable + I removed old upgrades processor and updated unit-tests to use the new one --------- Signed-off-by: tomg10 <[email protected]> * feat(prover): WG refactoring #3 (#2942) ## What ❔ Create WG struct with generic round param. Implement generic JobProcessor for all types of WG. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore(eth-sender): remove deprecated proof loading (#2958) ## What ❔ Removes deprecated proof loading for eth-sender ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(eth-watch): add missing check that from_block is not larger than finalized_block (#2969) Signed-off-by: tomg10 <[email protected]> * refactor: Deleted unnecessary files (#2959) ## What ❔ - ~`.cargo/config.toml` is a remnant from when we used private repos. It should no longer be necessary.~ - ~`configs/` doesn't seem to be used at all.~ - `building-from-scratch` just contains an old docker file that was used for Coinbase. No longer used. * fix: cargo-deny advisory due to structopt (#2972) ## What ❔ Adds https://rustsec.org/advisories/RUSTSEC-2024-0375 to `deny.toml`, which is due to `StructOpt`. <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ * Currently it's impossible to migrate to clap as [zkevm_test_harness](https://github.com/matter-labs/era-zkevm_test_harness/blob/v1.5.0/Cargo.toml#L46) still depends on structopt. * This [unblocks](https://github.com/matter-labs/zksync-era/actions/runs/11054218994/job/30710479095) the CI <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix: ignore unknown fields in rpc json response (#2962) In some previous pr, I've added a field to json RPC response (consensusGlobalConfig) and it turned out to be incompatible due to the default parsing behavior: an error was returned when unknown fields were present. In general, for programmatic communications the recommended behaviour is to silently ignore unknown fields, so that adding new fields is compatible. I've also weakened the restriction on accepting new consensus genesis by external nodes - when the unknown fields are silently dropped, it is possible that the node will perceive genesis with the same fork number differently after a binary update (because it receives more fields). * fix(zk_toolbox): Do not panic if mint is not successful (#2973) ## What ❔ Use governor for minting process and do not panic if minting is not successful ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Danil <[email protected]> * initial commit * feat: zk_toolbox command for setting attester committee (#2900) zk_toolbox command for setting attester committee in consensus registry contract. Fixes BFT-505 --------- Co-authored-by: Bruno França <[email protected]> * fix: chainstack block limit exceeded (#2974) ## What ❔ Added support for chainstack developer and growth plan. ## Why ❔ I was getting the below error with chainstack's growth plan when I tried run self deployed zksync hyperchain. To fix this, I have added appropriate changes to the code. ``` ERROR zksync_eth_watch: Failed to process new blocks: Eth client error: JSON-RPC request get_logs(filter=Filter { from_block: Some(Number(6732838)), to_block: Some(Number(6757429)), block_hash: None, address: Some(ValueOrArray([0xadd82b0c015be3fb7e8410061b4f15d577b4207f, 0xd6ee28b24f87c1d634540699b00e9c37fdc7b3bd, 0x41b75786db38a486c99972c8322fcb14c4f000a2, 0x5d300908338b85bee3f30f2e0ad196fc1587b70d])), topics: Some([Some(ValueOrArray([0x4531cd5795773d7101c17bdeb9f5ab7f47d7056017506f937083be5d6e77a382, 0x23bc9f5dc037eb49c162fd08c2a4d43dfe70063149e140d502273168da0a0625, 0xd50ef21701c8ef211433b140724b8d6de471e7d822c8a616c3d424fe2d0e98a9]))]), limit: None }) failed: ErrorObject { code: InvalidParams, message: "Block range limit exceeded. See more details at https://docs.chainstack.com/docs/limits#evm-range-limits", data: None } ``` ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(eth-sender): add a cap to time_in_mempool (#2978) Signed-off-by: tomg10 <[email protected]> * chore: update derive_more to stable version (#2961) ## What ❔ Updates `derive_more` to a stable version. <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ We're currently on beta and this is a requisite for supporting [foundry-zksync](https://github.com/matter-labs/foundry-zksync). <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * use Notify for a more deterministic approach * feat(da-clients): add secrets (#2954) ## What ❔ Add an opportunity to store the sensitive information of DA clients in the secrets. ## Why ❔ Some sensitive info is currently stored as plaintext, which is not secure, we should use secrets for those values ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Co-authored-by: Danil <[email protected]> * fix(api): Fix batch fee input for `debug` namespace (#2948) ## What ❔ Changes the `debug_traceCall` handler to use the fee provider for recent blocks (pending / latest / committed). ## Why ❔ Previously, the `debug` namespace used a static batch fee input retrieved at the node initialization. This looks incorrect. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): add fees integration test to toolbox (#2898) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> Add fees integration test to toolbox and CI. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> Co-authored-by: Danil <[email protected]> * feat: Expose http debug page (#2952) ## What ❔ Exposes the consensus http debug on port 5000. Fixes BFT-508 * feat(zk_inception): Add ecosystem ports scanner (#2849) ## What ❔ Added `EcosystemPortsScanner` tool to scan ecosystem directory config and find all values for ports. Returns `EcosystemPorts` object with ports found and custom description of a service. `EcosystemPorts` also contains methods to: check whether the port is assigned, allocate a new port in range, etc. ## Why ❔ With multiple chains and number of services (like explorer, etc.) growing - it is useful to have a tool (`EcosystemPorts`) that can be used to allocate new ports, thus, lowering the chance of port conflicts within created configuration. ## Checklist - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Co-authored-by: Matías Ignacio González <[email protected]> * build: Set consensus and pruning as default for EN (#2956) ## What ❔ ENs, started from docker compose, now have consensus and pruning enabled as default. Also, the EN docs were updated. * fix(ci): Remove deprecated artifacts dir in contracts (#2989) ## What ❔ Removing deprecated `artifacts` dir from build process ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore(zk_toolbox): Use load current chain function (#2990) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Danil <[email protected]> * perf(api): More efficient gas estimation (#2937) ## What ❔ Makes gas estimation more efficient by using heuristics similar to ones used by Ethereum clients: - Defines the lower gas limit bound as the gas consumed with an effectively infinite gas limit. - Defines an initial binary search pivot as a slightly larger value so that in the best / average case, most of the search space is immediately discarded. ## Why ❔ Improves gas estimation latency in the best and average case. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * test: Base token price client, fix: ratio calculation multiplicative inverse error (#2970) ## What ❔ - I introduce tests to external_price_api crate. Some work was taken from https://github.com/matter-labs/zksync-era/pull/2315 - fix multiplicative inverse ratio calculation bug - fix missing checks in get_fraction ## Why ❔ - We want to make sure the flow of data from API response to token ratio is correct ## Checklist - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [X] Tests for the changes have been added / updated. - [X] Documentation comments have been added / updated. - [X] Code has been formatted via `zk fmt` and `zk lint`. * chore(githooks): Use zk_toolbox in githooks (#2895) ## What ❔ Switch to zk_supervisor fmt in our githooks ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> * feat(zk_toolbox): Add SQL format for zk supervisor (#2950) ## What ❔ Add SQL format for zk supervisor * feat(ci): Add Makefile for easy local build (#2985) ## What ❔ Add Makefile for easier local build of all components ## Why ❔ ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk_toolbox): wrong configs path for prover binaries (#2922) ## What ❔ Right now, while running prover binaries inside of zksync-era, config paths are specified as path to config from chain config(which is `./chain/configs` - relative path, but not absolute). This breaks, because to run prover binaries, shell enters prover directory of era. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): Migrate db during update (#2995) ## What ❔ Update databases during update process, now it's a universal receipt how to pull changes ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. Signed-off-by: Danil <[email protected]> * fix(ci): Bring back hack for contracts build till full migration to foundry (#3000) ## What ❔ Creation of empty dirs, to support building from both new (foundry-built) and old contracts ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. * chore(main): release core 24.28.0 (#2968) :robot: I have created a release *beep* *boop* --- ## [24.28.0](https://github.com/matter-labs/zksync-era/compare/core-v24.27.0...core-v24.28.0) (2024-10-02) ### Features * **da-clients:** add secrets ([#2954](https://github.com/matter-labs/zksync-era/issues/2954)) ([f4631e4](https://github.com/matter-labs/zksync-era/commit/f4631e4466de620cc1401b326d864cdb8b48a05d)) * **eth-sender:** add a cap to time_in_mempool ([#2978](https://github.com/matter-labs/zksync-era/issues/2978)) ([650d42f](https://github.com/matter-labs/zksync-era/commit/650d42fea6124d80b60a8270a303d72ad6ac741e)) * **eth-watch:** redesign to support multiple chains ([#2867](https://github.com/matter-labs/zksync-era/issues/2867)) ([aa72d84](https://github.com/matter-labs/zksync-era/commit/aa72d849c24a664acd083eba73795ddc5d31d55f)) * Expose http debug page ([#2952](https://github.com/matter-labs/zksync-era/issues/2952)) ([e0b6488](https://github.com/matter-labs/zksync-era/commit/e0b64888aae7324aec2d40fa0cd51ea7e1450cd9)) * **zk_toolbox:** add fees integration test to toolbox ([#2898](https://github.com/matter-labs/zksync-era/issues/2898)) ([e7ead76](https://github.com/matter-labs/zksync-era/commit/e7ead760ce0417dd36af3839ac557f7e9ab238a4)) * **zk_toolbox:** Add SQL format for zk supervisor ([#2950](https://github.com/matter-labs/zksync-era/issues/2950)) ([540e5d7](https://github.com/matter-labs/zksync-era/commit/540e5d7554f54e80d52f1bfae37e03ca8f787baf)) ### Bug Fixes * **api:** Fix batch fee input for `debug` namespace ([#2948](https://github.com/matter-labs/zksync-era/issues/2948)) ([79b6fcf](https://github.com/matter-labs/zksync-era/commit/79b6fcf8b5d10a0ccdceb846370dd6870b6a32b5)) * chainstack block limit exceeded ([#2974](https://github.com/matter-labs/zksync-era/issues/2974)) ([4ffbf42](https://github.com/matter-labs/zksync-era/commit/4ffbf426de166c11aaf5d7b5ed7d199644fba229)) * **eth-watch:** add missing check that from_block is not larger than finalized_block ([#2969](https://github.com/matter-labs/zksync-era/issues/2969)) ([3f406c7](https://github.com/matter-labs/zksync-era/commit/3f406c7d0c0e76d798c2d838abde57ca692822c0)) * ignore unknown fields in rpc json response ([#2962](https://github.com/matter-labs/zksync-era/issues/2962)) ([692ea73](https://github.com/matter-labs/zksync-era/commit/692ea73f75a5fb9db2b4ac33ad24d20568638742)) ### Performance Improvements * **api:** More efficient gas estimation ([#2937](https://github.com/matter-labs/zksync-era/issues/2937)) ([3b69e37](https://github.com/matter-labs/zksync-era/commit/3b69e37e470dab859a55787f6cc971e7083de2fd)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <[email protected]> * feat(zk_toolbox): Add subcommands and flags for chain registration (#2946) ## What ❔ Added subcommands: `zki chain init configs` - just creates configs with an intention to run chain initialization manually via subcommands `zki chain register-chain` - runs steps from `RegisterHyperchain.s.sol` `zki chain accept-chain-ownership` - accepts ownership for `DiamondProxy` `zki chain genesis database` - initializes database only, performs migration (uses values from args or `secrets.yaml`) `zki chain genesis server` - runs server --genesis Added flags: `zki ecosystem init --ecosystem-only` - runs `init` only for ecosystem (skips `init` for chain) Other changes: * Fixed issue with `--wallet_path` value ignored * Nullify database names if `zki ecosystem init` is used for multiple chains * Zeroify some addresses in `contracts.yaml` when copying from ecosystem during init ## Why ❔ These changes allow us to run the chain registration process for externally hosted chains. Not ideal yet, but the process goes like this: L1 side: * Init ecosystem: `zki ecosystem create && zki ecosystem init --ecosystem-only && zki chain init configs` * Fill in wallets * Register chain: `zki chain register-chain` * Deploy L2 contracts: `zki chain deploy-l2-contracts` * Share `contracts.yaml` L2 side: * Init ecosystem: `zki ecosystem create && zki ecosystem init --ecosystem-only && zki chain init configs` * Fill in wallets * Copy `contracts.yaml` * Accept ownership: `zki chain accept-chain-ownership` * Initialize databases: `zki chain genesis database` * Run server genesis: `zki chain genesis server` ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk_toolbox): Correct secrets (#3004) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. Signed-off-by: Danil <[email protected]> * replace atomic for mutex * move const to config * feat(ci): Add external-node build to Makefile (#3002) ## What ❔ Add external-node build to Makefile ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. * fix(zk_toolbox): Remove prover db from server init (#3009) ## What ❔ Remove prover db from server init ## Why ❔ prover db is not needed in server init since prover db is initialized with prover init * fix(ci): Build zk-env with CUDA (#3013) ## What ❔ Ability to build zk-env with cuda not only from push to main branch ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. * Update contracts * Finalize merge * feat(en): periodically fetch bridge addresses (#2949) ## What ❔ Periodically fetch bridge addresses ## Why ❔ Addresses will be changed during gateway upgrade ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(eth-watch): catch another reth error (#3026) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. * test(api): Add more `TxSender` tests (#3001) ## What ❔ Covers more `TxSender` functionality with unit tests. ## Why ❔ More test coverage is good. Also, it will help with new VM integration (by running the added tests with a shadowed VM). ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. * feat(eth-sender): add time_in_mempool_cap config (#3018) Configuration parameter for time_in_mempool_cap, cap for time_in_mempool in eth-sender fee model + default values for parameter_a and parameter_b for GasAdjuster * fix(zkstack_cli): Fix fmt --check (#3027) ## What ❔ `fmt --check` wasn't returning an error code when failing * docs(zk_toolbox): remove generate-sk subcommand references (#3016) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> Remove `generate-sk` subcommand which doesn’t exist anymore. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. * chore(vm): Remove tests for old VM versions (#3015) ## What ❔ Removes unit test files for old VM versions in the `multivm` crate. ## Why ❔ These tests are not built (not included into the module tree of the crate), so they are just deadweight. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). * Update contracts * refactor(eth): Brush up `eth_signer` crate (#3014) ## What ❔ Brushes up the `eth_signer` crate so that its API is easier to use (e.g., it doesn't require an async runtime). ## Why ❔ Improved DevEx. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. * Update query jsons * fix(vm): Prepare new VM for use in API server and fix divergences (#2994) ## What ❔ Prepares the new VM for use in the API server (without tracers): - Fills in more statistics - Covers more statistic fields in divergence checks and fixes the corresponding divergence in `pubdata_published` - Fixes a divergence if revert reaches the bootloader call frame (happens in the call execution mode) - Fixes a panic in `ShadowVm` if the transaction hash is not set and makes `impl Debug for Transaction` non-panicking. (A hash is not set if a transaction is converted from a call.) ## Why ❔ Part of preparations for integrating the new VM into the API server. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. --------- Co-authored-by: Joonatan Saarhelo <[email protected]> * feat: Increased the timeout of waiting for db in tests (#3007) I don't know about other people's setup, but for me it is very flaky with just 3s * feat: add metric to track current cbt ratio (#3020) Add a new metric to track custom base token to ETH ratio * Format md * Refactor secrets --------- Signed-off-by: Danil <[email protected]> Signed-off-by: Harald Hoyer <[email protected]> Signed-off-by: tomg10 <[email protected]> Co-authored-by: Artem Makhortov <[email protected]> Co-authored-by: Alex Ostrovski <[email protected]> Co-authored-by: Danil <[email protected]> Co-authored-by: Nacho Avecilla <[email protected]> …
github-merge-queue bot
pushed a commit
that referenced
this pull request
Oct 31, 2024
🤖 I have created a release *beep* *boop* --- ## [16.6.0](prover-v16.5.0...prover-v16.6.0) (2024-10-31) ### Features * (DB migration) Rename recursion_scheduler_level_vk_hash to snark_wrapper_vk_hash ([#2809](#2809)) ([64f9551](64f9551)) * Add initial version prover_autoscaler ([#2993](#2993)) ([ebf9604](ebf9604)) * added seed_peers to consensus global config ([#2920](#2920)) ([e9d1d90](e9d1d90)) * attester committees data extractor (BFT-434) ([#2684](#2684)) ([92dde03](92dde03)) * Bump crypto and protocol deps ([#2825](#2825)) ([a5ffaf1](a5ffaf1)) * **circuit_prover:** Add circuit prover ([#2908](#2908)) ([48317e6](48317e6)) * **consensus:** Support for syncing blocks before consensus genesis over p2p network ([#3040](#3040)) ([d3edc3d](d3edc3d)) * **da-clients:** add secrets ([#2954](#2954)) ([f4631e4](f4631e4)) * gateway preparation ([#3006](#3006)) ([16f2757](16f2757)) * Integrate tracers and implement circuits tracer in vm2 ([#2653](#2653)) ([87b02e3](87b02e3)) * Move prover data to /home/popzxc/workspace/current/zksync-era/prover/data ([#2778](#2778)) ([62e4d46](62e4d46)) * Prover e2e test ([#2975](#2975)) ([0edd796](0edd796)) * **prover:** add CLI option to run prover with max allocation ([#2794](#2794)) ([35e4cae](35e4cae)) * **prover:** Add endpoint to PJM to get queue reports ([#2918](#2918)) ([2cec83f](2cec83f)) * **prover:** Add error to panic message of prover ([#2807](#2807)) ([6e057eb](6e057eb)) * **prover:** Add min_provers and dry_run features. Improve metrics and test. ([#3129](#3129)) ([7c28964](7c28964)) * **prover:** Add scale failure events watching and pods eviction. ([#3175](#3175)) ([dd166f8](dd166f8)) * **prover:** Add sending scale requests for Scaler targets ([#3194](#3194)) ([767c5bc](767c5bc)) * **prover:** Add support for scaling WGs and compressor ([#3179](#3179)) ([c41db9e](c41db9e)) * **prover:** Autoscaler sends scale request to appropriate agents. ([#3150](#3150)) ([bfedac0](bfedac0)) * **prover:** Extract keystore into a separate crate ([#2797](#2797)) ([e239260](e239260)) * **prover:** Optimize setup keys loading ([#2847](#2847)) ([19887ef](19887ef)) * **prover:** Refactor WitnessGenerator ([#2845](#2845)) ([934634b](934634b)) * **prover:** Update witness generator to zkevm_test_harness 0.150.6 ([#3029](#3029)) ([2151c28](2151c28)) * **prover:** Use query macro instead string literals for queries ([#2930](#2930)) ([1cf959d](1cf959d)) * **prover:** WG refactoring [#3](#3) ([#2942](#2942)) ([df68762](df68762)) * **prover:** WitnessGenerator refactoring [#2](#2) ([#2899](#2899)) ([36e5340](36e5340)) * Refactor metrics/make API use binaries ([#2735](#2735)) ([8ed086a](8ed086a)) * Remove prover db from house keeper ([#2795](#2795)) ([85b7346](85b7346)) * **tee:** use hex serialization for RPC responses ([#2887](#2887)) ([abe0440](abe0440)) * **utils:** Rework locate_workspace, introduce Workspace type ([#2830](#2830)) ([d256092](d256092)) * vm2 tracers can access storage ([#3114](#3114)) ([e466b52](e466b52)) * **vm:** Do not panic on VM divergence ([#2705](#2705)) ([7aa5721](7aa5721)) * **vm:** EVM emulator support – base ([#2979](#2979)) ([deafa46](deafa46)) * **vm:** Extract batch executor to separate crate ([#2702](#2702)) ([b82dfa4](b82dfa4)) * **zk_toolbox:** `zk_supervisor prover` subcommand ([#2820](#2820)) ([3506731](3506731)) * **zk_toolbox:** Add external_node consensus support ([#2821](#2821)) ([4a10d7d](4a10d7d)) * **zk_toolbox:** Add SQL format for zk supervisor ([#2950](#2950)) ([540e5d7](540e5d7)) * **zk_toolbox:** deploy legacy bridge ([#2837](#2837)) ([93b4e08](93b4e08)) * **zk_toolbox:** Redesign zk_toolbox commands ([#3003](#3003)) ([114834f](114834f)) * **zkstack_cli:** Build dependencies at zkstack build time ([#3157](#3157)) ([724d9a9](724d9a9)) ### Bug Fixes * allow compilation under current toolchain ([#3176](#3176)) ([89eadd3](89eadd3)) * **api:** Return correct flat call tracer ([#2917](#2917)) ([218646a](218646a)) * count SECP256 precompile to account validation gas limit as well ([#2859](#2859)) ([fee0c2a](fee0c2a)) * Fix Doc lint. ([#3158](#3158)) ([c79949b](c79949b)) * ignore unknown fields in rpc json response ([#2962](#2962)) ([692ea73](692ea73)) * **prover:** Do not exit on missing watcher data. ([#3119](#3119)) ([76ed6d9](76ed6d9)) * **prover:** fix setup_metadata_to_setup_data_key ([#2875](#2875)) ([4ae5a93](4ae5a93)) * **prover:** Run for zero queue to allow scaling down to 0 ([#3115](#3115)) ([bbe1919](bbe1919)) * **tee_verifier:** correctly initialize storage for re-execution ([#3017](#3017)) ([9d88373](9d88373)) * **vm:** Prepare new VM for use in API server and fix divergences ([#2994](#2994)) ([741b77e](741b77e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What ❔
Why ❔
Checklist
zk fmtandzk lint.